Projects metadata

TREC document-to-document relevance assessment

Started in 2022-06-01 Concluded in 2023-03-31

Description

TREC 2005 Genomics Track data provide document-to-topic relevance assessment. In this project we analyze a document-to-document relevance assessment for a subset of the TREC collection using manual annotation for the judgement. The inter-annotator agreement is evaluated with Fleiss' Kappa.

Keywords

document relevance, doc2doc relevance

Url

URL: https://zbmed-semtec.github.io/projects/2022_TREC_doc2doc

Previous project members

Maria Fernanda Cadena Visit Person
Tim Fellerhoff Visit Person
Lukas Geist Visit Person
Olga Giraldo Visit Person
Nelson Quiñones Visit Person
Andrea Robayo-Gama Visit Person
Rohitha Ravinder Visit Person
Dhwani Solanki Visit Person
Talha Muhammad Visit Person
Dietrich Rebholz-Schuhmann Visit Person
Leyla Jael Castro Visit Person

Department

Semantic Technologies team at ZB MED

Visit URL Visit ResearchOrganization

URL: https://zbmed-semtec.github.io/

Parent organization, consortium or research project

Deutsche Zentralbibliothek für Medizin (ZB MED) - Informationszentrum Lebenswissenschaften

Visit URL Visit ResearchOrganization

Alternatename: ZB MED Information Centre for Life Sciences
URL: https://zbmed.de/

STELLA Living Labs Project

Visit URL Visit ResearchProject

URL: https://stella-project.org

Funding

Visit URL Visit Grant

Identifier: 407518790
Description: Project no. 407518790 (corresponding to the STELLA project)

Outcomes

Document-to-document relevance assessment for TREC Genomics Track 2005

Visit URL Visit ScholarlyArticle

Identifier: CEUR:Vol-3415/paper-12
Cite as: Giraldo O, Cadena MF, Robayo-Gama A, Solanki D, Fellerhoff T, Geist L, et al. Document-to-document relevance assessment for TREC Genomics Track 2005. CEUR; 2023. Available: https://ceur-ws.org/Vol-3415/paper-12.pdf
Datepublished: 2023-06-22
License: http://spdx.org/licenses/CC-BY-4.0
URL: https://ceur-ws.org/Vol-3415/paper-12.pdf#
Authors: https://orcid.org/0000-0003-2978-8922. https://orcid.org/0000-0002-5915-8895. https://zbmed-semtec.github.io/previous_members/#andrea-robayo-gama. https://orcid.org/0009-0004-1529-0095. https://orcid.org/0000-0002-8725-1317. https://orcid.org/0000-0002-2910-7982. https://orcid.org/0009-0004-4484-6283. https://zbmed-semtec.github.io/previous_members/#muhammad-talha. https://orcid.org/0000-0002-1018-0370. https://orcid.org/0000-0003-3986-0510.

TREC-doc-2-doc-relevance assessment interface

Visit URL Visit SoftwareApplication

Identifier: DOI:10.5281/zenodo.7341391
Cite as: Talha M, Geist L, Fellerhof T, Ravinder R, Giraldo O, Rebholz-Schuhmann D, et al. TREC-doc-2-doc-relevance assessment interface. Zenodo; 2022. doi:10.5281/zenodo.7341391
Description: The code, data and docs at this release aim at facilitating the creation of a doc-2-doc relevance assessment on PMIDs used in the TREC 2005 Genomics track. A doc-2-doc relevance assessment takes one document as reference and assess a second document regarding its relevance to the reference one. This doc-2-doc collection will be used to evaluate the doc-2-doc recommendations approaches that we are working on.
URL: https://zenodo.org/records/7341391
Softwareversion: 1.0.0
Datepublished: 2022-11-21
License: http://spdx.org/licenses/MIT
Authors: https://zbmed-semtec.github.io/previous_members/#muhammad-talha. https://orcid.org/0000-0002-2910-7982. https://orcid.org/0000-0002-8725-1317. https://orcid.org/0009-0004-4484-6283. https://orcid.org/0000-0003-2978-8922. https://orcid.org/0000-0002-1018-0370. https://orcid.org/0000-0003-3986-0510.

TREC-doc-2-doc-relevance

Visit URL Visit SoftwareSourceCode

Cite as: Talha M, Geist L, Fellerhof T, Ravinder R, Giraldo O, Rebholz-Schuhmann D, et al. TREC-doc-2-doc-relevance [Software source code]. GitHub; 2022.
Description: This is the software source code facilitating the creation of a doc-2-doc relevance assessment on PMIDs used in the TREC 2005 Genomics track along with its metadata.
URL: https://github.com/zbmed-semtec/TREC-doc-2-doc-relevance
Release: https://doi.org/10.5281/zenodo.7341391
License: http://spdx.org/licenses/MIT
Authors: https://zbmed-semtec.github.io/previous_members/#muhammad-talha. https://orcid.org/0000-0002-2910-7982. https://orcid.org/0000-0002-8725-1317. https://orcid.org/0009-0004-4484-6283. https://orcid.org/0000-0003-2978-8922. https://orcid.org/0000-0002-1018-0370. https://orcid.org/0000-0003-3986-0510.

Fleiss kappa for doc-2-doc relevance assessment

Visit URL Visit Dataset

Conformsto: https://bioschemas.org/profiles/Dataset/1.1-DRAFT
Identifier: DOI:10.5281/zenodo.7338056
Cite as: Giraldo O, Solanki D, Rebholz-Schuhmann D, Castro LJ. Fleiss kappa for doc-2-doc relevance assessment. Zenodo; 2022. doi:10.5281/zenodo.7338056
Description: Fleiss' kappa measuring inter-annotator agreement on a document-to-document relevance assessment task. The table contains 7 columns, the first one presents the topics, 8 in total. The second column shows the “reference articles”, represented by their PubMed-ID and organized by topic. The third column shows the Fleiss’ Kappa results. The fourth column shows the interpretation of the Fleiss' Kappa results being: i) “Poor” results <0.20, ii) “Fair” results within 0.21 - 0.40, and iii) “Moderate” results within 0.41 - 0.60. The fifth column shows the PubMed-IDs of evaluation articles rated by the four annotators as “Relevant” regarding its corresponding “reference article”. The sixth column shows the PubMed-IDs of evaluation articles rated by the four annotators as “Partially relevant” regarding its corresponding “reference article”. The seventh column shows the PubMed-IDs of evaluation articles rated by the four annotators as “Non-relevant” regarding its corresponding “reference article”
Keywords: Fleiss' Kappa, Inter-annoator agreement, TREC Genomics Track 2005, relevance assessment
License: http://spdx.org/licenses/CC-BY-4.0
URL: https://zenodo.org/record/7338056
Datepublished: 2022-11-19
Authors: https://orcid.org/0000-0003-2978-8922. https://orcid.org/0009-0004-1529-0095. https://orcid.org/0000-0002-1018-0370. https://orcid.org/0000-0003-3986-0510.

Document-to-document relevant assessment for TREC Genomics Track 2005

Visit URL Visit Dataset

Identifier: DOI:10.5281/zenodo.7324822
Cite as: Giraldo O, Solanki D, Cadena F, Robayo-Gama A, Rebholz-Schuhmann D, Castro LJ. Document-to-document relevant assessment for TREC Genomics Track 2005. Zenodo; 2022. doi:10.5281/zenodo.7324822
Description: A CSV table with document-to-document relevance assessment judgements on a subset of the TREC Genomics Track 2005 produced by four annotators. The 'raw data document evaluation' contains six columns, first row consecutive id, second original TREC topic, third PubMed Id used as reference document, fourth PMID used to evaluate the relevance wrt the reference document, fifth the relevance score (2 definitely relevant, 1 partially relevant, 0 non-relevant), and sixth annotator id
Keywords: Document-to-document relevance, TREC GEnomics Track 2005, relevance assessment
License: http://spdx.org/licenses/CC-BY-4.0
URL: https://zenodo.org/record/7324822
Datepublished: 2022-11-15
Authors: https://orcid.org/0000-0003-2978-8922. https://orcid.org/0009-0004-1529-0095. https://orcid.org/0000-0002-5915-8895. https://zbmed-semtec.github.io/previous_members/#andrea-robayo-gama. https://orcid.org/0000-0002-1018-0370. https://orcid.org/0000-0003-3986-0510.