Title: Are test collections “real”? : mirroring real-world complexity in IR test collections
Authors : Imhof, Melanie
Braschler, Martin
Proceedings: Experimental IR meets Multilinguality, Multimodality, and Interaction
Pages : 241
Pages to: 247
Conference details: 6th International Conference of the CLEF Association, CLEF’15, Toulouse, September 8–11, 2015
Editors of the parent work: Mothe, Josiane
Savoy, Jacques
Kamps, Jaap
Pinel-Sauvagnat, Karen
Jones, Gareth
San Juan, Eric
Capellato, Linda
Ferro, Nicola
Publisher / Ed. Institution : Springer
Publisher / Ed. Institution: Cham
Issue Date: 2015
License (according to publishing contract) : Licence according to publishing contract
Series : Lecture Notes in Computer Science
Series volume: 9283
Type of review: Not specified
Language : English
Subjects : IE; INIT; Datalab; Collection complexity; Modality categorization; Evaluation campaigns
Subject (DDC) : 020: Library and information sciences
Abstract: Objective evaluation of effectiveness is a major topic in the field of information retrieval (IR), as emphasized by the numerous evaluation campaigns in this area. The increasing pervasiveness of information has lead to a large variety of IR application scenarios that involve different information types (modalities), heterogeneous documents and context-enriched queries. In this paper, we argue that even though the complexity of academic test collections has increased over the years, they are still too structurally simple in comparison to operational collections in real-world applications. Furthermore, research has brought up retrieval methods for very specific modalities, such as ratings, geographical coordinates and timestamps. However, it is still unclear how to systematically incorporate new modalities in IR systems. We therefore propose a categorization of modalities that not only allows analyzing the complexity of a collection but also helps to generalize methods to entire modality categories instead of being specific for a single modality. Moreover, we discuss how such a complex collection can methodically be built for the usage in an evaluation campaign.
Departement: School of Engineering
Organisational Unit: Institute of Applied Information Technology (InIT)
Publication type: Conference Paper
DOI : 10.1007/978-3-319-24027-5_23
ISBN: 978-3-319-24027-5
URI: https://digitalcollection.zhaw.ch/handle/11475/4202
Appears in Collections:Publikationen School of Engineering

Files in This Item:
There are no files associated with this item.


Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.