Title: Evaluation for operational IR applications – generalizability and automation
Authors : Imhof, Melanie
Braschler, Martin
Hansen, Preben
Rietberger, Stefan
Proceedings: LivingLab '13 Proceedings of the 2013 workshop on Living labs for information retrieval evaluation
Pages : 2557
Pages to: 2558
Conference details: Workshop on Living Labs for Information Retrieval Evaluation, San Francisco, November 1, 2013
Publisher / Ed. Institution : ACM
Publisher / Ed. Institution: New York
Issue Date: 2013
License (according to publishing contract) : Licence according to publishing contract
Type of review: Not specified
Language : English
Subject (DDC) : 020: Library and information sciences
Abstract: Black box information retrieval (IR) application evaluation allows practitioners to measure the quality of their IR application. Instead of evaluating specific components, e.g. solely the search engine, a complete IR application, including the user’s perspective, is evaluated. The evaluation methodology is designed to be applicable to operational IR applications. The black box evaluation methodology could be packaged into an evaluation and monitoring tool, making it usable for industry stakeholders. The tool should lead practitioners through the evaluation process and maintain the test results for the manual and automatic tests. This paper shows that the methodology is generalizable, even though the diversity of IR applications is high. The challenges in automating tests are the simulation of tasks that require intellectual effort and the handling of different visualizations of the same concept.
Departement: School of Engineering
Organisational Unit: Institute of Applied Information Technology (InIT)
Publication type: Conference Paper
DOI : 10.1145/2513150.2513160
ISBN: 978-1-4503-2420-5
URI: https://digitalcollection.zhaw.ch/handle/11475/4228
Appears in Collections:Publikationen School of Engineering

Files in This Item:
There are no files associated with this item.


Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.