1. ZHAW digitalcollection
2. School of Engineering
3. Publikationen
4. Publikationen School of Engineering

Publication type:	Conference paper
Type of review:	Peer review (publication)
Title:	Spot The Bot : a robust and efficient framework for the evaluation of conversational dialogue systems
Authors:	Deriu, Jan Milan Tuggener, Don von Däniken, Pius Campos, Jon Ander Rodrigo, Alvaro Belkacem, Thiziri Soroa, Aitor Agirre, Eneko Cieliebak, Mark
et. al:	No
DOI:	10.18653/v1/2020.emnlp-main.326 10.21256/zhaw-21229
Proceedings:	Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)
Pages:	3971
Pages to:	3984
Conference details:	Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, 16-20 November 2020
Issue Date:	Nov-2020
Publisher / Ed. Institution:	Association for Computational Linguistics
Language:	English
Subjects:	Natural language processing; Chatbot; Dialogue system
Subject (DDC):	410.285: Computational linguistics
Abstract:	The lack of time efficient and reliable evaluation methods is hampering the development of conversational dialogue systems (chat bots). Evaluations that require humans to converse with chat bots are time and cost intensive, put high cognitive demands on the human judges, and tend to yield low quality results. In this work, we introduce Spot The Bot, a cost-efficient and robust evaluation framework that replaces human-bot conversations with conversations between bots. Human judges then only annotate for each entity in a conversation whether they think it is human or not (assuming there are humans participants in these conversations). These annotations then allow us to rank chat bots regarding their ability to mimic conversational behaviour of humans. Since we expect that all bots are eventually recognized as such, we incorporate a metric that measures which chat bot is able to uphold human-like behavior the longest, i.e.Survival Analysis. This metric has the ability to correlate a bot’s performance to certain of its characteristics (e.g.fluency or sensibleness), yielding interpretable results. The comparably low cost of our framework allows for frequent evaluations of chatbots during their evaluation cycle. We empirically validate our claims by applying Spot The Bot to three domains, evaluating several state-of-the-art chat bots, and drawing comparisonsto related work. The framework is released asa ready-to-use tool.
URI:	https://digitalcollection.zhaw.ch/handle/11475/21229
Fulltext version:	Published version
License (according to publishing contract):	CC BY 4.0: Attribution 4.0 International
Departement:	School of Engineering
Organisational Unit:	Institute of Applied Information Technology (InIT)
Published as part of the ZHAW project:	LIHLITH - Learning to Interact with Humans by Lifelong Interaction with Humans
Appears in collections:	Publikationen School of Engineering

Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.