Bitte benutzen Sie diese Kennung, um auf die Ressource zu verweisen:
https://doi.org/10.21256/zhaw-21229| Publikationstyp: | Konferenz: Paper |
| Art der Begutachtung: | Peer review (Publikation) |
| Titel: | Spot The Bot : a robust and efficient framework for the evaluation of conversational dialogue systems |
| Autor/-in: | Deriu, Jan Milan Tuggener, Don von Däniken, Pius Campos, Jon Ander Rodrigo, Alvaro Belkacem, Thiziri Soroa, Aitor Agirre, Eneko Cieliebak, Mark |
| et. al: | No |
| DOI: | 10.18653/v1/2020.emnlp-main.326 10.21256/zhaw-21229 |
| Tagungsband: | Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) |
| Seite(n): | 3971 |
| Seiten bis: | 3984 |
| Angaben zur Konferenz: | Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, 16-20 November 2020 |
| Erscheinungsdatum: | Nov-2020 |
| Verlag / Hrsg. Institution: | Association for Computational Linguistics |
| Sprache: | Englisch |
| Schlagwörter: | Natural language processing; Chatbot; Dialogue system |
| Fachgebiet (DDC): | 410.285: Computerlinguistik |
| Zusammenfassung: | The lack of time efficient and reliable evaluation methods is hampering the development of conversational dialogue systems (chat bots). Evaluations that require humans to converse with chat bots are time and cost intensive, put high cognitive demands on the human judges, and tend to yield low quality results. In this work, we introduce Spot The Bot, a cost-efficient and robust evaluation framework that replaces human-bot conversations with conversations between bots. Human judges then only annotate for each entity in a conversation whether they think it is human or not (assuming there are humans participants in these conversations). These annotations then allow us to rank chat bots regarding their ability to mimic conversational behaviour of humans. Since we expect that all bots are eventually recognized as such, we incorporate a metric that measures which chat bot is able to uphold human-like behavior the longest, i.e.Survival Analysis. This metric has the ability to correlate a bot’s performance to certain of its characteristics (e.g.fluency or sensibleness), yielding interpretable results. The comparably low cost of our framework allows for frequent evaluations of chatbots during their evaluation cycle. We empirically validate our claims by applying Spot The Bot to three domains, evaluating several state-of-the-art chat bots, and drawing comparisonsto related work. The framework is released asa ready-to-use tool. |
| URI: | https://digitalcollection.zhaw.ch/handle/11475/21229 |
| Volltext Version: | Publizierte Version |
| Lizenz (gemäss Verlagsvertrag): | CC BY 4.0: Namensnennung 4.0 International |
| Departement: | School of Engineering |
| Organisationseinheit: | Institut für Informatik (InIT) |
| Publiziert im Rahmen des ZHAW-Projekts: | LIHLITH - Learning to Interact with Humans by Lifelong Interaction with Humans |
| Enthalten in den Sammlungen: | Publikationen School of Engineering |
Dateien zu dieser Ressource:
| Datei | Beschreibung | Größe | Format | |
|---|---|---|---|---|
| 2020_Deriu-etal_Spot-The-Bot_EMNLP.pdf | 871.97 kB | Adobe PDF |  Öffnen/Anzeigen |
Zur Langanzeige
Deriu, J. M., Tuggener, D., von Däniken, P., Campos, J. A., Rodrigo, A., Belkacem, T., Soroa, A., Agirre, E., & Cieliebak, M. (2020). Spot The Bot : a robust and efficient framework for the evaluation of conversational dialogue systems [Conference paper]. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 3971–3984. https://doi.org/10.18653/v1/2020.emnlp-main.326
Deriu, J.M. et al. (2020) ‘Spot The Bot : a robust and efficient framework for the evaluation of conversational dialogue systems’, in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, pp. 3971–3984. Available at: https://doi.org/10.18653/v1/2020.emnlp-main.326.
J. M. Deriu et al., “Spot The Bot : a robust and efficient framework for the evaluation of conversational dialogue systems,” in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Nov. 2020, pp. 3971–3984. doi: 10.18653/v1/2020.emnlp-main.326.
DERIU, Jan Milan, Don TUGGENER, Pius VON DÄNIKEN, Jon Ander CAMPOS, Alvaro RODRIGO, Thiziri BELKACEM, Aitor SOROA, Eneko AGIRRE und Mark CIELIEBAK, 2020. Spot The Bot : a robust and efficient framework for the evaluation of conversational dialogue systems. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Conference paper. Association for Computational Linguistics. November 2020. S. 3971–3984
Deriu, Jan Milan, Don Tuggener, Pius von Däniken, Jon Ander Campos, Alvaro Rodrigo, Thiziri Belkacem, Aitor Soroa, Eneko Agirre, and Mark Cieliebak. 2020. “Spot the Bot : A Robust and Efficient Framework for the Evaluation of Conversational Dialogue Systems.” Conference paper. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 3971–84. Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.emnlp-main.326.
Deriu, Jan Milan, et al. “Spot the Bot : A Robust and Efficient Framework for the Evaluation of Conversational Dialogue Systems.” Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Association for Computational Linguistics, 2020, pp. 3971–84, https://doi.org/10.18653/v1/2020.emnlp-main.326.
Alle Ressourcen in diesem Repository sind urheberrechtlich geschützt, soweit nicht anderweitig angezeigt.