Bitte benutzen Sie diese Kennung, um auf die Ressource zu verweisen:
https://doi.org/10.21256/zhaw-29351| Publikationstyp: | Konferenz: Paper |
| Art der Begutachtung: | Peer review (Publikation) |
| Titel: | Spider4SPARQL : a complex benchmark for evaluating knowledge graph question answering systems |
| Autor/-in: | Kosten, Catherine Cudré-Mauroux, Philippe Stockinger, Kurt |
| et. al: | No |
| DOI: | 10.1109/BigData59044.2023.10386182 10.21256/zhaw-29351 |
| Tagungsband: | 2023 IEEE International Conference on Big Data (BigData) |
| Angaben zur Konferenz: | IEEE International Conference on Big Data, Sorrento, Italy, 15-18 December 2023 |
| Erscheinungsdatum: | 22-Jan-2024 |
| Verlag / Hrsg. Institution: | IEEE |
| ISBN: | 979-8-3503-2445-7 |
| Sprache: | Englisch |
| Schlagwörter: | Knowledge graph; Machine learning; Natural language processing; Benchmark |
| Fachgebiet (DDC): | 005: Computerprogrammierung, Programme und Daten 006: Spezielle Computerverfahren |
| Zusammenfassung: | With the recent spike in the number and availability of Large Language Models (LLMs), it has become increasingly important to provide large and realistic benchmarks for evaluating Knowledge Graph Question Answering (KBQA) systems. So far the majority of benchmarks rely on pattern-based SPARQL query generation approaches. The subsequent natural language (NL) question generation is conducted through crowdsourcing or other automated methods, such as rule-based paraphrasing or NL question templates. Although some of these datasets are of considerable size, their pitfall lies in their pattern-based generation approaches, which do not always generalize well to the vague and linguistically diverse questions asked by humans in real-world contexts. In this paper, we introduce Spider4SPARQL - a new SPARQL benchmark dataset featuring 9,693 previously existing manually generated NL questions and 4,721 unique, novel, and complex SPARQL queries of varying complexity. In addition to the NL/SPARQL pairs, we also provide their corresponding 166 knowledge graphs and ontologies, which cover 138 different domains. Our complex benchmark enables novel ways of evaluating the strengths and weaknesses of modern KGQA systems. We evaluate the system with state-of-the-art KGQA systems as well as LLMs, which achieve only up to 45% execution accuracy, demonstrating that Spider4SPARQL is a challenging benchmark for future research. |
| URI: | https://digitalcollection.zhaw.ch/handle/11475/29351 |
| Volltext Version: | Akzeptierte Version |
| Lizenz (gemäss Verlagsvertrag): | Lizenz gemäss Verlagsvertrag |
| Departement: | School of Engineering |
| Organisationseinheit: | Institut für Informatik (InIT) |
| Publiziert im Rahmen des ZHAW-Projekts: | INODE – Intelligent Open Data Exploration (EU Horizon 2020) |
| Enthalten in den Sammlungen: | Publikationen School of Engineering |
Dateien zu dieser Ressource:
| Datei | Beschreibung | Größe | Format | |
|---|---|---|---|---|
| 2023_Kosten-etal_Spider4SPARQL_IEEE_BigData_2023.pdf | Accepted Version | 317.49 kB | Adobe PDF |  Öffnen/Anzeigen |
Zur Langanzeige
Kosten, C., Cudré-Mauroux, P., & Stockinger, K. (2024, January 22). Spider4SPARQL : a complex benchmark for evaluating knowledge graph question answering systems. 2023 IEEE International Conference on Big Data (BigData). https://doi.org/10.1109/BigData59044.2023.10386182
Kosten, C., Cudré-Mauroux, P. and Stockinger, K. (2024) ‘Spider4SPARQL : a complex benchmark for evaluating knowledge graph question answering systems’, in 2023 IEEE International Conference on Big Data (BigData). IEEE. Available at: https://doi.org/10.1109/BigData59044.2023.10386182.
C. Kosten, P. Cudré-Mauroux, and K. Stockinger, “Spider4SPARQL : a complex benchmark for evaluating knowledge graph question answering systems,” in 2023 IEEE International Conference on Big Data (BigData), Jan. 2024. doi: 10.1109/BigData59044.2023.10386182.
KOSTEN, Catherine, Philippe CUDRÉ-MAUROUX und Kurt STOCKINGER, 2024. Spider4SPARQL : a complex benchmark for evaluating knowledge graph question answering systems. In: 2023 IEEE International Conference on Big Data (BigData). Conference paper. IEEE. 22 Januar 2024. ISBN 979-8-3503-2445-7
Kosten, Catherine, Philippe Cudré-Mauroux, and Kurt Stockinger. 2024. “Spider4SPARQL : A Complex Benchmark for Evaluating Knowledge Graph Question Answering Systems.” Conference paper. In 2023 IEEE International Conference on Big Data (BigData). IEEE. https://doi.org/10.1109/BigData59044.2023.10386182.
Kosten, Catherine, et al. “Spider4SPARQL : A Complex Benchmark for Evaluating Knowledge Graph Question Answering Systems.” 2023 IEEE International Conference on Big Data (BigData), IEEE, 2024, https://doi.org/10.1109/BigData59044.2023.10386182.
Alle Ressourcen in diesem Repository sind urheberrechtlich geschützt, soweit nicht anderweitig angezeigt.