ScienceBenchmark : a complex real-world benchmark for evaluating natural language to SQL systems

Zhang, Yi; Deriu, Jan Milan; Katsogiannis-Meimarakis, George; Kosten, Catherine; Koutrika, Georgia; Stockinger, Kurt

doi:10.14778/3636218.3636225

Please use this identifier to cite or link to this item: https://doi.org/10.21256/zhaw-30173

Publication type:	Conference paper
Type of review:	Peer review (publication)
Title:	ScienceBenchmark : a complex real-world benchmark for evaluating natural language to SQL systems
Authors:	Zhang, Yi Deriu, Jan Milan Katsogiannis-Meimarakis, George Kosten, Catherine Koutrika, Georgia Stockinger, Kurt
et. al:	No
DOI:	10.14778/3636218.3636225 10.21256/zhaw-30173
Proceedings:	Proceedings of the VLDB Endowment
Volume(Issue):	17
Issue:	4
Page(s):	685
Pages to:	698
Conference details:	50th International Conference on Very Large Data Bases, Guangzhou, China, 25-29 August 2024
Issue Date:	Mar-2024
Publisher / Ed. Institution:	Association for Computing Machinery
ISSN:	2150-8097
Language:	English
Subjects:	Database system; Latural language processing; Machine learning; Large language model
Subject (DDC):	005: Computer programming, programs and data 006: Special computer methods
Abstract:	Natural Language to SQL systems (NL-to-SQL) have recently shown improved accuracy (exceeding 80%) for natural language to SQL query translation due to the emergence of transformer-based language models, and the popularity of the Spider benchmark. However, Spider mainly contains simple databases with few tables, columns, and entries, which do not reflect a realistic setting. Moreover, complex real-world databases with domain-specific content have little to no training data available in the form of NL/SQL-pairs leading to poor performance of existing NL-to-SQL systems. In this paper, we introduce ScienceBenchmark, a new complex NL-to-SQL benchmark for three real-world, highly domain-specific databases. For this new benchmark, SQL experts and domain experts created high-quality NL/SQL-pairs for each domain. To garner more data, we extended the small amount of human-generated data with synthetic data generated using GPT-3. We show that our benchmark is highly challenging, as the top performing systems on Spider achieve a very low performance on our benchmark. Thus, the challenge is many-fold: creating NL-to-SQL systems for highly complex domains with a small amount of hand-made training data augmented with synthetic data. To our knowledge, ScienceBenchmark is the first NL-to-SQL benchmark designed with complex real-world scientific databases, containing challenging training and test data carefully validated by domain experts.
URI:	https://digitalcollection.zhaw.ch/handle/11475/30173
Fulltext version:	Published version
License (according to publishing contract):	CC BY-NC-ND 4.0: Attribution - Non commercial - No derivatives 4.0 International
Departement:	School of Engineering
Organisational Unit:	Centre for Artificial Intelligence (CAI) Institute of Computer Science (InIT)
Published as part of the ZHAW project:	INODE – Intelligent Open Data Exploration (EU Horizon 2020)
Appears in collections:	Publikationen School of Engineering

Files in This Item:

File	Description	Size	Format
2024_Zhang-etal_ScienceBenchmark-PVLDB2024.pdf		608.6 kB	Adobe PDF	View/Open

Show full item record

Zhang, Y., Deriu, J. M., Katsogiannis-Meimarakis, G., Kosten, C., Koutrika, G., & Stockinger, K. (2024). ScienceBenchmark : a complex real-world benchmark for evaluating natural language to SQL systems [Conference paper]. Proceedings of the VLDB Endowment, 17(4), 685–698. https://doi.org/10.14778/3636218.3636225

Zhang, Y. et al. (2024) ‘ScienceBenchmark : a complex real-world benchmark for evaluating natural language to SQL systems’, in Proceedings of the VLDB Endowment. Association for Computing Machinery, pp. 685–698. Available at: https://doi.org/10.14778/3636218.3636225.

Y. Zhang, J. M. Deriu, G. Katsogiannis-Meimarakis, C. Kosten, G. Koutrika, and K. Stockinger, “ScienceBenchmark : a complex real-world benchmark for evaluating natural language to SQL systems,” in Proceedings of the VLDB Endowment, Mar. 2024, vol. 17, no. 4, pp. 685–698. doi: 10.14778/3636218.3636225.

ZHANG, Yi, Jan Milan DERIU, George KATSOGIANNIS-MEIMARAKIS, Catherine KOSTEN, Georgia KOUTRIKA und Kurt STOCKINGER, 2024. ScienceBenchmark : a complex real-world benchmark for evaluating natural language to SQL systems. In: Proceedings of the VLDB Endowment. Conference paper. Association for Computing Machinery. März 2024. S. 685–698

Zhang, Yi, Jan Milan Deriu, George Katsogiannis-Meimarakis, Catherine Kosten, Georgia Koutrika, and Kurt Stockinger. 2024. “ScienceBenchmark : A Complex Real-World Benchmark for Evaluating Natural Language to SQL Systems.” Conference paper. In Proceedings of the VLDB Endowment, 17:685–98. Association for Computing Machinery. https://doi.org/10.14778/3636218.3636225.

Zhang, Yi, et al. “ScienceBenchmark : A Complex Real-World Benchmark for Evaluating Natural Language to SQL Systems.” Proceedings of the VLDB Endowment, vol. 17, no. 4, Association for Computing Machinery, 2024, pp. 685–98, https://doi.org/10.14778/3636218.3636225.