Natural Language Querying of DUCKDB using LLM

References

[1] A. Vaswani et al., “Attention is all you need,” Advances in Neural Information Processing Systems, vol. 30, 2017.

[2] P. Lewis et al., “Retrieval-augmented generation for knowledge-intensive NLP tasks,” Advances in Neural Information Processing Systems, vol. 33, pp. 9459–9474, 2020.

[3] H. Touvron et al., “LLaMA 2: Open foundation and fine-tuned chat models,” arXiv preprint arXiv:2307.09288, 2023.

[4] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” arXiv:1810.04805, 2018.

[5] T. Brown et al., “Language models are few-shot learners,” Advances in Neural Information Processing Systems, vol. 33, pp. 1877–1901, 2020.

[6] D. Kakwani et al., “IndicNLPSuite: Monolingual corpora and evaluation benchmarks for Indian languages,” EMNLP Findings, pp. 4948–4961, 2020.

[7] A. Kulkarni and A. Shivananda, Natural Language Processing Recipes, Apress, 2019.

[8] D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” arXiv:1409.0473, 2014.

[9] I. Sutskever, O. Vinyals, and Q. Le, “Sequence-to-sequence learning with neural networks,” Advances in Neural Information Processing Systems, vol. 27, 2014.

[10] T. Dettmers et al., “QLoRA: Efficient fine-tuning of quantized LLMs,” arXiv:2305.14314, 2023.

[11] S. Bird, E. Klein, and E. Loper, Natural Language Processing with Python, O’Reilly, 2009.

[12] M. Honnibal and I. Montani, “spaCy 2: Natural language understanding with CNNs,” 2017.

[13] N. Reimers and I. Gurevych, “Sentence-BERT: Sentence embeddings using Siamese networks,” arXiv:1908.10084, 2019.

[14] P. Bojanowski, E. Grave, A. Joulin, and T. Mikolov, “Enriching word vectors with subword information,” TACL, vol. 5, pp. 135–146, 2017.

[15] T. Mikolov et al., “Efficient estimation of word representations in vector space,” arXiv:1301.3781, 2013.

[16] J. Pennington, R. Socher, and C. D. Manning, “GloVe: Global vectors for word representation,” EMNLP, 2014.

[17] L. Xue et al., “mT5: A multilingual pre-trained text-to-text transformer,” arXiv:2010.11934, 2020.

[18] A. Conneau et al., “Unsupervised cross-lingual representation learning at scale,” arXiv:1911.02116, 2019.

[19] S. Doddapaneni et al., “A primer on pretrained multilingual language models,” arXiv:2107.00676, 2021.

[20] J. Pfeiffer et al., “AdapterHub: A framework for adapting Transformers,” arXiv:2007.07779, 2020.

[21] E. J. Hu et al., “LoRA: Low-Rank adaptation of LLMs,” arXiv:2106.09685, 2021.

[22] T. Wolf et al., “Transformers: State-of-the-art NLP,” EMNLP System Demonstrations, 2020.

[23] A. Paszke et al., “PyTorch: An imperative style high-performance deep learning library,” NeurIPS, 2019.

[24] M. Abadi et al., “TensorFlow: Large-scale machine learning,” OSDI, 2016.

[25] F. Pedregosa et al., “Scikit-learn: Machine learning in Python,” Journal of Machine Learning Research, vol. 12, pp. 2825–2830, 2011.

[26] W. McKinney, “Data structures for statistical computing in Python,” SciPy Conference, pp. 51–56, 2010.

[27] C. R. Harris et al., “Array programming with NumPy,” Nature, vol. 585, pp. 357–362, 2020.

[28] M. Grinberg, Flask Web Development, O’Reilly Media, 2018.

[29] A. Ronacher, “Werkzeug,” Pallets Projects, 2020.

[30] M. Raasveldt and H. Mühleisen, “DuckDB: An embeddable analytical database,” ACM SIGMOD, 2020

@article{1716470,
      author = {Poojitha K D, Chandana M Pallegar, G K Harshitha, Rakshitha B M, Padmapriya H N},
      title = {Natural Language Querying of DUCKDB using LLM},
      journal = {Iconic Research And Engineering Journals},
      year = {2026},
      volume = {9},
      number = {10},
      pages = {2596-2622},
      issn = {2456-8880},
      url = {https://www.irejournals.com/formatedpaper/1716470.pdf},
      abstract = {Many users work with CSV files to store data, but analyzing this data usually requires knowledge of SQL, which can be difficult for non-technical users. To overcome this problem, this project presents a web-based application that allows users to interact with their data using simple English queries instead of writing SQL commands. The system converts natural language input into SQL queries, making data analysis faster and easier without requiring any programming skills. The proposed system runs completely offline and is developed using the Flask web framework along with a locally hosted pre-trained Large Language Model. This approach ensures data privacy and avoids dependency on cloud-based services. When a CSV file is uploaded, it is processed using Pandas to clean column names, identify data types, and dynamically generate a relational structure inside DuckDB, an in-memory analytical database. The user’s query is combined with the dataset schema and processed by the language model to generate accurate SQL statements. These queries are validated to prevent unsafe operations and are then executed on the dataset, with results returned in a structured JSON format. The application provides a simple and interactive user interface built using HTML, CSS, and JavaScript, supporting drag-and-drop file uploads, data preview, and real-time query responses. This project demonstrates an efficient and secure solution for querying structured data using natural language, making it useful for academic, business, and inventory-related datasets. Future improvements may include support for multiple datasets, graphical data visualization, and enhanced model optimization for handling more complex queries.},
      month = {April},
      doi = {https://doi.org/10.64388/IREV9I10-1716470}
  }