Home / Current Issue / Paper 1725280
RAG-Based Legal Document Question Answering using Large Language Models.
Subject area: Science,Engineering and Technology · Area of research: Artificial Intelligence, NLP and RAG
DOI: 10.64388/ire1725280
Abstract
The recent rapid development of LLMs (Large Language Models) has brought new opportunities to the intelligent analysis and question answering about legal documents. At the same time, the use of conventional LLMs could result in inaccurate answers because of the lack of knowledge completeness, freshness, and relevance to official sources. The issue of inadequate answers provided by LLMs can be solved with the help of RAG (Retrieval-Augmented Generation), which is capable of retrieving relevant documents from an external collection and passing this data as context information to the LLM during the answer generation process. This paper proposes the RAG framework for question answering about legal documents, including legal documents preprocessing, structure-aware text chunking, semantic retrieval, and answer generation with the use of LLM. The proposed approach is designed for retrieving relevant legal provisions, clauses, and cases passages and generating answers based on this evidence. The paper studies the framework through the lens of the following metrics: retrieval accuracy, answer relevance, factual faithfulness, citation correctness, and hallucination rate. The comparative analysis with conventional vector-based RAG and LLM-based question answering will be performed to test the effectiveness of the proposed framework.
Keywords
evidence grounding, hallucination reduction, information retrieval, large language models, legal artificial intelligence, legal document analysis, legal question answering, natural language processing, retrieval-augmented generation, semantic retrieval
References
[1] Lewis P, Perez E, Piktus A, Petroni F, Karpukhin V, Goyal N, Kuttler H, Lewis M, Yih WT, Rocktäschel T, Riedel S, Kiela D. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. In: Advances in Neural Information Processing Systems, vol. 33. 2020.
[2] Karpukhin V, Oğuz B, Min S, Lewis P, Wu L, Edunov S, Chen D, Yih WT. Dense Passage Retrieval for Open-Domain Question Answering. In: Proc. 2020 Conf. Empirical Methods in Natural Language Processing (EMNLP). 2020:6769–6781.
[3] Chalkidis I, Jana A, Hartung D, Bommarito M, Androutsopoulos I, Katz DM, Aletras N. LexGLUE: A Benchmark Dataset for Legal Language Understanding in English. In: Proc. 60th Annual Meeting of the Association for Computational Linguistics (ACL). 2022:4310–4330.
[4] Guha N, Nyarko J, Ho D, Ré C, Chilton A, Choi A, et al. LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models. In: Advances in Neural Information Processing Systems, vol. 36. 2023.
[5] Huang Q, et al. Lawyer LLaMA Technical Report. arXiv preprint arXiv:2305.15062. 2023.
[6] Yue S, et al. DISC-LawLLM: Fine-tuning Large Language Models for Intelligent Legal Services. arXiv preprint arXiv:2309.11325. 2023.
[7] Mavi V, Saparov A, Zhao C. Retrieval-Augmented Chain-of-Thought in Semi-Structured Domains. In: Proc. Natural Legal Language Processing Workshop. 2023:178–191.
[8] Ryu C, Lee S, Pang S, Choi C, Choi H, Min M, Sohn JY. Retrieval-based Evaluation for LLMs: A Case Study in Korean Legal QA. In: Proc. Natural Legal Language Processing Workshop. 2023.
[9] Shui R, Cao Y, Wang X, Chua TS. A Comprehensive Evaluation of Large Language Models on Legal Judgment Prediction. In: Findings of the Association for Computational Linguistics: EMNLP. 2023:7337–7348.
[10] Wiratunga N, Abeyratne R, Jayawardena L, Martin K, Massie S, Nkisi-Orji I, Weerasinghe R, Liret A, Fleisch B. CBR-RAG: Case-Based Reasoning for Retrieval Augmented Generation in LLMs for Legal Question Answering. arXiv preprint arXiv:2404.04302. 2024.
[11] Pipitone N, Houir Alami G. LegalBench-RAG: A Benchmark for Retrieval-Augmented Generation in the Legal Domain. arXiv preprint arXiv:2408.10343. 2024.
[12] Kalra R, Wu Z, Gulley A, Hilliard A, Guan X, Koshiyama A, Treleaven PC. HyPA-RAG: A Hybrid Parameter Adaptive Retrieval-Augmented Generation System for AI Legal and Policy Applications. In: Proc. NAACL-HLT Industry Track. 2025:1036–1054.
[13] Wiggers G, Hartz C. LeReRAG: Measuring Legal Relevance in Retrieval Augmented Generation Applications. In: Legal Knowledge and Information Systems. 2024.
[14] Redelaar F, van Drie R, Verberne S, de Boer M. Attributed Question Answering for Preconditions in the Dutch Law. In: Proc. Natural Legal Language Processing Workshop. 2024.
[15] Magesh V, Surani F, Dahl M, Suzgun M, Manning CD, Ho DE. Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools. Journal of Empirical Legal Studies. 2025;22(2):216–242.
[16] Hou AB, Weller O, Qin G, Yang E, Lawrie D, Holzenberger N, Blair-Stanek A, Van Durme B. CLERC: A Dataset for U.S. Legal Case Retrieval and Retrieval-Augmented Analysis Generation. In: Findings of NAACL. 2025:7913–7928.
[17] Kim D, Kang D, Kim J, Ryu S, Lee G. GuRE: Generative Query Rewriter for Legal Passage Retrieval. In: Proc. Natural Legal Language Processing Workshop. 2025.
[18] Reuter M, Lingenberg T, Liepina R, Lagioia F, Lippi M, Sartor G, Passerini A, Sayin B. Towards Reliable Retrieval in RAG Systems for Large Legal Datasets. In: Proc. Natural Legal Language Processing Workshop. 2025:17–30.
[19] Li A, Wu Y, Liu Y, Cai M, Qing L, Wang S, Kang Y, Liu C, Wu F, Kuang K. UniLR: Unleashing the Power of LLMs on Multiple Legal Tasks with a Unified Legal Retriever. In: Proc. 63rd Annual Meeting of the Association for Computational Linguistics (ACL). 2025:11953–11967.
[20] Nigam SK, Mishra SK, Mishra AK, Shallum N, Bhattacharya A. Legal Question-Answering in the Indian Context: Efficacy, Challenges, and Potential of Modern AI Models. arXiv preprint arXiv:2309.14735. 2023.
How to cite this paper
@article{1725280,
author = {Gautam Kumar, Manveer Singh, Paras Bansal, Yash Jaiswal},
title = {RAG-Based Legal Document Question Answering using Large Language Models.},
journal = {Iconic Research And Engineering Journals},
year = {2026},
volume = {10},
number = {5},
pages = {3796-3808},
issn = {2456-8880},
url = {https://www.irejournals.com/formatedpaper/1725280.pdf},
abstract = {The recent rapid development of LLMs (Large Language Models) has brought new opportunities to the intelligent analysis and question answering about legal documents. At the same time, the use of conventional LLMs could result in inaccurate answers because of the lack of knowledge completeness, freshness, and relevance to official sources. The issue of inadequate answers provided by LLMs can be solved with the help of RAG (Retrieval-Augmented Generation), which is capable of retrieving relevant documents from an external collection and passing this data as context information to the LLM during the answer generation process. This paper proposes the RAG framework for question answering about legal documents, including legal documents preprocessing, structure-aware text chunking, semantic retrieval, and answer generation with the use of LLM. The proposed approach is designed for retrieving relevant legal provisions, clauses, and cases passages and generating answers based on this evidence. The paper studies the framework through the lens of the following metrics: retrieval accuracy, answer relevance, factual faithfulness, citation correctness, and hallucination rate. The comparative analysis with conventional vector-based RAG and LLM-based question answering will be performed to test the effectiveness of the proposed framework.},
keywords = {evidence grounding, hallucination reduction, information retrieval, large language models, legal artificial intelligence, legal document analysis, legal question answering, natural language processing, retrieval-augmented generation, semantic retrieval},
month = {November},
doi = {https://doi.org/10.64388/ire1725280}
}