Home / Current Issue / Paper 1724784
Agent Based Framework For Automated Data Cleaning And Validation
Subject area: Science,Engineering and Technology · Area of research: Artificial Intelligence, Data Pre-processing
DOI: 10.64388/ire1724784
Abstract
Real-world datasets can contain missing values, duplicate records, inconsistent data, outliers, and invalid entries. Manual cleaning is time-consuming, and the result must be checked to determine whether it satisfies data-quality requirements. This paper presents an agent-based framework for automated data cleaning, validation, and merging implemented using Python, NumPy, and Pandas, with Flask and scikit-learn used in the prototype application. Different agents handle schema profiling, issue detection, reasoning, cleaning, and validation. The system detects candidate problems using type inference, pattern matching, hash matching, and IQR-based outlier detection. A Reasoning Agent proposes possible cleaning actions, which are presented to the user for approval or rejection before accepted transformations are applied. The Validation Agent then checks the output against configured quality rules. If validation identifies remaining problems, the issue can be returned to the reasoning stage for a further proposed correction and user decision. The framework also supports merging two datasets using schema matching. Screenshots from a customer-data demonstration show a reduction in reported missing values and duplicate rows, while the validation dashboard still reports failed checks. These observations describe prototype interface output rather than a controlled benchmark; they do not establish general cleaning accuracy or performance. The framework aims to reduce repetitive manual work while retaining human oversight and an explicit validation step before data are used for analysis or machine learning.
Keywords
agent-based framework; automated data cleaning; data validation; data merging; data quality; missing values; duplicate records; outlier detection; invalid data; type inference; pattern matching; hash matching; IQR; reasoning agent; validation agent; schema matching; NumPy; pandas; human-in-the-loop
References
[1] Rahm E, Do HH. Data cleaning: Problems and current approaches. IEEE Data Engineering Bulletin. 2000;23(4):3–13.
[2] Krishnan S, Wu E. AlphaClean: Automatic generation of data cleaning pipelines. arXiv:1904.11827. 2019.
[3] Qi D, Miao Z, Wang J. CleanAgent: Automating data standardization with LLM-based agents. In: Proc. VLDB 2025 Workshop on Data Driven AI (DDAI); 2025.
[4] Dong S, Sahri S, Palpanas T, Wang Q. Automated data quality validation in an end-to-end GNN framework. In: Proc. 28th International Conference on Extending Database Technology (EDBT); 2025. p. 874–880. https://doi.org/10.48786/EDBT.2025.71
[5] Li L, Fang L, Ludäscher B, Torvik VI. AutoDCWorkflow: LLM-based data cleaning workflow auto-generation and benchmark. In: Findings of the Association for Computational Linguistics: EMNLP 2025; 2025. p. 7766–7780. https://doi.org/10.18653/v1/2025.findings-emnlp.410
[6] Yasin HM, Khorsheed AK. Automated data cleaning in large databases using machine learning methods. Asian Journal of Research in Computer Science. 2025;18(5):364–386. https://doi.org/10.9734/ajrcos/2025/v18i5661
[7] Rekatsinas T, Chu X, Ilyas IF, Ré C. HoloClean: Holistic data repairs with probabilistic inference. Proc VLDB Endowment. 2017;10(11):1190–1201. https://doi.org/10.14778/3137628.3137631
[8] Chu X, Ilyas IF, Krishnan S, Wang J. Data cleaning: Overview and emerging challenges. In: Proc. ACM SIGMOD International Conference on Management of Data; 2016. p. 2201–2206. https://doi.org/10.1145/2882903.2912574
How to cite this paper
@article{1724784,
author = {Gnanashri J, Manya M N, Sakshi C Patil, Likitha B},
title = {Agent Based Framework For Automated Data Cleaning And Validation},
journal = {Iconic Research And Engineering Journals},
year = {2026},
volume = {10},
number = {5},
pages = {1736-1750},
issn = {2456-8880},
url = {https://www.irejournals.com/formatedpaper/1724784.pdf},
abstract = {Real-world datasets can contain missing values, duplicate records, inconsistent data, outliers, and invalid entries. Manual cleaning is time-consuming, and the result must be checked to determine whether it satisfies data-quality requirements. This paper presents an agent-based framework for automated data cleaning, validation, and merging implemented using Python, NumPy, and Pandas, with Flask and scikit-learn used in the prototype application. Different agents handle schema profiling, issue detection, reasoning, cleaning, and validation. The system detects candidate problems using type inference, pattern matching, hash matching, and IQR-based outlier detection. A Reasoning Agent proposes possible cleaning actions, which are presented to the user for approval or rejection before accepted transformations are applied. The Validation Agent then checks the output against configured quality rules. If validation identifies remaining problems, the issue can be returned to the reasoning stage for a further proposed correction and user decision. The framework also supports merging two datasets using schema matching. Screenshots from a customer-data demonstration show a reduction in reported missing values and duplicate rows, while the validation dashboard still reports failed checks. These observations describe prototype interface output rather than a controlled benchmark; they do not establish general cleaning accuracy or performance. The framework aims to reduce repetitive manual work while retaining human oversight and an explicit validation step before data are used for analysis or machine learning.},
keywords = {agent-based framework; automated data cleaning; data validation; data merging; data quality; missing values; duplicate records; outlier detection; invalid data; type inference; pattern matching; hash matching; IQR; reasoning agent; validation agent; schema matching; NumPy; pandas; human-in-the-loop},
month = {November},
doi = {https://doi.org/10.64388/ire1724784}
}