Home / Current Issue / Paper 1724942
Smart Data Quality and Auto Healing Pipeline
Subject area: Science,Engineering and Technology · Area of research: Data Engineering
DOI: 10.64388/ire1724942
Abstract
The Smart Data Quality and Auto-Healing Pipeline is an automated, configuration-driven system developed to improve the reliability and usability of structured data, particularly CSV datasets. Data may contain missing values, duplicate records, invalid formats, datatype inconsistencies and unusual observations, making manual cleaning time-consuming and difficult to scale. The pipeline combines ingestion, schema validation, profiling, rule-based validation, optional anomaly detection, safe automatic healing, quality scoring, audit metadata and reporting. YAML configuration separates dataset-specific rules from implementation code. The system validates data before and after healing and records unresolved issues rather than claiming every problem has been fixed. FastAPI exposes pipeline services and cleaned-data downloads, while a Streamlit dashboard presents quality metrics and healing outcomes. In the evaluated sample, the overall quality score increased from 77.02 to 94.13 with 61 automated tests passing.
Keywords
data quality, data cleaning, auto-healing, CSV, data profiling, schema validation, Isolation Forest, FastAPI, Streamlit, YAML
References
[1] Rahm E, Do HH. Data cleaning: problems and current approaches. IEEE Data Engineering Bulletin. 2000;23(4):3–13.
[2] Chu X, Ilyas IF, Krishnan S, Wang J. Data cleaning: overview and emerging challenges. In: Proceedings of the 2016 ACM SIGMOD International Conference on Management of Data; 2016. p. 2201–2206.
[3] Liu FT, Ting KM, Zhou ZH. Isolation forest. In: Proceedings of the 2008 Eighth IEEE International Conference on Data Mining; 2008. p. 413–422.
[4] McKinney W. Python for data analysis. 3rd ed. Sebastopol (CA): O'Reilly Media; 2022.
[5] Python Software Foundation. Python 3 documentation. Available from: https://docs.python.org/3/
[6] pandas development team. pandas: Python data analysis library. Documentation. Available from: https://pandas.pydata.org/docs/
[7] FastAPI. FastAPI: modern, fast web framework for building APIs with Python. Documentation. Available from: https://fastapi.tiangolo.com/
[8] Streamlit. Streamlit: a faster way to build and share data apps. Documentation. Available from: https://docs.streamlit.io/
[9] scikit-learn developers. IsolationForest. scikit-learn documentation. Available from: https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.IsolationForest.html
How to cite this paper
@article{1724942,
author = {Sure Mamatha, Anvitha Bandi, Swathi Srividya, Akhila Thirichanur, Akshat Tumkur},
title = {Smart Data Quality and Auto Healing Pipeline},
journal = {Iconic Research And Engineering Journals},
year = {2026},
volume = {10},
number = {5},
pages = {1989-1995},
issn = {2456-8880},
url = {https://www.irejournals.com/formatedpaper/1724942.pdf},
abstract = {The Smart Data Quality and Auto-Healing Pipeline is an automated, configuration-driven system developed to improve the reliability and usability of structured data, particularly CSV datasets. Data may contain missing values, duplicate records, invalid formats, datatype inconsistencies and unusual observations, making manual cleaning time-consuming and difficult to scale. The pipeline combines ingestion, schema validation, profiling, rule-based validation, optional anomaly detection, safe automatic healing, quality scoring, audit metadata and reporting. YAML configuration separates dataset-specific rules from implementation code. The system validates data before and after healing and records unresolved issues rather than claiming every problem has been fixed. FastAPI exposes pipeline services and cleaned-data downloads, while a Streamlit dashboard presents quality metrics and healing outcomes. In the evaluated sample, the overall quality score increased from 77.02 to 94.13 with 61 automated tests passing.},
keywords = {data quality, data cleaning, auto-healing, CSV, data profiling, schema validation, Isolation Forest, FastAPI, Streamlit, YAML},
month = {November},
doi = {https://doi.org/10.64388/ire1724942}
}