SmartClean-AI: Automated Data and Image Cleaning Libraries for Machine Learning Workflows

References

[1] Côté PO, Nikanjam A, Ahmed N, Humeniuk D, Khomh F. Data cleaning and machine learning: A systematic literature review. Automated Software Engineering. 2024;31(2):54. https://doi.org/10.1007/s10515-024-00453-w

[2] Ridzuan F, Wan Zainon WMN. A review on data cleansing methods for big data. Procedia Computer Science. 2019;161:731–738. https://doi.org/10.1016/j.procs.2019.11.177

[3] Haider SN, Zhao Q, Meran BK. Automated data cleaning for data centers: A case study. In: Proceedings of the 39th Chinese Control Conference (CCC). 2020. https://doi.org/10.23919/CCC50068.2020.9189357

[4] Shi X, et al. An automated data cleaning method for electronic health records by incorporating clinical knowledge. BMC Medical Informatics and Decision Making. 2021;21:267. https://doi.org/10.1186/s12911-021-01630-7

[5] Clemente F, et al. ydata-profiling: Accelerating data-centric AI with high-quality data. Neurocomputing. 2023;554:126585. https://doi.org/10.1016/j.neucom.2023.126585

[6] Shen X, Hu S, Zhang X, Han X, Meng X, Wei J, Liu Z, Sun M. AutoClean: LLMs can prepare their training corpus. In: Proceedings of NAACL-HLT 2025 (System Demonstrations). 2025. p. 85–95. https://doi.org/10.18653/v1/2025.naacl-demo.9

[7] Bare B, et al. A deep learning based no-reference image quality assessment model for single-image super-resolution. In: Proceedings of IEEE ICASSP. 2018. https://doi.org/10.1109/ICASSP.2018.8461931

[8] Li A, Li J, Lin Q, Ma C. Deep image quality assessment driven single image deblurring. In: Proceedings of IEEE ICME. 2020. https://doi.org/10.1109/ICME46284.2020.9102899

[9] Chen J, Li S, Lin L, Wan J, Li Z. No-reference blurred image quality assessment method based on structure of structure features. Signal Processing: Image Communication. 2023;118:117008. https://doi.org/10.1016/j.image.2023.117008

[10] Hu B, Li L, Wu J, Qian J. Subjective and objective quality assessment for image restoration: A critical survey. Signal Processing: Image Communication. 2020;85:115839. https://doi.org/10.1016/j.image.2020.115839

[11] Sang Q, Li Q, Liu L, Deng Z, Wu X, Bovik AC. No-reference image quality assessment leveraging GenAI images. IEEE Transactions on Image Processing. 2025. https://doi.org/10.1109/TIP.2025.3610238

[12] Dost S, Saud F, Shabbir M, Khan MG, Shahid M, Lovstrom B. Reduced reference image and video quality assessments: Review of methods. EURASIP Journal on Image and Video Processing. 2022;2022:1. https://doi.org/10.1186/s13640-021-00578-y

[13] McKinney W. Data structures for statistical computing in Python. In: Proceedings of the 9th Python in Science Conference. 2010. p. 56–61.

[14] Bradski G. The OpenCV library. Dr. Dobb’s Journal of Software Tools. 2000.

[15] ImageHash. Python package. https://pypi.org/project/ImageHash/ (accessed 14 October 2026).

[16] Tukey JW. Exploratory data analysis. Reading (MA): Addison-Wesley; 1977.

[17] Zauner C. Implementation and benchmarking of perceptual image hash functions [MSc thesis]. Hagenberg, Austria: Upper Austria University of Applied Sciences; 2010.

[18] Harris CR, et al. Array programming with NumPy. Nature. 2020;585:357–362.

[19] smartclean-ai 1.1.0. Python Package Index. https://pypi.org/project/smartclean-ai/ (accessed 14 October 2026).

[20] smartimageclean 0.1.2. Python Package Index. https://pypi.org/project/smartimageclean/ (accessed 14 October 2026).

[21] Parkhi OM, Vedaldi A, Zisserman A, Jawahar CV. Cats and dogs. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2012. p. 3498–3505. https://doi.org/10.1109/CVPR.2012.6248092

@article{1724488,
      author = {Sumit Palekar, Yash Talati, Aniket Kambale, Fayaj Shaikh, Ashwini Garkhedkar},
      title = {SmartClean-AI: Automated Data and Image Cleaning Libraries for Machine Learning Workflows},
      journal = {Iconic Research And Engineering Journals},
      year = {2026},
      volume = {10},
      number = {5},
      pages = {1-11},
      issn = {2456-8880},
      url = {https://www.irejournals.com/formatedpaper/1724488.pdf},
      abstract = {Data quality determines the reliability of analysis and machine learning, yet preparing raw data is repetitive and rarely documented. This paper presents SmartClean-AI, a research project that delivers two open-source Python libraries, smartclean-ai for structured data and smartimageclean for image datasets, both published on the Python Package Index under the MIT licence. smartclean-ai accepts a pandas DataFrame, profiles it, handles missing values, removes duplicate rows, and detects and caps outliers with the interquartile-range (IQR) rule, returning the cleaned data, before-and-after profiles and a cleaning report. smartimageclean validates image files, measures blur, brightness and resolution, detects exact and near-duplicate images, computes a dataset quality score, assigns KEEP, REVIEW or REMOVE decisions, standardizes eligible images, re-analyses them and writes CSV and JSON reports. In a recorded run on 45 images, 32 were initially acceptable and 13 were not; after standardization, seven of the 13 satisfied the requirements and six still needed review, and the image library’s 26 automated tests passed. The libraries are rule-based and make every change visible; accuracy against annotated data and comparison with existing tools remain future work.},
      keywords = {data cleaning; duplicate detection; image quality; outlier detection; Python library},
      month = {November},
      doi = {https://doi.org/10.64388/ire1724488}
  }