Ensuring Data Quality and Integrity in Machine Learning Pipelines: Strategies for Data Engineers

References

[1] Akidau, T., Balikov, A., Bekiroglu, K., Chernyak, S., Haberman, J., Lax, R., & Whittle, S. (2015). The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in Massive-Scale, Unbounded, Out-of-Order Data Processing. Proceedings of the VLDB Endowment, 8(12), 1792-1803.

[2] Zaharia, M., Das, T., Li, H., Hunter, T., Shenker, S., & Stoica, I. (2013). Discretized Streams: Fault-Tolerant Streaming Computation at Scale. Proceedings of the 24th ACM Symposium on Operating Systems Principles (SOSP), 423-438.

[3] Kreps, J., Narkhede, N., & Rao, J. (2011). Kafka: A Distributed Messaging System for Log Processing. Proceedings of the NetDB, 11(2011), 1-7.

[4] Carbone, P., Katsifodimos, A., Ewen, S., Markl, V., Haridi, S., & Tzoumas, K. (2015). Apache Flink: Stream and Batch Processing in a Single Engine. Bulletin of the IEEE Computer Society Technical Committee on Data Engineering, 36(4), 28-38.

[5] Grolinger, K., Capretz, M. A. M., Mezghani, E., & Exposito, E. (2014). Big Data Analytics: A Survey. Journal of Big Data, 1(1), 1-17.

[6] Xu, L. D., He, W., & Li, S. (2014). Internet of Things in Industries: A Survey. IEEE Transactions on Industrial Informatics, 10(4), 2233-2243.

[7] Villari, M., Celesti, A., Fazio, M., & Puliafito, A. (2016). Real-Time Big Data Processing for Smart Cities: The Smart Cloud Framework. IEEE Cloud Computing, 3(2), 32-41.

[8] Rajalakshmi, P., & Shahnasser, H. (2018). Predictive Analytics for IoT-Based Smart Transportation Systems. Future Generation Computer Systems, 88, 430-439.

[9] Krishnan, P. (2020). Data Engineering for Streaming Analytics: Challenges, Techniques, and Emerging Trends. ACM Computing Surveys, 53(4), 1-38.

[10] Dean, J., & Ghemawat, S. (2004). MapReduce: Simplified Data Processing on Large Clusters. Communications of the ACM, 51(1), 107-113.0p

@article{1707514,
      author = { Bhanu Prakash Reddy Rella},
      title = {Ensuring Data Quality and Integrity in Machine Learning Pipelines: Strategies for Data Engineers},
      journal = {Iconic Research And Engineering Journals},
      year = {2022},
      volume = {6},
      number = {2},
      pages = {331-339},
      issn = {2456-8880},
      url = {https://www.irejournals.com/formatedpaper/1707514.pdf},
      abstract = {Data quality and integrity are critical factors in ensuring the reliability and accuracy of machine learning (ML) models. Poor data quality?caused by missing values, inconsistencies, duplicate records, and biases?can lead to inaccurate predictions and unreliable insights. This paper explores key strategies that data engineers can implement to enhance data quality in ML pipelines. It covers data validation, data cleaning, automated anomaly detection, schema enforcement, and data governance frameworks. Additionally, it examines modern tools and frameworks, such as Great Expectations, TensorFlow Data Validation (TFDV), and Apache Deequ, which assist in maintaining high data integrity. The paper also highlights best practices for designing scalable and automated data quality monitoring systems to support real-time and batch ML workflows. By implementing these strategies, data engineers can ensure that ML models are trained on high-quality, trustworthy data, leading to more accurate and fair outcomes.},
      month = {August},
  }