Home / Current Issue / Paper 1705069
Automating ETL Workflows with CI/CD Pipelines for Machine Learning Applications
Subject area: Science,Engineering and Technology · Area of research: Machine Learning
Abstract
In today's fast-paced data-driven landscape, automating Extract, Transform, Load (ETL) workflows is crucial for enhancing the efficiency of Machine Learning (ML) applications. This research explores how Continuous Integration and Continuous Deployment (CI/CD) pipelines can automate and streamline ETL processes, reducing the time and manual intervention required for data preparation and deployment. Integrating CI/CD pipelines into ETL workflows ensures that the entire data lifecycle?from extraction, transformation, loading, to model training and deployment?operates seamlessly. The automation of these processes enables rapid iteration, minimizes errors, and accelerates time-to-market for ML models. This paper investigates key strategies for leveraging modern tools such as Jenkins, Apache Airflow, and Kubernetes to build scalable and efficient automated workflows. It also examines real-world case studies where CI/CD pipelines have optimized ML workflows, leading to enhanced productivity, accuracy, and cost savings. By adopting such automation techniques, organizations can better manage large-scale data pipelines, ensure model accuracy, and reduce operational complexities in machine learning projects.
Keywords
Automated ETL workflows, CI/CD pipelines, Machine Learning applications, data lifecycle automation, Jenkins, Apache Airflow, Kubernetes, model deployment.
References
[1] Goel, P. & Singh, S. P. (2009). Method and Process Labor Resource Management System. International Journal of Information Technology, 2(2), 506-512.
[2] Singh, S. P. & Goel, P., (2010). Method and process to motivate the employee at performance appraisal system. International Journal of Computer Science & Communication, 1(2), 127-130.
[3] Goel, P. (2012). Assessment of HR development framework. International Research Journal of Management Sociology & Humanities, 3(1), Article A1014348. https://doi.org/10.32804/irjmsh
[4] Goel, P. (2016). Corporate world and gender discrimination. International Journal of Trends in Commerce and Economics, 3(6). Adhunik Institute of Productivity Management and Research, Ghaziabad.
[5] Eeti, E. S., Jain, E. A., & Goel, P. (2020). Implementing data quality checks in ETL pipelines: Best practices and tools. International Journal of Computer Science and Information Technology, 10(1), 31-42. https://rjpn.org/ijcspub/papers/IJCSP20B1006.pdf
[6] "Effective Strategies for Building Parallel and Distributed Systems", International Journal of Novel Research and Development, ISSN:2456-4184, Vol.5, Issue 1, page no.23-42, January-2020. http://www.ijnrd.org/papers/IJNRD2001005.pdf
[7] "Enhancements in SAP Project Systems (PS) for the Healthcare Industry: Challenges and Solutions", International Journal of Emerging Technologies and Innovative Research (www.jetir.org), ISSN:2349-5162, Vol.7, Issue 9, page no.96-108, September-2020, https://www.jetir.org/papers/JETIR2009478.pdf
[8] Venkata Ramanaiah Chintha, Priyanshi, Prof.(Dr) Sangeet Vashishtha, "5G Networks: Optimization of Massive MIMO", IJRAR - International Journal of Research and Analytical Reviews (IJRAR), E-ISSN 2348-1269, P- ISSN 2349-5138, Volume.7, Issue 1, Page No pp.389-406, February-2020. (http://www.ijrar.org/IJRAR19S1815.pdf )
[9] Cherukuri, H., Pandey, P., & Siddharth, E. (2020). Containerized data analytics solutions in on-premise financial services. International Journal of Research and Analytical Reviews (IJRAR), 7(3), 481-491 https://www.ijrar.org/papers/IJRAR19D5684.pdf
[10] Sumit Shekhar, SHALU JAIN, DR. POORNIMA TYAGI, "Advanced Strategies for Cloud Security and Compliance: A Comparative Study", IJRAR - International Journal of Research and Analytical Reviews (IJRAR), E-ISSN 2348-1269, P- ISSN 2349-5138, Volume.7, Issue 1, Page No pp.396-407, January 2020. (http://www.ijrar.org/IJRAR19S1816.pdf )
[11] "Comparative Analysis OF GRPC VS. ZeroMQ for Fast Communication", International Journal of Emerging Technologies and Innovative Research, Vol.7, Issue 2, page no.937-951, February-2020. (http://www.jetir.org/papers/JETIR2002540.pdf )
[12] Eeti, E. S., Jain, E. A., & Goel, P. (2020). Implementing data quality checks in ETL pipelines: Best practices and tools. International Journal of Computer Science and Information Technology, 10(1), 31-42. https://rjpn.org/ijcspub/papers/IJCSP20B1006.pdf
[13] "Effective Strategies for Building Parallel and Distributed Systems". International Journal of Novel Research and Development, Vol.5, Issue 1, page no.23-42, January 2020. http://www.ijnrd.org/papers/IJNRD2001005.pdf
[14] "Enhancements in SAP Project Systems (PS) for the Healthcare Industry: Challenges and Solutions". International Journal of Emerging Technologies and Innovative Research, Vol.7, Issue 9, page no.96-108, September 2020. https://www.jetir.org/papers/JETIR2009478.pdf
[15] Venkata Ramanaiah Chintha, Priyanshi, & Prof.(Dr) Sangeet Vashishtha (2020). "5G Networks: Optimization of Massive MIMO". International Journal of Research and Analytical Reviews (IJRAR), Volume.7, Issue 1, Page No pp.389-406, February 2020. (http://www.ijrar.org/IJRAR19S1815.pdf)
[16] Cherukuri, H., Pandey, P., & Siddharth, E. (2020). Containerized data analytics solutions in on-premise financial services. International Journal of Research and Analytical Reviews (IJRAR), 7(3), 481-491. https://www.ijrar.org/papers/IJRAR19D5684.pdf
[17] Sumit Shekhar, Shalu Jain, & Dr. Poornima Tyagi. "Advanced Strategies for Cloud Security and Compliance: A Comparative Study". International Journal of Research and Analytical Reviews (IJRAR), Volume.7, Issue 1, Page No pp.396-407, January 2020. (http://www.ijrar.org/IJRAR19S1816.pdf)
[18] "Comparative Analysis of GRPC vs. ZeroMQ for Fast Communication". International Journal of Emerging Technologies and Innovative Research, Vol.7, Issue 2, page no.937-951, February 2020. (http://www.jetir.org/papers/JETIR2002540.pdf)
[19] Eeti, E. S., Jain, E. A., & Goel, P. (2020). Implementing data quality checks in ETL pipelines: Best practices and tools. International Journal of Computer Science and Information Technology, 10(1), 31-42. Available at: http://www.ijcspub/papers/IJCSP20B1006.pdf
[20] Chopra, E. P. (2021). Creating live dashboards for data visualization: Flask vs. React. The International Journal of Engineering Research, 8(9), a1-a12. Available at: http://www.tijer/papers/TIJER2109001.pdf
[21] Eeti, S., Goel, P. (Dr.), & Renuka, A. (2021). Strategies for migrating data from legacy systems to the cloud: Challenges and solutions. TIJER (The International Journal of Engineering Research), 8(10), a1-a11. Available at: http://www.tijer/viewpaperforall.php?paper=TIJER2110001
[22] Shanmukha Eeti, Dr. Ajay Kumar Chaurasia, Dr. Tikam Singh. (2021). Real-Time Data Processing: An Analysis of PySpark's Capabilities. IJRAR - International Journal of Research and Analytical Reviews, 8(3), pp.929-939. Available at: http://www.ijrar/IJRAR21C2359.pdf
[23] Kolli, R. K., Goel, E. O., & Kumar, L. (2021). Enhanced network efficiency in telecoms. International Journal of Computer Science and Programming, 11(3), Article IJCSP21C1004. rjpn ijcspub/papers/IJCSP21C1004.pdf
[24] Antara, E. F., Khan, S., & Goel, O. (2021). Automated monitoring and failover mechanisms in AWS: Benefits and implementation. International Journal of Computer Science and Programming, 11(3), 44-54. rjpn ijcspub/viewpaperforall.php?paper=IJCSP21C1005
[25] Antara, F. (2021). Migrating SQL Servers to AWS RDS: Ensuring High Availability and Performance. TIJER, 8(8), a5-a18. Tijer
[26] Bipin Gajbhiye, Prof.(Dr.) Arpit Jain, Er. Om Goel. (2021). "Integrating AI-Based Security into CI/CD Pipelines." International Journal of Creative Research Thoughts (IJCRT), 9(4), 6203-6215. Available at: http://www.ijcrt.org/papers/IJCRT2104743.pdf
[27] Aravind Ayyagiri, Prof.(Dr.) Punit Goel, Prachi Verma. (2021). "Exploring Microservices Design Patterns and Their Impact on Scalability." International Journal of Creative Research Thoughts (IJCRT), 9(8), e532-e551. Available at: http://www.ijcrt.org/papers/IJCRT2108514.pdf
[28] Voola, Pramod Kumar, Krishna Gangu, Pandi Kirupa Gopalakrishna, Punit Goel, and Arpit Jain. 2021. "AI-Driven Predictive Models in Healthcare: Reducing Time-to-Market for Clinical Applications." International Journal of Progressive Research in Engineering Management and Science 1(2):118-129. doi:10.58257/IJPREMS11.
[29] ABHISHEK TANGUDU, Dr. Yogesh Kumar Agarwal, PROF.(DR.) PUNIT GOEL, "Optimizing Salesforce Implementation for Enhanced Decision-Making and Business Performance", International Journal of Creative Research Thoughts (IJCRT), ISSN:2320-2882, Volume.9, Issue 10, pp.d814-d832, October 2021, Available at: http://www.ijcrt.org/papers/IJCRT2110460.pdf
[30] Voola, Pramod Kumar, Kumar Kodyvaur Krishna Murthy, Saketh Reddy Cheruku, S P Singh, and Om Goel. 2021. "Conflict Management in Cross-Functional Tech Teams: Best Practices and Lessons Learned from the Healthcare Sector." International Research Journal of Modernization in Engineering Technology and Science 3(11). DOI: https://www.doi.org/10.56726/IRJMETS16992.
[31] Salunkhe, Vishwasrao, Dasaiah Pakanati, Harshita Cherukuri, Shakeb Khan, and Arpit Jain. 2021. "The Impact of Cloud Native Technologies on Healthcare Application Scalability and Compliance." International Journal of Progressive Research in Engineering Management and Science 1(2):82-95. DOI: https://doi.org/10.58257/IJPREMS13.
[32] Salunkhe, Vishwasrao, Aravind Ayyagiri, Aravindsundeep Musunuri, Arpit Jain, and Punit Goel. 2021. "Machine Learning in Clinical Decision Support: Applications, Challenges, and Future Directions." International Research Journal of Modernization in Engineering, Technology and Science 3(11):1493. DOI: https://doi.org/10.56726/IRJMETS16993.
[33] Agrawal, Shashwat, Pattabi Rama Rao Thumati, Pavan Kanchi, Shalu Jain, and Raghav Agarwal. 2021. "The Role of Technology in Enhancing Supplier Relationships." International Journal of Progressive Research in Engineering Management and Science 1(2):96-106. DOI: 10.58257/IJPREMS14.
[34] Arulkumaran, Rahul, Shreyas Mahimkar, Sumit Shekhar, Aayush Jain, and Arpit Jain. 2021. "Analyzing Information Asymmetry in Financial Markets Using Machine Learning." International Journal of Progressive Research in Engineering Management and Science 1(2):53-67. doi:10.58257/IJPREMS16.
[35] Arulkumaran, Rahul, Dasaiah Pakanati, Harshita Cherukuri, Shakeb Khan, and Arpit Jain. 2021. “Gamefi Integration Strategies for Omnichain NFT Projects.” International Research Journal of Modernization in Engineering, Technology and Science 3(11). doi: https://www.doi.org/10.56726/IRJMETS16995.
[36] Agarwal, Nishit, Dheerender Thakur, Kodamasimham Krishna, Punit Goel, and S. P. Singh. 2021. "LLMS for Data Analysis and Client Interaction in MedTech." International Journal of Progressive Research in Engineering Management and Science (IJPREMS) 1(2):33-52. DOI: https://www.doi.org/10.58257/IJPREMS17.
[37] Agarwal, Nishit, Umababu Chinta, Vijay Bhasker Reddy Bhimanapati, Shubham Jain, and Shalu Jain. 2021. "EEG Based Focus Estimation Model for Wearable Devices." International Research Journal of Modernization in Engineering, Technology and Science 3(11):1436. doi: https://doi.org/10.56726/IRJMETS16996.
[38] Agrawal, Shashwat, Abhishek Tangudu, Chandrasekhara Mokkapati, Dr. Shakeb Khan, and Dr. S. P. Singh. 2021. "Implementing Agile Methodologies in Supply Chain Management." International Research Journal of Modernization in Engineering, Technology and Science 3(11):1545. doi: https://www.doi.org/10.56726/IRJMETS16989.
[39] Mahadik, Siddhey, Raja Kumar Kolli, Shanmukha Eeti, Punit Goel, and Arpit Jain. 2021. "Scaling Startups through Effective Product Management." International Journal of Progressive Research in Engineering Management and Science 1(2):68-81. doi:10.58257/IJPREMS15.
[40] Mahadik, Siddhey, Krishna Gangu, Pandi Kirupa Gopalakrishna, Punit Goel, and S. P. Singh. 2021. "Innovations in AI-Driven Product Management." International Research Journal of Modernization in Engineering, Technology and Science 3(11):1476. https://www.doi.org/10.56726/IRJMETS16994.
[41] Dandu, Murali Mohana Krishna, Swetha Singiri, Sivaprasad Nadukuru, Shalu Jain, Raghav Agarwal, and S. P. Singh. (2021). "Unsupervised Information Extraction with BERT." International Journal of Research in Modern Engineering and Emerging Technology (IJRMEET) 9(12): 1.
[42] Dandu, Murali Mohana Krishna, Pattabi Rama Rao Thumati, Pavan Kanchi, Raghav Agarwal, Om Goel, and Er. Aman Shrivastav. (2021). "Scalable Recommender Systems with Generative AI." International Research Journal of Modernization in Engineering, Technology and Science 3(11): [1557]. https://doi.org/10.56726/IRJMETS17269.
[43] Balasubramaniam, Vanitha Sivasankaran, Raja Kumar Kolli, Shanmukha Eeti, Punit Goel, Arpit Jain, and Aman Shrivastav. 2021. "Using Data Analytics for Improved Sales and Revenue Tracking in Cloud Services." International Research Journal of Modernization in Engineering, Technology and Science 3(11):1608. doi:10.56726/IRJMETS17274.
[44] Joshi, Archit, Pattabi Rama Rao Thumati, Pavan Kanchi, Raghav Agarwal, Om Goel, and Dr. Alok Gupta. 2021. "Building Scalable Android Frameworks for Interactive Messaging." International Journal of Research in Modern Engineering and Emerging Technology (IJRMEET) 9(12):49. Retrieved from www.ijrmeet.org.
[45] Joshi, Archit, Shreyas Mahimkar, Sumit Shekhar, Om Goel, Arpit Jain, and Aman Shrivastav. 2021. "Deep Linking and User Engagement Enhancing Mobile App Features." International Research Journal of Modernization in Engineering, Technology, and Science 3(11): Article 1624. doi:10.56726/IRJMETS17273.
[46] Tirupati, Krishna Kishor, Raja Kumar Kolli, Shanmukha Eeti, Punit Goel, Arpit Jain, and S. P. Singh. 2021. "Enhancing System Efficiency Through PowerShell and Bash Scripting in Azure Environments." International Journal of Research in Modern Engineering and Emerging Technology (IJRMEET) 9(12):77. Retrieved from http://www.ijrmeet.org.
[47] Tirupati, Krishna Kishor, Venkata Ramanaiah Chintha, Vishesh Narendra Pamadi, Prof. Dr. Punit Goel, Vikhyat Gupta, and Er. Aman Shrivastav. 2021. "Cloud Based Predictive Modeling for Business Applications Using Azure." International Research Journal of Modernization in Engineering, Technology and Science 3(11):1575. https://www.doi.org/10.56726/IRJMETS17271.
[48] Nadukuru, Sivaprasad, Dr S P Singh, Shalu Jain, Om Goel, and Raghav Agarwal. 2021. "Integration of SAP Modules for Efficient Logistics and Materials Management." International Journal of Research in Modern Engineering and Emerging Technology (IJRMEET) 9(12):96. Retrieved (http://www.ijrmeet.org).
[49] Nadukuru, Sivaprasad, Fnu Antara, Pronoy Chopra, A. Renuka, Om Goel, and Er. Aman Shrivastav. 2021. "Agile Methodologies in Global SAP Implementations: A Case Study Approach." International Research Journal of Modernization in Engineering Technology and Science 3(11). DOI: https://www.doi.org/10.56726/IRJMETS17272.
[50] Phanindra Kumar Kankanampati, Rahul Arulkumaran, Shreyas Mahimkar, Aayush Jain, Dr. Shakeb Khan, & Prof.(Dr.) Arpit Jain. (2021). Effective Data Migration Strategies for Procurement Systems in SAP Ariba. Universal Research Reports, 8(4), 250–267. https://doi.org/10.36676/urr.v8.i4.1389
[51] Rajas Paresh Kshirsagar, Raja Kumar Kolli, Chandrasekhara Mokkapati, Om Goel, Dr. Shakeb Khan, & Prof.(Dr.) Arpit Jain. (2021). Wireframing Best Practices for Product Managers in Ad Tech. Universal Research Reports, 8(4), 210–229. https://doi.org/10.36676/urr.v8.i4.1387
[52] Gannamneni, Nanda Kishore, Jaswanth Alahari, Aravind Ayyagiri, Prof.(Dr) Punit Goel, Prof.(Dr.) Arpit Jain, & Aman Shrivastav. (2021). "Integrating SAP SD with Third-Party Applications for Enhanced EDI and IDOC Communication." Universal Research Reports, 8(4), 156–168. https://doi.org/10.36676/urr.v8.i4.1384.
[53] Gannamneni, Nanda Kishore, Jaswanth Alahari, Aravind Ayyagiri, Prof.(Dr) Punit Goel, Prof.(Dr.) Arpit Jain, & Aman Shrivastav. 2021. "Integrating SAP SD with Third-Party Applications for Enhanced EDI and IDOC Communication." Universal Research Reports, 8(4), 156–168. https://doi.org/10.36676/urr.v8.i4.1384
[54] Mahika Saoji, Abhishek Tangudu, Ravi Kiran Pagidi, Om Goel, Prof.(Dr.) Arpit Jain, & Prof.(Dr) Punit Goel. 2021. "Virtual Reality in Surgery and Rehab: Changing the Game for Doctors and Patients." Universal Research Reports, 8(4), 169–191. https://doi.org/10.36676/urr.v8.i4.1385
[55] HR Efficiency Through Oracle HCM Cloud Optimization." International Journal of Creative Research Thoughts (IJCRT) 10(12).p. (ISSN: 2320-2882). Retrieved from https://ijcrt.org.
[56] Salunkhe, Vishwasrao, Umababu Chinta, Vijay Bhasker Reddy Bhimanapati, Shubham Jain, and Punit Goel. 2022. “Clinical Quality Measures (eCQM) Development Using CQL: Streamlining Healthcare Data Quality and Reporting.” International Journal of Computer Science and Engineering (IJCSE) 11(2):9–22.
[57] Khair, Md Abul, Kumar Kodyvaur Krishna Murthy, Saketh Reddy Cheruku, S. P. Singh, and Om Goel. 2022. "Future Trends in Oracle HCM Cloud." International Journal of Computer Science and Engineering 11(2):9–22.
[58] Arulkumaran, Rahul, Aravind Ayyagiri, Aravindsundeep Musunuri, Prof. (Dr.) Punit Goel, and Prof. (Dr.) Arpit Jain. 2022. "Decentralized AI for Financial Predictions." International Journal for Research Publication & Seminar 13(5):434. https://doi.org/10.36676/jrps.v13.i5.1511.
[59] Arulkumaran, Rahul, Aravind Ayyagiri, Aravindsundeep Musunuri, Arpit Jain, and Punit Goel. 2022. "Real-Time Classification of High Variance Events in Blockchain Mining Pools." International Journal of Computer Science and Engineering 11(2):9–22.
[60] Agarwal, Nishit, Rikab Gunj, Venkata Ramanaiah Chintha, Raja Kumar Kolli, Om Goel, and Raghav Agarwal. 2022. “Deep Learning for Real Time EEG Artifact Detection in Wearables.” International Journal for Research Publication & Seminar 13(5):402. https://doi.org/10.36676/jrps.v13.i5.1510.
[61] Ravi Kiran Pagidi, Nishit Agarwal, Venkata Ramanaiah Chintha, Er. Aman Shrivastav, Shalu Jain, Om Goel, "Data Migration Strategies from On-Prem to Cloud with Azure Synapse", IJRAR - International Journal of Research and Analytical Reviews (IJRAR), E-ISSN 2348-1269, P- ISSN 2349-5138, Volume.9, Issue 3, Page No pp.308-323, August 2022, Available at : http://www.ijrar.org/IJRAR22C3165.pdf.
[62] Tirupati, Krishna Kishor, Pattabi Rama Rao Thumati, Pavan Kanchi, Raghav Agarwal, Om Goel, and Aman Shrivastav. 2022. “Best Practices for Automating Deployments Using CI/CD Pipelines in Azure.” International Journal of Computer Science and Engineering 11(1):141–164. ISSN (P): 2278–9960; ISSN (E): 2278–9979.
[63] Sivaprasad Nadukuru, Rahul Arulkumaran, Nishit Agarwal, Prof.(Dr) Punit Goel, & Anshika Aggarwal. 2022. Optimizing SAP Pricing Strategies with Vendavo and PROS Integration. International Journal for Research Publication and Seminar, 13(5), 572–610. https://doi.org/10.36676/jrps.v13.i5.1529.
[64] Nadukuru, Sivaprasad, Pattabi Rama Rao Thumati, Pavan Kanchi, Raghav Agarwal, and Om Goel. 2022. "Improving SAP SD Performance Through Pricing Enhancements and Custom Reports." International Journal of General Engineering and Technology (IJGET) 11(1):9–48.
[65] Pagidi, Ravi Kiran, Raja Kumar Kolli, Chandrasekhara Mokkapati, Om Goel, Dr. Shakeb Khan, & Prof.(Dr.) Arpit Jain. (2022). Enhancing ETL Performance Using Delta Lake in Data Analytics Solutions. Universal Research Reports, 9(4), 473–495. https://doi.org/10.36676/urr.v9.i4.1381.
[66] Salunkhe, Vishwasrao, Venkata Ramanaiah Chintha, Vishesh Narendra Pamadi, Arpit Jain, and Om Goel. 2022. "AI-Powered Solutions for Reducing Hospital Readmissions: A Case Study on AI-Driven Patient Engagement." International Journal of Creative Research Thoughts 10(12):757-764.
[67] Agrawal, Shashwat, Digneshkumar Khatri, Viharika Bhimanapati, Om Goel, and Arpit Jain. 2022. "Optimization Techniques in Supply Chain Planning for Consumer Electronics." International Journal for Research Publication & Seminar 13(5):356. DOI: https://doi.org/10.36676/jrps.v13.i5.1507.
[68] Dandu, Murali Mohana Krishna, Archit Joshi, Krishna Kishor Tirupati, Akshun Chhapola, Shalu Jain, and Er. Aman Shrivastav. (2022). “Quantile Regression for Delivery Promise Optimization.” International Journal of Computer Science and Engineering (IJCSE) 11(1): 141–164. ISSN (P): 2278–9960; ISSN (E): 2278–9979.
[69] Vanitha Sivasankaran Balasubramaniam, Santhosh Vijayabaskar, Pramod Kumar Voola, Raghav Agarwal, & Om Goel. (2022). Improving Digital Transformation in Enterprises Through Agile Methodologies. International Journal for Research Publication and Seminar, 13(5), 507–537. https://doi.org/10.36676/jrps.v13.i5.1527.
[70] Mahadik, Siddhey, Kumar Kodyvaur Krishna Murthy, Saketh Reddy Cheruku, Prof. (Dr.) Arpit Jain, and Om Goel. 2022.
[71] “Agile Product Management in Software Development.” International Journal for Research Publication & Seminar 13(5):453. https://doi.org/10.36676/jrps.v13.i5.1512.
[72] Mahadik, Siddhey, Amit Mangal, Swetha Singiri, Akshun Chhapola, and Shalu Jain. 2022.
[73] "Risk Mitigation Strategies in Product Management." International Journal of Creative Research Thoughts (IJCRT) 10(12):665.
[74] Khair, Md Abul, Kumar Kodyvaur Krishna Murthy, Saketh Reddy Cheruku, Shalu Jain, and Raghav Agarwal. 2022. “Optimizing Oracle HCM Cloud Implementations for Global Organizations.” International Journal for Research Publication & Seminar 13(5):372. https://doi.org/10.36676/jrps.v13.i5.1508.
[75] Arulkumaran, Rahul, Sowmith Daram, Aditya Mehra, Shalu Jain, and Raghav Agarwal. 2022. "Intelligent Capital Allocation Frameworks in Decentralized Finance." International Journal of Creative Research Thoughts (IJCRT) 10(12):669. ISSN: 2320-2882.
[76] "Agarwal, Nishit, Rikab Gunj, Amit Mangal, Swetha Singiri, Akshun Chhapola, and Shalu Jain. 2022. “Self-Supervised Learning for EEG Artifact Detection.” International Journal of Creative Research Thoughts 10(12).p. Retrieved from https://www.ijcrt.org/IJCRT2212667."
[77] Murali Mohana Krishna Dandu, Venudhar Rao Hajari, Jaswanth Alahari, Om Goel, Prof. (Dr.) Arpit Jain, & Dr. Alok Gupta. (2022). Enhancing Ecommerce Recommenders with Dual Transformer Models. International Journal for Research Publication and Seminar, 13(5), 468–506. https://doi.org/10.36676/jrps.v13.i5.1526.
[78] Agarwal, N., Daram, S., Mehra, A., Goel, O., & Jain, S. (2022). Machine learning for muscle dynamics in spinal cord rehab. International Journal of Computer Science and Engineering (IJCSE), 11(2), 147–178. © IASET. https://www.iaset.us/archives?jname=14_2&year=2022&submit=Search.
[79] Salunkhe, Vishwasrao, Srikanthudu Avancha, Bipin Gajbhiye, Ujjawal Jain, and Punit Goel. 2022. "AI Integration in Clinical Decision Support Systems: Enhancing Patient Outcomes through SMART on FHIR and CDS Hooks." International Journal for Research Publication & Seminar 13(5):338. DOI: https://doi.org/10.36676/jrps.v13.i5.1506.
[80] Agrawal, Shashwat, Fnu Antara, Pronoy Chopra, A Renuka, and Punit Goel. 2022. "Risk Management in Global Supply Chains." International Journal of Creative Research Thoughts (IJCRT) 10(12):2212668.
[81] Agrawal, Shashwat, Srikanthudu Avancha, Bipin Gajbhiye, Om Goel, and Ujjawal Jain. 2022. "The Future of Supply Chain Automation." International Journal of Computer Science and Engineering 11(2):9–22.
[82] Voola, Pramod Kumar, Umababu Chinta, Vijay Bhasker Reddy Bhimanapati, Om Goel, and Punit Goel. 2022. "AI-Powered Chatbots in Clinical Trials: Enhancing Patient-Clinician Interaction and Decision-Making." International Journal for Research Publication & Seminar 13(5):323. https://doi.org/10.36676/jrps.v13.i5.1505.
[83] Ayyagiri, Aravind, Shalu Jain, and Anshika Aggarwal. 2022. "Leveraging Docker Containers for Scalable Web Application Deployment." International Journal of Computer Science and Engineering 11(1):69–86. ISSN (P): 2278–9960; ISSN (E): 2278–9979. Retrieved September 14, 2024 (https://iaset.us/download/archives/03-09-2024-1725362533-6-%20IJCSE-abstract-6.Abs.%20IJCSE_2022_Vol_11_Issue_1_Res.Paper_NO_299.%20Leveraging%20Docker%20Containers%20for%20Scalable%20Web%20Application%20Deployment.docx).
[84] Voola, Pramod Kumar, Shreyas Mahimkar, Sumit Shekhar, Prof. (Dr) Punit Goel, and Vikhyat Gupta. 2022. "Machine Learning in ECOA Platforms: Advancing Patient Data Quality and Insights." International Journal of Creative Research Thoughts (IJCRT) 10(12)
[85] Gajbhiye, B., Khan, S. (Dr.), & Goel, O. (2022). "Penetration testing methodologies for serverless cloud architectures." Innovative Research Thoughts, 8(4), Article 1456. https://doi.org/10.36676/irt.v8.14.1456
[86] Kolli, R. K., Chhapola, A., & Kaushik, S. (2022). Arista 7280 switches: Performance in national data centers. The International Journal of Engineering Research, 9(7), TIJER2207014. tijer tijer/papers/TIJER2207014.pdf
[87] Antara, F., Gupta, V., & Khan, S. (2022). Transitioning legacy HR systems to cloud-based platforms: Challenges and solutions. Journal of Emerging Technologies and Innovative Research (JETIR), 9(7), Article JETIR2207741. https://www.jetir.org
[88] FNU Antara, DR. PRERNA GUPTA, "Enhancing Data Quality and Efficiency in Cloud Environments: Best Practices", IJRAR - International Journal of Research and Analytical Reviews (IJRAR), Volume.9, Issue 3, pp.210-223, August 2022. http://www.ijrar IJRAR22C3154.pdf
[89] Pronoy Chopra, Akshun Chhapola, Dr. Sanjouli Kaushik. (February 2022). Comparative Analysis of Optimizing AWS Inferentia with FastAPI and PyTorch Models. International Journal of Creative Research Thoughts (IJCRT), 10(2), pp.e449-e463. Available at: http://www.ijcrt/IJCRT2202528.pdf
[90] Chopra, E. P., Gupta, E. V., & Jain, D. P. K. (2022). Building serverless platforms: Amazon Bedrock vs. Claude3. International Journal of Computer Science and Publications, 12(3), 722-733. Available at: http://www.ijcspub/viewpaperforall.php?paper=IJCSP22C1306
[91] Key Technologies and Methods for Building Scalable Data Lakes. (July 2022). International Journal of Novel Research and Development, 7(7), pp.1-21. Available at: http://www.ijnrd/IJNRD2207179.pdf
[92] Efficient ETL Processes: A Comparative Study of Apache Airflow vs. Traditional Methods. (August 2022). International Journal of Emerging Technologies and Innovative Research, 9(8), pp.g174-g184. Available at: http://www.jetir/JETIR2208624.pdf
[93] Balasubramaniam, Vanitha Sivasankaran, Archit Joshi, Krishna Kishor Tirupati, Akshun Chhapola, and Shalu Jain. 2022. "The Role of SAP in Streamlining Enterprise Processes: A Case Study." International Journal of General Engineering and Technology (IJGET) 11(1):9–48.
[94] Sivasankaran Balasubramaniam, Vanitha, S. P. Singh, Sivaprasad Nadukuru, Shalu Jain, Raghav Agarwal, and Alok Gupta. 2022. "Integrating Human Resources Management with IT Project Management for Better Outcomes." International Journal of Computer Science and Engineering 11(1):141–164. ISSN (P): 2278–9960; ISSN (E): 2278–9979.
[95] Joshi, Archit, Sivaprasad Nadukuru, Shalu Jain, Raghav Agarwal, and Om Goel. 2022. "Innovations in Package Delivery Tracking for Mobile Applications." International Journal of General Engineering and Technology 11(1):9–48.
[96] García, S., & Ramírez, A. (2015). "Towards automated data integration: A survey on ETL processes." Journal of Computer and System Sciences, 81(5), 926-945. DOI: 10.1016/j.jcss.2015.01.001
[97] Choudhary, A., & Kumar, V. (2016). "A framework for ETL processes automation using DevOps." International Journal of Computer Applications, 142(8), 1-5. DOI: 10.5120/pcas2016-2535
[98] O'Connor, M. J., & Le, P. (2017). "Automating ETL workflows: Bridging the gap between data engineering and machine learning." Data Science and Engineering, 2(2), 133-145. DOI: 10.1007/s41019-017-0040-5
[99] Khan, S., & Qureshi, M. R. (2018). "Continuous integration and delivery of data pipelines: A DevOps approach." Journal of Big Data, 5(1), 12. DOI: 10.1186/s40537-018-0128-6
[100] Mehta, H., & Narasimhan, S. (2018). "Implementing DataOps: A DevOps-inspired approach for data workflows." IEEE Transactions on Big Data, 5(2), 135-147. DOI: 10.1109/TBDATA.2018.2830539
[101] Feldman, R., & Yitzhaki, I. (2019). "Real-time data processing using CI/CD pipelines for machine learning applications." ACM Transactions on Intelligent Systems and Technology, 10(3), 1-22. DOI: 10.1145/3312324
[102] Garcia, C., & Almaraz, L. (2020). "Data quality enhancement through automated ETL processes in machine learning environments." Information Systems, 93, 101597. DOI: 10.1016/j.is.2020.101597
[103] Shah, M. A., & Ahmed, A. (2020). "The role of CI/CD in automating ETL workflows for big data analytics." International Journal of Information Management, 52, 102095. DOI: 10.1016/j.ijinfomgt.2020.102095
[104] Patel, N., & Banerjee, R. (2021). "Leveraging AI for automating ETL processes in machine learning pipelines." Expert Systems with Applications, 165, 113747. DOI: 10.1016/j.eswa.2020.113747
[105] Reddy, M. K., & Prasad, R. (2021). "Towards an automated ETL framework using CI/CD practices in data engineering." Data & Knowledge Engineering, 134, 101874. DOI: 10.1016/j.datak.2021.101874
[106] Soni, P., & Gupta, A. (2022). "Impact of CI/CD on the efficiency of ETL workflows: A quantitative analysis." Journal of King Saud University - Computer and Information Sciences. DOI: 10.1016/j.jksuci.2022.02.012
[107] Nair, S. K., & Singhal, V. (2022). "Emerging trends in DataOps: Automating data pipelines for enhanced analytics." Journal of Data Science, 20(1), 23-37. DOI: 10.6339/jds.202205_20(1).0003
How to cite this paper
@article{1705069,
author = {Antony Satya Vivek Vardhan Akisetty, Ashish Kumar, Murali Mohana Krishna Dandu, Prof. (Dr) Punit Goel, Prof. (Dr.) Arpit Jain; Er. Aman Shrivastav},
title = {Automating ETL Workflows with CI/CD Pipelines for Machine Learning Applications},
journal = {Iconic Research And Engineering Journals},
year = {2023},
volume = {7},
number = {3},
pages = {478-497},
issn = {2456-8880},
url = {https://www.irejournals.com/formatedpaper/1705069.pdf},
abstract = {In today's fast-paced data-driven landscape, automating Extract, Transform, Load (ETL) workflows is crucial for enhancing the efficiency of Machine Learning (ML) applications. This research explores how Continuous Integration and Continuous Deployment (CI/CD) pipelines can automate and streamline ETL processes, reducing the time and manual intervention required for data preparation and deployment. Integrating CI/CD pipelines into ETL workflows ensures that the entire data lifecycle?from extraction, transformation, loading, to model training and deployment?operates seamlessly. The automation of these processes enables rapid iteration, minimizes errors, and accelerates time-to-market for ML models. This paper investigates key strategies for leveraging modern tools such as Jenkins, Apache Airflow, and Kubernetes to build scalable and efficient automated workflows. It also examines real-world case studies where CI/CD pipelines have optimized ML workflows, leading to enhanced productivity, accuracy, and cost savings. By adopting such automation techniques, organizations can better manage large-scale data pipelines, ensure model accuracy, and reduce operational complexities in machine learning projects.},
keywords = {Automated ETL workflows, CI/CD pipelines, Machine Learning applications, data lifecycle automation, Jenkins, Apache Airflow, Kubernetes, model deployment.},
month = {September},
}