AI-Powered Data Engineering: Automating Pipelines for Real-Time Analytics and Quality

References

[1] Garcia, A. B., Babiceanu, R. F., & Seker, R. (2021). Artificial intelligence and machine learning approach for aviation cybersecurity: An overview. In Integrated Communications, Navigation and Surveillance Conference, ICNS (Vol. 2021-April). Institute of Electrical and Electronics Engineers Inc. https://doi.org/10.1109/ICNS52807.2021.9441594

[2] Liu, Y., Huang, A., Luo, Y., Huang, H., Liu, Y., Chen, Y., … Yang, Q. (2021). Federated learning-powered visual object detection for safety monitoring. AI Magazine, 42(2), 19–27. https://doi.org/10.1609/aimag.v42i2.15095

[3] Chen, C., Yang, J., Lu, M., Wang, T., Zheng, Z., Chen, Y., … Rudoff, A. (2021). Optimizing in-memory database engine for AI-powered online decision augmentation using persistent memory. Proceedings of the VLDB Endowment, 14(5), 799–812. https://doi.org/10.14778/3446095.3446102

[4] Vuppalapati, C. (2021). Democratization of Artificial Intelligence for the Future of Humanity. Democratization of Artificial Intelligence for the Future of Humanity. CRC Press. https://doi.org/10.1201/9781003057789

[5] Martin-Ducup, O., Mofack, G., Wang, D., Raumonen, P., Ploton, P., Sonké, B., … Pélissier, R. (2021). Evaluation of automated pipelines for tree and plot metric estimation from TLS data in tropical forest areas. Annals of Botany, 128(6), 753–766. https://doi.org/10.1093/aob/mcab051

[6] Fuqua, T., Jordan, J., Halavatyi, A., Tischer, C., Richter, K., & Crocker, J. (2021). An open-source semi-automated robotics pipeline for embryo immunohistochemistry. Scientific Reports, 11(1). https://doi.org/10.1038/s41598-021-89676-5

[7] Pascal Andreu, V., Augustijn, H. E., van den Berg, K., van der Hooft, J. J. J., Fischbach, M. A., & Medema, M. H. (2021). BiG-MAP: An Automated Pipeline To Profile Metabolic Gene Cluster Abundance and Expression in Microbiomes. MSystems, 6(5). https://doi.org/10.1128/msystems.00937-21

[8] Segalin, C., Williams, J., Karigo, T., Hui, M., Zelikowsky, M., Sun, J. J., … Kennedy, A. (2021). The mouse action recognition system (MARS) software pipeline for automated analysis of social behaviours in mice. ELife, 10. https://doi.org/10.7554/eLife.63720

[9] Banerjee, A., Camps, J., Zacur, E., Andrews, C. M., Rudy, Y., Choudhury, R. P., … Grau, V. (2021). A completely automated pipeline for 3D reconstruction of the human heart from 2D cine magnetic resonance slices. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 379(2212). https://doi.org/10.1098/rsta.2020.0257

[10] Naseer, A., Naseer, H., Ahmad, A., Maynard, S. B., & Masood Siddiqui, A. (2021). Real-time analytics, incident response process agility and enterprise cybersecurity performance: A contingent resource-based analysis. International Journal of Information Management, 59. https://doi.org/10.1016/j.ijinfomgt.2021.102334

[11] Dubuc, T., Stahl, F., & Roesch, E. B. (2021). Mapping the Big Data Landscape: Technologies, Platforms and Paradigms for Real-Time Analytics of Data Streams. IEEE Access, 9, 15351–15374. https://doi.org/10.1109/ACCESS.2020.3046132

[12] Jeba, N., & Rathi, S. (2021). Effective data management and real-time analytics in the Internet of Things. In International Journal of Cloud Computing (Vol. 10, pp. 112–128). Inderscience Publishers. https://doi.org/10.1504/IJCC.2021.113994

[13] Shim, J. P., O’Leary, D. E., & Nisar, K. (2021). REAL-TIME STREAMING TECHNOLOGY AND ANALYTICS FOR VALUE CREATION. Journal of Organizational Computing and Electronic Commerce, 31(4), 364–382. https://doi.org/10.1080/10919392.2021.2023943

[14] Kuznetsov, S. D., Velikhov, P. E., & Fu, Q. (2021). Real-Time Analytics, Hybrid Transactional/Analytical Processing, In-Memory Data Management, and Non-Volatile Memory. Proceedings of the Institute for System Programming of the RAS, 33(3), 171–198. https://doi.org/10.15514/ispras-2021-33(3)-13

[15] Sagmeister, P., Lebl, R., Castillo, I., Rehrl, J., Kruisz, J., Sipek, M., … Kappe, C. O. (2021). Advanced Real-Time Process Analytics for Multistep Synthesis in Continuous Flow**. AngewandteChemie - International Edition, 60(15), 8139–8148. https://doi.org/10.1002/anie.202016007

@article{1702759,
      author = {Allen R Chan},
      title = {AI-Powered Data Engineering: Automating Pipelines for Real-Time Analytics and Quality},
      journal = {Iconic Research And Engineering Journals},
      year = {2021},
      volume = {4},
      number = {7},
      pages = {145-155},
      issn = {2456-8880},
      url = {https://www.irejournals.com/formatedpaper/1702759.pdf},
      abstract = {Organizations are producing massive amounts of data at warp speed, yet classic data engineering approaches seldom keep pace. Slow updates, manual pipeline creation, and error prone data transformations limit the organization from getting real-time insights. On the other hand, AIpowered data engineering eliminates manual processes like pipeline configuration, real-time monitoring, and data quality checks. This automation allows data teams to focus their efforts away from these manual maintenance tasks and on high-level innovation instead. Studies suggest that by implementing AI, enterprises are able to convert crude data into actionable intelligence much quicker and more reliably. In this article, we look into how AI optimizes data pipelines, enhances real-time analytics, and improves data quality. It also looks to the future and discusses the changing landscape of AI-powered data engineering and offers up notable advantages, issues, and considerations.},
      keywords = {AI-Powered Data Engineering, Automated Data Pipelines, Real-Time Analytics, Data Quality, Machine Learning, Schema Evolution, Data Automation, Advanced Analytics, Artificial Intelligence, Data Processing.},
      month = {January},
  }