The Role of Feature Engineering in Machine Learning: Techniques, Challenges, and Automation with Data Engineering

References

[1] Géron, A. (2019). Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow. O'Reilly Media.

[2] Kuhn, M., & Johnson, K. (2019). Feature Engineering and Selection: A Practical Approach for Predictive Models. Chapman and Hall/CRC.

[3] Aggarwal, C. C. (2015). Data Mining: The Textbook. Springer.

[4] Provost, F., & Fawcett, T. (2013). Data Science for Business: What You Need to Know about Data Mining and Data-Analytic Thinking. O'Reilly Media.

[5] Bengio, Y., Courville, A., & Vincent, P. (2013). Representation Learning: A Review and New Perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8), 1798-1828.

[6] Guyon, I., & Elisseeff, A. (2003). An Introduction to Variable and Feature Selection. Journal of Machine Learning Research, 3, 1157-1182.

[7] He, H., & Garcia, E. A. (2009). Learning from Imbalanced Data. IEEE Transactions on Knowledge and Data Engineering, 21(9), 1263-1284.

[8] Li, Y., Wang, S., & Wang, W. (2018). A Survey on Feature Selection Methods for High-Dimensional Data Processing. Springer, Neural Computing and Applications, 30(1), 1-22.

[9] Molnar, C. (2022). Interpretable Machine Learning: A Guide for Making Black Box Models Explainable. Leanpub.

[10] Luo, W., Phung, D., Tran, T., & Venkatesh, S. (2016). Guided Feature Learning for Self-Supervised Feature Engineering and Classification. IEEE Transactions on Knowledge and Data Engineering, 28(10), 2633-2646.

[11] FeatureTools (2023). Feature Engineering Automation Library. Retrieved from https://featuretools.alteryx.com.

[12] TsFresh (2023). Time Series Feature Extraction Library. Retrieved from https://tsfresh.readthedocs.io. [Crossref]

[13] AutoFeat (2023). Automated Feature Engineering in Python. Retrieved from https://github.com/cod3licious/autofeat. [Crossref]

[14] Google AutoML (2023). AutoML Tables & Feature Engineering. Retrieved from https://cloud.google.com/automl.

[15] Databricks (2023). Feature Store Documentation. Retrieved from https://databricks.com/product/feature-store.

@article{1707510,
      author = {Bhanu Prakash Reddy Rella},
      title = {The Role of Feature Engineering in Machine Learning: Techniques, Challenges, and Automation with Data Engineering},
      journal = {Iconic Research And Engineering Journals},
      year = {2025},
      volume = {8},
      number = {10},
      pages = {805-823},
      issn = {2456-8880},
      url = {https://www.irejournals.com/formatedpaper/1707510.pdf},
      abstract = {Feature engineering is a crucial step in the machine learning (ML) pipeline, significantly impacting model performance by transforming raw data into meaningful features. This process involves selecting, creating, and transforming variables to enhance predictive accuracy and efficiency. Traditional feature engineering techniques include domain-specific feature selection, polynomial transformations, encoding categorical variables, and feature scaling. However, challenges such as high-dimensional data, data sparsity, and feature selection bias pose significant hurdles. With advancements in automation, feature engineering is increasingly integrated with data engineering workflows through tools like Feature Stores, AutoML, and deep learning-based feature extraction. Automated feature engineering streamlines the process, reducing manual effort and improving scalability, particularly in big data environments. This paper explores key techniques, challenges, and automation trends in feature engineering, highlighting its critical role in building robust machine learning models.},
      keywords = {Feature Engineering, Machine Learning, Data Engineering, Feature Selection, Automated Feature Engineering, Feature Stores, AutoML, High-Dimensional Data, Model Performance, Data Transformation},
      month = {April},
  }