Data Lakehouse Architecture: Bridging the Gap Between Data Lakes and Data Warehouses

References

[1] M. Armbrust et al., “Delta Lake: High-Performance ACID Table Storage over Cloud Object Stores,” PVLDB, vol. 13, no. 12, 2020. doi:10.14778/3415478.3415560

[2] J. Levandoski et al., “BigLake: BigQuery’s Evolution Toward a Multi Cloud Lakehouse,” SIGMOD, 2024. doi:10.1145/3626246.3653388

[3] T. B. Samwel et al., “Photon: A Fast Query Engine for Lakehouse Systems,” SIGMOD, 2022. doi:10.1145/3514221.3526054

[4] J. Camacho-Rodr´ıguez et al., “LST-Bench: Benchmarking Log-Structured Tables in the Cloud,” Proc. ACM on Management of Data, 2024. doi:10.1145/3639314

[5] J. Dean and S. Ghemawat, “MapReduce: Simplified Data Processing on Large Clusters,” Commun. ACM, 51(1), 2008. doi:10.1145/1327452.1327492

[6] S. Melnik et al., “Dremel: Interactive Analysis of Web-Scale Datasets,” VLDB, 2010. doi:10.1145/1953122.1953148

[7] A. Thusoo et al., “Hive: A Warehousing Solution over a Map-Reduce Framework,” PVLDB, 2009. doi:10.14778/1687553.1687609

[8] M. Zaharia et al., “Apache Spark: A Unified Engine for Big Data Processing,” Commun. ACM, 59(11), 2016. doi:10.1145/2934664

[9] X. Zeng et al., “An Empirical Evaluation of Columnar Storage Formats,” PVLDB, 17(1), 2023. doi:10.14778/3626292.3626298

[10] T. Ivanov et al., “The Impact of Columnar File Formats on SQL on-Hadoop,” Concurrency and Computation: Practice and Experience, 32(20), 2020. doi:10.1002/cpe.5523

[11] A. Okolnychyi et al., “Petabyte-Scale Row-Level Operations in Data Lakehouses,” PVLDB, 17(12), 2024. doi:10.14778/3685800.3685834

[12] S. Melnik et al., “Dremel: A Decade of Interactive SQL Analysis at Web Scale,” PVLDB, vol. 13, no. 12, 2020. doi:10.14778/3415478.3415568

[13] R. Hai et al., “Data Lakes: A Survey of Functions and Systems,” IEEE TKDE, 35(12), 2023. doi:10.1109/TKDE.2023.3270101

[14] P. Wieder and H. Nolte, “Toward Data Lakes as Central Building Blocks for Data Management and Analysis,” Frontiers in Big Data, 2022. doi:10.3389/fdata.2022.945720

[15] A. Nambiar and D. Mundra, “An Overview of Data Warehouse and Data Lake in Modern Enterprise Data Management,” Big Data and Cognitive Computing, 6(4), 2022. doi:10.3390/bdcc6040132

[16] A. Agrawal et al., “XTable in Action: Seamless Interoperability in Data Lakes,” arXiv, 2401.09621, 2024. doi:10.48550/arXiv.2401.09621

[17] Z. Bao et al., “Delta Tensor: Efficient Vector and Tensor Storage in Delta Lake,” arXiv, 2405.03708, 2024. doi:10.48550/arXiv.2405.03708

[18] R. Kienzler et al., “Tensor Lakehouse for Foundation Model Training,” arXiv, 2309.02094, 2023. doi:10.48550/arXiv.2309.02094

[19] A. Khatiwada et al., “Integrating Data Lake Tables,” PVLDB, 16(4), 2022. doi:10.14778/3574245.3574274

[20] S. M. Shaffi, S. Vengathattil, and J. Mehta, “Enhancing Cloud Security Through AI-Driven Anomaly Detection and Advanced Machine Learning Algorithms,” Proc. 8th IEEE Int. Symp. on Big Data and Applied Statistics (ISBDAS 2025), 2025. doi:10.1109/ISBDAS64762.2025.11116832

[21] N. Rey et al., “Nested Parquet Is Flat, Why Not Use It? How To Scan Nested Parquet Efficiently,” SIGMOD, 2025. doi:10.1145/3725329

[22] J. Dean and S. Ghemawat, “MapReduce: A Flexible Data Processing Tool,” Commun. ACM, 53(1), 2010. doi:10.1145/1629175.1629198

@article{1711258,
      author = {Maya Thomas, Lavanya Gonsalez, Ribin Jacob, Tincy Mathew},
      title = {Data Lakehouse Architecture: Bridging the Gap Between Data Lakes and Data Warehouses},
      journal = {Iconic Research And Engineering Journals},
      year = {2025},
      volume = {9},
      number = {4},
      pages = {593-600},
      issn = {2456-8880},
      url = {https://www.irejournals.com/formatedpaper/1711258.pdf},
      abstract = {The exponential growth of enterprise and scientific data has challenged longstanding assumptions in data management. Traditional data warehouses deliver mature relational semantics and predictable performance, but struggle with semi-structured modalities, iterative data science, and real-time signals. Data lakes, by contrast, scale elastically on commodity object stores and support diverse data types through schema-on-read, yet historically lacked transactional guarantees, strong governance, and consistent query performance. The Data Lakehouse architecture reconciles these trade-offs by layering warehouse-like ACID transactions, versioned metadata, and query optimization over open file formats in a cloud-native design. This paper provides a deep, holistic treatment of Lakehouse principles and practice. We (i) trace the intellectual lineage from MapReduce, Dremel, and Hive to modern log-structured table formats; (ii) formalize a reference architecture encompassing storage, transaction/metadata, and processing layers with a cross-cutting governance plane; (iii) present a comparative analysis of Delta Lake, Apache Iceberg, and Apache Hudi; (iv) synthesize performance considerations for vectorized execution, small-file mitigation, and streaming upsets; (v) examine governance and interoperability patterns for multi-cloud deployments; and (vi) explore emerging directions-  including vector/tensor extensions for AI, zero-ETL pipelines, semantic integration, and carbon-aware optimization. Throughout, we anchor discussion in peer-reviewed literature and production learnings, retaining resolvable DOIs for all referenced works. The result is a practitioner-ready, research-grounded blueprint for building resilient, interoperable, and AI-native data platforms.},
      keywords = {Data Lakehouse, Delta Lake, Apache Iceberg, Apache Hudi, Parquet, ORC, ACID Transactions, Metadata Governance, Big Data Architecture, Cloud Analytics, Machine Learning, Vector Databases},
      month = {October},
      doi = {https://doi.org/10.64388/IREV9I4-1711258-3166}
  }