Home / Current Issue / Paper 1709558
A Metadata-Driven Framework for Delta Lakehouse Integration in Healthcare Data Engineering
Subject area: Science,Engineering and Technology · Area of research: Data Engineering
Abstract
Healthcare data engineering faces significant challenges arising from the heterogeneity, volume, and regulatory complexity of clinical data. This paper proposes a metadata-driven framework to enhance the integration of Delta Lakehouse architecture within healthcare data systems, addressing critical needs for scalability, governance, and real-time reliability. By elevating metadata to a central operational role, the framework orchestrates data ingestion, transactional storage, policy enforcement, and analytics delivery, ensuring traceability, schema evolution, and compliance with regulations such as HIPAA and GDPR. Key metadata services, including schema registries, data catalogs, lineage trackers, and audit logs, are integrated to automate data quality checks, consent management, and security policies throughout the data lifecycle. The framework supports seamless integration of batch and streaming healthcare data standards (e.g., EHRs, HL7, FHIR), enabling continuous integration and deployment of data pipelines with embedded validation and anomaly detection. This approach enhances data trustworthiness, operational efficiency, and compliance readiness, addressing current gaps in metadata utilization within Delta Lakehouse deployments. The paper concludes by highlighting academic and practical implications and outlining future research directions involving semantic metadata modeling, machine learning integration, and empirical benchmarking. The proposed framework provides a strategic blueprint for healthcare organizations aiming to build resilient, compliant, and agile data ecosystems in an increasingly complex digital health landscape.
Keywords
Metadata-Driven Framework, Delta Lakehouse, Healthcare Data Engineering, Data Governance, Real-Time Data Integration, Regulatory Compliance
How to cite this paper
@article{1709558,
author = {Olanrewaju Oluwaseun Ajayi, Okeoma Onunka, Linda Azah},
title = {A Metadata-Driven Framework for Delta Lakehouse Integration in Healthcare Data Engineering},
journal = {Iconic Research And Engineering Journals},
year = {2020},
volume = {4},
number = {1},
pages = {257-269},
issn = {2456-8880},
url = {https://www.irejournals.com/formatedpaper/1709558.pdf},
abstract = {Healthcare data engineering faces significant challenges arising from the heterogeneity, volume, and regulatory complexity of clinical data. This paper proposes a metadata-driven framework to enhance the integration of Delta Lakehouse architecture within healthcare data systems, addressing critical needs for scalability, governance, and real-time reliability. By elevating metadata to a central operational role, the framework orchestrates data ingestion, transactional storage, policy enforcement, and analytics delivery, ensuring traceability, schema evolution, and compliance with regulations such as HIPAA and GDPR. Key metadata services, including schema registries, data catalogs, lineage trackers, and audit logs, are integrated to automate data quality checks, consent management, and security policies throughout the data lifecycle. The framework supports seamless integration of batch and streaming healthcare data standards (e.g., EHRs, HL7, FHIR), enabling continuous integration and deployment of data pipelines with embedded validation and anomaly detection. This approach enhances data trustworthiness, operational efficiency, and compliance readiness, addressing current gaps in metadata utilization within Delta Lakehouse deployments. The paper concludes by highlighting academic and practical implications and outlining future research directions involving semantic metadata modeling, machine learning integration, and empirical benchmarking. The proposed framework provides a strategic blueprint for healthcare organizations aiming to build resilient, compliant, and agile data ecosystems in an increasingly complex digital health landscape.},
keywords = {Metadata-Driven Framework, Delta Lakehouse, Healthcare Data Engineering, Data Governance, Real-Time Data Integration, Regulatory Compliance},
month = {July},
}