A Metadata-Driven Framework for Delta Lakehouse Integration in Healthcare Data Engineering

• Author(s): Olanrewaju Oluwaseun Ajayi ; Okeoma Onunka ; Linda Azah
• Paper ID: 1709558
• Page: 257-269
• Published Date: 31-07-2020
• Published In: Iconic Research And Engineering Journals
• Publisher: IRE Journals
• e-ISSN: 2456-8880
• Volume/Issue: Volume 4 Issue 1 July-2020

Abstract

Healthcare data engineering faces significant challenges arising from the heterogeneity, volume, and regulatory complexity of clinical data. This paper proposes a metadata-driven framework to enhance the integration of Delta Lakehouse architecture within healthcare data systems, addressing critical needs for scalability, governance, and real-time reliability. By elevating metadata to a central operational role, the framework orchestrates data ingestion, transactional storage, policy enforcement, and analytics delivery, ensuring traceability, schema evolution, and compliance with regulations such as HIPAA and GDPR. Key metadata services, including schema registries, data catalogs, lineage trackers, and audit logs, are integrated to automate data quality checks, consent management, and security policies throughout the data lifecycle. The framework supports seamless integration of batch and streaming healthcare data standards (e.g., EHRs, HL7, FHIR), enabling continuous integration and deployment of data pipelines with embedded validation and anomaly detection. This approach enhances data trustworthiness, operational efficiency, and compliance readiness, addressing current gaps in metadata utilization within Delta Lakehouse deployments. The paper concludes by highlighting academic and practical implications and outlining future research directions involving semantic metadata modeling, machine learning integration, and empirical benchmarking. The proposed framework provides a strategic blueprint for healthcare organizations aiming to build resilient, compliant, and agile data ecosystems in an increasingly complex digital health landscape.

Keywords

Metadata-Driven Framework, Delta Lakehouse, Healthcare Data Engineering, Data Governance, Real-Time Data Integration, Regulatory Compliance

Citations

IRE Journals:
Olanrewaju Oluwaseun Ajayi , Okeoma Onunka , Linda Azah "A Metadata-Driven Framework for Delta Lakehouse Integration in Healthcare Data Engineering" Iconic Research And Engineering Journals Volume 4 Issue 1 2020 Page 257-269

IEEE:
Olanrewaju Oluwaseun Ajayi , Okeoma Onunka , Linda Azah "A Metadata-Driven Framework for Delta Lakehouse Integration in Healthcare Data Engineering" Iconic Research And Engineering Journals, 4(1)