Achieving end-to-end lineage in Databricks while allowing external users to access raw data can be a challenging task. In Databricks, leveraging Unity Catalog for end-to-end lineage is a best practice. However, enabling external users to access raw data while maintaining security and lineage integrity requires a well-thought-out architecture. This blog outlines a reference architecture to achieve this balance.
Key Requirements
To meet the needs of both internal and external users, the architecture must:
- Maintain end-to-end lineage within Databricks using Unity Catalog.
- Allow external users to access raw data without compromising governance.
- Secure data while maintaining flexibility for different use cases.
Recommended Architecture
1. Shared Raw Data Lake (Pre-Bronze)
The architecture starts with a shared data lake as a landing zone for raw, unprocessed data from various sources. This data lake is located in external cloud storage, such as AWS S3 or Azure Data Lake, and is independent of Databricks. Access to this data is managed using IAM roles and policies, allowing both Databricks and external users to interact with the data without overlapping permissions.
Benefits:
- External users can access raw data without direct entry into the Databricks Lakehouse.
- Secure and isolated raw data management.
- Maintains data availability for non-Databricks consumers.
2. Bronze Layer (Managed by Databricks)
The bronze layer ingests raw data from the shared data lake into Databricks. Using Delta Live Tables (DLT), data is processed and stored as managed or external Delta tables. Unity Catalog governs these tables, enforcing fine-grained access control to maintain data security and lineage. End-to-end lineage and Databricks begins with the bronse layer and can be easily maintained throughout silver and gold by using DLTs.
Governance:
- Permissions are enforced through Unity Catalog.
- Data versioning and lineage tracking are maintained within Databricks.
3. Silver and Gold Layers (Processed Data)
Subsequent data processing transforms bronze data into refined (silver) and aggregated (gold) tables. These layers are exclusively managed within Databricks to ensure lineage continuity, leveraging Delta Lake’s optimization features.
Access:
- Internal users access data through Unity Catalog with appropriate permissions.
- External users do not have direct access to these curated layers, preserving data quality.
Access Patterns
- External Users: Access raw data from the shared data lake through configured IAM policies. No direct access to Databricks-managed bronze tables.
- Internal Users: Access the full data pipeline from bronze to gold within Databricks, leveraging Unity Catalog for secure and controlled access.
Why This Architecture Works
- Security: Separates raw data from managed bronze, reducing exposure.
- Governance: Unity Catalog maintains strict access control and lineage.
- Performance: Internal data processing benefits from Delta Lake optimizations, while raw data remains easily accessible for external systems.
End-to-end lineage in Databricks
This reference architecture offers a balanced approach to handling raw data access while maintaining governance and lineage within Databricks. By isolating raw data in a shared lake and managing processed data within Databricks, organizations can effectively support both internal analytics and external data sharing.
Contact us to learn more about how to empower your teams with the right tools, processes, and training to unlock Databricks’ full potential across your enterprise.
Source: Read MoreÂ
