The Layered Data Architecture
In data engineering, a layered architecture is a method of organizing data pipelines by separating data into logical stages based on its level of processing, trust, and intended use. This design pattern improves clarity, maintainability, and governance in modern data systems. Each layer represents a distinct phase in the data lifecycle, from raw ingestion to refined, analysis-ready outputs.Core Principles of Layered Data Models
- Separation of concerns: Each layer handles a specific part of the data journey—ingestion, transformation, or consumption.
- Data lineage and auditability: By preserving data at each stage, teams can trace changes and debug pipelines more easily.
- Scalability and maintainability: Teams can evolve pipelines independently within each layer.
- Quality enforcement: Layers provide natural checkpoints for validation, deduplication, and enrichment.
Typical Layers in Modern Data Systems
While naming conventions may vary, a common pattern includes:- Bronze Layer Stores data in its most original or minimally processed format, usually after ingestion from external sources.
- Silver Layer A workspace for applying business rules, cleansing, validation, and transformation—turning Bronze data into structured, reliable outputs.
- Gold Layer Final layer used by reporting tools, analytics, dashboards, and machine learning systems. Data here is curated, documented, and ready for consumption.
Why Adopt a Layered Approach?
Using layers provides a strong foundation for building scalable and trustworthy data systems. Benefits include:-
Improved Accessibility
Logical organization makes it easier to find and use the right data. -
Better Data Quality
Clear separation ensures transformations are applied methodically and transparently. -
Streamlined Collaboration
Different teams (data engineers, analysts, AI teams) can work in parallel across different layers.
How Nekt Implements Layers
In Nekt, the layered model is embedded in your workspace through the Catalog, which organizes your datasets by layers. This structure helps users manage data transformation pipelines intuitively, following best practices. By default, every Catalog comes with three Layers:- Bronze
- Silver
- Gold
Default Layers in Nekt
Bronze Layer
The Bronze Layer holds data that has already been converted into Delta Tables, following ingestion from external sources. It retains the original fidelity of the data while ensuring it is optimized for processing and querying.- Purpose: Store source-aligned, structured data that is ready for processing.
- Benefits: Maintains data integrity and audit trail while enabling efficient access.
Silver Layer
The Silver Layer is your workspace for using Queries and Notebooks to clean, deduplicate, validate, or standardize data from the Bronze Layer.- Purpose: Enable custom workflows to prepare reliable, analysis-ready datasets.
- Benefits: Centralizes logic and definitions, creating consistent, high-quality data.
Gold Layer
The Gold Layer is where refined datasets become actionable. It supports Integrations with BI tools and AI agents, as well as Destinations and business applications.- Purpose: Serve final datasets to business users, analytics tools, or AI systems.
- Benefits: Fast, reliable, and user-friendly access to key business data.
Customizing Layers in Nekt
If you’re on the Growth plan, you can tailor your data structure even further:- Create new layers for domain-specific or project-specific pipelines.
- Rename layers to align with your team’s terminology or internal standards.