Layers

​

The Layered Data Architecture

In data engineering, a layered architecture is a method of organizing data pipelines by separating data into logical stages based on its level of processing, trust, and intended use. This design pattern improves clarity, maintainability, and governance in modern data systems.

Each layer represents a distinct phase in the data lifecycle, from raw ingestion to refined, analysis-ready outputs.

​

Core Principles of Layered Data Models

• Separation of concerns: Each layer handles a specific part of the data journey—ingestion, transformation, or consumption.
• Data lineage and auditability: By preserving data at each stage, teams can trace changes and debug pipelines more easily.
• Scalability and maintainability: Teams can evolve pipelines independently within each layer.
• Quality enforcement: Layers provide natural checkpoints for validation, deduplication, and enrichment.

​

Typical Layers in Modern Data Systems

While naming conventions may vary, a common pattern includes:

• Raw Layer
  Stores data in its most original or minimally processed format, usually after ingestion from external sources.

• Staging/Trusted Layer
  A workspace for applying business rules, cleansing, validation, and transformation—turning raw inputs into structured, reliable outputs.

• Presentation/Service Layer
  Final layer used by reporting tools, analytics, dashboards, and machine learning systems. Data here is curated, documented, and ready for consumption.

​

Why Adopt a Layered Approach?

Using layers provides a strong foundation for building scalable and trustworthy data systems. Benefits include:

• Improved Accessibility
  Logical organization makes it easier to find and use the right data.

• Better Data Quality
  Clear separation ensures transformations are applied methodically and transparently.

• Streamlined Collaboration
  Different teams (data engineers, analysts, AI teams) can work in parallel across different layers.

​

How Nekt Implements Layers

In Nekt, the layered model is embedded in your workspace through the Catalog, which organizes your datasets by layers. This structure helps users manage data transformation pipelines intuitively, following best practices.

By default, every catalog comes with three layers:

• Raw
• Trusted
• Service

This structure is available to all customers, with expanded customization options on higher-tier plans.

​

Default Layers in Nekt

Raw Layer

The Raw Layer holds data that has already been converted into Delta tables, following ingestion from external sources. It retains the original fidelity of the data, while ensuring it’s optimized for processing and querying.

• Purpose: Store raw-but-structured data, ready for transformation.
• Benefits: Maintains data integrity and audit trail while enabling efficient access.

Trusted Layer

The Trusted Layer is your workspace for applying transformations—cleaning, deduplicating, validating, or standardizing data from the Raw Layer.

• Purpose: Enable custom workflows to prepare reliable, analysis-ready datasets.
• Benefits: Centralizes logic and definitions, creating consistent, high-quality data.

Service Layer

The Service Layer is where refined datasets become actionable. It supports integration with BI tools, dashboards, and ML pipelines.

• Purpose: Serve final datasets to business users, analytics tools, or AI systems.
• Benefits: Fast, trusted, and user-friendly access to key business data.

​

Customizing Layers in Nekt

If you’re on the Growth plan, you can tailor your data structure even further:

• Create new layers for domain-specific or project-specific pipelines.
• Rename layers to align with your team’s terminology or internal standards.

This flexibility lets you build a layered architecture that fits your team’s workflows and scales with your organization.

​

Explore Layers in Nekt

Want to see it in action? Head to the Catalog in Nekt and explore your workspace layers. On the Growth plan, you can fully customize this structure to support your team’s evolving data strategy.