​

Configuring Vault as a Source

​

1. Add account access

• Vault Address: The URL of your Vault instance (e.g., https://vault.<your-company>.com.br/). The connector will automatically extract the base URL and use the /v1 API endpoint.
• Auth Token: Your Vault authentication token (Bearer token) used for API access. This token should have appropriate permissions to read secrets from the KV v2 secrets engine.

​

2. Select streams

• kv_list_secrets: Lists all secret paths and keys recursively from your Vault instance
• kv_secrets: Retrieves the actual secret data (key-value pairs) for each secret
• kv_subkeys: Retrieves the subkey structure for each secret (keys only, values are null)

Tip: The stream can be found more easily by typing its name.

​

3. Configure data streams

• Layer: choose between the existing layers on your catalog. This is where you will find your new extracted tables as the extraction runs successfully.
• Folder: a folder can be created inside the selected layer to group all tables being created from this new data source.
• Table name: we suggest a name, but feel free to customize it. You have the option to add a prefix to all tables at once and make this process faster!
• Sync Type: you can choose between INCREMENTAL and FULL_TABLE.
  • Incremental: every time the extraction happens, we’ll get only the new data - which is good if, for example, you want to keep every record ever fetched.
  • Full table: every time the extraction happens, we’ll get the current state of the data - which is good if, for example, you don’t want to have deleted data in your catalog.

​

4. Configure data source

• Configure Delta Log Retention and determine for how long we should store old states of this table as it gets updated. Read more about this resource here.
• Determine when to execute an Additional Full Sync. This will complement the incremental data extractions, ensuring that your data is completely synchronized with your source every once in a while.

​

5. Check your new source

​

Streams and Fields

​

Data Model

• kv_list_secrets is the parent stream that discovers all secrets recursively
• Both kv_secrets and kv_subkeys are child streams that depend on kv_list_secrets
• Each record from kv_list_secrets provides a secret_path context that child streams use to fetch detailed data
• Child streams automatically skip folder records (where is_folder is true)

​

Use Cases for Data Analysis

​

1. Secret Inventory and Organization

• Maintain an inventory of all secrets across your organization
• Identify secrets by their location and path structure
• Track which secrets are folders vs. actual secret data
• Audit secret organization and naming conventions

SELECT
   path,
   key,
   is_folder,
   CASE
      WHEN is_folder THEN 'Folder'
      ELSE 'Secret'
   END AS type
FROM
   nekt_raw.vault_kv_list_secrets
ORDER BY
   path,
   key

SELECT
   path,
   key,
   is_folder,
   CASE
      WHEN is_folder THEN 'Folder'
      ELSE 'Secret'
   END AS type
FROM
   `nekt_raw.vault_kv_list_secrets`
ORDER BY
   path,
   key

​

2. Secret Data Analysis

• Understand what data is stored in each secret
• Track secret versions and creation times
• Identify secrets that have been deleted or destroyed
• Monitor custom metadata associated with secrets

SELECT
   path,
   version,
   data,
   metadata.created_time,
   metadata.destroyed,
   metadata.deletion_time,
   metadata.custom_metadata
FROM
   nekt_raw.vault_kv_secrets
WHERE
   metadata.destroyed = false
   AND metadata.deletion_time = ''
ORDER BY
   path,
   version DESC

SELECT
   path,
   version,
   data,
   metadata.created_time,
   metadata.destroyed,
   metadata.deletion_time,
   metadata.custom_metadata
FROM
   `nekt_raw.vault_kv_secrets`
WHERE
   metadata.destroyed = false
   AND metadata.deletion_time = ''
ORDER BY
   path,
   version DESC

​

3. Secret Structure Analysis

• Understand the schema and structure of secrets
• Identify common key patterns across secrets
• Track changes in secret structure over time
• Plan migrations or reorganizations based on structure

SELECT
   path,
   version,
   subkeys,
   metadata.created_time,
   metadata.version
FROM
   nekt_raw.vault_kv_subkeys
WHERE
   metadata.destroyed = false
ORDER BY
   path,
   version DESC

SELECT
   path,
   version,
   subkeys,
   metadata.created_time,
   metadata.version
FROM
   `nekt_raw.vault_kv_subkeys`
WHERE
   metadata.destroyed = false
ORDER BY
   path,
   version DESC

​

4. Secret Version Tracking

• Monitor secret version history
• Identify recently created or modified secrets
• Track secret lifecycle and changes over time
• Audit secret management practices

WITH secret_versions AS (
   SELECT
      path,
      version,
      metadata.created_time,
      metadata.destroyed,
      metadata.deletion_time,
      ROW_NUMBER() OVER (PARTITION BY path ORDER BY version DESC) AS version_rank
   FROM
      nekt_raw.vault_kv_secrets
)
SELECT
   path,
   MAX(version) AS latest_version,
   COUNT(*) AS total_versions,
   MIN(metadata.created_time) AS first_created,
   MAX(metadata.created_time) AS last_modified,
   SUM(CASE WHEN metadata.destroyed THEN 1 ELSE 0 END) AS destroyed_versions,
   SUM(CASE WHEN metadata.deletion_time != '' THEN 1 ELSE 0 END) AS deleted_versions
FROM
   nekt_raw.vault_kv_secrets
GROUP BY
   path
ORDER BY
   last_modified DESC

WITH
   secret_versions AS (
      SELECT
         path,
         version,
         metadata.created_time,
         metadata.destroyed,
         metadata.deletion_time,
         ROW_NUMBER() OVER (
            PARTITION BY path
            ORDER BY
               version DESC
         ) AS version_rank
      FROM
         `nekt_raw.vault_kv_secrets`
   )
SELECT
   path,
   MAX(version) AS latest_version,
   COUNT(*) AS total_versions,
   MIN(metadata.created_time) AS first_created,
   MAX(metadata.created_time) AS last_modified,
   SUM(CASE WHEN metadata.destroyed THEN 1 ELSE 0 END) AS destroyed_versions,
   SUM(CASE WHEN metadata.deletion_time != '' THEN 1 ELSE 0 END) AS deleted_versions
FROM
   `nekt_raw.vault_kv_secrets`
GROUP BY
   path
ORDER BY
   last_modified DESC

​

Implementation Notes

​

Data Quality Considerations

• The connector uses KV v2 secrets engine (secret/ mount path)
• Paths with spaces and special characters are automatically URL-encoded
• Secrets that return 404 are gracefully skipped (not treated as errors)
• Folder records are automatically filtered out from child streams
• The connector recursively traverses all folders and subfolders

​

API Limits & Performance

• A 1-second delay is applied between requests to avoid rate limiting
• For large Vault instances with many secrets, extraction may take some time
• The connector handles recursive folder traversal efficiently by tracking visited folders
• Child streams only process individual secrets (folders are automatically skipped)

​

Security Considerations

• Authentication tokens are stored securely and marked as secret fields
• The connector only reads secrets (does not modify or delete them)
• Ensure your authentication token has appropriate read permissions for the KV v2 secrets engine
• Consider using Vault policies to restrict access to only necessary paths