​

Configuring Vault as a Source

​

1. Add account access

• Vault Address: The URL of your Vault instance (e.g., https://vault.<your-company>.com.br/). The connector will automatically extract the base URL and use the /v1 API endpoint.
• Auth Token: Your Vault authentication token (Bearer token) used for API access. For KV v2 streams, the token needs list and read on the secret paths you use. For Identity streams, it needs list and read on /identity/entity/name, /identity/entity-alias/id, and/or /identity/group/name as needed.

​

2. Select streams

• kv_identity_list_entities: Lists all entity names; parent for entity details
• kv_identity_entity: Full entity data (aliases, policies, groups) for each entity name
• kv_identity_list_entity_aliases: Lists all entity-alias IDs; parent for alias details
• kv_identity_entity_alias: Full alias data (mount, canonical entity) for each alias ID
• kv_identity_list_groups: Lists all group names; parent for group details
• kv_identity_group: Full group data (members, policies, type) for each group name

• kv_list_secrets: Lists all secret paths and keys recursively from your Vault instance
• kv_secrets: Retrieves the actual secret data (key-value pairs) for each secret
• kv_subkeys: Retrieves the subkey structure for each secret (keys only, values are null)

Tip: The stream can be found more easily by typing its name.

​

3. Configure data streams

• Layer: choose between the existing layers on your catalog. This is where you will find your new extracted tables as the extraction runs successfully.
• Folder: a folder can be created inside the selected layer to group all tables being created from this new data source.
• Table name: we suggest a name, but feel free to customize it. You have the option to add a prefix to all tables at once and make this process faster!
• Sync Type: you can choose between INCREMENTAL and FULL_TABLE.
  • Incremental: every time the extraction happens, we’ll get only the new data - which is good if, for example, you want to keep every record ever fetched.
  • Full table: every time the extraction happens, we’ll get the current state of the data - which is good if, for example, you don’t want to have deleted data in your catalog.

​

4. Configure data source

• Configure Delta Log Retention and determine for how long we should store old states of this table as it gets updated. Read more about this resource here.
• Determine when to execute an Additional Full Sync. This will complement the incremental data extractions, ensuring that your data is completely synchronized with your source every once in a while.

​

5. Check your new source

​

Streams and Fields

​

Data Model

• kv_identity_list_entities lists entity names; kv_identity_entity uses each entity_name to fetch full entity data (aliases, policies, groups).
• kv_identity_list_entity_aliases lists entity-alias IDs; kv_identity_entity_alias uses each entity_alias_id to fetch full alias data (canonical entity, mount, metadata).
• kv_identity_list_groups lists group names; kv_identity_group uses each group_name to fetch full group data (members, policies, type).

• kv_list_secrets is the parent stream that discovers all secrets recursively.
• Both kv_secrets and kv_subkeys are child streams that depend on kv_list_secrets; each record provides a secret_path context.
• Child streams automatically skip folder records (where is_folder is true).

​

Use Cases for Data Analysis

​

Identity: Entity and Group Inventory

​

1. Secret Inventory and Organization

• Maintain an inventory of all secrets across your organization
• Identify secrets by their location and path structure
• Track which secrets are folders vs. actual secret data
• Audit secret organization and naming conventions

SELECT
   path,
   key,
   is_folder,
   CASE
      WHEN is_folder THEN 'Folder'
      ELSE 'Secret'
   END AS type
FROM
   nekt_raw.vault_kv_list_secrets
ORDER BY
   path,
   key

SELECT
   path,
   key,
   is_folder,
   CASE
      WHEN is_folder THEN 'Folder'
      ELSE 'Secret'
   END AS type
FROM
   `nekt_raw.vault_kv_list_secrets`
ORDER BY
   path,
   key

​

2. Secret Data Analysis

• Understand what data is stored in each secret
• Track secret versions and creation times
• Identify secrets that have been deleted or destroyed
• Monitor custom metadata associated with secrets

SELECT
   path,
   version,
   data,
   metadata.created_time,
   metadata.destroyed,
   metadata.deletion_time,
   metadata.custom_metadata
FROM
   nekt_raw.vault_kv_secrets
WHERE
   metadata.destroyed = false
   AND metadata.deletion_time = ''
ORDER BY
   path,
   version DESC

SELECT
   path,
   version,
   data,
   metadata.created_time,
   metadata.destroyed,
   metadata.deletion_time,
   metadata.custom_metadata
FROM
   `nekt_raw.vault_kv_secrets`
WHERE
   metadata.destroyed = false
   AND metadata.deletion_time = ''
ORDER BY
   path,
   version DESC

​

3. Secret Structure Analysis

• Understand the schema and structure of secrets
• Identify common key patterns across secrets
• Track changes in secret structure over time
• Plan migrations or reorganizations based on structure

SELECT
   path,
   version,
   subkeys,
   metadata.created_time,
   metadata.version
FROM
   nekt_raw.vault_kv_subkeys
WHERE
   metadata.destroyed = false
ORDER BY
   path,
   version DESC

SELECT
   path,
   version,
   subkeys,
   metadata.created_time,
   metadata.version
FROM
   `nekt_raw.vault_kv_subkeys`
WHERE
   metadata.destroyed = false
ORDER BY
   path,
   version DESC

​

4. Secret Version Tracking

• Monitor secret version history
• Identify recently created or modified secrets
• Track secret lifecycle and changes over time
• Audit secret management practices

WITH secret_versions AS (
   SELECT
      path,
      version,
      metadata.created_time,
      metadata.destroyed,
      metadata.deletion_time,
      ROW_NUMBER() OVER (PARTITION BY path ORDER BY version DESC) AS version_rank
   FROM
      nekt_raw.vault_kv_secrets
)
SELECT
   path,
   MAX(version) AS latest_version,
   COUNT(*) AS total_versions,
   MIN(metadata.created_time) AS first_created,
   MAX(metadata.created_time) AS last_modified,
   SUM(CASE WHEN metadata.destroyed THEN 1 ELSE 0 END) AS destroyed_versions,
   SUM(CASE WHEN metadata.deletion_time != '' THEN 1 ELSE 0 END) AS deleted_versions
FROM
   nekt_raw.vault_kv_secrets
GROUP BY
   path
ORDER BY
   last_modified DESC

WITH
   secret_versions AS (
      SELECT
         path,
         version,
         metadata.created_time,
         metadata.destroyed,
         metadata.deletion_time,
         ROW_NUMBER() OVER (
            PARTITION BY path
            ORDER BY
               version DESC
         ) AS version_rank
      FROM
         `nekt_raw.vault_kv_secrets`
   )
SELECT
   path,
   MAX(version) AS latest_version,
   COUNT(*) AS total_versions,
   MIN(metadata.created_time) AS first_created,
   MAX(metadata.created_time) AS last_modified,
   SUM(CASE WHEN metadata.destroyed THEN 1 ELSE 0 END) AS destroyed_versions,
   SUM(CASE WHEN metadata.deletion_time != '' THEN 1 ELSE 0 END) AS deleted_versions
FROM
   `nekt_raw.vault_kv_secrets`
GROUP BY
   path
ORDER BY
   last_modified DESC

​

Implementation Notes

​

Data Quality Considerations

• The connector uses both the Vault Identity API (/identity/entity/name, /identity/entity-alias/id, /identity/group/name) and the KV v2 secrets engine (secret/ mount path).
• Identity: Parent streams call list endpoints; child streams request each item by name or ID. Ensure the token has list and read on the Identity paths you use.
• KV v2: Paths with spaces and special characters are automatically URL-encoded. Secrets that return 404 are gracefully skipped (not treated as errors). Folder records are automatically filtered out from child streams. The connector recursively traverses all folders and subfolders.

​

API Limits & Performance

• A 1-second delay is applied between requests to avoid rate limiting
• For large Vault instances with many secrets, extraction may take some time
• The connector handles recursive folder traversal efficiently by tracking visited folders
• Child streams only process individual secrets (folders are automatically skipped)

​

Security Considerations

• Authentication tokens are stored securely and marked as secret fields
• The connector only reads secrets (does not modify or delete them)
• Ensure your authentication token has appropriate read permissions for the KV v2 secrets engine
• Consider using Vault policies to restrict access to only necessary paths