Amazon DynamoDB is a fully managed NoSQL database service provided by AWS that offers seamless scalability and high performance. It’s designed for applications that need consistent, single-digit millisecond performance at any scale, making it ideal for mobile, web, gaming, and IoT applications. By connecting DynamoDB to your Nekt catalog, you can centralize your NoSQL data alongside other data sources, enabling comprehensive analysis and data integration across your organization. This guide will walk you through the process of setting up DynamoDB as a data source, including the necessary AWS permissions configuration and connection settings. It’s important to note we support incremental sync for this source, making the whole process more efficient. For that to work, you need to have a valid replication key in your documents (either a date or timestamp field).

Configuring DynamoDB as a Source

In the Sources tab, click on the “Add source” button located on the top right of your screen. Then, select the DynamoDB option from the list of connectors. Click Next and you’ll be prompted to add your access.

1. Add account access

You need to define some permissions to allow Nekt to access your DynamoDB tables. Check the instructions below:

Setting permissions

Create role and add custom policy

Open the AWS Console using the account that hosts the DynamoDB table you’d like to extract data from.
Enter the IAM (Identity and Access Management) service page.
In the left panel under Access management, select Roles.
Click Create role.
In Trusted entity type, select Custom trust policy.
Change the Principal value to:

{
    "AWS": "arn:aws:iam::{AWS_ACCOUNT_ID}:role/nekt-ecs-task-role"
}

The variable must be replaced by the ID of the AWS account where the Nekt workspace is deployed. In order to find this information, click the dropdown in the top-right corner of the AWS console page after you log in with your account, then click on the “copy” icon to the right of your Account ID.

Check the full trust policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Statement1",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::{AWS_ACCOUNT_ID}:role/nekt-ecs-task-role"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

Click Next.
Click Next again without adding any permission. We’ll create an inline policy further on.
Under Role details, you can type any Role name as you want. Example: nekt-dynamodb-source.
Optionally, type a Description. Example: Used by Nekt to extract data from DynamoDB.
Click Create role .

You should get redirected back to the IAM roles page and a message of success should appear at the top.Add permissions to role

Open the role you just created by clicking View role in the right end of the green banner that appeared at the top of the page. If you dismissed it already, you can type the given Role name in the search field and then select it when it appears in the list.
Within Permission policies (0), click the Add permissions.
Select Create inline policy.
Within Policy editor, select JSON.
Paste the following policy in the Policy editor:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "dynamodb:BatchGetItem",
                "dynamodb:ConditionCheckItem",
                "dynamodb:DescribeContributorInsights",
                "dynamodb:Scan",
                "dynamodb:ListTagsOfResource",
                "dynamodb:Query",
                "dynamodb:DescribeTimeToLive",
                "dynamodb:PartiQLSelect",
                "dynamodb:DescribeTable",
                "dynamodb:GetItem",
                "dynamodb:DescribeContinuousBackups",
                "dynamodb:DescribeKinesisStreamingDestination",
                "dynamodb:DescribeTableReplicaAutoScaling"
            ],
            "Resource": "{DYNAMO_DB_ARN}"
        }
    ]
}

Replace the {DYNAMO_DB_ARN} part with the ARN of the DynamoDB table which you’d like to extract data from.
- If you don’t know it, open a new tab in your browser and enter the DynamoDB service page.
- In the left panel, click Tables.
- Open the desired table by clicking on its name in the list.
- Under General information, click Additional info.
- Click the “copy” icon just below Amazon Resource Name (ARN). A tool tip should appear containing the message ARN copied.
- Go back to the IAM policy editor that you left open in the previous tab and paste the ARN you just copied over {DYNAMO_DB_ARN}.

Full role policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "dynamodb:BatchGetItem",
                "dynamodb:ConditionCheckItem",
                "dynamodb:DescribeContributorInsights",
                "dynamodb:Scan",
                "dynamodb:ListTagsOfResource",
                "dynamodb:Query",
                "dynamodb:DescribeTimeToLive",
                "dynamodb:PartiQLSelect",
                "dynamodb:DescribeTable",
                "dynamodb:GetItem",
                "dynamodb:DescribeContinuousBackups",
                "dynamodb:DescribeKinesisStreamingDestination",
                "dynamodb:DescribeTableReplicaAutoScaling"
            ],
            "Resource": "arn:aws:dynamodb:{AWS_REGION}:{AWS_ACCOUNT_ID}:table/{TABLE_NAME}"
        }
    ]
}

Click Next.
Under Policy details type any Policy name as you want. Example: nekt-dynamodb-source-policy.
Click Create policy.

You should get redirected back to the role you just created and a message of success should appear at the top.

When setting up the connector at Nekt, you will use the ARN of the role you’ve just created.

The following configurations are available:

Table names: Provide the name of the tables you want to extract. Write them exactly as you see in your Dynamo DB.
Assume role ARN (AWS): The ARN role you’ve just created in the permissions setup step.
Extraction mode: Choose how records are extracted. Envelope extracts each item as a JSON document stored in a single column, while Infer schema (default) tries to infer a fixed schema based on a sample of records. Only use Infer schema if the records in your table have consistent types and fields across all documents.
Infer schema sample size: (Default: 500, only applies to Infer schema mode) Defines how many records you want to use to infer your table’s schema. The more consistent your schema is, the smaller your sample can be.
Incremental lookback window (days): (Default: 7) For tables replicated incrementally, how many days to re-scan before the last saved replication key value. Protects against DynamoDB’s eventually consistent reads silently skipping records that were mid-write during a previous scan. Set to 0 to disable.

Additionally, the following advanced configurations are available:

Per-table Scan overrides: Map a table name to boto3 Scan kwargs that override the default when reading that table. Example: {"MyTable": {"ConsistentRead": true, "ProjectionExpression": "id, updated_at"}}.
Incremental GSI partition key values: For a table with a Global Secondary Index (GSI) whose sort key matches its configured incremental replication key, provide the full list of partition key values on that index (e.g. shard values like SHARD#1,SHARD#2,SHARD#3). When set, incremental extraction Querys this GSI once per value instead of Scanning the whole table. DynamoDB has no way to enumerate these values automatically, so this list must be kept up to date manually — the connector warns if the data contains a value missing from it.

Once you’re done, click Next.

2. Select streams

Choose which data streams you want to sync. For DynamoDB, these correspond to the tables you configured in the previous step.

Tip: The stream can be found more easily by typing its name.

Select the streams and click Next.

3. Configure data streams

Customize how you want your data to appear in your catalog. Select the desired layer where the data will be placed, a folder to organize it inside the layer, a name for each table (which will effectively contain the fetched data) and the type of sync.

Layer: choose between the existing layers on your catalog. This is where you will find your new extracted tables as the extraction runs successfully.
Folder: a folder can be created inside the selected layer to group all tables being created from this new data source.
Table name: we suggest a name, but feel free to customize it. You have the option to add a prefix to all tables at once and make this process faster!
Sync Type: you can choose between INCREMENTAL and FULL_TABLE.
- Incremental: every time the extraction happens, we’ll get only the new data - which is good if, for example, you want to keep every record ever fetched.
- Full table: every time the extraction happens, we’ll get the current state of the data - which is good if, for example, you don’t want to have deleted data in your catalog.

If you define INCREMENTAL as the sync type for your table, you will have to add an incremental key. Incremental keys must be of integer (epoch seconds or milliseconds) or string types (formatted as ISO8601 dates, using either T or a space to separate date and time). To ensure consistency between extractions, make sure the field you select represents the last modification date of a document.

Once you are done configuring, click Next.

4. Configure data source

Describe your data source for easy identification within your organization, not exceeding 140 characters. To define your Trigger, consider how often you want data to be extracted from this source. This decision usually depends on how frequently you need the new table data updated (every day, once a week, or only at specific times). Optionally, you can define some additional settings:

Configure Delta Log Retention and determine for how long we should store old states of this table as it gets updated. Read more about this resource here.
Determine when to execute an Additional Full Sync. This will complement the incremental data extractions, ensuring that your data is completely synchronized with your source every once in a while.

Once you are ready, click Next to finalize the setup.

5. Check your new source

You can view your new source on the Sources page. If needed, manually trigger the source extraction by clicking on the arrow button. Once executed, your data will appear in your Catalog.

For you to be able to see it on your Catalog, you need at least one successful source run.

If you encounter any issues, reach out to us via Slack, and we’ll gladly assist you!

Streams and Fields

The DynamoDB connector dynamically exposes streams corresponding to your selected tables. The shape of each stream depends on the configured Extraction mode.

Infer schema mode (default)

Each selected DynamoDB table is mapped to a stream whose schema is inferred from a sample of records.

The schema is inferred based on a sample of records from each table (size controlled by Infer schema sample size).
The properties captured match the fields present in the sampled documents.
Primary keys come from the DynamoDB table’s key schema.
If you are running incremental syncs, make sure the configured replication key is present in all records.

Envelope mode

Each item is serialized into a standardized envelope, which is robust to inconsistent document shapes.Key fields:

Field	Type	Description
`_hash_id`	String	Unique identifier for the item, generated by hashing the primary keys of the DynamoDB record.
`document`	String	A JSON string containing the full serialized DynamoDB item.

If an incremental replication key is configured, it is also exposed as its own top-level column alongside _hash_id and document.Serialization Notes:The entire DynamoDB item is flattened into a single JSON string within the document field. During this process, specific native data types are serialized to ensure proper formatting downstream:

Decimals: Automatically converted to standard numeric formats (int or float).
Dates/Timestamps: Serialized to ISO 8601 standard strings.
Sets: Converted to standard JSON Arrays (lists).
Bytes: Decoded into UTF-8 strings.

Implementation Notes

Eventual Consistency and Incremental Syncs

To optimize AWS costs and reduce Read Capacity Units (RCU) usage, the connector performs eventually consistent reads by default. When using incremental syncs, a trailing lookback window (configurable via the Incremental lookback window (days) option) is applied to your replication key. This ensures that records that were mid-write during previous scans are caught in subsequent runs and not silently skipped, safely bridging the gap introduced by eventual consistency. For string replication keys, the lookback shifts only the calendar date — the time of day, fractional seconds, timezone, and the original date/time separator (T per ISO 8601, or a space) are preserved verbatim, so the shifted value always matches the exact format used by your source table.

Query-based incremental extraction via GSI

By default, incremental syncs Scan the whole table and filter on the replication key. If a table has a Global Secondary Index whose sort key matches the configured replication key, the connector auto-detects it; when you also configure Incremental GSI partition key values, incremental runs Query that index once per partition key value instead of Scanning the entire table, reading only changed records. The auto-detected GSI must have a projection type of ALL: a GSI only carries the attributes it projects, so Querying one that projects less than ALL would return incomplete records — the connector detects this and falls back to Scan instead. If a matching GSI exists but the partition key values aren’t configured, the connector also keeps using Scan and logs a hint. The very first run for a stream (no bookmark yet) always uses Scan. If dynamodb:Query is denied on the index ARN (a separate IAM resource from the table ARN), the connector falls back to a full-table Scan and reports it in the end-of-run summary.

Transformation example: extracting fields from the document

When using Envelope mode, the document column stores all event attributes as a single JSON string. To analyze specific properties in Explorer or downstream models, you can parse the JSON and expose the keys as separate columns.

SQL transformation (AWS Athena / GCP BigQuery)

AWS (Athena)
GCP (BigQuery)

SELECT
   _hash_id,
   json_extract_scalar(document, '$.item_name') AS item_name,
   json_extract_scalar(document, '$.status') AS status,
   CAST(json_extract_scalar(document, '$.price') AS DOUBLE) AS price
FROM
   nekt_raw.dynamodb_your_table

Replace nekt_raw.dynamodb_your_table with your actual layer and table name. Use json_extract_scalar() for string properties; for numeric properties use CAST(json_extract_scalar(document, '$.key') AS DOUBLE).

SELECT
   _hash_id,
   JSON_VALUE(document, '$.item_name') AS item_name,
   JSON_VALUE(document, '$.status') AS status,
   SAFE_CAST(JSON_VALUE(document, '$.price') AS FLOAT64) AS price
FROM
   `nekt_raw.dynamodb_your_table`

Replace nekt_raw.dynamodb_your_table with your actual dataset and table name. Use JSON_VALUE() for string properties; for numeric properties use SAFE_CAST(JSON_VALUE(document, '$.key') AS FLOAT64).

Skills for agents

Download DynamoDB skills file

DynamoDB connector documentation as plain markdown, for use in AI agent contexts.

Documentation

Get started

Context

Activate

Workspace

Resources

DynamoDB as a data source

Configuring DynamoDB as a Source

1. Add account access

2. Select streams

3. Configure data streams

4. Configure data source

5. Check your new source

Streams and Fields

Implementation Notes

Eventual Consistency and Incremental Syncs

Query-based incremental extraction via GSI

Transformation example: extracting fields from the document

Skills for agents

Download DynamoDB skills file

​Configuring DynamoDB as a Source

​1. Add account access

​2. Select streams

​3. Configure data streams

​4. Configure data source

​5. Check your new source

​Streams and Fields

​Implementation Notes

​Eventual Consistency and Incremental Syncs

​Query-based incremental extraction via GSI

​Transformation example: extracting fields from the document

​Skills for agents

Download DynamoDB skills file

Configuring DynamoDB as a Source

1. Add account access

2. Select streams

3. Configure data streams

4. Configure data source

5. Check your new source

Streams and Fields

Implementation Notes

Eventual Consistency and Incremental Syncs

Query-based incremental extraction via GSI

Transformation example: extracting fields from the document

Skills for agents