> ## Documentation Index
> Fetch the complete documentation index at: https://docs.nekt.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Azure Cosmos DB as a data source
> Bring data from Azure Cosmos DB for NoSQL to Nekt.
Azure Cosmos DB is Microsoft's fully managed NoSQL database. This connector reads accounts that use the **NoSQL API** (also called the Core or SQL API), Cosmos DB's native API, which stores schemaless JSON documents and queries them with a SQL-like dialect.
Cosmos DB also offers APIs that emulate other databases. If your account uses the MongoDB API, connect it with the [MongoDB](/sources/mongo-db) source instead; if it uses the PostgreSQL API, use the [PostgreSQL](/sources/postgres) source. This connector is only for accounts on the NoSQL API.
An account organizes data as `databases → containers → items`. A container holds JSON documents; an item is one document. This connector discovers your databases and containers at run time and creates one stream per container.
## Configuring Azure Cosmos DB as a Source
In the [Sources](https://app.nekt.ai/sources) tab, click on the "Add source" button located on the top right of your screen. Then, select the Azure Cosmos DB option from the list of connectors.
Click **Next** and you'll be prompted to add your access.
### 1. Add account access
Check the instructions next to each configuration option to discover where you can find the required parameters for the connection. These are the available configurations for this source:
* **Account endpoint:** The URI of your Cosmos DB account. In the [Azure portal](https://portal.azure.com), open the account and copy the value of **URI** from the Overview page. It looks like `https://my-account.documents.azure.com:443/`.
* **Authentication type:** How Nekt authenticates against the account. Choose **Account key** unless your organization has disabled key-based access on the account, in which case choose **Microsoft Entra ID**.
* **Account key:** Required for account key authentication. In the Azure portal, open the account, go to **Keys**, and copy a key from the **Read-only Keys** tab. Nekt only reads data, so a read-only key is all it needs.
* **Directory (tenant) ID**, **Application (client) ID** and **Client secret:** Required for Microsoft Entra ID authentication. These come from the app registration you create for Nekt.
* **Databases:** Extract only these databases. Leave empty to discover every database the credentials can read.
* **Containers:** Extract only containers with these names, across the selected databases. Leave empty to discover every container.
* **Start date:** The earliest document modification date to extract on a stream's first incremental sync.
* **Max item count:** How many documents Cosmos DB returns per page. Lower it if extraction times out on containers with large documents.
* **Max throttling wait (seconds)** and **Max throttling retries:** How long, and how many times, to keep retrying while Cosmos DB is throttling the extraction because a container has run out of provisioned throughput.
* **Connection timeout (seconds):** How long to wait for Cosmos DB to answer a single request.
* **Preferred regions:** For accounts replicated across several Azure regions, the regions to read from in order of preference. Leave empty to read from the account's write region.
For Microsoft Entra ID authentication, the service principal needs the **Cosmos DB Built-in Data Reader** role on the account. This is a *data-plane* role and has to be assigned separately: the control-plane **Reader** role lets a principal see the account but not read any documents, which is the most common cause of a permission error here.
Once you're done, click **Next**.
### 2. Select streams
The next step is letting us know which streams you want to bring. Each stream in that list corresponds to one container in your account, named `{database}_{container}`. You can select entire groups of streams or only a subset of them.
Containers that hold no documents are not listed, since a stream with no documents would have nothing to write.
> Tip: The stream can be found more easily by typing its name.
Select the streams and click **Next**.
### 3. Configure data streams
Customize how you want your data to appear in your catalog. Select the desired layer where the data will be placed, a folder to organize it inside the layer, a name for each table (which will contain the fetched data) and the type of sync.
* **Layer**: choose between the existing layers on your catalog. This is where you will find your new extracted tables as the extraction runs successfully.
* **Folder**: a folder can be created inside the selected layer to group all tables being created from this new data source.
* **Table name**: we suggest the same name as the container, but feel free to customize it. You have the option to add a **prefix** to all tables at once and make this process faster!
* **Sync Type**: you can choose between INCREMENTAL and FULL\_TABLE.
* **Incremental**: every time the extraction happens, we'll get only the new data - which is good if, for example, you want to keep every record ever fetched.
* **Full table**: every time the extraction happens, we'll get the current state of the data - which is good if, for example, you don't want to have deleted data in your catalog. However, keep in mind this increases resource usage such as computing time and storage.
For incremental syncs, the connector defaults to `_ts`, a timestamp Cosmos DB maintains on every document. See [Choosing a replication key](#choosing-a-replication-key) before selecting a different field.
Once you are done configuring, click **Next**.
### 4. Configure data source
Describe your data source for easy identification within your organization, not exceeding 140 characters.
To define your [Trigger](https://docs.nekt.com/runs/scheduling-and-triggers), consider how often you want data to be extracted from this source. This decision usually depends on how frequently you need the new table data updated (every day, once a week, or only at specific times).
Optionally, you can define some additional settings:
* Configure Delta Log Retention and determine for how long we should store old states of this table as it gets updated. Read more about this resource [here](https://docs.nekt.com/get-started/core-concepts/resource-control).
* Determine when to execute an **Additional [Full Sync](https://docs.nekt.com/get-started/core-concepts/types-of-sync#additional-full-sync)**. This will complement the incremental data extractions, ensuring that your data is completely synchronized with your source every once in a while.
Once you are ready, click **Next** to finalize the setup.
### 5. Check your new source
You can view your new source on the [Sources](https://app.nekt.ai/sources) page. If needed, manually trigger the source extraction by clicking on the arrow button. Once executed, your data will appear in your Catalog.
For you to be able to see it on your [Catalog](https://app.nekt.ai/catalog), you need at least one successful source run.
# Streams and Fields
Streams correspond dynamically to the containers present in your account. Two documents in the same container can share nothing but their `id`, so a container has no fixed schema to map into columns. Every stream therefore has the same shape: the fields Cosmos DB maintains itself become columns, and the document travels whole in a JSON string.
All extracted streams have the following standard properties:
| Field | Type | Description |
| :---------------- | :--------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------- |
| `id` | String | The document identifier set by your application. Unique within a logical partition, but not necessarily within the container — see the note below. |
| `_rid` | String | The resource identifier assigned by Cosmos DB. Unique across the container and stable for the document's lifetime. This is the table's primary key. |
| `_etag` | String | A version tag Cosmos DB changes on every write, which can be used to detect updates. |
| `_ts` | Integer | The time of the document's last write, in seconds since 1970-01-01 UTC. Maintained by Cosmos DB and used as the default incremental replication key. |
| `partition_key` | String | The document's value for the container's partition key. Containers with a hierarchical partition key report the levels as a JSON array, in order. |
| `document` | String | A stringified JSON version of the document payload. |
| `replication_key` | String / Integer | When you choose a document field as the incremental key, that field is also extracted into its own column under its own name. |
# Implementation Notes
## Why `_rid` is the primary key
Cosmos DB guarantees `id` to be unique **within a logical partition**, not within a container. A container partitioned by `/tenant_id` can legitimately hold a document with `id` `order-1` for one tenant and another with the same `id` for a different tenant. Keying the table on `id` would treat those as the same record and keep only one of them, so the connector keys on `_rid`, which Cosmos DB guarantees to be unique across the container.
`_rid` is stable for as long as a document exists. It does change if a document is deleted and recreated, or if a container is migrated or restored from a backup — after any of those, run a full sync so the table matches the source again.
## Choosing a replication key
`_ts` is the default and the recommended choice. It exists on every document, and Cosmos DB updates it itself on every write, so no application behavior can leave it stale.
Any other top-level document field can be used instead, but two things are worth knowing before you do:
**Documents missing the field are never extracted.** An incremental query filters on the replication key, so a document that does not have that field never matches and never reaches your catalog. Cosmos DB also compares values by type: if some documents store a date as a string and others as a number, one incremental query only returns the ones matching the type of the stored bookmark, and the rest are silently skipped. The connector samples the field when discovering streams and warns you when it is missing or inconsistently typed.
The connector also re-reads the boundary of each incremental window. `_ts` has one-second resolution, so many documents can share a value; the query uses "greater than or equal to" the last value seen, which means the final second of the previous run is read again. Those records are simply overwritten in your catalog. The alternative — a strict comparison — would permanently drop every document written in that second.
## Full sync reads the whole container
A full sync ignores the **Start date** setting and always reads every document. A full sync replaces the destination table, so filtering it would not make the read cheaper — it would delete every older document from your catalog. **Start date** applies only to a stream's first incremental run.
## Request units and throttling
Cosmos DB bills by **request unit** (RU) rather than by request, and reading a whole container costs RUs in proportion to the data read. When a container runs out of provisioned throughput, Cosmos DB rejects requests until throughput frees up, and the extraction waits and retries.
If runs fail with a throughput error, you have three options: raise the container's provisioned throughput, switch the container to autoscale, or schedule the source outside your application's peak hours. Raising **Max throttling wait (seconds)** makes the extraction more patient, which helps when throttling is brief but does not help when a container is permanently short of throughput.
### Best practices
* **Filter to the containers you need.** Every container you extract costs request units on your Azure bill. Use the **Databases** and **Containers** settings to skip the ones you do not analyze; you can always add more streams later.
* **Prefer a read-only key.** Cosmos DB accounts have separate read-only keys. The connector never writes, so giving it a read-write key grants access it does not use.
* **Use `_ts` for incremental syncs** unless you have a specific reason to use an application field, and make sure that field exists with a consistent type on every document if you do.
## Skills for agents
Azure Cosmos DB connector documentation as plain markdown, for use in AI agent contexts.