> ## Documentation Index
> Fetch the complete documentation index at: https://docs.nekt.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Apify as a data source

> Bring data from Apify to Nekt.

Apify is a web scraping and automation platform that allows you to build, run, and scale web scrapers, crawlers, and automation tools called Actors. It stores results in datasets that can be accessed via API, making it easy to collect and process data from any website.

<img width="200" src="https://mintlify.s3.us-west-1.amazonaws.com/nekt/assets/logo/logo-apify.png" />

## Configuring Apify as a Source

In the [Sources](https://app.nekt.ai/sources) tab, click on the "Add source" button located on the top right of your screen. Then, select the Apify option from the list of connectors.

Click **Next** and you'll be prompted to add your access.

### 1. Add account access

You'll need to provide your Apify API token for authentication. You can find your API token in the Apify Console under **Settings > API & Integrations**.

The following configurations are available:

* **API Key**: Your Apify API token used for authentication. This is required.

* **Dataset IDs** *(optional)*: A list of dataset IDs to extract. You can find dataset IDs in the Apify Console under **Storage > Datasets**, or in the output tab of your Actor runs. Note that most Apify datasets (especially those created by Actor runs) are unnamed and won't appear in generic listings, so you must provide their IDs explicitly.

* **Actor IDs** *(optional)*: A list of Actor IDs to extract run data from. The tap will incrementally sync succeeded runs and their dataset items. You can find Actor IDs in the Apify Console under **Actors**, or use the format `username~actor-name`. This is ideal when you have scheduled Actors running periodically and want to automatically sync their results without manually tracking dataset IDs.

<Note>You must provide at least one of **Dataset IDs** or **Actor IDs**. You can also use both at the same time.</Note>

Once you're done, click **Next**.

### 2. Select streams

Choose which data streams you want to sync. For faster extractions, select only the streams that are relevant to your analysis.

> Tip: The stream can be found more easily by typing its name.

Select the streams and click **Next**.

### 3. Configure data streams

Customize how you want your data to appear in your catalog. Select the desired layer where the data will be placed, a folder to organize it inside the layer, a name for each table and the type of sync.

* **Layer**: choose between the existing layers on your catalog. This is where you will find your new extracted tables as the extraction runs successfully.
* **Folder**: a folder can be created inside the selected layer to group all tables being created from this new data source.
* **Table name**: we suggest a name, but feel free to customize it. You have the option to add a **prefix** to all tables at once and make this process faster!
* **Sync Type**: you can choose between INCREMENTAL and FULL\_TABLE.
  * Incremental: every time the extraction happens, we'll get only the new data.
  * Full table: every time the extraction happens, we'll get the current state of the data.

Once you are done configuring, click **Next**.

### 4. Configure data source

Describe your data source for easy identification within your organization, not exceeding 140 characters.

To define your [Trigger](https://docs.nekt.com/runs/scheduling-and-triggers), consider how often you want data to be extracted from this source.

Optionally, you can define some additional settings:

* Configure Delta Log Retention and determine for how long we should store old states of this table as it gets updated. Read more about this resource [here](https://docs.nekt.com/get-started/core-concepts/resource-control).
* Determine when to execute an **Additional [Full Sync](https://docs.nekt.com/get-started/core-concepts/types-of-sync#additional-full-sync)**.

Once you are ready, click **Next** to finalize the setup.

### 5. Check your new source

You can view your new source on the [Sources](https://app.nekt.ai/sources) page. If needed, manually trigger the source extraction by clicking on the arrow button. Once executed, your data will appear in your Catalog.

<Warning>For you to be able to see it on your [Catalog](https://app.nekt.ai/catalog), you need at least one successful source run.</Warning>

# Streams and Fields

Below you'll find all available data streams from Apify and their corresponding fields:

<AccordionGroup>
  <Accordion title="Datasets">
    Lists all datasets in your Apify account. This stream supports incremental sync based on the `modifiedAt` field.

    | Field            | Type      | Description                                  |
    | :--------------- | :-------- | :------------------------------------------- |
    | `id`             | String    | Unique identifier of the dataset             |
    | `name`           | String    | Name of the dataset                          |
    | `createdAt`      | Date-time | Timestamp when the dataset was created       |
    | `modifiedAt`     | Date-time | Timestamp when the dataset was last modified |
    | `accessedAt`     | Date-time | Timestamp when the dataset was last accessed |
    | `itemCount`      | Integer   | Total number of items in the dataset         |
    | `cleanItemCount` | Integer   | Number of clean (deduplicated) items         |
    | `actId`          | String    | ID of the Actor that created the dataset     |
    | `actRunId`       | String    | ID of the Actor run that created the dataset |
    | `stats`          | String    | Dataset statistics as a JSON string          |
    | `fields`         | String    | Dataset field definitions as a JSON string   |
  </Accordion>

  <Accordion title="Dataset Items">
    Retrieves all items (records) stored in each dataset. Each item is serialized as a JSON string in the `data` field, preserving the original structure from the Actor run. This stream supports incremental sync based on the `run_started_at` field.

    | Field            | Type      | Description                                     |
    | :--------------- | :-------- | :---------------------------------------------- |
    | `dataset_id`     | String    | ID of the parent dataset                        |
    | `actor_id`       | String    | ID of the Actor that generated the dataset      |
    | `run_id`         | String    | ID of the Actor run                             |
    | `run_started_at` | Date-time | Timestamp when the Actor run started            |
    | `_row_index`     | Integer   | Sequential index of the item within the dataset |
    | `data`           | String    | The dataset item serialized as a JSON string    |
  </Accordion>

  <Accordion title="Actor Runs">
    Lists all succeeded runs for each configured Actor, with incremental sync based on the `startedAt` field. Each run includes metadata such as cost, schedule info, and a reference to its default dataset.

    | Field                    | Type      | Description                                                                     |
    | :----------------------- | :-------- | :------------------------------------------------------------------------------ |
    | `id`                     | String    | Unique identifier of the Actor run                                              |
    | `actId`                  | String    | ID of the Actor                                                                 |
    | `userId`                 | String    | ID of the user who triggered the run                                            |
    | `actorTaskId`            | String    | ID of the Actor task (if run via a task)                                        |
    | `status`                 | String    | Run status (always SUCCEEDED for this stream)                                   |
    | `startedAt`              | Date-time | Timestamp when the run started                                                  |
    | `finishedAt`             | Date-time | Timestamp when the run finished                                                 |
    | `buildId`                | String    | ID of the Actor build used                                                      |
    | `buildNumber`            | String    | Semantic version of the build                                                   |
    | `usageTotalUsd`          | Number    | Total cost of the run in USD                                                    |
    | `defaultKeyValueStoreId` | String    | ID of the run's default key-value store                                         |
    | `defaultDatasetId`       | String    | ID of the run's default dataset                                                 |
    | `defaultRequestQueueId`  | String    | ID of the run's default request queue                                           |
    | `meta`                   | Object    | Run metadata including origin, scheduleId, scheduledAt, clientIp, and userAgent |
  </Accordion>

  <Accordion title="Actor Run Dataset Items">
    Retrieves all items from each Actor run's default dataset. This is a child stream of Actor Runs, meaning it automatically fetches dataset items for every new run synced. Each item is serialized as a JSON string in the `data` field. This stream supports incremental sync based on the `run_started_at` field.

    | Field            | Type      | Description                                                                                |
    | :--------------- | :-------- | :----------------------------------------------------------------------------------------- |
    | `dataset_id`     | String    | ID of the dataset (from the Actor run)                                                     |
    | `actor_id`       | String    | ID of the Actor that produced this data                                                    |
    | `run_id`         | String    | ID of the Actor run                                                                        |
    | `run_started_at` | Date-time | Timestamp when the parent Actor run started (used as replication key for incremental sync) |
    | `_row_index`     | Integer   | Sequential index of the item within the dataset                                            |
    | `data`           | String    | The dataset item serialized as a JSON string                                               |
  </Accordion>

  <Accordion title="Dataset Statistics">
    Retrieves detailed metadata and statistics for each dataset. This stream supports incremental sync based on the `modifiedAt` field.

    | Field            | Type      | Description                                  |
    | :--------------- | :-------- | :------------------------------------------- |
    | `id`             | String    | Unique identifier of the dataset             |
    | `name`           | String    | Name of the dataset                          |
    | `createdAt`      | Date-time | Timestamp when the dataset was created       |
    | `modifiedAt`     | Date-time | Timestamp when the dataset was last modified |
    | `accessedAt`     | Date-time | Timestamp when the dataset was last accessed |
    | `itemCount`      | Integer   | Total number of items in the dataset         |
    | `cleanItemCount` | Integer   | Number of clean (deduplicated) items         |
    | `actId`          | String    | ID of the Actor that created the dataset     |
    | `actRunId`       | String    | ID of the Actor run that created the dataset |
  </Accordion>
</AccordionGroup>

# Data Model

The following diagram illustrates the relationships between the core data streams in Apify.

```mermaid theme={null}
erDiagram
    DATASETS {
        string id PK
        string actId
        string actRunId
    }
    ACTOR_RUNS {
        string id PK
        string actId
        string defaultDatasetId FK
    }
    DATASET_ITEMS {
        string dataset_id FK
        integer _row_index PK
    }
    ACTOR_RUN_DATASET_ITEMS {
        string dataset_id FK
        string run_id FK
        integer _row_index PK
    }
    DATASET_STATISTICS {
        string id PK
    }

    ACTOR_RUNS ||--|| DATASETS : generates
    DATASETS ||--o{ DATASET_ITEMS : contains
    DATASETS ||--|| DATASET_STATISTICS : has
    ACTOR_RUNS ||--o{ ACTOR_RUN_DATASET_ITEMS : has
```

## Skills for agents

<Snippet file="agent-skills-intro.mdx" />

<Card title="Download Apify skills file" icon="wand-magic-sparkles" href="/sources/apify.md">
  Apify connector documentation as plain markdown, for use in AI agent contexts.
</Card>