Skip to main content
Apify is a web scraping and automation platform that allows you to build, run, and scale web scrapers, crawlers, and automation tools called Actors. It stores results in datasets that can be accessed via API, making it easy to collect and process data from any website.

​
Configuring Apify as a Source

In the Sources tab, click on the “Add source” button located on the top right of your screen. Then, select the Apify option from the list of connectors. Click Next and you’ll be prompted to add your access.

​
1. Add account access

You’ll need to provide your Apify API token for authentication. You can find your API token in the Apify Console under Settings > API & Integrations. The following configurations are available:
  • API Key: Your Apify API token used for authentication. This is required.
  • Dataset IDs (optional): A list of dataset IDs to extract. You can find dataset IDs in the Apify Console under Storage > Datasets, or in the output tab of your Actor runs. Note that most Apify datasets (especially those created by Actor runs) are unnamed and won’t appear in generic listings, so you must provide their IDs explicitly.
  • Actor IDs (optional): A list of Actor IDs to extract run data from. The tap will incrementally sync succeeded runs and their dataset items. You can find Actor IDs in the Apify Console under Actors, or use the format username~actor-name. This is ideal when you have scheduled Actors running periodically and want to automatically sync their results without manually tracking dataset IDs.
You must provide at least one of Dataset IDs or Actor IDs. You can also use both at the same time.
Once you’re done, click Next.

​
2. Select streams

Choose which data streams you want to sync. For faster extractions, select only the streams that are relevant to your analysis.
Tip: The stream can be found more easily by typing its name.
Select the streams and click Next.

​
3. Configure data streams

Customize how you want your data to appear in your catalog. Select the desired layer where the data will be placed, a folder to organize it inside the layer, a name for each table and the type of sync.
  • Layer: choose between the existing layers on your catalog. This is where you will find your new extracted tables as the extraction runs successfully.
  • Folder: a folder can be created inside the selected layer to group all tables being created from this new data source.
  • Table name: we suggest a name, but feel free to customize it. You have the option to add a prefix to all tables at once and make this process faster!
  • Sync Type: you can choose between INCREMENTAL and FULL_TABLE.
    • Incremental: every time the extraction happens, we’ll get only the new data.
    • Full table: every time the extraction happens, we’ll get the current state of the data.
Once you are done configuring, click Next.

​
4. Configure data source

Describe your data source for easy identification within your organization, not exceeding 140 characters. To define your Trigger, consider how often you want data to be extracted from this source. Optionally, you can define some additional settings:
  • Configure Delta Log Retention and determine for how long we should store old states of this table as it gets updated. Read more about this resource here.
  • Determine when to execute an Additional Full Sync.
Once you are ready, click Next to finalize the setup.

​
5. Check your new source

You can view your new source on the Sources page. If needed, manually trigger the source extraction by clicking on the arrow button. Once executed, your data will appear in your Catalog.
For you to be able to see it on your Catalog, you need at least one successful source run.

​
Streams and Fields

Below you’ll find all available data streams from Apify and their corresponding fields:
Lists all datasets in your Apify account. This stream supports incremental sync based on the modifiedAt field.
Retrieves all items (records) stored in each dataset. Each item is serialized as a JSON string in the data field, preserving the original structure from the Actor run. This stream supports incremental sync based on the run_started_at field.
Lists all succeeded runs for each configured Actor, with incremental sync based on the startedAt field. Each run includes metadata such as cost, schedule info, and a reference to its default dataset.
Retrieves all items from each Actor run’s default dataset. This is a child stream of Actor Runs, meaning it automatically fetches dataset items for every new run synced. Each item is serialized as a JSON string in the data field. This stream supports incremental sync based on the run_started_at field.
Retrieves detailed metadata and statistics for each dataset. This stream supports incremental sync based on the modifiedAt field.

​
Data Model

The following diagram illustrates the relationships between the core data streams in Apify.

​
Skills for agents

Download Apify skills file

Apify connector documentation as plain markdown, for use in AI agent contexts.