Configuring Redshift as a Source
In the Sources tab, click on the “Add source” button located on the top right of your screen. Then, select the Redshift option from the list of connectors. Click Next and you’ll be prompted to add your access.1. Add account access
You can connect to Redshift using the following parameters:- Host: Hostname for your Redshift instance. Use only the host endpoint instead of the full connection string (for example:
nekt.us-east-1.redshift-serverless.amazonaws.com). - Port: The port on which Redshift is awaiting connection (default: 5439).
- User: Username used to authenticate.
- Password: Password used to authenticate.
- Database name: The name of the database to connect to.
- Filter schemas: Filter the schemas you want to sync from your database. Using this filter helps speed up the discovery process, since schemas that aren’t included in the filter will be ignored.
You must ensure the Nekt Public IP for your workspace is allow-listed in to access your Redshift instance. You can find the IP in the Redshift source configuration page.
2. Select streams
After configuring the connection, the connector will automatically discover all available tables in your Redshift database (based on your schema filter settings). This is how the discovery process works:- Connects to your Redshift instance using the provided credentials
- Queries the system catalogs to identify available schemas and tables
- For each table:
- Retrieves column definitions and data types
- Identifies primary keys and indexes
- Determines appropriate replication methods
- Generates a catalog of all discoverable streams
Tip: Tables can be found more easily by typing their name.Select the tables and click Next.
3. Configure data streams
Customize how you want your data to appear in your catalog. Select a name for each table (which will contain the fetched data) and the type of sync.- Table name: we suggest a name, but feel free to customize it. You have the option to add a prefix and make this process faster!
-
Sync Type: you can choose between INCREMENTAL and FULL_TABLE.
- Incremental: every time the extraction happens, we’ll get only the new data - which is good if, for example, you want to have more efficient extractions and keep every record ever fetched.
- Full table: every time the extraction happens, we’ll get the current state of the data - which is good if, for example, you don’t want to have deleted data in your catalog.
For incremental syncs, you must provide an incremental key to be used as a query filter during the data extraction process. The recommended approach is using a date column that indicates when the record was last modified. You can also use IDs or creation date columns, however it only works for records that are not modified after creation.Once you are done configuring, click Next.
4. Configure data source
Describe your data source for easy identification within your organization, not exceeding 140 characters. To define your Trigger, consider how often you want data to be extracted from this source. This decision usually depends on how frequently you need the new table data updated (every day, once a week, or only at specific times). Optionally, you can determine when to execute a full sync. This will complement the incremental data extractions, ensuring that your data is completely synchronized with your source every once in a while. Once you are ready, click Next to finalize the setup.5. Check your new source
You can view your new source on the Sources page. If needed, manually trigger the source extraction by clicking on the arrow button. Once executed, your data will appear in your Catalog.For you to be able to see it on your Catalog, you need at least one successful source run.