PostgreSQL as a destination
Send your data from the Catalog to PostgreSQL.
PostgreSQL is a powerful open-source object-relational database system that extends the SQL language with advanced features. It’s known for its reliability, data integrity, and support for complex queries, making it ideal for enterprise applications, data warehousing, and web services.
1. Add your PostgreSQL access
-
In the Destinations tab, click on the “Add destination” button located on the top right of your screen. Then, select the Postgres option from the list of connectors.
-
Click Next and you’ll be prompted to add your access: host, port, user, password, database name and target schema.
-
Define your load method: append, upsertor overwrite.
Load methods
- Append: This method will simply add all records from the incoming dataset to the existing table at the destination, without removing or updating any existing data. You can only use this method if a primary key was not defined for the input table. - Upsert: This method will check for records with matching IDs in the destination table. If a match is found, the existing record will be updated with the new data. If no match is found, the new record will be inserted. In other words, it merges the incoming dataset with the existing table based on the record ID. - Overwrite: This method will completely replace all existing data in the destination table with the incoming dataset. Any data that was previously in the table will be deleted.
-
Decide about:
-
Enabling SSH Tunnel: When this is turned on, the system will route the connection through the configured SSH tunnel to reach the PostgreSQL database. The SSH tunnel is an intermediary machine required when the PostgreSQL is not directly accessible over the internet.
-
Enabling Interactive Tunnel Authentication: In some environments, logging into the SSH tunnel may require interactive authentication steps. Enabling this option allows the connection process to simulate such interactive behavior, which is sometimes necessary for the tunnel to be established successfully.
We do not recommend enabling it initially. If your extraction fails, edit the source and activate this option to try again.
-
Enabling SSL: Activates SSL encryption for the database connection. This is required when the PostgreSQL server enforces secure connections using SSL certificates. It ensures encrypted communication between your system and the database, improving security.
-
Adding record metadata: If active, it will add metadata columns with timestamps related to the loading process. Read more about each column below.
Record metadata
_sdc_extracted_at
: Timestamp indicating when record was extracted from the source -_sdc_received_at
: Timestamp indicating when the destination received the record for loading -_sdc_batched_at
: Timestamp indicating when the destination loaded the batch the record was a part of into the destination -_sdc_sequence
: A Unix epoch (in nanoseconds) that indicates the order in which data points were considered for loading -_sdc_table_version
: Indicates the version of the table. Used to determine when to issue TRUNCATE commands during loading, where applicable.
- Click Next.
2. Select your catalog data to send
-
The next step is letting us know which data you want to send to the database. Select the layer and then the desired table.
Tip: The table can be found more easily by typing its name.
-
Define which column from the selected table should be used as primary key on the destination database.
-
Click Next.
3. Configure your PostgreSQL data destination
-
Describe your destination for easy identification within your organization. You can inform things like what data it sends, to which team it belongs, etc.
-
To define your Trigger, consider how often you want data to be sent to this destination. This decision usually depends on how frequently you need the new table data updated (every day, once a week, only at specific times, etc.).
-
Click Done.
Check your new destination!
-
Once completed, you’ll receive confirmation that your new destination is set up!
-
You can view your new destination on the Destinations page. Now, for you to be able to see it on your PostgreSQL database, you have to wait for the pipeline to run. You can monitor it on the Destinations page to see its execution and completion. If needed, manually trigger the pipeline by clicking on the refresh icon. Once executed, your data should be seen on PostgreSQL.
If you encounter any issues, reach out to us via Slack, and we’ll gladly assist you!