Parquet AWS S3 as a destination

​

1. Add your AWS S3 access

1. In the Destinations tab, click on the “Add destination” button located on the top right of your screen. Then, select the Parquet (AWS S3) option from the list of connectors.
2. Click Next and you’ll be prompted to add your access:

​

Required Configuration

• Bucket name: The AWS S3 bucket name where you want to store the Parquet files.

​

AWS Authentication

1. Access keys: provide an Access key ID and Secret access key created in the AWS IAM console.
2. IAM role: provide an Assume IAM Role ARN (e.g. arn:aws:iam::123456789012:role/MyRole). Nekt assumes the role via STS with automatically refreshed credentials, so you don’t need to share access keys.
3. None: leave all authentication fields empty to use the AWS account where your pipelines run (e.g. for Nekt-managed buckets, pattern: nekt-lakehouse-{company-name}).

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:GetObject",
        "s3:ListBucket",
        "s3:DeleteObject"
      ],
      "Resource": [
        "arn:aws:s3:::your-bucket-name",
        "arn:aws:s3:::your-bucket-name/*"
      ]
    }
  ]
}

​

File Configuration

• Output folder path: Folder path where to store the output tables inside the S3 bucket. Leave empty if the tables should be stored in the bucket’s root folder.
  • Example: data/exports or reports/daily
• Table folder name: A custom folder name for the output table. If empty, the name of the table being sent is used. When several tables are sent, use the per-table folder name instead.
• Compression: (Advanced, default: snappy) Compression codec used when writing the Parquet files (snappy or gzip).

3. Click Next.

​

Output layout

s3://{bucket}/{output folder path}/{table folder}/_nekt_sync_at={sync timestamp}/*.parquet

​

2. Select your data to send

1. The next step is letting us know which data you want to export to S3. Select the layer and then the desired table(s).

   Tip: Tables can be found more easily by typing their names.

2. Click Next.

​

3. Configure your Parquet (AWS S3) data destination

1. Describe your destination for easy identification within your organization. You can inform things like what data it exports, to which team it belongs, etc.
2. To define your Trigger, consider how frequently you need to have the data updated in S3 (every day, once a week, only at specific times, etc.).
3. Click Done.

​

4. Check your new destination

1. Once completed, you’ll receive confirmation that your new destination is set up!
2. You can view your new destination on the Destinations page. The pipeline will run according to your configured trigger schedule. You can monitor it on the Destinations page to see its execution and completion.
3. If needed, manually trigger the pipeline by clicking on the refresh icon. Once executed, your Parquet files will be available in your S3 bucket at the path you specified.

If you encounter any issues, reach out to us via Slack, and we’ll gladly assist you!

​

Implementation Notes

​

Error Logging

• Sample records: Logs a sample of up to 5 records from the failed batch.
• Record truncation: Each logged record is truncated if it exceeds 4,000 characters.
• Schema truncation: The table schema is truncated if it exceeds 64,000 characters.