Skip to main content
Cloud Storage Files reads data files from a Google Cloud Storage bucket into tables in your Catalog. Each table you configure reads a set of files from the bucket, selected by folder and file name pattern. The first supported format is Avro, including the files a Pub/Sub Cloud Storage subscription writes to a bucket. A common use is change data capture: database change events published to Pub/Sub, written to a bucket, and loaded into Nekt as one table of events.

​
Prerequisites

You need a Google Cloud service account that can read the bucket, and a JSON key for it.
1

Create a service account

In the Google Cloud console, go to IAM & Admin > Service Accounts in the project of your choice and create a service account, for example nekt-reader.
2

Grant read access on the bucket

Open the bucket in Cloud Storage > Buckets, go to Permissions, click Grant access, and give the service account the Storage Object Viewer role (roles/storage.objectViewer). Granting it on the bucket, rather than on the project, limits Nekt to that bucket.
3

Create a JSON key

In the service account, go to Keys > Add key > Create new key, choose JSON, and download the file. You will upload it in the next section. Google’s guide: Create and delete service account keys.
Nekt only lists and downloads files. The connector requests a read-only Cloud Storage scope even if the service account has broader roles.

​
Configuring Cloud Storage Files as a Source

In the Sources tab, click on the “Add source” button located on the top right of your screen. Then, select the Cloud Storage Files option from the list of connectors. Click Next and you’ll be prompted to add your access.

​
1. Add your access

  • Storage: Google Cloud Storage.
  • Bucket: the bucket name, with or without the gs:// prefix.
  • Google Cloud service account key (JSON): upload the JSON key file from the prerequisites.

​
Tables

Click Add a table once for each table you want. Each table reads a set of files from the bucket.
  • Table name: name of the table the files load into. Lowercase letters, digits, and underscores, starting with a letter, and unique within the source.
  • Folder: only files whose name starts with this prefix are read, e.g. exports/cdc/. Leave empty to read from the root of the bucket.
  • File pattern: (Default: .*) a regular expression searched in each file name, folder included. Only matching files are read. .* reads every file under the folder; \.avro$ reads only files ending in .avro.
  • File format: Avro.
  • Envelope: (Default: None) choose Pub/Sub Cloud Storage subscription when the files were written by a Pub/Sub Cloud Storage subscription. Each record in those files holds one message, and the message body becomes the payload. Choose None when each record in the file is the data itself.
  • Message body format: (Default: JSON) with the Pub/Sub envelope, JSON parses each message body, so its fields can be promoted to columns. Text keeps the body as plain text.
  • Schema mode: (Default: Raw) each record is stored whole, as JSON text, in the payload column, plus the promoted columns. Use it when records do not share one schema, for example change events of several collections in the same files.
  • Promoted columns: values copied out of each record into their own columns, so the table can be filtered without parsing JSON. Key is the column name (lowercase snake_case) and Value is the JSONPath of the value in the record, e.g. $.collection or $.documentKey._id. Promoted columns are always stored as text; objects and arrays are stored as JSON text.
  • Primary keys: (Default: _nekt_file_path, _nekt_file_row) columns that identify a record. The default makes re-reading a file update its rows instead of duplicating them. Use _nekt_record_hash to also collapse identical records delivered in different files, as Pub/Sub does.
  • Skip if empty: (Default: off) when no file matches the folder and pattern, skip this table with a warning instead of failing the run.
Change events published to Pub/Sub and written by a Cloud Storage subscription under cdc/, one row per event:Update events usually carry only the fields that changed. To rebuild the current state of each collection, use a Notebook or a Query that applies the events in cluster_time order.

​
Advanced settings

  • Start date: only files written to the bucket at or after this moment are read. Leave empty to read every file.
  • Maximum files per run: (Default: 10000) the most new files one run reads per table, oldest first. A long history is loaded over several runs, each small enough to finish.
  • Parallel downloads: (Default: 8, up to 64) how many files are downloaded at the same time.
Once you’re done, click Next. Nekt checks that the service account can list the bucket before moving on.

​
2. Select streams

You will see one stream per table you configured. Select the ones you want and click Next.

​
3. Configure data streams

Customize how you want your data to appear in your catalog. Select the desired layer where the data will be placed, a folder to organize it inside the layer, a name for each table (which will effectively contain the fetched data) and the type of sync.
  • Layer: choose between the existing layers on your catalog. This is where you will find your new extracted tables as the extraction runs successfully.
  • Folder: a folder can be created inside the selected layer to group all tables being created from this new data source.
  • Table name: we suggest a name, but feel free to customize it. You have the option to add a prefix to all tables at once and make this process faster!
  • Sync Type: INCREMENTAL, based on when each file was written to the bucket. Read more about Sync Types here.
Once you are done configuring, click Next.

​
4. Configure data source

Describe your data source for easy identification within your organization, not exceeding 140 characters. To define your Trigger, consider how often you want data to be extracted from this source. This decision usually depends on how frequently new files land in the bucket. Once you are ready, click Next to finalize the setup.

​
5. Check your new source

You can view your new source on the Sources page. If needed, manually trigger the source extraction by clicking on the arrow button. Once executed, your data will appear in your Catalog.
For you to be able to see it on your Catalog, you need at least one successful source run.

​
Incremental sync

  • Nekt tracks the time each file was written to the bucket, not the content of its records. A file that arrives late is still read, even if its records are old.
  • Each run reads new files oldest first, up to Maximum files per run per table. When there are more, the run log says how many are left, and the next runs continue from where it stopped. A bucket with a long history is loaded over several runs.
  • The newest files already read are read again on the next run. Their rows are updated by primary key, not duplicated.
  • If a run fails, the next one reads the same files again. No file is skipped.
For a large history, schedule the source to run frequently until it catches up, then move it to the cadence you need.

​
Streams and Fields

Each table you configure becomes one stream, named after its Table name.
File metadata:
  • _nekt_file_path - Location of the file the record came from, as gs://bucket/path
  • _nekt_file_last_modified - When the file was written to the bucket; used as the incremental replication key
  • _nekt_file_row - Position of the record in its file, starting at 1
Record:
  • _nekt_record_hash - SHA-256 of the record’s original bytes (with the Pub/Sub envelope, of the message body). Identical records have the same hash, even in different files
  • payload - The record as JSON text. With the Pub/Sub envelope, the message body exactly as it was published
Pub/Sub message metadata (only with the Pub/Sub envelope, and filled only when the subscription has Write metadata enabled):
  • pubsub_message_id - ID Pub/Sub assigned to the message
  • pubsub_publish_time - When the message was published
  • pubsub_attributes - Message attributes, as JSON text
Promoted columns:
  • One text column per entry in Promoted columns, named after its Key

​
Troubleshooting

​
Skills for agents

Download Cloud Storage Files skills file

Cloud Storage Files connector documentation as plain markdown, for use in AI agent contexts.