Bring data from REST API to your Lakehouse.
https://api.example.com/v1
.
Example
$.pagination.cursor
. For detailed instructions on JSONPath syntax, click here.limit
.
page
.
Example of limit and next page params
50
, Pagination limite per page API param to limit
and Pagination next page/offset API param to offset
. The request will created the following query params: ?limit=50&offset=0
.offset
is dynamic and will be incremented automatically by the paginator.0
for offset based pagination and 1
for page number based pagination, but it can vary from API to API.
query params
for a GET
method. Pagination parameters don’t need to be provided as they are automatically added by the connector.
users
, projects
, boards
and tasks
). Each of these entities correspond to a data stream and has specific configurations.
This is the list of stream configurations available:
users
/users
Example
user_id
as the primary key since it uniquely identifies each record. For a composite key example, if you had a purchases
stream with fields user_id
and order_id
, you might want to set both as primary keys since the combination uniquely identifies each purchase.$[*]
for responses that return an array of items directly.
Example
$.data.users[*]
to extract the array of user records.updated_at
Example
updated_at
would be a good replication key since it changes whenever a record is modified.last-updated
Example
$last_run_date
as the variable representing the last time the execution happened, which will be replaced at runtime. Example: > 2021-01-01
Example
DDD-MM-YYYY
params
for a GET
method. Pagination parameters don’t need to be provided as they are automatically added by the connector. Stream level params will be merged with top-level params with stream level params overwriting top-level params with the same key.
If you encounter any issues, reach out to us via AWS S3 Parquet, and we’ll gladly assist you!