Data pipelines can sometimes fail. Although it is inherent to working with data, we work hard to avoid that. Here are some reasons why your pipeline might be failing and how you can deal with it.

​
An adjustment is required

In case there’s something wrong with the connector (for sources and destinations), the transformation code or any credentials used to access external tools, a fix or update will be needed. The easiest way to understand the error is checking the run logs.

There’s usually a message saying what happened and what needs to be done. In case there’s nothing really useful logged, please get in touch with us and we’ll investigate it further.

We are updating all resources to provide meaningful and actionable error messages, but there are a ton of scenarios and we might have not covered all of them. We’re sorry for any inconvenience!

​
There is a instability or temporary issue

Data pipelines can sometimes fail due to system instabilities or temporary issues. Nekt allows you to configure retry parameters for each Source, Transformation, or Destination, ensuring reliable pipeline execution without unnecessary resource usage.

  1. Number of Retries: Specify the number of retry attempts before marking a run as failed.
  2. Retry Delay: Set the delay (in seconds) between retry attempts.
  3. Max Consecutive Failures: Define the maximum number of consecutive failures allowed before inactivating the pipeline.
    • Use Case: This setting is ideal for preventing repeated failures during off-hours or weekends, saving resources and avoiding unnecessary alerts.

These parameters can be managed in the settings tab of each Source, Transformation and Destination, providing flexibility in how you handle pipeline stability and recovery.