> ## Documentation Index > Fetch the complete documentation index at: https://docs.nekt.com/llms.txt > Use this file to discover all available pages before exploring further. # Webhook as a data source > Receive real-time data from any external system via webhooks. The Webhook connector lets you receive event data from any external system without polling. You point your external system to a Nekt-provisioned URL, and every event it sends is automatically captured and queued on your cloud infrastructure. When the source runs, all queued events are processed and loaded into the Catalog at once. ## How Webhooks Work in Nekt When you create a Webhook source, Nekt provisions a unique URL endpoint on your cloud infrastructure. You configure your external system to send HTTP POST requests to that URL whenever a relevant event occurs. Incoming events are not processed immediately. Instead, each request is stored in a queue. When the source runs — triggered manually, on a schedule, or by an event — it processes all queued events and loads them into the Catalog. This means no events are missed between runs: everything sent to the endpoint is held in the queue until the next extraction. ```mermaid theme={null} sequenceDiagram participant External System participant API Gateway participant Lambda participant Amazon SQS participant Nekt Connector participant Nekt Data Lake External System->>API Gateway: POST event data API Gateway->>Lambda: Forward request Lambda->>Amazon SQS: Queue message Note over Amazon SQS: Data is queued for processing Nekt Connector->>Amazon SQS: Poll messages Nekt Connector->>Nekt Data Lake: Extract & load data Note over Nekt Data Lake: Data available in catalog ``` ## Configuring Webhook as a Source In the [Sources](https://app.nekt.ai/sources) tab, click on the "Add source" button located on the top right of your screen. Then, select the **Webhook** option from the list of connectors. Click **Next** and you'll be prompted to configure your webhook. ### 1. Configure webhook security (optional) You can protect your webhook endpoint with an API key to ensure only authorized systems can send data: * **Webhook API Key**: Enable this option if you want to require an API key for incoming requests. * **API Key Header**: The HTTP header name where the API key will be sent (e.g., `x-api-key`, `Authorization`). * **API Key Value**: The expected value of the API key. Requests without this key (or with an incorrect key) will be rejected. While API key authentication is optional, it's **strongly recommended** for production use to prevent unauthorized data from being sent to your webhook endpoint. Once you're done, click **Next**. ### 2. Get your Webhook URL After creating the source, Nekt will generate a unique webhook URL for you. This URL is where external systems should send their data. Please ask Nekt support for your webhook URL after the source is created. Keep your webhook URL secure. Anyone with access to this URL (and the API key, if configured) can send data to your Nekt account. ### 3. Configure your external system Configure your external system, application, or service to send HTTP POST requests to the Nekt webhook URL. The request should: 1. Use the **POST** method 2. Send data as **JSON** in the request body 3. Include the **API key header** (if you configured authentication) 4. Set `Content-Type: application/json` **Example request:** ```bash theme={null} curl -X POST "https://your-webhook-url.nekt.ai/webhook/abc123" \ -H "Content-Type: application/json" \ -H "x-api-key: your-api-key" \ -d '{ "event_type": "order_created", "order_id": "12345", "customer_email": "customer@example.com", "total": 99.99, "items": [ {"sku": "PROD-001", "quantity": 2} ] }' ``` ### 4. Configure data streams Customize how you want your data to appear in your catalog. Select the desired layer where the data will be placed, a folder to organize it inside the layer, and a name for the table. * **Layer**: Choose the layer in your catalog where the webhook data will be stored. * **Folder**: Optionally create a folder to organize your webhook data. * **Table name**: Name for the table that will contain the webhook events. The sync type for webhooks is always **INCREMENTAL** since webhooks are event-driven and each event is a new record. Once you are done configuring, click **Next**. ### 5. Configure data source Describe your data source for easy identification within your organization, not exceeding 140 characters. To define your [Trigger](https://docs.nekt.com/get-started/core-concepts/triggers), consider how often you want the queued webhook data to be processed and loaded into Nekt. Common configurations: * **Every 15 minutes**: Near real-time processing for time-sensitive data * **Every hour**: Balanced approach for most use cases * **Daily**: Cost-effective for non-urgent data Once you are ready, click **Next** to finalize the setup. ### 6. Check your new source You can view your new source on the [Sources](https://app.nekt.ai/sources) page. Once your external system starts sending webhooks and the extraction trigger runs, your data will appear in your Catalog. For you to be able to see it on your [Catalog](https://app.nekt.ai/catalog), you need at least one successful source run with data. # Stream and Fields The Webhook connector provides a single stream that captures all incoming webhook events: | Field | Type | Description | | :-------------------- | :------- | :------------------------------------------------------------------------------ | | `message_id` | String | Unique identifier for the webhook message. | | `payload` | String | The complete JSON payload sent by the external system, stored as a JSON string. | | `webhook_received_at` | DateTime | Timestamp when Nekt received the webhook. | The `payload` field contains the raw JSON data sent to your webhook. You can parse this field in your SQL queries to extract specific values. See the [Use Cases](#use-cases-for-data-analysis) section for examples. # Use Cases for Data Analysis The webhook connector stores your data as a JSON string in the `payload` field. Here's how to extract and analyze your data using SQL. ### 1. Parse JSON Payload Extract specific fields from your webhook payload for analysis. ```sql theme={null} SELECT message_id, webhook_received_at, JSON_EXTRACT_SCALAR(payload, '$.event_type') AS event_type, JSON_EXTRACT_SCALAR(payload, '$.order_id') AS order_id, JSON_EXTRACT_SCALAR(payload, '$.customer_email') AS customer_email, CAST(JSON_EXTRACT_SCALAR(payload, '$.total') AS DOUBLE) AS total_amount FROM nekt_raw.webhooks WHERE webhook_received_at >= CURRENT_DATE - INTERVAL '7' DAY ORDER BY webhook_received_at DESC ``` ```sql theme={null} SELECT message_id, webhook_received_at, JSON_VALUE(payload, '$.event_type') AS event_type, JSON_VALUE(payload, '$.order_id') AS order_id, JSON_VALUE(payload, '$.customer_email') AS customer_email, CAST(JSON_VALUE(payload, '$.total') AS FLOAT64) AS total_amount FROM `nekt_raw.webhooks` WHERE DATE(webhook_received_at) >= DATE_SUB(CURRENT_DATE(), INTERVAL 7 DAY) ORDER BY webhook_received_at DESC ``` ### 2. Event Type Distribution Analyze the distribution of different event types received via webhooks. ```sql theme={null} SELECT JSON_EXTRACT_SCALAR(payload, '$.event_type') AS event_type, COUNT(*) AS event_count, MIN(webhook_received_at) AS first_received, MAX(webhook_received_at) AS last_received FROM nekt_raw.webhooks WHERE webhook_received_at >= CURRENT_DATE - INTERVAL '30' DAY GROUP BY JSON_EXTRACT_SCALAR(payload, '$.event_type') ORDER BY event_count DESC ``` ```sql theme={null} SELECT JSON_VALUE(payload, '$.event_type') AS event_type, COUNT(*) AS event_count, MIN(webhook_received_at) AS first_received, MAX(webhook_received_at) AS last_received FROM `nekt_raw.webhooks` WHERE DATE(webhook_received_at) >= DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY) GROUP BY event_type ORDER BY event_count DESC ``` ### 3. Create a Materialized View For frequently accessed webhook data, consider creating a transformation to parse the payload into structured columns. ```sql theme={null} -- Create a structured view of your webhook data SELECT message_id, webhook_received_at, JSON_EXTRACT_SCALAR(payload, '$.event_type') AS event_type, JSON_EXTRACT_SCALAR(payload, '$.order_id') AS order_id, JSON_EXTRACT_SCALAR(payload, '$.customer_email') AS customer_email, JSON_EXTRACT_SCALAR(payload, '$.customer_name') AS customer_name, CAST(JSON_EXTRACT_SCALAR(payload, '$.total') AS DOUBLE) AS total_amount, JSON_EXTRACT_SCALAR(payload, '$.currency') AS currency, JSON_EXTRACT_SCALAR(payload, '$.status') AS status, JSON_EXTRACT(payload, '$.items') AS items_json FROM nekt_raw.webhooks ``` ```sql theme={null} -- Create a structured view of your webhook data SELECT message_id, webhook_received_at, JSON_VALUE(payload, '$.event_type') AS event_type, JSON_VALUE(payload, '$.order_id') AS order_id, JSON_VALUE(payload, '$.customer_email') AS customer_email, JSON_VALUE(payload, '$.customer_name') AS customer_name, CAST(JSON_VALUE(payload, '$.total') AS FLOAT64) AS total_amount, JSON_VALUE(payload, '$.currency') AS currency, JSON_VALUE(payload, '$.status') AS status, JSON_QUERY(payload, '$.items') AS items_json FROM `nekt_raw.webhooks` ``` ## Implementation Notes ### Data Format * All webhook data is stored as a JSON string in the `payload` field * Use your data warehouse's JSON functions to extract specific fields * Consider creating transformations to parse frequently accessed data into structured tables ### Message Handling * Each webhook request creates one record in the stream * Messages are processed incrementally on each extraction run * The `message_id` field uniquely identifies each webhook event * The `webhook_received_at` timestamp reflects when Nekt received the webhook (not when the event occurred in the source system) ### Best Practices 1. **Include timestamps in your payload**: Add an `event_timestamp` or similar field in your webhook payload to track when the event actually occurred in the source system 2. **Use consistent payload structures**: Maintain a consistent JSON structure across all webhook events to simplify data parsing 3. **Include event types**: Add an `event_type` field to categorize different types of events 4. **Add idempotency keys**: Include a unique identifier in your payload to detect and handle duplicate webhook deliveries 5. **Test with sample data**: Send test webhooks to verify your integration before enabling production webhooks ### Troubleshooting | Issue | Possible Cause | Solution | | :---------------- | :------------------------------------------------ | :------------------------------------------------------------------------------- | | No data appearing | Webhook not configured correctly in source system | Verify the webhook URL and test with a sample request | | 401/403 errors | API key mismatch | Check that the API key header name and value match your configuration | | Data not updating | Extraction trigger hasn't run | Check your trigger schedule or manually trigger an extraction | | Malformed data | Invalid JSON in request body | Ensure your source system sends valid JSON with `Content-Type: application/json` | ## Skills for agents Webhook connector documentation as plain markdown, for use in AI agent contexts.