​

How Webhooks Work in Nekt

​

Configuring Webhook as a Source

​

1. Configure webhook security (optional)

• Webhook API Key: Enable this option if you want to require an API key for incoming requests.
  • API Key Header: The HTTP header name where the API key will be sent (e.g., x-api-key, Authorization).
  • API Key Value: The expected value of the API key. Requests without this key (or with an incorrect key) will be rejected.

​

2. Get your Webhook URL

​

3. Configure your external system

1. Use the POST method
2. Send data as JSON in the request body
3. Include the API key header (if you configured authentication)
4. Set Content-Type: application/json

curl -X POST "https://your-webhook-url.nekt.ai/webhook/abc123" \
  -H "Content-Type: application/json" \
  -H "x-api-key: your-api-key" \
  -d '{
    "event_type": "order_created",
    "order_id": "12345",
    "customer_email": "customer@example.com",
    "total": 99.99,
    "items": [
      {"sku": "PROD-001", "quantity": 2}
    ]
  }'

​

4. Configure data streams

• Layer: Choose the layer in your catalog where the webhook data will be stored.
• Folder: Optionally create a folder to organize your webhook data.
• Table name: Name for the table that will contain the webhook events.

​

5. Configure data source

• Every 15 minutes: Near real-time processing for time-sensitive data
• Every hour: Balanced approach for most use cases
• Daily: Cost-effective for non-urgent data

​

6. Check your new source

​

Stream and Fields

​

Use Cases for Data Analysis

​

1. Parse JSON Payload

SELECT
  message_id,
  webhook_received_at,
  JSON_EXTRACT_SCALAR(payload, '$.event_type') AS event_type,
  JSON_EXTRACT_SCALAR(payload, '$.order_id') AS order_id,
  JSON_EXTRACT_SCALAR(payload, '$.customer_email') AS customer_email,
  CAST(JSON_EXTRACT_SCALAR(payload, '$.total') AS DOUBLE) AS total_amount
FROM
  nekt_raw.webhooks
WHERE
  webhook_received_at >= CURRENT_DATE - INTERVAL '7' DAY
ORDER BY
  webhook_received_at DESC

SELECT
  message_id,
  webhook_received_at,
  JSON_VALUE(payload, '$.event_type') AS event_type,
  JSON_VALUE(payload, '$.order_id') AS order_id,
  JSON_VALUE(payload, '$.customer_email') AS customer_email,
  CAST(JSON_VALUE(payload, '$.total') AS FLOAT64) AS total_amount
FROM
  `nekt_raw.webhooks`
WHERE
  DATE(webhook_received_at) >= DATE_SUB(CURRENT_DATE(), INTERVAL 7 DAY)
ORDER BY
  webhook_received_at DESC

​

2. Event Type Distribution

SELECT
  JSON_EXTRACT_SCALAR(payload, '$.event_type') AS event_type,
  COUNT(*) AS event_count,
  MIN(webhook_received_at) AS first_received,
  MAX(webhook_received_at) AS last_received
FROM
  nekt_raw.webhooks
WHERE
  webhook_received_at >= CURRENT_DATE - INTERVAL '30' DAY
GROUP BY
  JSON_EXTRACT_SCALAR(payload, '$.event_type')
ORDER BY
  event_count DESC

SELECT
  JSON_VALUE(payload, '$.event_type') AS event_type,
  COUNT(*) AS event_count,
  MIN(webhook_received_at) AS first_received,
  MAX(webhook_received_at) AS last_received
FROM
  `nekt_raw.webhooks`
WHERE
  DATE(webhook_received_at) >= DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY)
GROUP BY
  event_type
ORDER BY
  event_count DESC

​

3. Create a Materialized View

-- Create a structured view of your webhook data
SELECT
  message_id,
  webhook_received_at,
  JSON_EXTRACT_SCALAR(payload, '$.event_type') AS event_type,
  JSON_EXTRACT_SCALAR(payload, '$.order_id') AS order_id,
  JSON_EXTRACT_SCALAR(payload, '$.customer_email') AS customer_email,
  JSON_EXTRACT_SCALAR(payload, '$.customer_name') AS customer_name,
  CAST(JSON_EXTRACT_SCALAR(payload, '$.total') AS DOUBLE) AS total_amount,
  JSON_EXTRACT_SCALAR(payload, '$.currency') AS currency,
  JSON_EXTRACT_SCALAR(payload, '$.status') AS status,
  JSON_EXTRACT(payload, '$.items') AS items_json
FROM
  nekt_raw.webhooks

-- Create a structured view of your webhook data
SELECT
  message_id,
  webhook_received_at,
  JSON_VALUE(payload, '$.event_type') AS event_type,
  JSON_VALUE(payload, '$.order_id') AS order_id,
  JSON_VALUE(payload, '$.customer_email') AS customer_email,
  JSON_VALUE(payload, '$.customer_name') AS customer_name,
  CAST(JSON_VALUE(payload, '$.total') AS FLOAT64) AS total_amount,
  JSON_VALUE(payload, '$.currency') AS currency,
  JSON_VALUE(payload, '$.status') AS status,
  JSON_QUERY(payload, '$.items') AS items_json
FROM
  `nekt_raw.webhooks`

​

Implementation Notes

​

Data Format

• All webhook data is stored as a JSON string in the payload field
• Use your data warehouse’s JSON functions to extract specific fields
• Consider creating transformations to parse frequently accessed data into structured tables

​

Message Handling

• Each webhook request creates one record in the stream
• Messages are processed incrementally on each extraction run
• The message_id field uniquely identifies each webhook event
• The webhook_received_at timestamp reflects when Nekt received the webhook (not when the event occurred in the source system)

​

Best Practices

1. Include timestamps in your payload: Add an event_timestamp or similar field in your webhook payload to track when the event actually occurred in the source system
2. Use consistent payload structures: Maintain a consistent JSON structure across all webhook events to simplify data parsing
3. Include event types: Add an event_type field to categorize different types of events
4. Add idempotency keys: Include a unique identifier in your payload to detect and handle duplicate webhook deliveries
5. Test with sample data: Send test webhooks to verify your integration before enabling production webhooks

​

Troubleshooting