> ## Documentation Index
> Fetch the complete documentation index at: https://docs.nekt.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Elasticsearch (Kibana) as a data source

> Bring log data from Elasticsearch, accessed through Kibana, to Nekt.

Elasticsearch is a distributed search and analytics engine commonly used to store and query application logs, and Kibana is its visualization and management interface. This connector extracts log documents from your Elasticsearch indices — filtered by time period, service and free-text search — so you can analyze them alongside the rest of your data.

## Configuring Elasticsearch as a Source

In the [Sources](https://app.nekt.ai/sources) tab, click on the "Add source" button located on the top right of your screen. Then, select the Elasticsearch option from the list of connectors.

Click **Next** and you'll be prompted to add your access.

### 1. Add account access

The following configurations are available:

* **Kibana URL**: The base URL of your Kibana instance (e.g. `https://my-project.kb.us-central1.gcp.elastic.cloud`). Search requests are routed through Kibana, so this is the only endpoint you need.

* **API Key**: A base64-encoded Elasticsearch API key. You can create one in Kibana under **Stack Management > API Keys**. The same key authenticates both Kibana and Elasticsearch requests.

* **Log searches**: The list of searches to extract. Each search becomes one table in your catalog, with the following options:
  * **Stream name**: The table name for this search (letters, numbers and underscores — e.g. `checkout_api_logs`).
  * **Index pattern**: The Elasticsearch index pattern to search (e.g. `logs-*`). Check the `data_views` stream or your Kibana Discover page if you're unsure which patterns exist.
  * **Service name** (optional): An exact filter on the `service.name` field (e.g. `Lastlink.Checkout.Api`).
  * **Search query** (optional): A Lucene query applied to the `message` field by default. Use `field:value` syntax to target other fields (e.g. `event.category:antifraud`).

Optionally, you can define some advanced settings:

* **Elasticsearch URL**: When provided, log searches query the official Elasticsearch `_search` API directly instead of going through the Kibana console proxy. Recommended if your Kibana instance has the Dev Tools console disabled.

* **Start Date**: The earliest log timestamp (`@timestamp`) to extract on the first sync.

* **Page size**: Number of documents fetched per request (1000–5000 recommended).

Once you're done, click **Next**.

### 2. Select streams

Choose which data streams you want to sync. For faster extractions, select only the streams that are relevant to your analysis. You can select entire groups of streams or pick specific ones.

> Tip: The stream can be found more easily by typing its name.

Select the streams and click **Next**.

### 3. Configure data streams

Customize how you want your data to appear in your catalog. Select the desired layer where the data will be placed, a folder to organize it inside the layer, a name for each table (which will effectively contain the fetched data) and the type of sync.

* **Layer**: choose between the existing layers on your catalog. This is where you will find your new extracted tables as the extraction runs successfully.
* **Folder**: a folder can be created inside the selected layer to group all tables being created from this new data source.
* **Table name**: we suggest a name, but feel free to customize it. You have the option to add a **prefix** to all tables at once and make this process faster!
* **Sync Type**: you can choose between INCREMENTAL and FULL\_TABLE.
  * Incremental: every time the extraction happens, we'll get only the new data - which is good if, for example, you want to keep every record ever fetched.
  * Full table: every time the extraction happens, we'll get the current state of the data - which is good if, for example, you don't want to have deleted data in your catalog.

Once you are done configuring, click **Next**.

### 4. Configure data source

Describe your data source for easy identification within your organization, not exceeding 140 characters.

To define your [Trigger](https://docs.nekt.com/runs/scheduling-and-triggers), consider how often you want data to be extracted from this source. This decision usually depends on how frequently you need the new table data updated (every day, once a week, or only at specific times).

Optionally, you can define some additional settings:

* Configure Delta Log Retention and determine for how long we should store old states of this table as it gets updated. Read more about this resource [here](https://docs.nekt.com/get-started/core-concepts/resource-control).
* Determine when to execute an **Additional [Full Sync](https://docs.nekt.com/get-started/core-concepts/types-of-sync#additional-full-sync)**. This will complement the incremental data extractions, ensuring that your data is completely synchronized with your source every once in a while.

Once you are ready, click **Next** to finalize the setup.

### 5. Check your new source

You can view your new source on the [Sources](https://app.nekt.ai/sources) page. If needed, manually trigger the source extraction by clicking on the arrow button. Once executed, your data will appear in your Catalog.

<Warning>For you to be able to see it on your [Catalog](https://app.nekt.ai/catalog), you need at least one successful source run.</Warning>

# Streams and Fields

Below you'll find the available data streams from Elasticsearch and their corresponding fields:

<AccordionGroup>
  <Accordion title="Log searches (one stream per configured search)">
    Log documents matching the search's index pattern and filters, synced incrementally by `@timestamp`.

    | Field          | Type     | Description                                                           |
    | -------------- | -------- | --------------------------------------------------------------------- |
    | `id`           | string   | Elasticsearch document `_id`.                                         |
    | `index`        | string   | Concrete index the document lives in.                                 |
    | `timestamp`    | datetime | Document `@timestamp`, normalized to UTC.                             |
    | `service_name` | string   | Value of `service.name`, when present.                                |
    | `message`      | string   | Log message, when present.                                            |
    | `source`       | string   | The full document `_source` as a JSON string, preserving every field. |
  </Accordion>

  <Accordion title="Data Views">
    Data views registered in Kibana (formerly index patterns). Useful to discover which index patterns exist when configuring your log searches.

    | Field             | Type   | Description                             |
    | ----------------- | ------ | --------------------------------------- |
    | `id`              | string | Data view unique identifier.            |
    | `name`            | string | Display name of the data view.          |
    | `title`           | string | Index pattern the data view targets.    |
    | `type`            | string | Data view type, when present.           |
    | `time_field_name` | string | Default time field of the data view.    |
    | `namespaces`      | array  | Kibana spaces the data view belongs to. |
  </Accordion>
</AccordionGroup>

If there is any data you need that is not mapped here, just reach out to us through our [Slack](https://join.slack.com/t/nektcommunity/shared_invite/zt-2k3v2t1jq-cZG~ZWXzNZJF~Q1bAnyTsQ) channel and we'll evaluate adding it to the connector.