Slack as a data source
Bring your data from your Slack to your catalog.
1. Add your Slack access
-
In the Sources tab, click on the “Add source” button located at the top right of your screen. Then, select the Slack option from the list of connectors.
-
Click Next and you’ll be prompted to add the connector configuration:
Default configs
-
Authorization: first of all, authorize Nekt through the Slack Authorization button. You must be an App Manager to authorize the installation of our app into your workspace.
You’ll see the following warning when installing the app:
This app is not approved by Slack. Apps are reviewed to ensure a quality experience.
This occurs because our app is private and not yet available in the Slack marketplace.
-
Channel types: the types of conversations the tap will attempt to extract data from. Options are
public_channel
orprivate_channel
. Currently, fetching messages from DMs with a bot user is not allowed. -
Auto join public channels: whether the bot user should attempt to join public channels that it has not yet joined. The bot user must be a member of the channel to be able to retrieve messages. For private channels, you have to manually add the bot to the channel. Here’s the step by step:
- Type the command
/add
in the message input box of the channel you want to add the bot. - Choose
Add apps to your channel
. - Select the
Nekt Source
app. - Done, you’ll be able to extract messages from this channel.
Repeat this process for all channels you want to get data from.
- Type the command
-
Start date: defines from which date we should start syncing messages.
Advanced configs
-
Thread lookback (days): the number of days to look in the past for new thread replies to existing messages.
-
Selected channels: an array of channel IDs you want to select for extraction. This helps reduce the extraction time by only selecting channels of interest.
-
Excluded channels: an array of channel IDs you want to exclude from extraction. This helps reduce the extraction time by excluding channels that contain bots or flooding messages.
The channel ID can be found by clicking with the right button on the channel name and ‘View details’. The channel ID will be available at the bottom of the modal.
-
-
Click Next.
2. Select your Slack streams
-
The next step is letting us know which streams you want to bring. You can select entire groups of streams or only a subset of them.
-
Click Next.
3. Configure your Slack data streams
- Customize how you want your data to appear in your catalog. Select the desired layer where the data will be placed, a folder to organize it inside the layer, a name for each table (which will effectively contain the fetched data) and the type of sync.
- Layer: choose between the existing layers on your catalog. This is where you will find your new extracted tables as the extraction runs successfully.
- Folder: a folder can be created inside the selected layer to group all tables being created from this new data source.
- Table name: we suggest a name, but feel free to customize it. You have the option to add a prefix to all tables at once and make this process faster!
- Sync Type: depending on the data you are bringing to the lake, you can choose between INCREMENTAL and FULL_TABLE. Read more about Sync Types here.
- Click Next.
4. Configure your Slack data source
-
Describe your data source for easy identification within your organization. You can inform things like what data it brings, to which team it belongs, etc.
-
To define your Trigger, consider how often you want data to be extracted from this source. This decision usually depends on how frequently you need the new table data updated (every day, once a week, or only at specific times).
Check your new source!
-
Click Next to finalize the setup. Once completed, you’ll receive confirmation that your new source is set up!
-
You can view your new source on the Sources page. Now, for you to be able to see it on your Catalog, you have to wait for the pipeline to run. You can now monitor it on the Sources page to see its execution and completion. If needed, manually trigger the pipeline by clicking on the refresh icon. Once executed, your new table will appear in the Catalog section.
If you encounter any issues, reach out to us via Slack, and we’ll gladly assist you!