Connectors
Oracle as a data source
Bring data from Oracle to your Lakehouse.
Oracle is a multinational technology company that provides database management systems, cloud services, and enterprise software solutions. Its database platform is widely used for enterprise applications, data warehousing, and business intelligence, offering robust data management and analytics capabilities.
0. Required pre work
In order to connect Nekt to a database, you have to do some pre work to ensure access is granted in a secure way. So,If Oracle DB is hosted on AWS
If Oracle DB is hosted on AWS
- Stablish a peering connection between Nekt VPC and database VPC
- On your AWS Console, access the VPC service and go to Peering Connection. Make sure you are logged in with the account that contains the database you want to connect with.
- Select Create peering connection
- Setup as requested
- Give your connection a name (something as ‘Nekt <> Database’)
- Select
nekt-vpcID as requester (on VPC ID field) - Choose ‘Another account’ on the Select another VPC to peer with section. Inform the Account ID.
- To get the Account ID, go to the RDS service (you’ll find it searching on the AWS Console).
- Click on BD Instances
- Select the desired database
- Copy the Account ID associated with this database.
- Inform the VPC ID (Accepter)
- On the desired database details, on the Connectivity and Security section, click on the VPC.
- Copy the VPC ID
- Click on Create peering connection. You’ll notice the status is ‘Pending acceptance’.
- Go to Peering Connection again and your new peering connection should be listed, yet still pending acceptance.
- On the Actions menu, click on Accept Request and confirm it.
- You should edit your Peering Connection name to ‘Nekt <> Database’, to keep the pattern.
- Nekt VPC
- Access the created Peering Connection - that now should have the status ‘Active’ and a value on Accepter CIDRs. Copy this value, it will be the Nekt VPC IP.
- In the VPC dashboard menu, go to Route Tables
- In every route table with ‘nekt’ in its name, follow these steps:
- Click on the Routes tab
- Click on Edit routes
- Click on Add route
- On the ‘Destination’ column, paste the Nekt VPC IP (Accepter CIDRs previously copied)
- On the ‘Target’ column, choose ‘Peering Connection’ and the
Nekt <> Databaseoption in the field that will open. - Keep ‘Status’ and ‘Propagated’ colmns as default
- Target: Peering connection stablished between Nekt and database
- Save changes
- Database VPC
- Repeat the process done for Nekt VPC, but now use the Nekt VPC IP as Destination
- Search for RDS on your AWS Console and access it.
- Select your database and go to Connectivity & Security.
- Click on VPC security groups.
- Selcting your DB security group, go to the Inbound Rules tab
- Click on Edit inbound rules
- Add the following inbound rule to the security group:
- Type: Oracle-RDS
- Source:
Customwith the Nekt VPC IP as value - Add a description to better identify it. Something like ‘Nekt’
- Save rule
- Add the following inbound rule to the security group:
if Oracle DB is NOT hosted on AWS
if Oracle DB is NOT hosted on AWS
- Ask Nekt to create a fixed IP in your AWS infra.
- In your database provider, give access to the IP provided by Nekt.
1. Add your Oracle access
- Once you have done the pre work defined in section 0, you can inform your database accesses. In the Sources tab, click on the “Add source” button located on the top right of your screen. Then, select the Oracle option from the list of connectors.
- Click Next and you’ll be prompted to add your database access.
- Click Next.
2. Select your Oracle DB streams
-
The next step is letting us know which streams you want to bring. You can select entire groups of streams or only a subset of them.
Tip: The stream can be found more easily by typing its name.
- Click Next.
3. Configure your Oracle DB data streams
- Customize how you want your data to appear in your catalog. Select a name for each table (which will contain the fetched data) and the type of sync.
- Table name: we suggest a name, but feel free to customize it. You have the option to add a prefix and make this process faster!
- Sync Type: you can choose between INCREMENTAL and FULL_TABLE.
- Incremental: every time the extraction happens, we’ll get only the new data - which is good if, for example, you want to keep every record ever fetched.
- Full table: every time the extraction happens, we’ll get the current state of the data - which is good if, for example, you don’t want to have deleted data in your catalog.
- Click Next.
4. Configure your Oracle DB data source
- Describe your data source for easy identification within your organization. You can inform things like what data it brings, to which team it belongs, etc.
- To define your Trigger, consider how often you want data to be extracted from this source. This decision usually depends on how frequently you need the new table data updated (every day, once a week, or only at specific times).
Check your new source!
- Click Done to finalize the setup. Once completed, you’ll receive confirmation that your new source is set up!
- You can view your new source on the Sources page. Now, for you to be able to see it on your Catalog, you have to wait for the pipeline to run. You can now monitor it on the Sources page to see its execution and completion. If needed, manually trigger the pipeline by clicking on the refresh icon. Once executed, your new table will appear in the Catalog section.
If you encounter any issues, reach out to us via Slack, and we’ll gladly assist you!