Notebooks

​

Choosing the Right Notebook Type

​

PySpark

• Aggregating or joining large tables from your Lakehouse
• Complex business logic applied to high-volume data
• ETL pipelines that would be slow or memory-intensive in plain Python

import nekt
from pyspark.sql import functions as F

# Load a large orders table
df = nekt.load_table(layer_name="raw", table_name="orders")

# Aggregate revenue by customer
result = df.groupBy("customer_id").agg(
    F.sum("amount").alias("total_revenue"),
    F.count("order_id").alias("order_count")
)

nekt.save_table(
    df=result,
    layer_name="trusted",
    table_name="customer_revenue"
)

​

Python — Base Image

• Calling external REST APIs and enriching your data with the responses
• Transformations on smaller tables that fit comfortably in memory
• Prototyping logic before scaling it to PySpark

import nekt
import requests

# Load a contacts table
df = nekt.load_table(layer_name="raw", table_name="contacts")

# Enrich each record with data from an external API
def enrich_contact(row):
    response = requests.get(f"https://api.example.com/contacts/{row['id']}")
    data = response.json()
    row["score"] = data.get("score", 0)
    return row

df = df.apply(enrich_contact, axis=1)

nekt.save_table(
    df=df,
    layer_name="trusted",
    table_name="contacts_enriched"
)

​

Python — Browser Image

• Scraping dynamic web pages (React, Vue, etc.) that don’t expose a public API
• Automating form submissions or login flows to collect data
• Extracting structured data from web UIs that require interaction

import nekt
import os
import pandas as pd
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options


def create_driver():
    options = Options()
    options.add_argument("--headless=new")
    options.add_argument("--no-sandbox")
    options.add_argument("--disable-dev-shm-usage")
    options.add_argument("--window-size=1920,1080")

    possible_paths = [
        "/usr/local/bin/chromedriver",
        "/usr/bin/chromedriver",
        "/usr/bin/chromium-driver",
    ]
    for path in possible_paths:
        if os.path.exists(path):
            return webdriver.Chrome(service=Service(executable_path=path), options=options)

    return webdriver.Chrome(options=options)


def scrape_holidays():
    driver = create_driver()
    results = []

    try:
        driver.get("https://www.timeanddate.com/holidays/brazil/2025")
        wait = WebDriverWait(driver, 15)

        wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, "table.table")))

        rows = driver.find_elements(By.CSS_SELECTOR, "table.table tbody tr")
        for row in rows:
            cols = row.find_elements(By.TAG_NAME, "td")
            if len(cols) >= 3:
                results.append({
                    "date": cols[0].text.strip(),
                    "name": cols[1].text.strip(),
                    "type": cols[2].text.strip(),
                })
    finally:
        driver.quit()

    return pd.DataFrame(results)


df = scrape_holidays()

nekt.save_table(
    df=df,
    layer_name="raw",
    table_name="brazil_holidays_2025"
)

​

Creating a Notebook

​

Step 1: Generate a Token

1. Navigate to the Notebooks module.
2. Generate a token with appropriate access permissions.
   • You can create multiple tokens, each with specific access levels for better security.
3. Select a notebook template to get started.

​

Step 2: Develop and Test Your Transformation

• Google Colab Notebook: Run a Jupyter notebook in the cloud with minimal setup. Requires a Google account.
• GitHub Codespaces: Use a cloud-hosted Jupyter notebook powered by GitHub for easy setup. Requires a GitHub account.
• Local Dev Container: Set up a Jupyter notebook on your local machine using an isolated environment with all dependencies pre-installed.
• Local Jupyter Notebook: Manually configure a Jupyter notebook on your local machine. Best for users with advanced knowledge of Python environments.

1. Copy your token and paste it into the template as instructed.
2. Use the Nekt SDK to load the data you need for your transformation.
   Copy

   Ask AI

   import nekt
   
   df = nekt.load_table(
      layer_name="layer_name",
      table_name="table_name"
   )
   
3. Develop and test your transformation code within the notebook environment.

​

Step 3: Deploy Your Notebook to Nekt

1. Navigate back to the Add Notebook page in Nekt.
2. Select the Notebook type: PySpark or Python.
   • If you selected Python, also choose the Python image: Base for standard notebooks, or Browser if your code requires a headless browser.
3. Copy your notebook code into Nekt.
4. Use the Nekt SDK .save_table method to save transformed dataframes as new tables in your Lakehouse. You can call it multiple times to save several dataframes in the same notebook.
   Copy

   Ask AI

   import nekt
   
   nekt.save_table(
      df=df,
      layer_name="layer_name",
      table_name="table_name",
      folder_name="folder_name"  # optional
   )
   
5. If you need external dependencies, add them in the Define dependencies section.
6. Save the notebook to make it available for execution.

​

Best Practices

1. Choose the right runtime: Use PySpark for large-scale data, Python (Base) for API-heavy or moderate workloads, and Python (Browser) only when a real browser is strictly necessary — it carries more overhead than the Base image.
2. Development: Use Notebook Templates to reduce setup time and start with simple notebooks before scaling complexity.
3. Testing: Validate your notebook logic locally before deployment. Test with sample data of various sizes and verify edge cases.
4. Performance:
   • For PySpark: leverage built-in Spark functions and avoid .collect() on large datasets.
   • For Python/Browser: keep external requests minimal and handle timeouts gracefully.
5. Maintenance: Document complex logic, keep dependencies up to date, and use version control for your notebooks. Direct integration with Github is available on the Growth plan.

​

Need Help?

• Use our example notebooks as templates
• Contact our support team for additional guidance