PySpark transformations in Nekt combine the distributed computing power of Apache Spark with the flexibility of Python to help you build sophisticated data pipelines. Whether you’re implementing complex business logic, handling large-scale data processing, or creating custom transformations, our Jupyter notebook integration makes it easy to develop and deploy your code.

​
Creating PySpark Transformations

Nekt makes it simple to create, test, and deploy transformations using Jupyter Notebooks. Whether you’re new to data engineering or an experienced user, our templates provide an easy starting point for building transformations.

​
Step 1: Generate a Token

To allow Jupyter Notebooks to access your data:

  1. Navigate to the Add Transformation page.
  2. Select the data tables you want to include in your transformation.
  3. Generate a token with appropriate access permissions.
    • You can create multiple tokens, each with specific access levels for better security.

Generate a Token

​
Step 2: Develop and Test Your Transformation

Nekt provides several environment options for running Jupyter Notebooks. Choose the option that best fits your development preferences:

  • Google Colab Notebook: Run a Jupyter notebook in the cloud with minimal setup. Requires a Google account.
  • GitHub Codespaces: Use a cloud-hosted Jupyter notebook powered by GitHub for easy setup. Requires a GitHub account.
  • Local Dev Container: Set up a Jupyter notebook on your local machine using an isolated environment with all dependencies pre-installed.
  • Local Jupyter Notebook: Manually configure a Jupyter notebook on your local machine. Best for users with advanced knowledge of Python environments.

How to Start:

  1. Copy your token and paste it into the template as instructed.
  2. Add the INPUT_TABLES list, which specifies the tables for the transformation.
  3. Develop and test your transformation code within the notebook environment.

View Environment Options

​
Step 3: Add Your Transformation to Nekt

Once your transformation is ready:

  1. Run and test your transformation in Jupyter to ensure everything works as expected.
  2. Navigate back to the Add Transformation page in Nekt.
  3. Copy your transformation code into Nekt and configure the necessary trigger preferences.
  4. Save the transformation to make it available for execution.

Add Transformation

​
Best Practices

  1. Development:

    • Use Notebook Templates to reduce setup time
    • Start with simple transformations and scale complexity
    • Leverage PySpark’s built-in functions when possible

  2. Testing:

    • Validate transformation logic in Jupyter before deployment
    • Test with sample data of various sizes
    • Verify edge cases and error handling

  3. Performance:

    • Monitor resource usage and execution times
    • Optimize Spark configurations for your workload
    • Use appropriate partitioning strategies

  4. Maintenance:

    • Document complex logic and dependencies
    • Keep dependencies up to date
    • Use version control for your notebooks

​
Need Help?

If you encounter challenges during the transformation process, our team is ready to assist. You can:

  • Check our PySpark documentation for reference
  • Use our example notebooks as templates
  • Contact our support team for additional guidance