Dependencies

Every notebook runs with the Nekt SDK and its runtime’s preinstalled libraries. Any other Python package your code imports must be added as a dependency. Nekt installs the dependencies before your code runs. You add dependencies in the Define dependencies step when you create a notebook, or in the Dependencies section when you edit one. Each line holds one dependency.

Public packages

Add packages from PyPI with their name and version:

holidays==0.48
openpyxl==3.1.5

Pin an exact version (==) so every run installs the same code. A package without a pinned version is installed once and reused until the notebook’s dependencies change, so runs don’t pick up its new releases. When you save, Nekt checks that each package exists on PyPI and rejects the ones it can’t find.

Private packages

To share code across notebooks, put it in a Python package in a private Git repository and install it as a dependency. Private repositories need credentials, which you store as a Secret and reference by key instead of writing them into the dependency. Reference a secret as ${KEY}, where KEY is the secret’s key:

git+https://${GITHUB_TOKEN}@github.com/acme/data-utils.git@v1.0.0

Credentials must always come from a secret. A dependency with a token or password written into its URL is rejected when you save the notebook, and when it is synced from GitHub.

Create a token for the repository

Use a token that can only read the repository. On GitHub, create a fine-grained personal access token with access to that repository only and the Contents: Read-only repository permission.

Store the token as a secret

In Secrets, create a secret whose value is the token, for example with the key GITHUB_TOKEN. A key used in a reference may contain only letters, digits, and underscores, and must not start with a digit.

Add the dependency

Add the dependency in the git+https://${KEY}@host/owner/repository.git@ref format. Set ref to a tag, branch, or commit.

Save the notebook

Nekt checks that the secret exists, that you can read it, and that the repository and ref can be reached with its value.

Once the dependency installs, your notebook imports the package like any other:

from data_utils import normalize_column_names

Branches, tags, and commits

What ref points at decides when the notebook picks up new code:

`ref`	Example	What each run installs
A commit	`@4f2c9e1...`	Always that commit.
A tag	`@v1.0.0`	The commit the tag points at.
A branch	`@main`	The latest commit on the branch.
None	`utils.git`	The latest commit on the repository’s default branch.

For a branch, a tag, or no ref, Nekt checks the repository for a new commit when each run starts. A run after a push installs the new commit. On a PySpark notebook, a new commit makes the run rebuild the notebook’s image before it starts, which adds a few minutes to that run only. PyPI packages are not checked: they install at the versions listed. The run log shows the check as one line when it starts and one summary when it ends:

Dependencies: checking 1 package installed from a git repository for new commits. PyPI packages are not checked: they install at the versions listed.
Dependencies: checked 1 package; this run installs:
- github.com/acme/data-utils@main: commit 4f2c9e1a7b3d, new since 9b81d0c4e2f7.

When the check fails

What happens depends on whether the problem can fix itself:

Problem	What the run does
The Git host doesn’t answer, times out, or returns a server error	Tries again after 30 s, 60 s, and 120 s. If the host still doesn’t answer, the run starts with the commit it last installed and the summary warns about it. If no commit was installed before, the run fails.
The token is invalid or can’t read the repository, the repository or `ref` doesn’t exist, or a referenced secret has no value	The run fails before it starts, without retrying. The run log says which package can’t be installed and what to fix.

A failed check stops the run before its code runs, so no data is written.

Point production notebooks at a tag (@v1.0.0) or a commit. Code reaches them only when you change the dependency, so a push to the repository can’t break a notebook that is already running. Use a branch while you develop the package.

How secrets in dependencies are handled

Access: You can only reference secrets you have permission to read. Saving a notebook that references any other secret fails, whether the secret is missing or you lack permission.
Linking: A secret referenced in a dependency is added to the notebook’s secrets, the same way as a secret your code loads with load_secret.
Deletion: A secret a notebook uses can’t be deleted. The error lists the notebooks that use it. Remove the reference from their code and dependencies first.
Storage: The notebook stores only the ${KEY} reference. The value is read when dependencies are installed, so rotating the secret needs no change to the notebook.
Exposure: The secret’s value is used only to install the package. Your notebook code can’t read it from the environment, and the value is masked in run logs.

The git+https:// scheme is the only one supported for Git dependencies. SSH URLs (git+ssh://) are rejected.

Other Git hosts

Any Git host that accepts a token over HTTPS works. Put the token where the host expects it in the URL. For example, on GitLab:

git+https://oauth2:${GITLAB_TOKEN}@gitlab.com/acme/data-utils.git@v1.0.0

Spark jars

PySpark notebooks can also add Spark jars: Maven Central coordinates in the group:artifact:version format, added to Spark’s classpath.

org.apache.sedona:sedona-spark-shaded-3.5_2.12:1.7.1

Transitive dependencies of a jar are not downloaded, so prefer shaded jars. If the jar has a Python API, add its package under Dependencies too.

Troubleshooting

Error	What to check
`Following modules are invalid: <package>`	The package name or version doesn’t exist on PyPI.
`These secrets do not exist or you do not have permission to read them: <KEY>`	The secret’s key matches the reference exactly (keys are case-sensitive), and you have access to the secret.
`Invalid secret references`	The reference uses the `${KEY}` format, with only letters, digits, and underscores in the key.
`Repository not found, or the credentials cannot read it.`	The repository URL is correct, and the token in the secret is valid and can read the repository.
`Ref '<ref>' not found in the repository.`	The tag, branch, or commit after `@` exists in the repository.
`Git dependencies must not contain a token or password`	Store the token as a secret and reference it as `${KEY}`.
`Cannot delete secret because it is used by transformations`	Remove the secret from the listed notebooks’ code and dependencies, then delete it.
Run log: `the package cannot be installed, so the run stopped before it started`	Follow the reason after the package name, for example renew the token in the secret or fix the `ref`.

Get started

Context

Activate

Organizations

Workspaces

Resources

Dependencies

Public packages

Private packages

Branches, tags, and commits

When the check fails

How secrets in dependencies are handled

Other Git hosts

Spark jars

Troubleshooting

​Public packages

​Private packages

​Branches, tags, and commits

​When the check fails

​How secrets in dependencies are handled

​Other Git hosts

​Spark jars

​Troubleshooting

Public packages

Private packages

Branches, tags, and commits

When the check fails

How secrets in dependencies are handled

Other Git hosts

Spark jars

Troubleshooting