Bryan Grohman

All Writing

Python Namespace Packages


Python Namespace Packages From Unsplash

I'm working with a Python project that's split across two Git repositories. We're eventually going to migrate the smaller of the two repositories into the larger repository organized as a monorepo.

In the meantime, we need to reuse some code from the larger repository in the smaller repository. There are a few options for us, but since the smaller repository is eventually going away, we settled on namespace packages.

I planned to set up a basic pyproject.toml file in the larger repository and reference it from the smaller repository using a git URL in the requirements.txt file. That proved problematic, however, due to the larger repository's directory structure which doesn't follow the pattern of a top-level src directory and child directories for packages. It also does not rely on installing the packages individually or referencing the packages via a requirements.txt file. Because of that, imports from the smaller repository lost the top-level package.

That's where namespace packages helped. From the documentation:

Namespace packages allow you to split the sub-packages and modules within a
single package across multiple, separate distribution packages.

With the small script below, imports from the smaller package now work correctly.

import setuptools


From the smaller package, we can install the larger package in editable mode using a file system reference so that development changes can be used without a re-install. Using pip:

python -m pip install --editable "/path/to/larger/repo"

And finally, we can install the larger package's dependencies:

python -m pip install -r "/path/to/larger/repo/requirements.txt"

This approach works well as a short-term solution and doesn't require a lot of refactoring in either repository.