Python Namespace Packages

2023-05-27

From Unsplash

I'm working with a Python project that's split across two Git repositories. We're eventually going to migrate the smaller of the two repositories into the larger repository organized as a monorepo.

In the meantime, we need to reuse some code from the larger repository in the smaller repository. There are a few options for us, but since the smaller repository is eventually going away, we settled on namespace packages.

I planned to set up a basic pyproject.toml file in the larger repository and reference it from the smaller repository using a git URL in the requirements.txt file. That proved problematic, however, due to the larger repository's directory structure which doesn't follow the pattern of a top-level src directory and child directories for packages. It also does not rely on installing the packages individually or referencing the packages via a requirements.txt file. Because of that, imports from the smaller repository lost the top-level package.

That's where namespace packages helped. From the documentation:

Namespace packages allow you to split the sub-packages and modules within a
single package across multiple, separate distribution packages.

With the small setup.py script below, imports from the smaller package now work correctly.

import setuptools

setuptools.setup(
    name='apps',
    packages=setuptools.find_namespace_packages(include=['apps']),
)

From the smaller package, we can install the larger package in editable mode using a file system reference so that development changes can be used without a re-install. Using pip:

python -m pip install --editable "/path/to/larger/repo"

And finally, we can install the larger package's dependencies:

python -m pip install -r "/path/to/larger/repo/requirements.txt"

This approach works well as a short-term solution and doesn't require a lot of refactoring in either repository.