How we use PyPI¶
We publish some of our Python packages on pypi.org. This is
partially to be a good Python citizen but also because it means that we can make
pip to install them easily where we use them in other projects.
This page explains more about PyPI and why we do things the way we do.
We have a common CI/CD pipeline which can be used to publish Python packages.
The common pipeline template replaces our previous dedicated template for PyPI publishing. The information on this page is relevant to both pipeline templates.
Task-focussed and reference documentation for creating and publishing Python packages can be found in the following places:
- How to add the common pipeline to a Python project
- How to make a new release of a Python package
- How to create PyPI API tokens
- Reference documentation for the Python-specific portions of the common CI/CD pipeline
A tale of two PyPIs¶
Publishing a package is a one-way street; you cannot re-upload a package to PyPI once it has been uploaded. As such we always use the test PyPI as a check that our package metadata is suitable for upload and won't be rejected by PyPI.
Although you cannot re-upload a published package, you can delete a release which contains serious errors and upload a newer release. This can be done from the "releases" page for the package if you are signed in as our bot user.
Signing in to PyPI¶
In 1password we have account credentials for two "uis-devops-bot" accounts, one for each PyPI. Each are set up with 2FA and the 2FA tokens are present in 1password. They are both configured with firstname.lastname@example.org as the recovery email address.
This account is the primary owner of all of our published Python packages.
We do this to avoid individuals becoming single points of failure for package ownership and publication. It also enforces the use of automation for package releases which, in turn, ensures that test suites are run, etc.
We use API tokens to upload new packages to PyPI. PyPI allows API tokens to have "scopes" limiting them to certain projects. This presents us with a bit of a chicken-and-egg problem: we want to use an API token limited in scope to the project we're releasing but we can't select that scope until we've published the project.
We have to do a bit of token gymnastics to get around this issue. As documented in the how-to guide, we firstly create a temporary token with the ability to work with all projects. This scope is required to create new projects which will happen implicitly on the first release.
Once we have successfully made a release, we replace the API token with one whose scope is limited only to the project in question.
Fortunately this work only needs to be done once.
What makes a good PyPI release¶
Technically we can make a new release for a package providing only the name and
the version. Good packages include a bit more than this. At a minimum they
should include a long description. Poetry makes this easy by setting the
readme field in
pyproject.toml. The markdown-formatted README will be
uploaded to PyPI along with the release and will be present on the page.
Try to write your README from the point of view of someone coming across the project for the first time. You should include:
- what your project does,
- what problem it is trying to solve,
- a brief usage example, and
- where to find out more, for example a link to any API documentation.
LICENSE.txt file. See how to add the MIT licence to a
project for more.