Skip to content

How we use PyPI

We publish some of our Python packages on pypi.org. This is partially to be a good Python citizen but also because it means that we can make use of pip to install them easily where we use them in other projects.

This page explains more about PyPI and why we do things the way we do.

We have a common CI/CD pipeline which can be used to publish Python packages.

Info

The common pipeline template replaces our previous dedicated template for PyPI publishing. The information on this page is relevant to both pipeline templates.

Task-focussed and reference documentation for creating and publishing Python packages can be found in the following places:

A tale of two PyPIs

There are two publicly available instances of PyPI which we make use of. One is the "main" PyPI found at https://pypi.org/ and the second is the "test" instance at https://test.pypi.org/.

Publishing a package is a one-way street; you cannot re-upload a package to PyPI once it has been uploaded. As such we always use the test PyPI as a check that our package metadata is suitable for upload and won't be rejected by PyPI.

Tip

Although you cannot re-upload a published package, you can delete a release which contains serious errors and upload a newer release. This can be done from the "releases" page for the package if you are signed in as our bot user.

Signing in to PyPI

In 1password we have account credentials for two "uis-devops-bot" accounts, one for each PyPI. Each are set up with 2FA and the 2FA tokens are present in 1password. They are both configured with devops-account-recovery@uis.cam.ac.uk as the recovery email address.

This account is the primary owner of all of our published Python packages.

We do this to avoid individuals becoming single points of failure for package ownership and publication. It also enforces the use of automation for package releases which, in turn, ensures that test suites are run, etc.

Token "scopes"

We use API tokens to upload new packages to PyPI. PyPI allows API tokens to have "scopes" limiting them to certain projects. This presents us with a bit of a chicken-and-egg problem: we want to use an API token limited in scope to the project we're releasing but we can't select that scope until we've published the project.

We have to do a bit of token gymnastics to get around this issue. As documented in the how-to guide, we firstly create a temporary token with the ability to work with all projects. This scope is required to create new projects which will happen implicitly on the first release.

Once we have successfully made a release, we replace the API token with one whose scope is limited only to the project in question.

Fortunately this work only needs to be done once.

What makes a good PyPI release

Technically we can make a new release for a package providing only the name and the version. Good packages include a bit more than this. At a minimum they should include a long description. Poetry makes this easy by setting the readme field in pyproject.toml. The markdown-formatted README will be uploaded to PyPI along with the release and will be present on the page.

Try to write your README from the point of view of someone coming across the project for the first time. You should include:

  • what your project does,
  • what problem it is trying to solve,
  • a brief usage example, and
  • where to find out more, for example a link to any API documentation.

Include a LICENSE.txt file. See how to add the MIT licence to a project for more.