Skip to content

Publishing Python packages

We publish some of our Python packages on pypi.org. This is partially to be a good Python citizen but also because it means that we can make use of pip to install them easily where we use them in other projects.

A dedicated template is available in our CI templates project which can be used to automate publication of packages to PyPI.

A tale of two PyPIs

There are two publicly available instances of PyPI which we will use in this guide. One is the "main" PyPI found at https://pypi.org/ and the second is the "test" instance at https://test.pypi.org/. We have jobs which can publish to either. Publishing to the "test" instance is useful if you're testing packaging, etc. and don't want to actually make a "true" release.

Signing in to PyPI

In 1password we have account credentials for two "uis-devops-bot" accounts, one for each PyPI. Each are set up with 2FA and the 2FA tokens are present in 1password. They are both configured with devops-account-recovery@uis.cam.ac.uk as the recovery email address.

This account is the primary owner of all of our published Python packages.

Terminology

The steps in this guide assume that the configuration described below is present on the "master" branch or, at least, on some other protected branch. When we use the term "release branch" we mean the branch which has the PyPI-related GitLab CI configuration. For already published packages this is likely to be "master". For packages which have not yet been published this may be the branch that contains the to-be-reviewed configuration.

Publishing a brand new package

There is a small amount of setup required to publish brand new Python package to PyPI.

Make sure setup.py is suitable

Ensure that the package has an appropriate setup.py which allows it to be installed via pip. An example is below:

# setup.py contains configuration allowing this package to be installed into
# the Python environment.

import os
from setuptools import setup, find_packages

# Customisable fields placed here for easy editing. PyPI does not allow multiple
# uploads with the same name so the version number *must* be incremented on each
# publish. Try to use the semver convention when deciding which part of the
# version to increment.
PACKAGE_NAME="frobnicate"
PACKAGE_DESCRIPTION="Frobnicate wuggs and kikis"
PACKAGE_VERSION="0.0.1"
PACKAGE_URL="https://gitlab.developers.cam.ac.uk/uis/devops/lib/frobnicate/"

def load_requirements():
    """
    Load requirements file and return non-empty, non-comment lines with leading
    and trailing whitespace stripped.

    """
    with open(os.path.join(os.path.dirname(__file__), 'requirements.txt')) as f:
        return [
            line.strip() for line in f
            if line.strip() != '' and not line.strip().startswith('#')
        ]


with open("README.md") as fobj:
    long_description = fobj.read()


setup(
    name=PACKAGE_NAME,
    version=PACKAGE_VERSION,
    author="University of Cambridge Information Services",
    author_email=f"devops+{PACKAGE_NAME}@uis.cam.ac.uk",
    description=PACKAGE_DESCRIPTION,
    long_description=long_description,
    long_description_content_type="text/markdown",
    url=PACKAGE_URL,
    packages=find_packages(),
    install_requires=load_requirements(),
    classifiers=[
        "Programming Language :: Python :: 3",

        # Note: this is our standard license for open projects.
        "License :: OSI Approved :: MIT License",
    ],
)

This should be suitable for most of our Python packages. It assumes there is a README.md file which documents the product and a requirements.txt file which specifies the dependencies.

Add PyPI jobs to GitLab CI

We make use of GitLab CI to publish packages. We have a standard template available which may be included into the project's .gitlab-ci.yml file as follows:

# .gitlab-ci.yml contains configuration for GitLab CI jobs.

include:
  # ... other includes

  # Support uploading to PyPI
  - project: 'uis/devops/continuous-delivery/ci-templates'
    file: '/pypi-release.yml'

Create a temporary test PyPI API token

The PyPI publication jobs won't be added to pipelines unless the appropriate PYPI_API_TOKEN or TEST_PYPI_API_TOKEN variables are defined in the GitLab CI configuration and so the next step is to create an API token.

PyPI allows API tokens to have "scopes" limiting them to certain projects. This presents us with a bit of a chicken-and-egg problem: we want to use an API token limited in scope to the project we're releasing but we can't select that scope until we've published the project. We'll have to do a bit of token gymnastics to get around this.

Sign into the test PyPI instance using the uis-devops-bot account credentials. Create a new API token with the following settings:

  • Name: Temporary all scopes token to publish [PACKAGE NAME]
  • Scope: Entire account (all projects)

Make sure to copy the API token somewhere safe since it will only be shown to you once.

Add the temporary test PyPI API token as a GitLab CI variable

In the CI/CD settings for the target project on GitLab, add a new variable:

  • Key: TEST_PYPI_API_TOKEN
  • Value: paste in API token
  • Protect variable: yes
  • Mask variable: yes

Important

We have configured the variable as being protected. This means that it will only be available to CI jobs running on protected branches. By default only "master" is a protected branch in our standard project configuration.

Make the first release

Trigger a CI/CD pipeline run for the release branch. (This can be done via the Run Pipeline button on the CI/CD pipelines page for the GitLab project.) A new manual job is created called test-pypi-release.

Visit the pipeline page for the triggered pipeline and click the "play" button next to test-pypi-release. Confirm that an initial release has been made to the Test PyPI by looking in the log for the job.

Info

If there is no test-pypi-release job, make sure that the branch the pipeline is running on is protected.

Replace the API token with a scoped token

On the account management page for test PyPI, remove the temporary token created above and create a new one:

  • Name: GitLab CI for [PACKAGE NAME]
  • Scope: Select the appropriate project

Copy the new token and replace the value for the TEST_PYPI_API_TOKEN in the GitLab CI/CD settings.

Test the new token

Trigger a pipeline for the release branch and start the test-pypi-release job as before. The publication will fail because PyPI does not allow uploading new packages with the same version number but you shouldn't get a failure due to bad authentication.

Repeat for the production PyPI

Once you are happy publication is working well for the test PyPI, repeat the instructions for the production PyPI instance. The steps are the same except that tokens should be created in the main PyPI instance, the GitLab CI/CD variable should be PYPI_API_TOKEN and the job is called pypi-release.

Publishing a previously published package

This is far simpler because some kind soul will already have performed the configuration above.

Bump the version number if necessary

Before you publish make sure that the version of the package as set in setup.py is greater than the version currently published on PyPI. We use semver when versioning our packages.

Tip

Python packages support adding "devN" to a version to indicate a development release which should not be installed unless explicitly asked for. As such, if the most recent published version is "1.2.3", you can publish "1.2.3dev4" without it becoming the automatically installed version.

Publish to the test PyPI

On the GitLab project page, click on the "pipeline status" badge to get to the most recently run pipeline on "master". There should be a manual job waiting for you there called test-pypi-release. Click the "play" button and check the job logs to see if the package was successfully published.

Publish to production PyPI

If the test publication succeeds, click the "play" button next to the pypi-release job in the same pipeline. This should publish the package to the production PyPI site.