Publishing Python packages¶
We publish some of our Python packages on pypi.org. This is
partially to be a good Python citizen but also because it means that we can make
use of pip
to install them easily where we use them in other projects.
A dedicated template is available in our CI templates project which can be used to automate publication of packages to PyPI.
A tale of two PyPIs¶
There are two publicly available instances of PyPI which we will use in this guide. One is the "main" PyPI found at https://pypi.org/ and the second is the "test" instance at https://test.pypi.org/. We have jobs which can publish to either. Publishing to the "test" instance is useful if you're testing packaging, etc. and don't want to actually make a "true" release.
Signing in to PyPI¶
In 1password we have account credentials for two "uis-devops-bot" accounts, one for each PyPI. Each are set up with 2FA and the 2FA tokens are present in 1password. They are both configured with devops-account-recovery@uis.cam.ac.uk as the recovery email address.
This account is the primary owner of all of our published Python packages.
Terminology¶
The steps in this guide assume that the configuration described below is present on the "master" branch or, at least, on some other protected branch. When we use the term "release branch" we mean the branch which has the PyPI-related GitLab CI configuration. For already published packages this is likely to be "master". For packages which have not yet been published this may be the branch that contains the to-be-reviewed configuration.
Publishing a brand new package¶
There is a small amount of setup required to publish brand new Python package to PyPI.
Make sure setup.py
is suitable¶
Ensure that the package has an appropriate setup.py
which allows it to be
installed via pip
. An example is below:
# setup.py contains configuration allowing this package to be installed into
# the Python environment.
import os
from setuptools import setup, find_packages
# Customisable fields placed here for easy editing. PyPI does not allow multiple
# uploads with the same name so the version number *must* be incremented on each
# publish. Try to use the semver convention when deciding which part of the
# version to increment.
PACKAGE_NAME="frobnicate"
PACKAGE_DESCRIPTION="Frobnicate wuggs and kikis"
PACKAGE_VERSION="0.0.1"
PACKAGE_URL="https://gitlab.developers.cam.ac.uk/uis/devops/lib/frobnicate/"
def load_requirements():
"""
Load requirements file and return non-empty, non-comment lines with leading
and trailing whitespace stripped.
"""
with open(os.path.join(os.path.dirname(__file__), 'requirements.txt')) as f:
return [
line.strip() for line in f
if line.strip() != '' and not line.strip().startswith('#')
]
with open("README.md") as fobj:
long_description = fobj.read()
setup(
name=PACKAGE_NAME,
version=PACKAGE_VERSION,
author="University of Cambridge Information Services",
author_email=f"devops+{PACKAGE_NAME}@uis.cam.ac.uk",
description=PACKAGE_DESCRIPTION,
long_description=long_description,
long_description_content_type="text/markdown",
url=PACKAGE_URL,
packages=find_packages(),
install_requires=load_requirements(),
classifiers=[
"Programming Language :: Python :: 3",
# Note: this is our standard license for open projects.
"License :: OSI Approved :: MIT License",
],
)
This should be suitable for most of our Python packages. It assumes there is a
README.md
file which documents the product and a requirements.txt
file which
specifies the dependencies.
Add PyPI jobs to GitLab CI¶
We make use of GitLab CI to publish packages. We have a standard template
available which may be included into the project's .gitlab-ci.yml
file as
follows:
# .gitlab-ci.yml contains configuration for GitLab CI jobs.
include:
# ... other includes
# Support uploading to PyPI
- project: 'uis/devops/continuous-delivery/ci-templates'
file: '/pypi-release.yml'
Create a temporary test PyPI API token¶
The PyPI publication jobs won't be added to pipelines unless the appropriate
PYPI_API_TOKEN
or TEST_PYPI_API_TOKEN
variables are defined in the GitLab CI
configuration and so the next step is to create an API token.
PyPI allows API tokens to have "scopes" limiting them to certain projects. This presents us with a bit of a chicken-and-egg problem: we want to use an API token limited in scope to the project we're releasing but we can't select that scope until we've published the project. We'll have to do a bit of token gymnastics to get around this.
Sign into the test PyPI instance using the
uis-devops-bot
account credentials. Create a new API
token with the following settings:
- Name: Temporary all scopes token to publish [PACKAGE NAME]
- Scope: Entire account (all projects)
Make sure to copy the API token somewhere safe since it will only be shown to you once.
Add the temporary test PyPI API token as a GitLab CI variable¶
In the CI/CD settings for the target project on GitLab, add a new variable:
- Key: TEST_PYPI_API_TOKEN
- Value: paste in API token
- Protect variable: yes
- Mask variable: yes
Important
We have configured the variable as being protected. This means that it will only be available to CI jobs running on protected branches. By default only "master" is a protected branch in our standard project configuration.
Make the first release¶
Trigger a CI/CD pipeline run for the release branch. (This can be done via the
Run Pipeline button on the CI/CD pipelines page for the GitLab project.) A
new manual job is created
called test-pypi-release
.
Visit the pipeline page for the triggered pipeline and click the "play" button
next to test-pypi-release
. Confirm that an initial release has been made to
the Test PyPI by looking in the log for the job.
Info
If there is no test-pypi-release
job, make sure that the branch the
pipeline is running on is protected.
Replace the API token with a scoped token¶
On the account management page for test PyPI, remove the temporary token created above and create a new one:
- Name: GitLab CI for [PACKAGE NAME]
- Scope: Select the appropriate project
Copy the new token and replace the value for the TEST_PYPI_API_TOKEN
in the
GitLab CI/CD settings.
Test the new token¶
Trigger a pipeline for the release branch and start the test-pypi-release
job
as before. The publication will fail because PyPI does not allow uploading new
packages with the same version number but you shouldn't get a failure due to bad
authentication.
Repeat for the production PyPI¶
Once you are happy publication is working well for the test PyPI, repeat the
instructions for the production PyPI instance. The steps
are the same except that tokens should be created in the main PyPI instance,
the GitLab CI/CD variable should be PYPI_API_TOKEN
and the job is called
pypi-release
.
Publishing a previously published package¶
This is far simpler because some kind soul will already have performed the configuration above.
Bump the version number if necessary¶
Before you publish make sure that the version of the package as set in
setup.py
is greater than the version currently published on PyPI. We use
semver when versioning our packages.
Tip
Python packages support adding "devN" to a version to indicate a development release which should not be installed unless explicitly asked for. As such, if the most recent published version is "1.2.3", you can publish "1.2.3dev4" without it becoming the automatically installed version.
Publish to the test PyPI¶
On the GitLab project page, click on the "pipeline status" badge to get to the
most recently run pipeline on "master". There should be a manual job waiting for
you there called test-pypi-release
. Click the "play" button and check the job
logs to see if the package was successfully published.
Publish to production PyPI¶
If the test publication succeeds, click the "play" button next to the
pypi-release
job in the same pipeline. This should publish the package to the
production PyPI site.