GitLab CI/CD Component Best Practice¶

This guide covers some aspects of best practice for writing GitLab CI/CD components for the Unified DevOps Platform. It is worth reading this guide from top-to-bottom at least once before starting to write a new component. In addition, read and learn from the patterns used in the existing CI/CD components provided by the Unified DevOps Platform.

Follow the how-to guide for stop-by-step instructions for creating a new CI/CD component for the Unified DevOps Platform.

General guidelines¶

This section covers general guidelines for CI/CD components.

A CI/CD component should perform one well defined action. E.g "build a container image", "run tests via pytest", "publish a Python package to the GitLab package registry". Components which relate to the same artifact type should be collected together in the same project.

For example, the Python package "build" and "publish" components both live in the same project but the container image "build" and Python package "build" components each live in separate projects.

Avoid baking in "DevOps-isms"¶

Remember that components are meant to be shared with the wider University. Avoid "DevOps"-isms which require a particular project layout. You may choose defaults which assuming the layout matches templates provided by the Unified DevOps Platform. Avoid assumptions on the runner environment such as any particular workload identity or network environment. Do not assume that any particular runner tags are available. Do not assume that a GitLab token secret is present and readable.

Document project requirements¶

If a project needs a particular CI variable present or if you need to make use of a CI_JOB_TOKEN which can push changes to the repository, clearly document this fact and have informative error messages if the variable is missing or if the job token does not have push access.

Drink our own Champagne¶

Make use of the CI/CD components we have already published. For example the pre-commit check component uses the container image build component to build the pre-commit and prek containers used to run jobs.

Respect semantic versioning¶

We encourage people to include components via path/to/component:x so that minor bug-fixes and forward compatible features are automatically used without needing to be explicitly managed by renovate. This, in turn, means one needs to be very careful to mark breaking changes as such.

Breaking changes include:

Changing the naming or count of added jobs.
Removing a dotenv artifact variable or radically changing its semantics.
Adding a new required input.
Removing an existing input.
Requiring extra permissions for the CI_JOB_TOKEN.
Requiring downstream users to provide additional authentication for jobs, e.g. because of a change of docker registry.

Ensure the component version in example usage matches¶

The release-it configuration in our CI/CD component copier template includes configuration to automatically update the version number of the component if it appears after @ in a markdown file.

Make use of this to have example usage in READMEs which can be copied and pasted to make use of the most recent container version. See the container-image/build README for an example.

Project setup¶

This section covers how to configure your CI/CD component project.

Use the copier template¶

Use the CI/CD component copier template as a base for your component. It contains a set of linting jobs and a pre-commit configuration for keeping the component READMEs up-to-date.

Place components in their own directories¶

Even if your component is a single file, place it in the templates/{component-name}/template.yml file. This allows you to have per-component READMEs and to have per-component CI tests.

Enable Merge Request pipelines¶

The CI/CD component copier template configures the use of Merge Request pipelines. This allows non-DevOps members to open MRs while allowing the option to run the CI testing pipeline for that Merge Request using the component project’s runners. This avoids the need for all external contributors to have runners enabled for their projects.

Use the Merge Request release-it flow¶

The CI/CD component copier template configures the MR-based release-it flow for you but you will also need to ensure that ast_enable_mr_pipelines = true is present in the GitLab Project factory configuration.

Build container images as part of the CI/CD component project¶

If your CI/CD component requires a container image to run CI/CD jobs, build that container image within the CI/CD component project and have the container image match the version of your component. See the pre-commit CI/CD component for an example.

Set container registry expiration policies¶

If you build container images as part of your component, make sure to configure a container expiration policy in the GitLab Project Factory.

Use renovatebot to keep defaults for template inputs fresh¶

Our default DevOps renovatebot configuration ensures that the default values for inputs with the following special names are kept fresh:

...-docker-image specifies a docker image used by a component. E.g. python:3.14.
...-ci-component specifies a CI/CD component used by a component. E.g. uis/devops/platform/ci-components/ci-component/trigger-pipelines@0.6.5.
...-pip-requirement specifies a Python package used by a component. E.g. requests==2.34.2.

Testing components¶

This section covers testing your CI/CD components.

Write tests for your components¶

Follow the GitLab guidelines for testing components. At a minimum you should test that your component successfully runs with the default configuration. Where inputs can be used to enable or disable jobs, you should test that they work as expected.

Use child pipelines to test components¶

Make use of the trigger component pipelines CI/CD component to run your CI/CD component unit and integration tests in a separate child pipeline. This ensures that job names will not inadvertently clash between components in the same project.

Consider an integration test¶

If you have a project consisting of multiple components, for example a build and publish component, consider adding an integration test pipeline to the component project which tests both components working together.

Test outputs from your components¶

If your components produce dotenv artifacts, write dependent test jobs which check that the variables are set to expected values. See, for example, the dotenv-artifacts-are-set job from the container image build component.

Writing components¶

This section covers the actual content of the CI/CD component repository.

Name components as "{singular noun phrase}/{verb phrase}"¶

When possible, name the project holding components after the tool which they run or thing which they manage or produce. Name the component template itself with a verb indicating what it will do in the context of the project name.

Examples:

python-package/build and python-package/publish (two different components within the same project)
container-image/build
release/release (here, "release" is both a noun and a verb and the component will "release" a new "release")
pre-commit/check (here the noun is the tool or tool family being run)
ci-component/trigger-pipeline ("CI component" is a noun phrase, "trigger pipeline" is a verb phrase)

DO NOT name components like:

python-packages/build (plural noun for project name)
python-package/uv (using a noun, "uv", for component name instead of a verb)
docker/images (using a plural noun for component name instead of a verb)
build-ts-package/publish (using a verb phrase for project name)
ci-component/pipeline-triggering (using a noun phrase for the component name)

Use pre-commit hooks to keep the READMEs up-to-date¶

The CI/CD component copier template includes pre-commit hooks to automatically include tables of inputs in the per-component READMEs and to collect the tables for each component together into the top-level README. Keep that configuration in place.

Use inputs rather than variables¶

Avoid using CI variables to configure CI/CD components and prefer using CI/CD inputs. An exception can be made if the CI/CD component is intended to provide a backwards-compatible replacement for an existing well-known template. Even then, include a set of inputs which can be used by new users of the component.

Consider if component can be included multiple times¶

Generally a component should perform one task and performing multiple tasks is done by including the component multiple times. For example, the container-image/build component builds one container image. If a project needs to build multiple container images, it should be ergonomic to include the component multiple times.

If a CI/CD component must not be included multiple times, that must be made very clear in the documentation.

Be monorepo friendly¶

For build or packaging jobs, do not assume that the artifact needing to be built exists at the root of the repository. It is OK for that to be the default. For example, the python-package/build component allows building a Python package within a subdirectory. This is often very useful for writing tests of the CI/CD component itself if nothing else.

Treat adding, removing or renaming jobs as a breaking change¶

Make sure that the names of jobs added by your template are documented. Changing these names or changing the number of jobs added should be signalled as a breaking change. This is because users of your component need to know the names of jobs so that they may conditionally enable jobs by setting rules on the jobs rather than where the component is include-ed. They may want to do this because the CI variables available for rules and the directory which exists rules operate on can differ between include-based rules and job-based rules.

Allow customisation of the job names and stages¶

Users may wish to include your component multiple times. As such, make sure to include a job-name or job-name-prefix input to allow customising the name of added jobs. Add a job-stage input to allow users to run your jobs in a different stage if required.

Consider whether to allow customisation of the job image¶

If your job needs to run in a particular image, consider allowing overriding the image via an appropriately named ...-docker-image input. This allows downstream users of your tool to add additional requirements into the image. Do not allow customisation when the image is intended to be a "black box" which should not be customised.

Avoid using `tags` if possible¶

Avoid setting tags on jobs if possible since we cannot assume a particular runner environment.

Avoid using job `rules` if possible¶

Avoid using rules on jobs if possible since downstream users may wish to customise job rules directly. Instead use conditional include statements. See, for example, how the container image template conditionally includes container image scanning.

Avoid multiple stages¶

Do not use multiple stages to control job execution order. Prefer needs:job if you need to represent dependencies between jobs.

Avoid using `dependencies`¶

GitLab’s dependencies and needs configurations do not play well together. Avoid using dependencies if you can.

Publish `dotenv` artifacts for jobs which have outputs¶

Provide explicit dotenv artifacts when jobs create outputs. This allows for easy re-use of artifacts by dependent jobs which use needs. For example, the python-package/build job sets PYTHON_PACKAGE_{NAME,VERSION,ARTIFACT_DIR} variables for dependent jobs to use to locate the packages and to know what version was built. This allows customising the output directory in the build job without breaking downstream publish jobs. It also documents the interface used by downstream publish jobs so users can write their own if they need to publish the packages to a specific repository.

Expose pip requirements as inputs¶

If you make use of specific pip packages, expose the exact requirement, with version, as the default for an input named ...-pip-requirement. Format it as package-name==version. As noted above, renovatebot will keep the default fresh but downstream projects may need to pin a particular version to deal with regressions.

Expose docker image dependencies as inputs¶

If you make use of external docker images in your jobs, expose the exact tagged image as the default for an input named ...-docker-image. Format it as image-name:tag. As noted above, renovatebot will keep the default fresh but downstream projects may need to pin a particular version to deal with regressions.

Note

Image digest pinning is not possible at the moment due to a limitation with renovate’s jsonata manager.

Expose other CI/CD components as inputs¶

If you make use of CI/CD components from outside the parent project, expose the exact path, name and version used as the default for an input named ...-ci-component. Format it as path/to/project/component@x.y.z. As noted above, renovatebot will keep the default fresh but downstream projects may need to pin a particular version to deal with regressions.

Assume users will annotate your jobs¶

Assume users will want to add needs, dependencies, rules or tags. If this is not possible make this clear in the documentation.

Use `:` as a namespace separator¶

If your CI component must add multiple jobs, expose a common job name prefix as an input and use : as a namespace separator. E.g. the container image build component adds jobs named {prefix} and {prefix}:container-scanning.

Use namespaced infinitive verb phrases and kebab case for CI job names¶

Use the infinitive or "base dictionary entry" form of verbs to name CI jobs. Use kebab case. Make sure that job-names are prefixed with a noun phrase which describes what the CI/CD component is operating on.

Examples:

container-image:build
python-package:publish
python-package:scan-dependencies
container-image:push-to-artifact-registry

DO NOT use:

java:maven-packages (not a verb phrase)
Python:ScanDependencies or scan_dependencies (not kebab case)
python-package:runs-tests (not an infinitive verb)

There is an exception for jobs named after a well-known tool, existing CI/CD component or CI/CD template related to the action of the component. For example:

terraform-module:trivy
container-image:container-scanning
release:release-it

Use GitLab-provided components¶

Make use of components provided by GitLab themselves at https://gitlab.com/components when appropriate. You will need to make a mirror of the component project in order to do so. Add that mirror to uis/devops/external. Name the mirrored project gitlab-com-components-{...}-mirror. See, for example, the mirror of the container-scanning component.

Important

ALWAYS add new GitLab projects via the GitLab project factory, even mirroring projects.

Minimise the number of required inputs¶

Prefer defaults-based conventions over mandatory configuration. For example, for a project with a .pre-commit-config.yaml file in the root of the repository, one needs only include the pre-commit/check component to run pre-commit checks with no further configuration. Consider if you can add defaults for inputs which cover the common case.

Use `cache` when possible¶

Use GitLab CI/CD’s caching support when possible. In particular note that setting the XDG_CACHE_HOME environment variable to a cached directory is often sufficient to configure tools to use it as a cache. Allow the location of the cache directory to be customised but default to a .cache/{component-name} directory in the root of the repository.

Ensure that the cache is component, version and job specific¶

Avoid sharing caches between different versions of your component and different instances of the component within the same pipeline. At a minimum set the cache prefix to $[[ component.name ]]-$[[ component.sha ]]-${CI_JOB_NAME}.

Disable unrelated kics vulnerabilities in Docker images¶

Use kics-scan disable lines to disable warnings about missing HEALTHCHECK and USER configuration in images used to run CI/CD jobs. By design CI/CD jobs do not have a healthcheck and they will run as a runner-defined user. See the pre-commit component Dockerfile for an example.

Be aware of `include:rules` being special¶

The set of CI variables available to include:rules and {job}:rules differ. Make sure to carefully check the GitLab CI variable documentation when using include:rules.

When using include:rules:exists be mindful that it may not work as you expect with respect to which project is examined to see if a file exists. See the GitLab CI documentation for details.

Summary¶

This guide covered some best practice for writing CI/CD components intended to be re-used as part of the Unified DevOps Platform.