Container image cleanup policies¶
Our CI pipelines and build jobs push a lot of container images to both GitLab container registry and Google Cloud Artifact Registry. Cleanup policies delete old images from these registries automatically. This page explains the policies we apply, what each one deletes and keeps, and how to change it.
Why we clean up images¶
Projects built from our boilerplates push a new image every time CI runs, on every branch. Renovate merge requests add to this churn, so a busy product can push dozens of images a week that are never deployed again.
Before we had cleanup policies these images were kept forever. In Artifact Registry alone they added up to several terabytes across all products, most of it more than 90 days old. That storage costs money, and the clutter makes tagged releases harder to find.
Artifact Registry¶
The GCP product factory
applies a cleanup policy to each product's Docker repositories in Artifact Registry. Every product
gets a default Docker repository in its meta project, and optionally a public one. For
background on how products and their meta projects are created, read service
infrastructure.
What the policy does¶
The policy has three rules. Artifact Registry evaluates them in the background, not at push time, so an image can stay for a while after it becomes eligible for deletion.
The first rule deletes any image version older than 180 days, whether it is tagged or not.
The second rule keeps any version with a tag that looks like a release. By default that means
semantic version tags such as 1.2.3 or v1.2.3, and latest.
The third rule keeps the five most recent versions of each image, however old they are. This protects a service that is still running an image which has not been rebuilt in a long time. Without it, Cloud Run could be left pointing at an image that no longer exists.
Artifact Registry always gives keep rules precedence over delete rules. An image version is only deleted if it is older than 180 days and matches neither keep rule.
How release tags are matched¶
Artifact Registry matches tags by prefix and has no regular expression support. We approximate
semantic versions with the prefixes 0. to 99., plus v and latest. The trailing dot keeps
commit SHAs out, because a SHA never contains a dot. A bare 1 prefix would match a large share of
SHA tags and keep them forever.
Warning
The default prefixes only cover major versions 0 to 99. A release tagged 100.0.0 is not
protected and becomes eligible for deletion after 180 days, although v100.0.0 is still kept
by the v prefix. If you expect your product to release a major version above 99, add the
extra prefixes to keep_tag_prefixes in your product's policy, as described in changing the
policy for a product.
Any other tag starting with v or latest is kept too, for example latest-dev. That mistake
keeps images that could have gone, which is the safer way to be wrong.
Changing the policy for a product¶
Each product can tune the policy through the docker_repository_cleanup_policy variable in its
.tfvars file in the gcp-product-factory. The same settings apply to both the default and
public repositories. For example, a product that needs a longer history could keep images for a
year.
docker_repository_cleanup_policy = {
delete_older_than_days = 365
}
The variable also lets a product change how many recent versions are kept, replace the list of tag
prefixes, switch the policy to dry run, or disable it entirely. Setting keep_tag_prefixes
replaces the default list rather than adding to it, so include the semantic version prefixes if you
still want them. The variable
description
lists every field and its default.
In dry run, Artifact Registry writes what it would delete to the project's audit logs and deletes nothing. Use it to check the effect of a change before you enforce it.
GitLab container registry¶
We are working on a cleanup policy for GitLab container registry. We will announce it once it has been through the Tech Lead Forum.
Summary¶
Product Docker repositories in Artifact Registry delete image versions older than 180 days by
default. Versions with a release tag and the five most recent versions of each image are always
kept. Products can tune or disable the policy through docker_repository_cleanup_policy in the
product factory. A policy for GitLab container registry is in progress.
See also¶
- Service infrastructure, which describes the meta project that holds the Artifact Registry repositories.
- The IaC factory pattern, which explains how the product factory manages product configuration.
- Configure cleanup policies, Google's documentation for Artifact Registry cleanup policies.