How to tune GKE-hosted runner memory and CPU settings for GitLab CI jobs

This guide explains how to configure resource requests and limits for GitLab CI jobs that run on GKE-hosted GitLab runners. These settings help optimize performance and control resource usage for CI pipelines.

Note

This guide is particularly useful when CI jobs are terminated due to Out Of Memory (OOMKilled) errors (typically seen as exit code 137) in jobs such as pre-commit or python:tox.

Supported variables

The following variables are available for fine-tuning resource allocation for jobs:

Variable	Description	Example
KUBERNETES_MEMORY_REQUEST	Defines the amount of memory requested for the job’s build container.	512Mi
KUBERNETES_MEMORY_LIMIT	Defines the maximum amount of memory the build container can use.	1Gi
KUBERNETES_CPU_REQUEST	Defines the amount of CPU requested for the job’s build container.	500m
KUBERNETES_SERVICE_MEMORY_REQUEST	Defines the amount of memory requested for each service container, such as docker:dind.	1Gi
KUBERNETES_SERVICE_MEMORY_LIMIT	Defines the maximum amount of memory each service container can use.	3Gi
KUBERNETES_HELPER_MEMORY_REQUEST	Defines the amount of memory requested for the helper container.	256Mi
KUBERNETES_HELPER_MEMORY_LIMIT	Defines the maximum amount of memory the helper container can use.	2Gi

Each job pod has a build container, a helper container that clones the repository and handles artifacts and the cache, and one container for each entry in services. The first three variables only affect the build container. See Resource requests and limits for job pods for how the containers' values add up, and the GKE GitLab Runners reference for the default values.

For more details, see the official GitLab Runner documentation: Overwrite container resources

Note

The runner sets a ceiling for each overwrite, and a job cannot set a value above it. The request ceiling for a container is the same as its limit ceiling, because a request cannot be larger than its limit. As of October 2026 the production ceilings are 2000m CPU and 8Gi memory for the build container, 4Gi for the service containers and 4Gi for the helper container. The request and limit variables for the helper container and the service containers are new, and the platform team can change the ceilings per runner. Ask in the Platforms Team channel if a job genuinely needs more.

Warning

Use these variables only when necessary, and increase resource values incrementally and responsibly. Setting excessively high limits (for example, 8Gi memory for every job) can cause inefficient cluster utilization, prevent autoscaling from working correctly, and increase costs. Always profile and measure before raising limits.

When to use

Only use these variables if:

• A job is repeatedly failing due to OOMKilled (e.g. exit code 137).
• A job runs significantly slower than expected due to CPU throttling.
• Profiling shows that the job requires more resources than the default configuration provides.

Jobs that use docker-in-docker

When a job uses the docker:dind service, everything you run with docker (test suites, image builds, docker compose stacks) runs inside the service container, not the build container. Set the KUBERNETES_SERVICE_MEMORY_* variables for such jobs, not KUBERNETES_MEMORY_*. See Using docker in CI jobs for how to enable dind.

The service container has its own memory limit. If the workload inside it goes over the limit, the service container is OOMKilled and the job typically fails with Docker connection errors or exit code 137. Only that job's pod is affected, and other jobs on the node carry on.

Tools that start one worker per CPU are a common cause, because the container sees every CPU of the node (16 on the production nodes) whatever its limits. Capping the number of workers in the tool's own configuration is usually a better first step than raising the limit.

.gitlab-ci.yml

build-image:
  image: docker:24-git
  services:
    - docker:24-dind
  variables:
    DOCKER_HOST: tcp://docker:2376
    DOCKER_TLS_CERTDIR: "/certs"
    DOCKER_TLS_VERIFY: 1
    DOCKER_CERT_PATH: "/certs/client"
    KUBERNETES_SERVICE_MEMORY_REQUEST: "1Gi"
    KUBERNETES_SERVICE_MEMORY_LIMIT: "3Gi"
  before_script:
    - until docker info > /dev/null 2>&1; do sleep 1; done
  script:
    - docker build .
  tags:
    - $GKE_RUNNER_TAG

Estimating appropriate values

Before changing resource settings, profile the job locally or in a controlled environment to understand its actual requirements. An example profiling workflow is shown below:

1. Run the same job locally in a Docker container:

2. Monitor resource usage while the job runs:

   docker container top <container_id>
   docker stats <container_id>
   

3. Note the peak CPU and memory usage.

Steps

1. Define the resource tuning variables in a CI job configuration file. For example:

   resource-tuned-job:
     stage: test
     script:
       - echo "Running with tuned Kubernetes resource settings"
     variables:
       KUBERNETES_MEMORY_REQUEST: "512Mi"
       KUBERNETES_MEMORY_LIMIT: "1Gi"
       KUBERNETES_CPU_REQUEST: "500m"
     tags:
       - $GKE_RUNNER_TAG
   

2. Commit and push the configuration changes. Subsequent CI jobs will use the specified memory and CPU settings on the Kubernetes runner.

Summary

This guide described how to control memory and CPU resources for CI jobs running on a GKE-hosted GitLab runner using the KUBERNETES_MEMORY_REQUEST, KUBERNETES_MEMORY_LIMIT, and KUBERNETES_CPU_REQUEST variables, and the KUBERNETES_SERVICE_MEMORY_* and KUBERNETES_HELPER_MEMORY_* variables for service and helper containers. These settings should be used sparingly, only when default configurations are insufficient, and should always be based on profiling data to maintain efficient use of cluster resources.