Protobuf client generation from protobuf specification files¶

This page provides a reference on customising the protobuf-based client generation described in the how to automatically generate protobuf clients guide.

Customising the generated clients¶

The protobuf generator uses Buf to generate clients. You can customise the language-specific generation by configuring a buf.gen.yaml file in your repository. For example, to generate Python stubs:

version: v2
managed:
  enabled: true
plugins:
  - remote: buf.build/community/danielgtaylor-betterproto:v1.2.5
    out: "gen/python"

Python packaging metadata¶

The protobuf code generation does not include packaging metadata (such as pyproject.toml). To successfully build Python packages using the build tool, you must supply this metadata yourself.

These files must be placed in the directory defined by the PROTOBUF_GENERATOR_PYTHON_STATIC_FILES_DIR variable (default: generator-static/python). They will be copied into the generated source directory before building the package.

A minimal example using Poetry as the build backend:

[tool.poetry]
name = "example-protobuf-client"
version = "0.0.1"
description = ""
authors = [ ]

[[tool.poetry.packages]]
include = "./**/*.py"
to = "example_protobuf_client"

[tool.poetry.dependencies]
betterproto = "^1.2.5"

[build-system]
requires = [ "poetry-core" ]
build-backend = "poetry.core.masonry.api"

The build tool used in the CI pipeline follows PEP 517. Your pyproject.toml must define a compatible build backend such as poetry-core, or setuptools.build_meta.

Warning

If these files are missing or invalid, the Python package build step will fail.

Keeping versions in sync¶

When used in conjunction with release-it to manage releases, and your repository has a root-level pyproject.toml, it is recommended to keep the version in your protobuf client package (e.g. generator-static/python/pyproject.toml) in sync with the main project version.

For example, using:

    "@release-it/bumper": {
      "out": [
        {
          "file": "pyproject.toml",
          "type": "text/toml",
          "path": "tool.poetry.version"
        },
        {
          "file": "generator-static/python/pyproject.toml",
          "type": "text/toml",
          "path": "tool.poetry.version"
        }
      ]
    }

Using with release-it¶

You can publish the generated protobuf packages to your GitLab release using release-it.

For example, the following release-it configuration ensures that everything inside the protobuf_artifact_dir is uploaded as release assets:

  "gitlab": {
    "release": true,
    "releaseName": "${version}",
    "useIdsForUrls": true,
    "assets": [
      "protobuf_artifact_dir/**"
    ]
  }

Important

release-it support in the common pipeline is only supported when using the merge request release flow. You must explicitly enable this by setting the following CI variable in your project:

variables:
  USE_MERGE_REQUEST_RELEASE_FLOW: "1"

Disabling generation¶

Client generation can be disabled by setting the following CI variables:

Variable	Effect
`PROTOBUF_GENERATOR_DISABLED`	Disables all protobuf generation jobs.
`PROTOBUF_GENERATOR_PYTHON_DISABLED`	Disables Python client generation.

CI job artefact locations¶

Variable	Description
`PROTOBUF_GENERATOR_PYTHON_OUTPUT_DIR`	Directory where the generated Python files are placed.
`PROTOBUF_GENERATOR_PACKAGE_ARTIFACT_DIR`	Directory where the built package artifacts are stored.
`PROTOBUF_GENERATOR_PYTHON_STATIC_FILES_DIR`	Directory where static files like `pyproject.toml` and `__init__.py` are located.

Publication of packages¶

By default, packages are only published when changes are pushed to the default branch ($CI_DEFAULT_BRANCH). You can customise this behaviour by overriding the rules in the .protobuf:publish:base job in your CI configuration.

For example, to publish only on version tags:

.protobuf:publish:base:
  rules:
    - if: $CI_COMMIT_TAG =~ /v?[0-9]+\.[0-9]+\.[0-9]+/

You can also disable publication using the following CI variables:

Variable	Description
`PROTOBUF_GENERATOR_PUBLISH_DISABLED`	Disable all publication jobs.
`PROTOBUF_GENERATOR_PUBLISH_PYTHON_DISABLED`	Disable publication of the Python package.
`PROTOBUF_GENERATOR_PUBLISH_PYTHON_GITLAB_DISABLED`	Disable publication of the Python package to GitLab's registry.

CI Variables¶

Variable	Default	Description
`PROTOBUF_GENERATOR_BUF_CONFIGURATION_PATH`	`buf.yaml`	Path to your Buf configuration file.
`PROTOBUF_GENERATOR_PROTO_FILES_DIR`	`proto`	Directory containing your `.proto` files.
`PROTOBUF_GENERATOR_BUF_PYTHON_CONFIGURATION_PATH`	`buf.gen.yaml`	Buf generation template for Python clients.
`PROTOBUF_GENERATOR_PYTHON_OUTPUT_DIR`	`gen/python`	Output directory for generated Python code.
`PROTOBUF_GENERATOR_PYTHON_STATIC_FILES_DIR`	`generator-static/python`	Directory containing Python packaging metadata like `pyproject.toml`.
`PROTOBUF_GENERATOR_PACKAGE_ARTIFACT_DIR`	`protobuf_artifact_dir`	Directory in which built package artifacts are stored.
`PROTOBUF_GENERATOR_PUBLISH_REF_NAME`	`$CI_DEFAULT_BRANCH`	Branch from which packages should be published.

Summary¶

This guide explained how to customise, build, and publish protobuf-based client libraries using Buf and GitLab CI, with a focus on Python packaging. Be sure to supply valid PEP 517-compatible packaging metadata to ensure successful builds.

See the how-to guide for a walkthrough on setting up a new project.