Skip to content

How to automatically generate protobuf client libraries from protobuf specification files

Some of our services communicate using protocol buffers (protobuf). Rather than manually maintaining client libraries to decode and interact with these services, we can generate them automatically using Buf in our CI pipelines.

This how-to guide describes how to generate message libraries automatically using features in the common pipeline.

Prerequisites

In order to configure protobuf client generation you will need the following:

  • A project configured to use the common pipeline.
  • A protobuf configuration file named buf.yaml in the root of your repository.
  • Static protobuf specification files in a directory named proto/ in the root of your repository.
  • (Optional) A buf.gen.yaml file defining how code should be generated for each supported language.
  • For Python packaging:
    • A directory containing static packaging files (e.g. pyproject.toml) at the path defined by PROTOBUF_GENERATOR_PYTHON_STATIC_FILES_DIR (default: generator-static/python).
    • These files must include a valid pyproject.toml with PEP 517 configuration compatible with the build tool.

Deliverables

By enabling protobuf client generation you will get:

  • Protobuf client libraries built for one or more languages. Currently, only Python is supported, but more languages may be added in the future.
  • Distribution packages for the libraries. For example, wheels are built for the Python client libraries.
  • For commits on the default branch, protobuf client libraries published in the GitLab package registry.

Basic usage

Add a file named buf.yaml to the root of your repository with the protobuf configuration. This will trigger the protobuf client generator in the CI pipeline. No additional CI configuration is needed.

Required: Add static packaging files

The Python build process expects packaging metadata to be present, such as pyproject.toml, which defines the package name, version, dependencies, and build backend.

These files must be placed in the directory defined by the PROTOBUF_GENERATOR_PYTHON_STATIC_FILES_DIR variable (default: generator-static/python). They will be copied into the generated source directory before the packaging step.

Here’s a minimal example using Poetry:

[tool.poetry]
name = "example-protobuf-client"
version = "0.0.1"
description = ""
authors = [ ]

[[tool.poetry.packages]]
include = "./**/*.py"
to = "example_protobuf_client"

[tool.poetry.dependencies]
betterproto = "^1.2.5"

[build-system]
requires = [ "poetry-core" ]
build-backend = "poetry.core.masonry.api"

Warning

If no pyproject.toml or equivalent build metadata is present in the static files directory, Python package generation will fail.

Tip

If you're using release-it, it is recommended to keep the version in generator-static/python/pyproject.toml in sync with the root pyproject.toml.

For details on how to integrate with release-it and publish your generated packages as GitLab release assets, see the relevant section in the reference guide.

Summary

In this guide you learned how to generate and publish protobuf client libraries from protobuf specification files either stored in your project's repository.

Next steps

Read the full reference guide for further customisation, or explore testing generated protobuf clients with the auto-factory tool.