Skip to content

Terraform

This page discusses how we use terraform to configure our deployments.

Our standard terraform configurations assume that we have a product folder, product admin service account and product configuration Cloud Storage bucket created for us. See the Google Cloud projects and folders page for information on these.

A live example of a service configured using terraform is the API Gateway configuration (University member only access).

Logan

It can be a challenge making sure that everyone is running the same version of terraform and that they are running it as the correct Google Cloud IAM identity. This is particularly acute if we want to make use of a newer version of terraform for some products and not others.

We have developed an internal tool named logan which automates some of the common tasks and helps make sure that we're running the correct version of terraform.

Logan has three main responsibilities:

  • Make sure that the correct terraform workspace is being used (see below).
  • Decrypt any secrets required for deployment.
  • Run a specific version of terraform within a Docker container.

Logan is configured via a .logan.yaml file in the root of the configuration. It is designed to wrap the terraform command.

So, for example, to initialise the local terraform state:

$ logan terraform init

To create a new workspace:

$ logan terraform workspace new development

To use a specific terraform workspace:

$ logan --workspace production terraform apply

If no --workspace option is provided, logan defaults to using the development workspace.

Workspaces

We use terraform workspaces to allow us to use the same configuration to deploy different environments. This helps make sure that any differences between production and staging are explicit in the terraform configuration.

Generally there will be at least three workspaces for each product:

  • production represents the "live" service.
  • staging represents the "test" service. This should be as close to production as possible.
  • development represents a deployment which may be broken due to development/testing or be destroyed and re-created freely.

Occasionally if an Engineer is working on a feature which may take several iterations to land, they'll create a workspace named after their CRSid which contains their own deployment of the product which they can improve with extreme prejudice.

Usually the terraform configuration will be identical between workspaces but we may make use of the terraform.workspace variable to specialise configuration for each workspace.

State

A product-wide configuration Cloud Storage bucket is created for each product. We use this bucket to store the terraform state:

terraform {
  backend "gcs" {
    bucket = "my-product-config-abc1234"
    prefix = "terraform/my-product"
  }
}

This is configured within our boilerplate in backend.tf.

Product Configuration

In our terraform we specify the location of the configuration bucket and the location of the configuration JSON document within the bucket:

locals {
  config_bucket = "my-product-config-abc1234"
  config_path   = "config.json"
}

This is used by config.tf to load further configuration which is made available in a variety of locals. For example:

locals {
  # Project id of product-specific meta project.
  product_meta_project = local.gcp_config.product_meta_project

  # Name of DNS zone in product-specific meta project
  product_dns_zone_name = local.gcp_config.product_dns_zone_name

  # Parent GCP folder to create product folders within.
  product_folder = "folders/${local.gcp_config.product_folder}"

  # Billing account to associate with product projects.
  billing_account = local.gcp_config.billing_account
}

See locals.tf for the full list.

Providers

In providers.tf we configure a variety of providers which are used by the subsequent config:

  • The google and google-beta providers are configured to use the project admin service account which is discussed below.
  • The google.product_admin and google-beta.product_admin providers are configured to use the product admin service account. These should only be used to configure product-folder level resources such as the Google project itself.
  • The google.monitoring and google-beta.monitoring providers are configured with credentials which allow Cloud Monitoring resources to be created wihtin the Cloud Monitoring workspace associated with the product. See the dedicated monitoring and alerting page page for information on this provider.

Project creation

We use a custom terraform module to create an environment-specific project within the product folder. This makes use of the google.product_admin provider to configure the project.

Within our boilerplate, this is configured in main.tf.

The project id is generated at random. The name and id both contain the terraform workspace so it is easy at a glance to determine which project corresponds to production, staging, etc.

The additional_services local in locals.tf lists the Google APIs which should be enabled for the project. The list of APIs required will depend on which Google services the project is using. Generally we have an "add as needed" approach to this list. If a service isn't enabled, the terraform provider gives us a helpful error message saying which API needs enabling and we add it to additional_services.

Our Google Cloud project module also creates a project admin service account. Unlike the product admin service account, the project admin only has owner rights over the project specific to the current environment. As such it is far safer to use the project admin to configure resources since it is unable to affect things in different environments; the "development" project admin service account has no rights over resources in the "production" project.

As it is generally safer to use the project admin service account to configure resources, the default google and google-beta providers use the project admin service account.

The environment-specific project id is available in the project local. It is also the default project for the google and google-beta providers and so often one does not need to explicitly provide the project when configuring resources.

Project-specific configuration

Just as it is useful to have a Cloud Storage bucket configured for storing product-wide configuration it is occasionally useful to have a per-project bucket which can store configuration specific to the current environment. This is created in main.tf:

# A bucket used to store general configuration of resources.
resource "random_id" "configuration_bucket_name" {
  byte_length = 4
  prefix      = "${local.slug}-config-${terraform.workspace}-"
}

resource "google_storage_bucket" "configuration" {
  name     = random_id.configuration_bucket_name.hex
  location = local.region

  versioning {
    enabled = true
  }
}

We do not have any fixed use for this bucket and generally we find that uses suggest themselves on a per-product basis. For example, it can be useful for putting large, non-secret configuration files in.

Summary

In summary,

  • We use logan to make sure that we're always running the correct version of terraform and to decrypt any secrets required for deployment.
  • We use one terraform workspace per environment. These are usually named production, staging and development.
  • Terraform state is stored in a configuration Cloud Storage bucket within the meta project.
  • The terraform configuration fetches some configuration dynamically at runtime from the configuration JSON document provided to us by gcp-infra.
  • We create one Google project per environment.
  • The default terraform provider uses a service account which only has rights over the environment-specific project, not the entire product folder.