Terraform¶

This page discusses how we use terraform to configure our deployments.

Our standard terraform configurations assume that we have a product folder, meta project, multiple environment projects, and a product configuration Cloud Storage bucket created for us. See the Google Cloud projects and folders page for information on these.

A live example of a service configured using terraform is the API Gateway configuration (University member only access).

Logan¶

It can be a challenge making sure that everyone is running the same version of terraform and that they are running it as the correct Google Cloud IAM identity. This is particularly acute if we want to make use of a newer version of terraform for some products and not others.

We have developed an internal tool named logan which automates some of the common tasks and helps make sure that we're running the correct version of terraform.

Logan has three main responsibilities:

Make sure that the correct terraform workspace is being used (see below).
Decrypt any secrets required for deployment.
Run a specific version of terraform within a Docker container.

Logan is configured via a .logan.yaml file in the root of the configuration. It is designed to wrap the terraform command.

So, for example, to initialise the local terraform state:

logan --workspace default terraform init

To create a new workspace:

logan --workspace default terraform workspace new development

To use a specific terraform workspace:

logan --workspace production terraform apply

If no --workspace option is provided, logan defaults to using the development workspace.

Workspaces¶

We use terraform workspaces to allow us to use the same configuration to deploy different environments. This helps make sure that any differences between production and staging are explicit in the terraform configuration.

Generally there will be at least three workspaces for each product:

production represents the "live" service.
staging represents the "test" service. This should be as close to production as possible.
development represents a deployment which may be broken due to development/testing or be destroyed and re-created freely.

Occasionally if an Engineer is working on a feature which may take several iterations to land, they'll create a workspace named after their CRSid which contains their own deployment of the product which they can improve with extreme prejudice.

Usually the terraform configuration will be identical between workspaces but we may make use of the terraform.workspace variable to specialise configuration for each workspace.

State¶

A product-wide configuration Cloud Storage bucket is created for each product. We use this bucket to store the terraform state:

terraform {
  backend "gcs" {
    bucket = "my-product-config-abc1234"
    prefix = "terraform/my-product"
  }
}

This is configured within our boilerplate in backend.tf.

Product Configuration¶

In our terraform we specify the location of the configuration bucket and the location of the configuration JSON document within the bucket:

locals {
  config_bucket = "my-product-config-abc1234"
  config_path   = "config_v1.json"
}

This is used by config.tf to load further configuration which is made available in a variety of locals. For example:

locals {
  domain_verification = local.workspace_config.domain_verification

  # True if this is a "production-like" workspace.
  is_production = terraform.workspace == "production"

  # Project id of product-specific meta project.
  product_meta_project = local.gcp_config.product_meta_project

  # Project id for workspace-specific project.
  project = local.workspace_config.project_id
}

See locals.tf for the full list.

Providers¶

In providers.tf we configure the following providers which are used by the subsequent config.

The google.impersonation provider runs as the user who executed terraform. This provider is only used to generate short-lived access tokens to impersonate other, more privileged service accounts.
The google and google-beta providers are configured to impersonate the terraform-deploy service account for the specific environment. These providers are used to do most of the environment specific resource creation.
The gitlab provider is configured to authenticate using an access token stored in a secret in the meta project named gitlab-access-token. This is used to configure CI/CD variables on the infrastructure project.
The docker provider is also configured to authenticate using the gitlab-access-token secret. This provider is used to copy webapp images from Gitlab to the Google Container Registry.

Project-specific configuration¶

Just as it is useful to have a Cloud Storage bucket configured for storing product-wide configuration it is occasionally useful to have a per-project bucket which can store configuration specific to the current environment. This is created by the gcp-product-factory.

We do not have any fixed use for this bucket and generally we find that uses suggest themselves on a per-product basis. For example, it can be useful for putting large, non-secret configuration files in.

Summary¶

In summary,

We use logan to make sure that we're always running the correct version of terraform and to decrypt any secrets required for deployment.
We use one terraform workspace per environment. These are usually named production, staging and development.
Terraform state is stored in a configuration Cloud Storage bucket within the meta project.
The terraform configuration fetches some configuration dynamically at runtime from the configuration JSON document provided to us by gcp-project-factory.
We create one Google project per environment via the gcp-project-factory.
The default terraform provider impersonates a service account which only has rights over the environment-specific project, not the entire product folder.