GCP Projects and Folders¶

This page describes how we make use of Google Cloud projects and folders to structure our deployments.

Centrally managed infrastructure¶

We use the gcp-infra and gcp-product-factory projects (University members only) to configure and provision Google Cloud for the wider University. This repository contains terraform configuration for the scheme described below.

We use a series of Google Folders within Google Cloud to reflect a rough institution/group hierarchy similar to the hierarchy which has developed on the Developer Hub.

At the bottom of this hierarchy are products. At the Google Cloud management level we care only about products and each product is given its own product folder.

For example, the "Information Services" folder contains sub-folders for groups within the institution and each group has a sub-folder for each product:

Each product is associated with a finance cost code and a team name. These are added to projects within the product folder as a label. These labels are intended to be used to automate finance reporting.

In addition to the labels, products are allocated a budget and a budget alert is configured to notify us when a product is likely to go over budget for a month.

Individual user accounts can be designated as "product owners". These accounts will be given rights on projects within the product folder. More information can be found on the permissions and roles page.

The product folder and projects¶

When a new product is provisioned in Google Cloud, we take the budget, finance code and list of owners and add them to the gcp-product-factory project. This terraform will create a product folder, a meta project, and multiple environment projects for the product.

The product folder contains all Google Cloud projects which relate to a given product. We generally use completely separate Google Cloud projects for each of our production, test and development environments. Occasionally we'll provision additional projects within the product folder to develop larger features which are likely to require multiple iterations before they can be merged into the product as a whole.

The meta project is a project within the product folder which contains resources which are specific to the product but which are shared by all environments.

For example, here are the projects within the "Applicant Portal" product folder:

You can see how the meta project is tagged with the team name and cost-centre code. A running total of costs can also be seen.

Examples of resources which belong within the meta project are:

A Cloud DNS zone for the product. Usually of the form [product].gcp.uis.cam.ac.uk. Projects within the product folder will generally contain environment-specific zones such as devel.[product].gcp.uis.cam.ac.uk.
Product-wide secrets such as GitLab access tokens or credentials for external services which are shared by all environments.
A Cloud Storage bucket which hosts the terraform state.

Admin Accounts¶

The personal accounts of each product owner are added to the product folder with a specific set of roles/permissions to allow them to examine and modify configuration within the Google Cloud console. More information can be found on the permissions and roles page.

In addition a service account is added to the meta project which is also given specific roles/permissions on the product folder. This service account is called the product admin service account and is only used by the gcp-product-factory project.

Configuration¶

The gcp-product-factory configuration will create a generic "configuration" Cloud Storage bucket in the meta project which can be used for terraform state. It will also generate two objects in this bucket: a human-readable HTML configuration "cheat sheet" and a machine-readable JSON document with the same content. The HTML document is simply easier to read while the JSON document is ingested by the gcp-deploy-boilerplate.

An example of the configuration cheat sheet is as below:

Summary¶

In summary:

We manage Google Cloud as a collection of products and each product has a dedicated product folder and meta project created for it in Google Cloud.
Within the meta project we create:
- A product admin service account which has rights to create projects within the product folder and to associate them with the University billing account.
- A general configuration Cloud Storage bucket which can be used to store product-wide configuration.
- A cheat sheet and configuration JSON document providing human-readable and machine-readable information on the Google Cloud resources provided for the product.
- A managed DNS zone of the form [product].gcp.uis.cam.ac.uk which is pre-wired in to the University DNS.
Each environment, such as "production", "staging" and "development" should be a separate Google Cloud project created within the product folder.
The individual accounts for each product owner are granted permissions on the product folder so they can manage all environments.