Skip to content

How to start developing a new product

This guide describes how to bootstrap the Google, GitLab and GitLab CI infrastructure for a new product. It covers the common case of there also being only a single "product" in the Google, GitLab and GitLab CI senses.

Other sources of information

This guide is intentionally brief. If you have questions when reading, see the following resources for additional context:

Decide on the development team

Look at the team_data.json file and identify which development team should be assigned to the product. Ideally there should already be a team which matches your needs.

What does "admin", "view" and "deploy" mean in this file?

The team_data.json file defines development teams. Each team has "admin" users with full access to Google Cloud resources, "view" users who can see Google Cloud resources in the Google Cloud console and "deploy" users who can run terraform locally to deploy products but not necessarily see things in the Google Cloud console.

If no existing team is appropriate, open an issue in the team-data project to add your team. Tag the issue teamCloud. If possible, also provide a Merge Request which adds the data you require.

Use "gcloudadmin" accounts

We're moving over to having users with rights over projects in Google Cloud use dedicated accounts with names of the form {crsid}@gcloudadmin.g.apps.cam.ac.uk. Adding new "gcloudadmin" accounts to team_data.json will cause teamCloud to create those accounts and share credentials securely with the corresponding users.

For our example, we'll assume there is a team called business-apps defined which is managing this service.

Create the Google Cloud projects

To create Google Cloud projects for your product you will need:

  • a monthly budget, in Euros, used to set up billing alerts,
  • a cost centre code, for example VCMA ANCB,
  • a short human-readable display name, for example "Punt Leasing",
  • a short machine-friendly slug name, for example punt-leasing, and
  • the name of the team in team_data.json who will be managing the product.

The budget and cost code should have been provided for you when you were asked to start the product. If not, ask for them before proceeding. A typical budget alert for a small web application would be €150/month.

Open an issue in gcp-product-factory and tag the issue teamCloud.

If possible, also provide a Merge Request which adds the data you require. See the Merge Request adding the Regent House Ballots application as an example.

The Merge Request should add the tfvars file. For the punt leasing example, this should look like the following:

product-vars/punt-leasing/punt-leasing.tfvars
budget        = "..."  # TODO: fill in budget
cost_centre   = "..."  # TODO: fill in cost centre
team_name     = "..."  # TODO: fill in team name from team_data.json
display_name  = "Punt Leasing"
slug          = "punt-leasing"

parent_folder = "folders/433726114664"  # == "Information Services/DevOps"

workspace_names = [
  "development",
  "staging",
  "production"
]

view_users = [
  # These groups are created by gcp-workspace-management.
  "group:business-apps-deploy@gcloudadmin.g.apps.cam.ac.uk",
  "group:business-apps-view@gcloudadmin.g.apps.cam.ac.uk",
]

# ONLY add this if deploy users should be able to run terraform on their local machines. If you are
# running terraform in CI, this should not be necessary.
additional_deploy_users = [
  "group:business-apps-deploy@gcloudadmin.g.apps.cam.ac.uk",
]

teamCloud will create Google Cloud resources and provide you with a configuration bucket in Google Cloud. This contains metadata about your product in machine- and human-readable form.

You will need information from the Google Cloud configuration bucket for later steps.

Create the GitLab projects

To create GitLab projects for your product you will need:

  • a display name name for your product's group, for example "Punt Leasing",
  • a short machine-friendly slug name for your product's group, for example punt-leasing,
  • a list of projects you wish to create in your product group, and
  • the name of the "meta project" created in Google Cloud as provided in the Google Cloud configuration bucket.
How do I find the meta project name?

When teamCloud created your Google Cloud projects, they also created a HTML page with configuration data in human-readable form. The meta project can be found on that page.

We have an issue open to change GitLab project factory to take the location of the configuration bucket given to you by teamCloud directly so as to avoid this step.

Usually you'll want to create two projects to start with:

We are open by default and so the visibility of webapp can be public and should not be less than internal. As the infrastructure project can be externally sensitive, we recommend having the visibility of infrastructure be at most internal.

Open an issue in gitlab-project-factory tag the issue teamCloud.

If possible, also provide a Merge Request which adds the data you require. See the Merge Request adding the Regent House Ballots application as an example.

The Merge Request should add the tfvars file. For the punt leasing example, this should look like the following:

product-vars/punt-leasing/punt-leasing.tfvars
parent_group_id    = 5  # == "uis/devops"
group_name         = "punt-leasing"
group_display_name = "Punt Leasing"

product_meta_project = "punt-leasing-meta-abcdef01"  # <- provided by team::Cloud

projects = {
  "infrastructure" = {
    display_name     = "Cloud Infrastructure"
    description      = "Terraform configuration for punt leasing"
    visibility_level = "internal"
  }
  "webapp" = {
    display_name     = "Web Application"
    description      = "Django Web Application for punt leasing"
    visibility_level = "public"

    container_expiration_policy = {
      cadence           = "1d"
      enabled           = true
      keep_n            = 1
      older_than        = "7d"
      name_regex_delete = ".*"
      name_regex_keep   = "[0-9]+\\.[0-9]+\\.[0-9]+"
    }
  }
}
What is container_expiration_policy?

Our common CI pipeline will automatically build container images. If you are using release automation, images corresponding to released versions will be tagged with names of the form {x}.{y}.{z} and automatically pushed to Google Cloud for later deployment.

Since only images corresponding to releases will ordinarily be deployed, images built for branches or merge requests are only needed for tings like running tests or security scanning. The container expiration policy ensures that GitLab keeps container images corresponding to releases but is free to "garbage collect" other images to save storage space.

For an example of this in use, see the container registry for the Regent House ballots application which contains only images corresponding to released versions.

Once teamCloud have addressed your issue, you should have empty GitLab projects created and ready to go.

Create GitLab CI runners

All CI jobs run in GitLab need to run somewhere. We have a shared set of runners for DevOps projects but we are moving to per-product runners hosted in Google Cloud. These runners are needed for more advanced CI jobs such as those powering release automation.

To create CI runners you will need:

  • the name of the Google Cloud configuration bucket given to you when teamCloud created your Google projects, and
  • the numeric "group id" of the GitLab group containing all of your product projects which can be found on the group's page in GitLab.

Open an issue in gitlab-runner-infrastructure and tag the issue teamCloud.

If possible, also provide a Merge Request which adds the data you require. See the Merge Request adding the Regent House Ballots application as an example.

The Merge Request should add an entry to the product_runners.production map defined in locals.tf. For the punt leasing example, this should look like the following:

locals.tf
locals {
  # ...
  product_runners = {
    # ...
    production = {
      # ...
      "punt-leasing" = {
        config_bucket   = "punt-leasing-12345678"  # <- provided by team::Cloud
        gitlab_group_id = 987654 # <- from the product group's page in GitLab
      }
      # ...
    }
  }
  # ...
}

Once teamCloud have addressed your issue, you can create a new application.

Summary

In this guide you learned where configuration surrounding our development team membership, Google Cloud projects, GitLab projects and GitLab CI runners lives. You saw how to bootstrap a simple new product in all three places ready for development to start.

Next steps