How to add AWS to your product

We usually make use of Google Cloud services for products but some products or features we need are only available in Amazon Web Services (AWS). This guide covers how to add AWS to your product so that you may start using terraform to configure AWS resources.

We make use of the existing permission model and associated infrastructure which we use with Google Cloud and so your product will also need to have a set of Google Cloud projects configured even if you only intend to use AWS.

If you're interested in the rationale and background for how we integrate with AWS, there is a dedicated explainer guide.

Differences between AWS and Google Cloud

AWS and Google have many similar concepts but often use different names for them. This table summarises the different concepts you'll need to understand in order to follow this guide:

Google	AWS	Description
Project	Account	Collection of all the resources corresponding to a single environment such as "production" or "staging"
Folder	Organizational Unit (OU)	Collection of all of the projects/accounts for a single product
User	IAM User	An identity corresponding to a "real person" using password authentication
Service account	IAM Role	An identity corresponding to a machine or process using other means of authentication
IAM Policy	IAM Policy	A document granting a set of permissions to one or more identities

In Google Cloud the namespace for Users is global; there can be only one spqr2@cam.ac.uk user across all projects. In AWS the namespace for IAM Users is per-account; each account can have its own spqr2@cam.ac.uk user which is independent of any others.

In addition, AWS has the concept of a "root user" which has no real analogue in Google. There is exactly one root user per AWS account and the root user namespace is global.

Ordinary IAM users in AWS sign in via an account-specific sign in page whereas root users sign in via a shared sign in page for all accounts.

Generally you only ever sign in as the root user once in order to configure the account as described in this guide and then never again. Subsequent sign ins are done by impersonating a per-AWS account IAM Role.

Requirements and deliverables

Before you start you will need:

• your product's name,
• an existing terraform deployment using Google Cloud based on our template,
• access to the Google Cloud console for your product,
• a billing code which will usually match the one used for Google Cloud, and
• a 1Password vault to store secrets.

At the end of this process you will get:

• one AWS account for each environment which mirrors the one Google project for each environment,
• an AWS IAM Role corresponding to the "terraform deploy" Google service account,
• the ability for AWS resources to be managed in terraform alongside Google ones,
• an AWS IAM Role corresponding to an "admin" AWS console user,
• a mechanism whereby users which can access the Google Cloud console with Editor rights will also be able to access the AWS console.

Create AWS accounts for your product

We use a separate Google project for each of the environments we deploy to such as "production" or "staging". Similarly, we will create one AWS account for each environment.

This process is covered in a dedicated guide. If you do not have access to the appropriate AWS organisation management account credentials, you can ask teamCloud to create the accounts for you.

At the end of the process you should have an AWS account for each environment and "root user" credentials for each account. AWS provide documentation on how to sign in using these credentials.

Remember to enable Multi-Factor Authentication (MFA)

1Password has support for TOTP MFA. If not already configured, please follow the AWS documentation to enable MFA for the root user.

Create an IAM Role for the terraform deployment service account

In order to enable seamless integration between AWS and terraform, we need to create an IAM Role in AWS which corresponds to the Google Service Account used by terraform to manage resources.

For each Google Cloud project:

1. Access the service accounts page in the Google console.
2. Click on the service account named terraform-deploy@....
3. Make a note of the numeric "Unique ID" for the service account.

Service account details page in the Google Cloud console with "Unique ID" highlighted.

Warning

Make sure you don't mix up which service account id corresponds with which environment. From now on we'll only be using the numeric ids and you'll make sure you keep track of which id corresponds to which environment.

For each AWS Account:

1. Sign in to the AWS Console with the root user credentials.
2. Open the IAM page by searching "IAM" at the top of the page.
3. Click Roles in the left-hand menu.
4. Click Create role.
5. Select Custom trust policy.

6. Enter the following policy replacing the numeric Google service account and numeric AWS account ids as appropriate. MAKE SURE THE SERVICE ACCOUNT ID IS FOR THE CORRECT ENVIRONMENT. The numeric AWS account id can be found by clicking the account dropdown in the top-right of the AWS console and using the "copy" button next to the account id.

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Principal": {
                   "Federated": "accounts.google.com"
               },
               "Action": "sts:AssumeRoleWithWebIdentity",
               "Condition": {
                   "StringEquals": {
                       "accounts.google.com:sub": "{REPLACE WITH NUMERIC SERVICE ACCOUNT ID}",
                       "accounts.google.com:oaud": "arn:aws:iam::{REPLACE WITH NUMERIC AWS ACCOUNT ID}:role/TerraformDeploy"
                   }
               }
           }
       ]
   }
   

7. Click Next.

8. Do not select any permissions policies. Instead, click Next again.
9. For Role name enter "TerraformDeploy".
10. For Description enter "Role assumed by terraform when deploying".
11. Click Create role.
12. In the list of roles, click the newly created TerraformDeploy role's name.
13. Click Add permissions > Create inline policy.
14. Click JSON.

15. Replace the policy with the following:

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Action": [
                    "iam:*"
                ],
                "Resource": "*"
            }
        ]
    }
    

    YOU WILL NEED TO UPDATE THIS POLICY OVER TIME AND THIS DEFAULT POLICY MAY BE OVERLY BROAD. See the explainer guide section on the TerraformDeploy role for more information. It is strongly recommended that you include an up-to-date copy of the ServiceDeployer policy in your infrastructure project.

16. Click Next.

17. For Policy name enter "ServiceDeployer".
18. Click Create policy.

You may need to add extra permissions to the ServiceDeployer policy at a later date

Depending on which resources your terraform deployment will need to manage, you may need to add additional permissions. For example, if your terraform deployment manages AWS Simple Email Service (SES) resources you will need to add ses:* to the Action list in the inline policy.

Generally it's best to add these as needed. To do so, find the "TerraformDeploy" role in the list of roles, click its name and follow the steps above to edit the inline policy as needed.

Advanced users may wish to be even finer grained in permissions granted to the "TerraformDeploy" role or make use of separate permission policies rather than inline ones. The policy above is designed to maximise convenience but might be considered over-broad in some cases.

This should be the last bit of manual configuration needed in the AWS console.

Add AWS to an existing terraform deployment

We have configured an AWS account for each environment and added a "TerraformDeploy" IAM role within that AWS account configured with a trust relationship to the terraform deploy service account used by our terraform deployments.

Now we need to add the plumbing to terraform to make use of this. The guide below assumes you are using our standard deployment template.

A real-world example

The postbox infrastructure project (DevOps only link) provides a real-world example of the configuration described below.

In our example below we we assume a "staging" environment in AWS account "111111222222", a "production" environment in AWS account "333333444444" and a "development" environment in AWS account "555555666666". Replace the account ids as appropriate.

Configure the AWS provider

In locals.tf, add the following local definitions, replacing the numeric AWS account ids as necessary:

locals.tf

# These locals define common configuration parameters for the deployment.
locals {
  # ... other locals ...

  aws_workspace_config = lookup({
    staging = {
      account_id = "111111222222"
    }
    production = {
      account_id = "333333444444"
    }
    development = {
      account_id = "555555666666"
    }
  }, terraform.workspace, {
    account_id = "THIS_WORKSPACE_DOES_NOT_HAVE_AN_ASSOCIATED_AWS_ACCOUNT"
  })

  # Region for AWS resources.
  aws_region = "eu-west-2"
  # Account id for AWS account containing all resources.
  aws_account_id = local.aws_workspace_config.account_id
  # Account ids for accounts we are allowed to manage resources in.
  aws_allowed_account_ids = [local.aws_account_id]
  # Role ARN for the role to assume in AWS when deploying.
  aws_deploy_role_arn = "arn:aws:iam::${local.aws_account_id}:role/TerraformDeploy"
}

In versions.tf, add the AWS provider making sure to use the latest version:

versions.tf

# versions.tf specifies minimum versions for providers and terraform.

terraform {
  # ... other config ...

  required_providers {
    # ... other providers ...

    aws = {
      source  = "hashicorp/aws"
      version = "~> REPLACE_WITH_LATEST_PROVIDER_VERSION"
    }
  }
}

In providers.tf, configure the AWS provider:

providers.tf

# ... other config ...

# Generate an id token for the terraform-deploy service account for the current workspace used to
# authenticate to AWS.
data "google_service_account_id_token" "aws_terraform_deploy" {
  target_service_account = local.workspace_config.terraform_sa_email
  target_audience        = local.aws_deploy_role_arn

  provider = google.impersonation
}

# AWS is authenticated using a role corresponding to the terraform deploy service account.
provider "aws" {
  region              = local.aws_region
  allowed_account_ids = local.aws_allowed_account_ids

  assume_role_with_web_identity {
    role_arn           = local.aws_deploy_role_arn
    web_identity_token = data.google_service_account_id_token.aws_terraform_deploy.id_token
  }

  default_tags {
    tags = {
      Project     = local.gcp_config.product_display_name
      Environment = terraform.workspace
    }
  }
}

Test the changes via a terraform apply. It should do nothing but also should not raise any errors.

Enable AWS console access for admin users

The configuration below enables users which can sign in to the Google Cloud console as editors to sign in to the AWS console as admins.

In locals.tf add the following locals:

locals.tf

# These locals define common configuration parameters for the deployment.
locals {
  # ... other locals ...

  # Cloud IAM principals who can, by means of impersonating the AWS console admin service account,
  # access the AWS console. For consistency we align these with the IAM principals given full
  # read-write access in the Google Cloud console.
  aws_admin_iam_principals = local.editor_iam_principals
}

# These locals are derived from resources, data sources or other locals.
locals {
  # ... other locals ...

  # IAM principals associated with various roles.
  role_iam_principals = lookup(local.gcp_config, "iam_principals", {})

  # "Editor" IAM principals. Editors have full read-write access to the Google Cloud Console, for
  # example.
  editor_iam_principals = lookup(local.role_iam_principals, "editor", [])
}

Create a file named aws-admin.tf with the following content:

aws-admin.tf

# aws-console.tf describes resources related to AWS console acces.

# A service account which is able to assume a role in AWS allowing console sign in and full admin
# rights.
resource "google_service_account" "aws_admin" {
  account_id   = "aws-admin"
  display_name = "AWS Admin"
  description  = <<-EOI
    Google-side representation of an AWS role which has console sign in rights and full admin
    permissions.
  EOI
}

# Configure who can impersonate the console service account. These people effectively have the
# ability to sign in to the AWS console as an administrator.
resource "google_service_account_iam_member" "aws_admin" {
  for_each = toset(local.aws_admin_iam_principals)

  service_account_id = google_service_account.aws_admin.name
  role               = "roles/iam.serviceAccountTokenCreator"
  member             = each.value
}

resource "aws_iam_role" "admin" {
  name = "Admin"

  assume_role_policy = jsonencode({
    "Version" : "2012-10-17",
    "Statement" : [
      {
        "Effect" : "Allow",
        "Principal" : {
          "Federated" : "accounts.google.com"
        },
        "Action" : "sts:AssumeRoleWithWebIdentity",
        "Condition" : {
          "StringEquals" : {
            "accounts.google.com:oaud" : "arn:aws:iam::${local.aws_account_id}:role/Admin",
            "accounts.google.com:sub" : google_service_account.aws_admin.unique_id,
          }
        }
      }
    ]
  })
}

# An AWS role which can sign in to the console and has full admin rights. We tell trivy to ignore
# that this is a super-powered IAM role because that is the point.
# trivy:ignore:AVD-AWS-0057
resource "aws_iam_role_policy" "console_admin" {
  name = "ConsoleAdmin"
  role = aws_iam_role.admin.id

  policy = jsonencode({
    "Version" : "2012-10-17",
    "Statement" : [
      {
        "Effect" : "Allow",
        "Action" : "*",
        "Resource" : "*",
      }
    ]
  })
}

Gather the Google project names and AWS account numeric ids for each environment. In our example, we assume that the Google project names are punt-booker-test-456def, punt-booker-prod-789abc and punt-booker-devel-123abc.

Create the file .aws-helper.yaml in the project root with the following content, replacing the AWS account ids and Google project names as necessary:

.aws-helper.yaml

environments:
  staging:
    google_service_account_email: aws-admin@punt-booker-test-456def.iam.gserviceaccount.com
    google_quota_project_id: punt-booker-test-456def
    aws_role_arn: arn:aws:iam::111111222222:role/Admin
  production:
    google_service_account_email: aws-admin@punt-booker-prod-789abc.iam.gserviceaccount.com
    google_quota_project_id: punt-booker-prod-789abc
    aws_role_arn: arn:aws:iam::333333444444:role/Admin
  development:
    google_service_account_email: aws-admin@punt-booker-devel-123abc.iam.gserviceaccount.com
    google_quota_project_id: punt-booker-devel-123abc
    aws_role_arn: arn:aws:iam::555555666666:role/Admin

Run terraform apply as per usual and verify that terraform creates an admin AWS IAM Role and Google service account.

Follow the steps in the how to access the AWS console guide to test accessing the console in each environment.

Summary

In this guide we covered:

• how to create AWS accounts for each deployment environment for your product,
• how to allow terraform to manage resources within those accounts, and
• how to enable sign in to the AWS console for privileged users.

Next steps

• How to access the AWS console
• How to audit access to the AWS console
• How to use the aws CLI tool