Skip to content

How to enable backups of a GCS bucket to AWS S3

This guide describes how to enable or disable the automatic backup synchronisation serice for a Google Cloud Storage (GCS) bucket.

Overview

The Data Backup Service provides an automated way to enroll a GCS bucket for regular backups to an AWS S3 bucket. This is useful for disaster recovery purposes, ensuring that critical data stored in GCS is also available in a different cloud provider.

Enabling the bucket backup service is as simple as adding a label to the GCS bucket you wish to back up.

Disabling the backup service is equally straightforward, involving the removal of the label from the GCS bucket. This doesn't remove the existing backups in S3, but it does stop any future backups from being created.

Prerequisites

  • Your GCP product folder / project(s) are under the following folders in GCP:
    • DevOps
  • You have the necessary permissions to add or remove labels on GCS buckets you wish to back up, preferably automating the label management using Terraform. The role required for this is:
    • roles/storage.admin
    • Use your gcloudadmin account to perform these operations which will require deploy or admin permissions.

Enable backup of a GCS bucket to AWS S3

Add the following label to the GCS bucket you wish to back up:

  • ucam-devops-backup-destination

Set the value of the label to:

  • production – to enable backups to the production S3 bucket.

Disable backup of a GCS bucket to AWS S3

To disable backups for a GCS bucket, simply remove the ucam-devops-backup-destination label from the bucket. This will stop any future backups from being created, but it will not delete any existing backups in S3.

Configure backup retention

By default, once enabled, backups are kept in S3 indefinitely — the backup process never deletes objects itself, so backups accumulate forever unless you opt in to automatic expiry.

To automatically expire old backups, add a second label to the GCS bucket:

  • ucam-devops-backup-retention-days

Set the value of the label to the number of days after which backups should expire, e.g.:

  • "90" – expire backup objects 90 days after they were copied to S3.

Warning

The label value must be a positive whole number provided as a string (e.g. "90", not 90). A non-numeric value will cause the Terraform plan for the backup infrastructure to fail.

Tip

Set the value to be at least as long as the source bucket's own retention or lifecycle deletion age. The S3 retention clock starts when DataSync syncs the object, not when it was created in GCS, so using the exact same number as the source can still cause the backup to be deleted before the corresponding source object is. Add a buffer above the source's own retention/lifecycle age to be safe.

To go back to indefinite retention, remove the ucam-devops-backup-retention-days label from the bucket.

Info

Label changes aren't picked up immediately — GCP Cloud Asset Inventory can take a few minutes to index a label change, so a newly (re)labelled bucket may not appear in the next Terraform plan straight away.

See also