Backups and Restoring Data¶
This page documents how our services hosted in GCP are backed up, and how to restore data from a backup.
Backups¶
Our services are backed up in two ways, through the GCP native backup configuration and through a scheduled job which creates exports of the database and stores them in GCP buckets in a separate project.
Cloud SQL Backups¶
Google's documentation describes their managed backup service for Cloud SQL. For all of our services using Cloud SQL, the SQL instances should have automatic backups enabled, and they should retain at least 7 backups.
Scheduled Database Exports¶
The infrastructure project Data Backup deploys a GCP scheduled job to export data from our databases and store them in GCP buckets.
The backups can be viewed in the Data Backup - Production project in GCP.
Restoring Data¶
If a Cloud SQL Backup is available for the time frame you want to restore to, that backup should be used. Cloud SQL backups are kept for 7 days, if the time frame you want to restore to is longer than 7 days a Database Export must be used instead.
Restoring from Cloud SQL Backups¶
To restore from the native Cloud SQL backups follow the process outlined in the GCP documentation.
Note that the SQL instance is restarted as part of the restore process, so some service instability is expected when the restore operation is carried out. It may be necessary to notify users of this instability if the service being restored is undergoing active use.
Google's documentation contains additional information about the restore process.
Restoring from Database Exports¶
First the correct database export should be found in the Data Backup - Production
project, and the gs:// URL to the SQL export should be noted.
Within the backup bucket, the backups are stored in the following hierarchy:
<gcp-project-id><sql-instance-id><database-name><YYYYMMDD-hhmmss>.sql.gz
Backups are named with the date that the backup was created, and should be found in a folder matching the project, SQL instance and database's name.
The database exports are expecting to run from a blank slate, so before
importing the data the existing databases should be deleted from the SQL
instance. Navigate to the SQL instance in the GCP console, select
'Databases' from the navigation bar and delete all the databases in the
instance. Alternatively, the gcloud utility can be used to delete databases
using the command:
gcloud sql databases delete <DATABASE> --instance=<INSTANCE>
Once this is done, recreate each deleted database (using the same name as it had previously) in the instance:
gcloud sql databases create <DATABASE> --instance=<INSTANCE>
You can then restore data from a database export following the process outlined in the GCP documentation.
Finally, many deployments have a post-deployment configuration which may need to be re-run. Instructions for this will be found in the readme for the deployment repository of the service. In most cases, the specific instructions required will depend on the service being restored, but will typically be the configuration of database users and database access rights.
Recovery Objectives (RTO and RPO)¶
This page defines how we express recovery objectives for the services we run, and records the agreed objectives for each service. It exists so that, for any service, it is clear how much data we are prepared to lose in a disruption and how long we expect recovery to take.
Definitions¶
Recovery Point Objective (RPO) — the maximum amount of data, expressed as a length of time, that we are prepared to lose in a disruption. It is determined by how often a recoverable copy of the data is created (through backups, exports, replication or built-in versioning). An RPO of 24 hours means that, in the worst case, up to 24 hours of changes made since the last recovery point could be lost.
Recovery Time Objective (RTO) — the maximum acceptable time to restore a service to a working state after a disruption, measured from the point at which recovery is invoked to the point at which the service is usable again. It is determined by the restore mechanism and the volume of data to be recovered.
Key principle: objectives depend on the scenario¶
A single service can have more than one RTO/RPO pair. The objective that applies depends on the failure scenario and the recovery mechanism available for it. We distinguish two broad classes of recovery:
- Operational (business-as-usual) recovery — recovery from everyday events such as an accidental deletion, a bad change or user error. This is often handled by a built-in product feature and may be self-service.
- Disaster recovery — recovery from the loss of an instance, platform, cloud zone or data store. This is handled by our backups and exports.
The two can produce very different figures for the same service. Operational recovery figures must not be quoted as the service's disaster-recovery objectives.
How we derive the objectives¶
- RPO is set by the interval between recoverable copies for the scenario in question. Continuous, built-in versioning gives an RPO close to zero; a daily backup gives an RPO of up to 24 hours.
- RTO is set by the time to detect, decide and execute the restore for that scenario. It is usually dominated by the restore mechanism and the amount of data to be recovered, not by detection time.
Recording and approving objectives¶
- Each service records its recovery objectives on its own service page (or in its Non-Functional Requirements).
- The Service Owner approves the objectives and is accountable for them. They are reviewed at least annually and on any major change to the service or its architecture.
- Where the capability we can currently achieve does not meet a required objective, this is recorded as a gap with a remediation plan or a formal exception, in line with our vulnerability and exception-handling process.
Recorded objectives by service¶
The scenario column is important: read each row as "for this service, in this scenario, the objectives are…".
| Service | Scenario | RPO | RTO | Recovery mechanism | Notes |
|---|---|---|---|---|---|
| Database instances (Cloud SQL) | Data store loss / corruption (DR) | 24 hours | 30 minutes | Restore from daily backup / export | Baseline from the once-daily backup. |
| GitLab | Operational — user error, accidental change or deletion | ~0 (immediate) | ~0 (immediate) | Built-in Git / GitLab versioning | Self-service; no team intervention required. |
| GitLab | Disaster — loss of the GitLab instance / platform | Up to 24 hours | ~15 hours | Restore from daily full backup | Full-instance restore is time-consuming. |
Worked examples¶
Database instances¶
Our database instances are backed up once per day.
- RPO — 24 hours. Because a recoverable copy is created once daily, the worst case is the loss of up to 24 hours of changes made since the last backup.
- RTO — 30 minutes. Restoring an instance from its backup returns the service to a working state within roughly half an hour.
Point-in-time recovery
Where Cloud SQL point-in-time recovery (PITR) is enabled, the effective RPO is lower than 24 hours — down to minutes within the log-retention window (7 days by default) — because the database can be recovered to a chosen moment rather than only to the last daily backup. The 24-hour figure above is the guaranteed baseline provided by the daily backup alone. Remove this note if PITR is not in scope for the instances you are describing.
GitLab¶
GitLab is the clearest example of a service with two distinct recovery profiles, and it is worth setting out in full to avoid confusion.
Operational recovery (the common case). In normal use, GitLab's built-in version control is the recovery mechanism. Git retains the full history of every change, and previous versions of code and configuration can be restored by the user directly.
- RPO — effectively zero. Every committed change is already a recoverable point; nothing is lost.
- RTO — effectively zero. Recovery is immediate and self-service. A user can revert to an earlier version themselves, with no team intervention.
This near-instant recovery is a property of the technology itself, not of our backup arrangements.
Disaster recovery (loss of the instance). The picture is different if the GitLab instance or platform itself is lost. Here we rely on the full backup we take once per day.
- RPO — up to 24 hours. As with any once-daily backup, up to 24 hours of changes made since the last backup could be lost.
- RTO — approximately 15 hours. Restoring a full backup of the GitLab instance is a large operation and takes around 15 hours to complete.
Don't conflate the two
GitLab's instant, self-service recovery applies only to ordinary version-control operations. It does not protect against the loss of the instance. For that scenario the objectives are a 24-hour RPO and an approximately 15-hour RTO, and these are the figures that should be used for business-continuity and disaster-recovery planning.