DevSecOps standards for Ongoing Development¶

Introduction¶

This document outlines a strategy for extending the principle of DevSecOps to ongoing software development. DevSecOps refers to a software development philosophy which has two principal tenets:

Good security practice applies to all parts of software development, including ongoing “business as usual” development, rather than being something solely to be layered on after the fact at deployment time or before the fact at design time.
Security practices which are amenable to automation should be automated in order to increase developer productivity and ensure consistency of approach.

The holistic approach to security embodied in DevSecOps means that each stage of development should have processes and practices added to contribute to the overall security of a system.

Applicability¶

This standard should be implemented for all services that the university develops, whether internally, or through third parties. It is particularly relevant to services that handle sensitive data or support important processes, where a security breach could have significant negative impacts.

Exceptions can be submitted following the exemption process within the Systems Management Policy informing and seeking approval of the Tech Leads Forum first.

Scope¶

This document focuses specifically on DevSecOps practices applicable to ongoing software development that can be automated or semi-automated. It doesn't cover:

Security-first architectural design decisions (e.g., zero-trust communication).
Practical security measures for deployments (e.g., principle of least privilege, strong audit records).
Non-automated DevSecOps practices (e.g., ongoing engineer training).

These topics are left for a separate document.

Terminology¶

Within DevOps we manage a number of services which are formed of software components along with deployment configurations which connect these components together using Cloud resources. Software components have dependencies which are other software components which a given software component uses to implement functionality.

A specific example of a service/product is the Postgraduate Funding Portal. An example of a software component within that service is the Postgraduate Funding Portal Frontend user interface. The frontend, in turn, makes use of the PDFBox software library which is maintained by the Apache Software Foundation. As of writing, the specific version of PDFBox in use is “2.0.24”. PDFBox is a dependency of the Postgraduate Funding Portal Frontend. The PDFBox library may itself make use of other components. These components are indirect dependencies of the Postgraduate Funding Portal.

The full list of indirect and direct dependencies along with their versions is colloquially termed a bill of materials (BoM) and the locations where those components were downloaded from is similarly referred to as the supply chain. Both of these terms are borrowed from physical manufacturing.

A bug in a software component which may have security implications is termed a vulnerability.

For software, a release will have an associated version number and build artefact. The build artefact is generally speaking a container image or a programming language specific package. In either case the build artefacts are uploaded to an appropriate artefact registry where they may be retrieved given the package name and version number.

For deployments, the code which specifies the deployments will contain the exact version numbers of each software component. In this case, we tend to use a version identifier assigned by git itself called the commit SHA. For the purposes of this document, the commit SHA is considered to act like a version number. The version number of a deployment uniquely identifies both the way that the components are deployed and the components themselves. For deployments which use GitLab’s inbuilt automation to deploy, once changes from a MR are merged track which version is deployed to each of the production, staging and development environments. This information is available from a dedicated “environments” page in GitLab.

Standards¶

This section contains the standard practices to be adopted.

Protected Branches¶

A protected branch is a special branch in a code project that's locked down to keep it safe. This is important because it often contains the final, working version of the code, also known as production code.

What happens when a branch is protected?

No accidental changes: No one can accidentally delete or change the code in this branch by pushing to it.
Review needed: In order to make changes to this branch a Merge Requests needs to be opened. This ensures that other people, with rights to approve and merge, have to look over and approve any changes before they're added to the protected branch. More on this later.
Extra checks: Before any changes can be made, the code has to pass special tests that are executed when the Merge Request is opened. More on this later.

This helps make sure that the code is always working correctly and doesn't get broken by mistakes.

Merge Requests and the review process¶

Software, documentation on our practices and processes and Infrastructure as Code (IaC) used to deploy services must be version controlled and reviewable. Manual deployments and manual configuration of services must be avoided so that we retain confidence that the machine-readable description of how services are deployed and configured continue to reflect reality.

Use git source code management (SCM) and the university’s GitLab instance to manage versions of code over time. Code committed to our Gitlab instance allows other members to see, propose and review changes. When a team member wishes to propose a change to some code, they open a Merge Request (MR) which should contains the following information:

the changes they wish to make, split into separately reviewable chunks known as commits,
a description of each commit which focuses on the why rather than the what of the change, and
an overall description of the change which may make reference to existing issues which describe the agreed objectives of the change.

The team member who created the MR is termed the author. Other(s) team member(s), the reviewer( s), will look over the proposed change. As part of that process, the reviewer(s) may ask questions or suggest changes. Once the reviewer(s) is/are satisfied that they understand the change and that it meets the objectives in the related issue, they mark the MR as approved. Once approved, the MR is “merged” into the main body of the code. Teams differ on whether they prefer the reviewer or author to instigate the merge once approved but for the purposes of this document there is little difference between the two approaches.

The GitLab project where the code is hosted will be configured to require a reviewer(s) approval of a MR before it can be merged. “Branch protection” will also be enabled so that changes to the code may only be made via MRs

Common Continuous Integration pipeline configurations¶

GitLab allows automated tasks to be triggered when changes are merged or proposed for merging. These are called continuous integration (CI) pipelines. We have a common pipeline configuration which runs a number of security-related tasks, most of them inherited from GitLab’s AutoDevOps.

Some processes below require the common CI pipeline be enabled for repositories and to follow Continuous Integration and Continuous Delivery (CI/CD) standards.

Dependency pinning¶

Historically we’ve tolerated, but not recommended, software dependencies being specified with loose version requirements. E.g. “anything later than 2.3”, etc. This has meant that it was not always possible to determine uniquely which version of each dependency was deployed alongside a particular version of our software components.

We’ve tightened up our policy recently and now require not only that exact language-specific dependency versions be “pinned” but also, where packaging frameworks allow, that cryptographic hashes of dependency packages be recorded in repositories so that we may verify at build time that the supply chain has provided us the version of the dependency we expect.

For imports in HTML files, make sure that hashes are included when possible.

Dependency and container scanning¶

The common CI pipeline includes a task which examines dependency versions which are included in the build artefacts. For artefacts which are containers, this scanning extends to Operating System packages. The use of dependency pinning now means that build artefacts for a given version will have identical dependencies installed, even if they are re-built at a later date.

When a MR introduces new dependencies with known vulnerabilities, this is reported in the MR UI for the reviewer to review.

`Dependency scanning results presented in the MR UI.`

The full report links each vulnerability to a corresponding CVE and provides means for reviewers to get more information, to open issues so that fixes can be applied at a later date when available or to manually dismiss issues which they deem to be irrelevant.

`The full security report for a real MR.`

New Critical or High vulnerability reports should never be ignored. A solution for them, if these exist, should be applied as soon as possible, unless there is a strong reason not to. If after reviewing these a false positive is detected, it should be noted in GitLab when dismissing the vulnerability report.

Automated dependency updates¶

Vulnerabilities may be found in existing dependencies. A common DevSecOps practice is to have an automated process which periodically compares dependencies in software components to vulnerability databases and propose updates to dependency versions when appropriate. UIS DevOps use Renovate Bot. This automatically opens MRs to update dependency versions when appropriate.

Renovate Bot is enabled for all DevOps repositories. Renovate Bot merge requests need to be reviewed and applied, if deemed necessary, periodically.

Automated release and versioning¶

Our goal of being able to state with confidence whether a given deployed environment of a service contains a given vulnerability requires that we be consistent in how we release, version and package our software. UIS DevOps has developed a release automation process which takes away much of the manual process of versioning new releases. This removes opportunities for human error. All services must adopt this practice.

`A real MR generated by our release automation process.`

Automated deployments¶

In order to be able to tie versions of dependencies to deployed environments, UIS DevOps has developed automation around deployments. Currently, staging environments automatically track the current version of service and automated deployment of a selected version of a service to production may be manually triggered by a sufficiently privileged team member.

When deployed to an environment, the MR corresponding to the version deployed shows which environments it has been deployed to and when.

`UI within a real MR showing the deployment history of a version.`

The currently deployed version can be seen on a dashboard. Thus if we know that a particular version contains a critical vulnerability, we can check that it is not currently deployed.

`Dashboard showing current deployments of a real service.`

All services must adopt this practice.

Security Scanning¶

In addition to container and dependency scanning, GitLab supports proactive scanning for new vulnerabilities.

Static¶

Use our common CI pipeline to enable Static Application Security Testing (SAST) within GitLab. This is a process which analyses the code for common vulnerability patterns. For example, it is easy to detect when one calls a SQL database API passing a non-constant string value and this can be indicative of a potential SQL injection vulnerability.

Because we are using Infrastructure as Code, this means we can also apply static security scanning to this code. Use our common CI pipeline to enable Infrastructure as Code Scanning which currently uses KICS to identify vulnerabilities in Ansible, Terraform, Docker, or Kubernetes configuration files.

As part of our common pipeline, SAST and IaCS are commonplace in DevOps and SAST and IaCS reports are present in the MR UI.

New Critical or High vulnerability reports should never be ignored. A solution for them, if these exist, should be applied as soon as possible, unless there is a strong reason not to. If after reviewing these a false positive is detected, it should be noted in GitLab when dismissing the vulnerability report.

`An example of a SAST report in the MR UI taken from the GitLab documentation.`

Dynamic¶

GitLab supports Dynamic Application Security Testing which runs common automated attacks against a deployed environment.

`Example configuration of DAST for a service.`

You should enable DAST scans on deployed instances.

GitLab supports the concept of review apps which are instances of an application which exist for the lifetime of a MR which have the proposed changes incorporated. DAST can be configured to run against review apps in order to provide additional security reporting within MRs.

Review apps are not yet supported by our common pipeline and they are under investigation.

Secret detection¶

GitLab supports detecting leaked secrets in MRs and git branches. This is enabled as part of our common pipeline by using GitLab’s AutoDevOps. While GitLab supports auto revocation of secrets by triggering some action, the wide variety of secrets which could be inadvertently added to a repository means that a “one size fits all” approach is not yet feasible. Instead, you must use secret detection in pre-commit checks as well as adopting a security policy which alerts MR reviewers of leaked secrets in a timely manner.

Secret¶

Local developer machines should not have copies of secrets. While this is not always possible, whenever possible secrets should be “fetch-on-need” where the secret is fetched from 1Password on need using our 1Password CLI module and does not persist on disk.

Proactive triage of vulnerabilities¶

GitLab generates a vulnerability report for groups and projects. The data from this report comes from SAST, dependency and container scanning CI tasks. As such these must be enabled and have periodic scheduled scans for the dashboards to reflect all new vulnerabilities and those who have been resolved. Within this report, vulnerabilities may be triaged and thence classified into one of the following statuses provided by GitLab:

Confirm. The vulnerability is a true positive and requires a fix. This includes opening an issue.
Resolve. The vulnerability has been mitigated.
Acceptable risk. The vulnerability has not been mitigated but is deemed an acceptable business risk.
False positive. Upon examination, the vulnerability is not present.
Mitigating control. The vulnerability has not been directly mitigated but controls surrounding the service ensure that the circumstances of the vulnerability cannot occur. For example, a vulnerability may relate to non-signed in users but a service which mandates sign-in would not be vulnerable.
Used in tests. Usually for SAST vulnerabilities, this marks a potential vulnerability in code as being intentionally used as part of automated testing.
Not applicable. The vulnerability is known and not mitigated but relates to an unused part of the dependency.

`The vulnerability triage board for a UIS DevOps service.`

Security policies must be enabled which mandate the triage of new vulnerabilities.

The dashboard should be reviewed in every work iteration (e.g. sprint) to make sure that any potential Critical or High vulnerability is resolved in a timely manner. This needs to be embedded in all teams' cultures, including non-technical members of staff.

Security metrics dashboard / reporting¶

GitLab provides group- and project-level security dashboards which allow one to gain a sense of how a particular service is doing in terms of dealing with vulnerabilities.

`A real example of a security dashboard from a UIS DevOps service`

Common configuration policies for GitLab projects¶

Some of the practices in this document involve automation surrounding MR approval policies. These practices have no teeth unless it is mandatory that a) all MRs must be approved and b) the only way changes can be merged is through MRs.

Although this has long been the recommended configuration for GitLab projects in UIS DevOps, until recently we have had no way of enforcing it; GitLab project creation and configuration was a manual process.

We have now automated the provisioning of GitLab projects and have begun importing existing projects into this automation. This allows us to centralise and standardise the configuration of GitLab projects. At a minimum GitLab projects are now created with mandatory MR approval and branch protection. Deviations from this policy are detected and corrected by our IaC tooling. As of writing, detection and correction of deviations is performed automatically but triggered manually. As we gain confidence in the robustness of our automation, we plan to run this process automatically and frequently as a scheduled job.

Having laid the groundwork to be able to specify and automatically apply Division-wide policies on GitLab project configuration, we are now in a position to roll out additional DevSecOps practices surrounding security policies as detailed below.

Security policies¶

GitLab supports configuring projects with security policies which enable enforcement of DevSecOps policies. The common configuration policy described above allows us to do so.

Scheduled scan execution policies¶

Dependency, container and SAST, and IaCS scanning are by default only being performed for MRs so that security reports can be generated in the UI. A further scan is run once when the change is merged. If no changes are made to a software component, scans will not be repeated meaning that new vulnerabilities may not be detected.

GitLab security policies allow scheduled scans to be performed which keep the vulnerability information up-to-date in the security dashboard. Dependency and container scanning will run at least once a week.

Merge Request approval policies¶

Security policies will enforce the following practice:

MRs which have newly detected vulnerabilities which are classed as “medium” or “low” risk require additional approval, possibly from a small group of trusted approvers.
MRs which have newly detected vulnerabilities which are classed as “high” or “critical” risk require that the vulnerabilities be examined by a trusted approver and must be explicitly marked as not-relevant before a MR can be merged.

Discussion¶

Vulnerability databases¶

Not all bugs are vulnerabilities. The Common Vulnerabilities and Exposures (CVE) database is one effort to collect and categorise security-related bugs and to provide stable identifiers for them. Said identifiers, or CVE numbers, are issued by a family of CVE Numbering Authorities (CNAs). It is possible, in principle, to use a complete BoM for a service along with the CVE database to enumerate all CVE bugs which are outstanding for that service.

Not all vulnerabilities are equal in severity or applicability. Determining the effect that a given vulnerability has on a service often requires in-depth knowledge of that service. Although the Common Vulnerability Scoring System (CVSS) may help in that determination, there is still some nuance of understanding required.

CVEs and their associated CVSS scores provide quantifiable metrics for services which can be used to track the waxing and waning of vulnerabilities in a service over time. In accordance with Goodhart’s Law it is important that any DevSecOps strategy recognises the distinction between a metric and a target.

As a specific example of the above, it is tempting but ultimately counterproductive to have a target that zero CVEs be present in a service. Three reasons why this is not desirable are as follows:

A CVE may not yet have mitigations which involve simply updating dependency versions. In such cases, the CVEs will remain “present” in a service but may have been mitigated by disabling some functionality or introducing additional configuration.
A CVE may be bogus. There is some controversy about the quality of entries in the CVE database and many vendors have sought to become CNAs for their product simply to retain control over the quality of CVEs issued. As a recent example, the curl project became a CNA in January 2024 in order to deal with bogus CVEs. Even as a CNA, the curl project cannot remove a bogus CVE from the database. Early career security engineers have started including “number of CVEs reported” on their CVs which has led to a notable increase in poor-quality CVEs or even ones which have been fabricated by Large Language Models.
A CVE may affect only a non-applicable use case or relate to part of a dependency which is not in use. As a recent example, the bulk of our software makes use of the Linux kernel and in February 2024 the Linux kernel project became a CNA. This is notable because of how the Linux kernel developers intend to issue CVEs. Their opinion is that, due to the low-level nature of the kernel, virtually all bugs may have a security impact and so they intend to be extremely inclusive in what bugs will be assigned CVEs. A consequence of this decision is that each stable version of the Linux kernel will likely be released with several thousand new CVEs. It is not appropriate to use engineering resources reviewing each and every one of those CVEs for potential mitigation steps since the vast majority of them will not apply to our use cases. Even if we were to be able to review several thousand CVEs for each new release, we are limited to indirect mitigation rather than update. This is because for the majority of our deployments we do not control the exact version of the kernel as it forms part of the hosting platform provided by our Cloud vendor. Vulnerabilities may not directly affect us due to the nature of deployment. For example, denial of service or resource exhaustion vulnerabilities in JavaScript libraries can have limited scope; if the JavaScript is purely frontend an individual user’s session may be affected but no service-wide disruption is likely as no server-side JavaScript code is present.

Counts of CVEs and severity scoring systems such as CVSS are useful when used as metrics. For example, it is an appropriate use of CVEs to flag MRs if they significantly increase the number of CVEs present and to require additional review and triage of the new CVEs. Similarly it is a reasonable use of resources to triage newly issued CVEs which have been assigned a “high” impact at time of detection but to leave triaging of CVEs which are “low” impact to a process performed once per sprint.

Summary¶

Practices to be implemented:

Enable dependency scanning.
Enable container scanning for projects which build container images.
Enable SAST security scanning for software and IaCS for infrastructure as code projects.
Enable secret detection.
Pin all dependency versions via “lock” files. When possible these should include cryptographic hashes of the dependency packages.
Ensure all components have stable version numbers and automate version numbering.
Automate release and changelog generation.
Automate deployment via GitLab so that a record is kept of which versions are deployed to what environments.
Bring all GitLab projects under the control of our GitLab project automation so that consistent configuration can be enforced.
Enable security policies for all GitLab projects to ensure scheduled security scans are run.
Enable security policies for all GitLab projects to place additional approval and triage requirements on MRs which introduce newly detected vulnerabilities.
Configure Renovate Bot to open MRs for all projects when vulnerabilities are found in existing dependencies and those vulnerabilities can be mitigated by updating the dependency.
Enable proactive dynamic security scanning against staging instances of services.
Newly detected vulnerabilities for MRs must be triaged and categorised.
Project Managers should factor the security dashboard output into project management decisions.
Existing vulnerabilities should be triaged as a background process.

DevOps Continuous Integration and Continuous Delivery (CI/CD) standards

DevSecOps standards for Continuous Deployment