poe Tests Pipeline - Currently Wilson Team POC

The poe test pipeline is currently in POC state and being used and developed by the Wilson team. It is a alternative way to run all types of software tests for the unified DevOps platform, although currently it is primarily being used for tests that don't currently have existing off the shelf test runners.

Background

Primary Goals

• Support non-python (and python) based tests e.g. TS, cypress, selenium, etc.
  • Easily add multiple test types within a single repo, e.g. python and TS tests
• Cheaply add docker based dependencies, e.g. pubsub emulator
• Greater consistency between CI and local development - ability to run the same command

Secondary Goals

• Able to test built docker images, e.g. make API call on built docker image
• Reduction of sources of truth, e.g. a single file that defines the database version tests run against
• Concise declaration of test commands
• Pipeline level parallelism

Guide

In order to use the poe based test pipeline the first step is to include the pipeline template.

- project: "uis/devops/continuous-delivery/ci-templates"
  file:
    "/auto-devops/poe-tests.gitlab-ci.yml"
    #ref: poe-tests-poc # at the time of writing the code is not yet merged

To run a test command you need to add an entry that starts with test- or tests- to the list of poe commands in your repos pyproject.toml file. Non-python projects need not have any other entry in pyproject.toml than commands or you can use the slimed down poe_tasks.yaml/json/toml which follow slightly different syntax, see without_poetry. That is all that is needed for a child pipeline to execute.

E.g.

[tool.poe.tasks."test-hello-world"]
cmd = "echo hello"

An example test command using docker compose run.

[tool.poe.tasks."test-cypress"]
cmd = "docker compose run cypress"

However, you will need to make sure that your test commands only use the tools listed in the guidebook as the minimum developers tools. Otherwise the pipeline is likely to fail with missing dependencies. The test commands should bring their own dependencies beyond the minimal set of tooling defined here: prepare-your-system (at time of writing this link has an open MR that brings these values up to date).

This generally means running tests through docker compose. This also includes when the tests do only need tooling available in the pipeline docker image (e.g. python) as the tests should be incorporating pinning of tooling/libraries versions used with the tests. The availability of python is only intended for tooling/scripting that helps to run the tests and should not to be used run the test cases themselves. I.e. make sure python tests run in a container defined in the docker compose file.

The main pipeline template uses dynamically generated child pipelines in order to run each command in parallel. This means that in the gitlab UI there are multiple pipeline entries and nested child pipelines. E.g. an expanded view with multiple tests will look something like:

In this example one of the two poe test commands has failed (test-cypress-last-name-disable) and the other has passed. The job poe-tests-generator job should generally always pass as this is only is responsible for generating the child pipelines template. poe-tests-child-trigger will general reflect fail if any child pipelines fail and pass otherwise.