Webapp Developer Environment

The majority of our web applications make use of the same method to automate the developer experience. Our aim is to get developers up an running in the shortest possible time and to provide a unified experience across our applications.

The README of an application may refer to this document as a "getting started" guide since it documents features common to most of our applications and then only document features unique to that application.

Developer machine requirements

We assume the following software to be installed on developer machines:

  1. Docker.
  2. Some modern Python and a matching pip3 tool.
  3. docker-compose.
  4. Some programmer's editor or IDE.
  5. git

Our developers use both Mac OS X and Linux environments. Where possible the developer experience should support either OS.

Note

We do not mandate any particular IDE or editor. Cross-tool solutions such as editorconfig should be used to configure indentation, etc on a per-project basis.

Getting started with a project

The steps required to get a development instance of an application up and running are usually the same:

  1. git clone the web application repository.
  2. Some applications require credentials to be configured before running. If there is a secrets.env.in file, copy it to secrets.env and complete it using the instructions in the file.
  3. Ensure all of your container images are current via ./compose.sh development build --pull.
  4. Run ./compose.sh development up
  5. Visit http://localhost:8000/ in a browser.

Our standard application usually has the following services running:

  • http://localhost:8000/ is the web application itself
  • http://localhost:8025/ is a MailHog instance configured to show email sent by the application.
  • http://localhost:7000/ is a Swagger UI instance configured for the web application's API.
  • http://localhost:6060/ is a styleguidist instance which show interactive examples of UI components used by the application.

When running the developer instance both the frontend and backend are configured to re-compile or re-import when files on disk change so code-changes can often be tested by simply re-loading the browser page.

Important

The development container uses Django's hot-reload facility which can't handle new requirements. The use of ./compose.sh build will make sure the latest version of the container has been built.

Creating an initial "superuser"

Generally our applications allow auto-enrolment via Raven login but a new user has no permission. A "superuser" can be created which has all permissions via:

$ ./manage_development.sh createsuperuser

The command will ask you for a username, email and password for the superuser. You can then log in by visiting http://localhost:8000/admin.

Note

Generally we avid creating a superuser which matches our crsid. This makes it easier to test permissions in development by keeping our personal accounts "vanilla". If you are feeling uncreative, a common pattern is to name suerusers sys-{crsid} where {crsid} is the developer's crsid.

Secrets

Some of our applications require secrets to function, even in development. This section lists some secrets common to multiple applications.

Google OAuth2 credentials

Our newer applications make use of OAuth2 for login rather than the legacy UCam WebAuth protocol. This login method requires an OAuth2 "client id" and "client secret". Google provides documentation on creating OAuth2 credentials but we have a shared set of credentials within the team which allow access only for applications running on http://localhost:8000. These are available in our secrets repository (visible only to members of uis/devops on GitLab) or in the divisional 1password account.

Lookup credentials

Some applications make use of the Lookup API. Outside of the CUDN, this requires that you authenticate as a group. Shared group credentials for "bot" access are available in our secrets repository (visible only to members of uis/devops on GitLab) or in the divisional 1password account.

Running tests

There is a handy ./tox.sh wrapper script which can be used to launch the tox test runner within a special testing environment. Any arguments passed to the script are passed to the tox program itself.

Miscellaneous development "recipes"

This section describes who to perform some tasks which often come up when developing our applications.

Running a "production" version of the application

The development environment is configured to hot-load code from the current working copy and does not perform the full frontend compilation and minification steps performed in production.

You can build a production image via ./compose.sh production build --pull. This can then be run via ./compose.sh production up. Just like with the development environment, the web application is available at http://localhost:8000/ but it will be running the same container as would be deployed in production.

This can be very useful for checking browser support for the compiled frontend.

Running migrations

The Django web framework includes functionality to perform schema migrations of databases. These can be performed via the ./manage_development.sh migrate command. This command works just like the Django migrate command.

Creating migrations

The Django web framework includes functionality to automatically generate schema migrations when the database models change. To help developers keep the application stateless, the development instance is run in a container with a read-only filesystem. To use the Django makemigrations command, one first needs to edit compose/development.yml and comment out the read_only: true configuration for the development server volume mount. One can then create migrations as usual via ./manage_development.sh makemigrations.

Important

The generated migrations will be owned by root so make sure to sudo chown them to your own account before trying to edit them. Also don't forget to restore the read_only: true configuration.

Using a debugger

It is a little fiddly wiring a debugger into the containerised application. Within the development instance, the full-screen terminal-based debugger pudb has been included to allow you to run a debugger.

Use import pdb; pdb.set_trace() to mark a breakpoint in your code and then and attach to the container with docker attach {project-name}_development_app_1

For a fuller description of how to debug follow, read a guide to debugging with pdb and Docker which applies equally to pudb.

Destroying the database

We configure docker-compose to store tox artefacts and the database on persistent volumes. These can be deleted via ./compose.sh development down --volume.

A tour of our setup

For those interested in how our development environment or wanting to extend it, this section provides an overview of how it works.

All of the development and production code is run within a docker container. This allows us to specify exact versions of python, libraries, etc for a given project without having to have that list be identical between projects. It also allows for a degree of "agility" for developers in that one can quickly swap between environments for different projects without worrying about things like having the right database running.

We use docker-compose to orchestrate containers but support different "flavours" of environments: "development" for a developer-focussed environment, "production" for an environment designed to mirror the deployment environment and "tox" for an environment intended to run the test suite.

These environments are configured via the various docker-compose files in compose/. The correct set of files is then selected by the ./compose.sh script based on its first argument.

  • compose/base.yml describes services which should be run in all environments. There are not many of these. Usually it is just a Postgres database and a MailHog instance. The Postgres database is configured to use persistent containers so that data is not lost if any environment is taken down.
  • compose/development.yml describes services which should be run in the developer environment. These are usually a Django development server, a process which watches for changes in frontend code and re-compiles the frontend, a styleguidist instance and a Swagger UI instance. The current working directory is mounted within the Django development server and frontend compiler containers as a read-only volume so code-changes are reflected in the application.
  • compose/production.yml describes services which should be run in the production environment. This is usually a single service which is an instance of the application run using a container built from the Dockerfile in the repository root.
  • compose/tox.yml describes services which should be run when running the test suite. Usually this is simply a container built from the Dockerfile in the root of the repository with the working directory mounted as a read-only volume within it. The ./tox.sh wrapper script uses this environment to run the test suite.

These environments are configured through various .env files in compose/ which specify environment variables to run in different containers.

  • compose/base.env specifies environment variables common to production and development and environment variables for the database container.
  • compose/development.env specifies environment variables loaded into the Django development server.
  • compose/production.env specifies environment variables loaded into the production container.
  • compose/tox.env specifies environment variables required to run the test suite.

The ./manage_development.sh is simply a wrapper which uses ./compose.sh to run the Django management command within the development server container.