E2E + CI testing

[jamesgorrie]

We've been E2E tests for our frontend using playwright.

Where these tests are run is important and also defines what we build.

Currently we have the local tests running that will run against your localhost:3000.

There are a few other places that feel desirable to have the tests run:

1. PRs
2. Stage deployments
3. Prod deployments

1. PRs

As we don't deploy to any http environment we would need to test against a docker container built in CI, and make it available over HTTP (localhost)

You could then docker compose that with the playwright docker container.

Pros

• Confidence that branches won't break future deployment steps leading to reactive fixes
• Can be triggered in CI

Cons

• Could slow down branch builds as docker building takes a while

Alternatives

• Deploy to an http environment e.g. www-2efe2c8dfd90b1cfa40a04f8493fa62b00f5276a.wellcomecollection.org and tests against that similar to netlify / vercel etc

Stage deployments

As we deploy to a staging environment we can just set the NODE_ENV, which in turn sets the root domain for testing, and the tests can run against that.

Pros

• We know something is broken before releasing to prod
• Can be triggered via CI
• It's super simple
• No build time considerations

Cons

• 🤷

Prod deployments

Same as above, but we would need to find a way to trigger on deployment as prod deployments are done by humans at the moment - and we could add a Block Step to deploy to prod.

Pros

• Ultimate confidence in what we have deployed

Cons

• Need to find a way to trigger post human deploy

---

Thoughts

It feels as thought starting with testing stage would be a good place to start as we have the most setup to be able to do this.

Testing both PRs & Prod could b e done by trying to fiddle with what we have (docker / weco-deploy), or we could try the alternatives, which might be more work, but would allow us to preview all steps of the development / deployment cycle.

I recommend starting with stage, and then breaking out the other steps where we want the tests to run as separate discussions.

Testing at stage/prod are doubly important as this would be testing master, which could be an accumulation of feature branches, which is exactly what E2E/Integration testing is for.

[kenoir]

I think we could deploy to prod from CI with a Block Step requiring a human to push a button to deploy.

When you have a step that runs the step against a particular NODE_ENV, just adapt that for the prod deployment pipeline and put it before the "Block Step". If we are satisfied we can automate deploying to prod - simply remove the block.

Being able to deploy to prod in CI now without tests with the block step is also useful without tests by simply making it easier and more accessible to make a prod deployment.

[jamesgorrie]

This makes tonnes of sense - this has been reflected in the description.

[kenoir]

more "modern" frontend deployment cycle of develop/preview/tests/ship.

Beware of making value judgements about what is "modern" or not - IMO it is entirely a case of choosing what fits our situation best.

[jamesgorrie]

Good point, it's why I use " as I couldn't think of a better way to put it - I've updated to say that I see a benefit in having a preview for each step (which we have bar for PRs).

[kenoir]

It's also worth remembering we do not merge master into our branches before we build them for PRs, in stage master is merged so there is a difference.

[jamesgorrie]

Updated in thoughts.

It's a really good point, do you think that this negates or makes it less important to run E2E tests on PRs?

[kenoir]

I agree getting the E2E testing running in staging should be the priority.