From the beginning, our goal has been to help developers improve the speed and quality of their release pipelines—including our own team’s. We ship to production about 80 times a week across all our systems, so it’s important for us to cut back on any manual processes and checklists that stand in the way of catching bugs. But that’s hard to do when only three or four people can set up repositories or workflows. A year ago, very few people in our product organization could make the builds and add the required quality steps and gates. There was a lot of added complexity and surface area.

We like to think of ourselves as our first customer, and we use mabl as an integral part of our processes in building mabl. So something had to change. Migrating to a built-in continuous integration and continuous delivery (CI/CD) solution was the first step to more democratized repositories and workflows for our developers. There’s no more back and forth, and now everyone is able to contribute. Even our business analysts are tweaking queries, shipping them out to analytics tools, and changing builds. It's something that everybody is participating in, not just the bash command line experts.

Streamlining containerized workflows

As one of the senior engineers here at mabl—an intelligent test automation platform built for CI/CD–I focus on all of our backend systems, especially on automating all the things. I led most of the effort last year to transition all of our builds from our prior CI/CD provider to GitHub Actions, across more than 50 repositories. We were doing a lot of your typical containerized workflows, but all the container-based solutions out there were rather complex. With our prior solution, we didn’t get those pre-built runners you get out of the box with Actions; the majority of our effort was spent creating the build machine image itself. We always had to build the thing that was going to build the build. When we made a change in GitHub before, we’d have to go to a third-party CI tool and then keep switching back and forth to make things work.

Keeping developers in their flow

During development, we use Actions to help enforce basic code quality and code style. We try not to have a lot of procedures and checklists for our developers. If you did it right, it’s going to compile and pass all the GitHub checks.

We also integrate other tools with GitHub to keep everyone informed. For example, we integrate with Jira and note the ticket numbers, pull requests, and commits there to help non-committers and support personnel check in on projects without pulling our developers away. We also have Slack channels to help notify team members of new pull requests, pull requests that mention them, and when pull requests ship (like “Shipped UI to purrd 😺!”).

Best practices

1. Test early to catch problems before they’re integrated with the main branch.

2. Run CI and keep code in the same place to reduce management overhead and security footprint.

3. Reduce the burden on developers with automated checks and third-party integrations to stay in flow.

4. Empower everyone even business teams, to contribute to improving processes and workflows.

5. Learn from and give back to the community the best code is code you don’t have to write.

End-to-end testing for every build

One of our core properties is our web application, where our users do their account management, run tests, and view their results. It probably has the most contributors across our entire product team, so having comprehensive tests there is especially important. We make sure to do extensive testing on every single commit and branch. We run complex, end-to-end browser tests on each commit to catch problems as early as possible. A lot of companies might wait to run a similar set of tests on an integration environment or as part of a nightly batch job, but in those circumstances you don’t find out that you’ve broken something until after you’ve already merged the change.

For each pull request, we run through a typical build. Everyone’s going to build their code and run their unit tests, then we push every commit out to a preview environment. We end up having hundreds of these environments at once but we can run end-to-end tests against that environment for our application.

At this stage, we point mabl at the development branch. That means we have full-blown browsers in a container, not just a unit test. Our tests will click all the buttons and do everything a user would do. That'll make sure that the user interface for the web application works perfectly.

We’ve built our own Actions workflows to use in our testing. One action sets up the mabl CLI. The action is public and published on GitHub Marketplace. Our customers use it too. Using the CLI, you can start running tests headlessly, without needing UI access. We use Headless Chrome and run our tests on a GitHub runner. We also have a public action to trigger and run tests on the mabl cloud. We use that for ourselves, and our customers use it as well.

Finally, we also have a mabl GitHub App, which lets you get custom checks within your pull request. Full, rich results from the cloud run will show up live within GitHub. The app plus those two actions are the core pieces that let us do the end-to-end testing.

When we commit changes to the main branch or deploy to production, the commits are just pushing to different environments; it’s essentially the same workflow. The main branch pushes to an integration environment where we run the same end-to-end tests. Deployment tags go to a production environment.

Key indicators

• Build time

• Queue time for jobs

• Overall cycle time

• Change failure rate

• Cost

Standardizing and scaling best practices

The workflow to deliver our mabl CLI tool is very similar to that for our web application. Since we’ve created our own actions, it’s very easy to reuse them in other workflows by writing just a few dozen lines of YAML. This allows us to use consistent CI/CD and DevOps logic across all of our tooling.

The CLI workflow still requires the usual automated checks to enforce code quality and style, as well as builds and unit tests. Just like the web application workflow, the CLI workflow follows a similar branch development, integration, and production deployment process.

Where the CLI workflow differs is that it needs to run in many different end-user environments. It must work on Linux, Mac, and Windows laptops and servers. Since our CLI is written in Node.js, we also want to make sure it works on all of the possible versions of Node.js that we might deploy it on. Using the matrix build feature, we can install and test our CLI on all these operating systems and Node.js combinations, for every commit. Being able to use matrix builds gives us a lot of confidence in our ability to ship on multiple platforms.

Accelerating and learning with the community

There’s a great community around GitHub Marketplace. I would say 80 to 90 percent of the actions running our workflows are from Marketplace. But more importantly, GitHub itself is full of great communities. There are millions of repositories on GitHub; you can just go find someone else’s example build and adapt it to your own. There are these prominent open source projects, and it’s right there for you to see exactly how the big players that you look up to are doing the same thing you want to do.