Parallelize end-to-end tests

[2opremio]

The parallelization happens by running tests in multiple Kubernetes clusters (created with Kind).

I've had to fork bats in order to wire the clusters with the tests. I am waiting on bats-core/bats-core#255 to be merged before using upstream-bats again.

Let's see what results we get in CircleCI ...

Fixes #2646

[2opremio]

With two jobs the end-to-end tests take 11:00 minutes instead of 16:40, which is a 44% time reduction!

Let's see if things keep improving with 3 clusters or the overhead is too high.

[2opremio]

3 clusters doesn't really make it faster (it takes 11 minutes all the same), so let's keep it at 2.

[2opremio]

It turns out that the free tier of CircleCI only provides two CPU cores (medium resource_class): https://circleci.com/docs/2.0/configuration-reference/#resource_class

That explains it, let's see if I can make the switch to a large resource_class.

[2opremio]

I created a ticket at the CNCF support desk to upgrade the fluxcd org to CircleCI's performance plan.

However, I just realized that there is a per-user charge, which doesn't really make sense in an open source project with external contributors, so I will clarify this with CircleCI first. I just created a support ticket with them as well.

[2opremio]

I just left a call with some CircleCI guys and it turns out that the per-user charge only applies to users making use of private repos (which we have none).

The CircleCI stats are currently broken but it seems we are using 50K credits per month. Payment goes in blocks of 25K creates so it shouldn't translate in more than $30 per month.

I am looking into getting into a trial.

[2opremio]

It seems we weren't put on the trial plan yet. I am in conversations with the CNCF support desk so that the upgrade the plan.

[2opremio]

Let's get this merged first and then change the number of clusters once we are upgraded to the performance plan (which @caniszczyk agreed to do).

[2opremio]

I have now upgraded the e2e tests and CircleCI to 4 cores, which brings the e2e tests execution time to ~9 minutes (from 11 minutes with 2 cores). This is way better than the initial 16 minutes, but it's still far from the theoretical 50% improvement, which should had brought us down to 5:30.

Most of the overhead comes from the kind clusters creation, which, although done in parallel, still takes a long time (~4 minutes). This overhead will be amortized when we add more tests.

[2opremio]

The total execution time of build has gone down to ~13 minutes from ~22 minutes.

[2opremio]

I think we can get even better by:

• running certain steps in the background
• enabling docker layer caching

But I will do that in a separate PR

[squaremo]

On the basis that:

• it's already demonstrated as working
• it's a modest change, limited to the build system

I approve this PR with the following endorsements 🏎️