Invert build tags to improve DX

[t0yv0]

Consider inverting build tags.

Currently providers rely on build tags to split tests by language so that the CI test matrix gains some parallelism and finishes faster.

An example of this can be found in:

https://github.com/pulumi/pulumi-gcp/blob/master/examples/examples_go_test.go#L2

It reads:

//go:build go || all
// +build go all

So unless the user passes -tags all or -tags go, the code in this file is excluded from compilation.

Unfortunately this sacrifices the DX for new contributors. If you open these files in an IDE it likely will not be able to quickly run these tests until you reconfigure Go test flags. This is a bad default for developer experience.

The proposal is to switch these tags to negative tags:

// +build !nodejs !python !yaml !java !dotnet

The CI splitting work just as before, while IDEs would be able to pick these tests up without configuration.

Consider implementing the change as a quick source migration in this repo so it can auto-apply to all ci-mgmt managed providers overnight.

[t0yv0]

@pulumi/providers I can't recall where I heard this idea, I think it might have been in a discussion with @danielrbradley ; noting here.

[blampe]

Currently providers rely on build tags to split tests by language so that the CI test matrix gains some parallelism and finishes faster.

Build tags aren't the best way to speed up CI because they're static, and sharding on language means we tend to end up with one very "heavy" (usually nodejs) shard, meanwhile the java shard does literally nothing. We're already in a situation where adding a new shard for yaml tests is non-trivial.

Instead, it would be better to assign tests randomly to some number N workers. (Currently N=5 for go/nodejs/dotnet/python/java.)

1. Collect all tests: go test -tags all ./... -list=.
   • Note: this is another case where we make life harder for ourselves by having multiple go.mod files.
2. Segment tests into N groups.
3. Execute one group of tests per worker: go test -tags all -run "^(A|B|C)$" ./....

Steps 1 and 2 are fast and can be performed independently and deterministically by each worker.

This could be rolled out in a backwards-compatible way by building with -tags all on each shard.

[blampe]

@corymhall found https://github.com/hashicorp-forge/go-test-split-action which seems to do what I described above.