single entry point for test runners (ofborg, hydra, github actions, etc)

[ghost]

Currently we have been adding extra entry points for test runners in a very haphazard way.

Worse, we've been adding them in multiple repositories, which means that adding a new class of tests requires synchronizing with a different repository that has its own (usually smaller and therefore less-responsive) committer group.

As an example, here is the out-of-repo list of test suite entrypoints in nixpkgs that is hardwired into ofborg:

• https://github.com/nixos/ofborg

Adding a new test that doesn't obviously belong in any of these existing locations is an enormous pain.

We should create a single entry point into nixpkgs for test runners. It should not be specific to any particular type of infrastructure (i.e. ofborg, github runners, etc). Essentially it should just be a shell script which is invoked in an environment which has nix.stable in its $PATH and the to-be-tested nixpkgs checkout as its current working directory. That script, which is part of nixpkgs, should be responsible for "fanning out" to all the different tests in all the different places where we keep them.

A rough (but problematic) sketch of a solution was included in one of the later versions of

• Ensure all jobset attributes evaluate, and check that in CT #269356

See also the useful but not yet implemented feedback here. Ultimately that sketch was pushing the scope of #269356 way too much. We should still do something like it, but as a standalone PR.

[roberth]

Does Nixpkgs package set evaluation still need to hijack the lib tests?
It creates an impossibly long iteration cycle for working on lib.

[lf-]

https://github.com/nixos/nixpkgs/blob/2b4e18f3d4a7b80af21b640c0970f83b34efceff/lib/tests/release.nix#L27 This line of code referencing this issue is broken because it doesn't pass through pkgs, which then gets reimported, which then in turn explodes if running in pure eval mode.