Add support for picking RD_LOCATION for BATS

[jandubois]

Choose from system/user/dist/npm or pick first available. dist and npm only work when running from inside a git clone. npm is not yet supported on Windows.

Also move "load ../helpers/load" from setup() to top level, so we can abort on fatal errors.

Predefine teardown_file() with rdctl shutdown so it doesn't have to be repeated in every test file.

Partially implements #4542 but more work is needed.

Additional comments

./bats-core/bin/bats tests/* passed successfully on macOS Intel for both "moby" and "containerd": 158 tests, 0 failures, 5 skipped.

You can tell BATS to use the package in the dist directory even when RD is also installed globally, or per user:

RD_LOCATION=dist ./bats-core/bin/bats tests/*

I'm not really happy with the RD_LOCATION name, but couldn't think of anything better. I briefly considered RD_INSTALL, but that seems even worse.

Using RD_LOCATION=npm has been tested with the registry/creds.bats test, which does run a second factory-reset in the middle:

$ RD_LOCATION=npm ./bats-core/bin/bats tests/registry/creds.bats
creds.bats
[...]

19 tests, 0 failures

The factory reset logic should be made more robust (e.g. I've seen it once that limactl/qemu were not killed), and needs to be implemented for Windows. This should happen in a separate PR.

The apify_arg() function should be auto-generated from the schema in command-api.yaml. This too should happen in a separate PR.

[mook-as]

We should consider making npm run dev take the same arguments as rdctl start instead. That's better for our sanity. :)

[jandubois]

That works for me too, but should be a separate PR.

[mook-as]

Yeah, definitely not as part of this PR.

[ericpromislow]

I see the same breakage on the main branch in macOS, so something else is going on outside this PR.

I get this breakage on macOS after running `bats bats/tests/k8s/wordpress.bats bats/tests/registry/creds.bats` in both `main` and this branch with `~/.docker/config.json` set to ``` { "credsStore": "none" } ```

Breakage:

/Users/ericp/workspace/rancher/desktop/bats/tests/k8s/wordpress.bats
 ✓ factory reset
 ✓ add helm repo
 ✓ start rancher desktop
 ✓ deploy wordpress
 ✓ verify wordpress was deployed
/Users/ericp/workspace/rancher/desktop/bats/tests/registry/creds.bats
 ✗ factory reset
   (from function `factory_reset' in file bats/tests/helpers/vm.bash, line 17,
    in test file bats/tests/registry/creds.bats, line 82)
     `factory_reset' failed
   kill -9 90658
   kill -9 90668
   kill -9 90672
   kill -9 90853
   91287
   91293
   91303
   91304
   91307
   91311
   91315
   92382
 ✓ start container engine
 ✓ wait for container engine
 ✗ verify credential is set correctly
   (from function `assert_output' in file bats/bats-assert/src/assert_output.bash, line 194,
    from function `verify_default_credStore' in file bats/tests/registry/creds.bats, line 107,
    in test file bats/tests/registry/creds.bats, line 100)
     `verify_default_credStore' failed

   -- output differs --
   expected : osxkeychain
   actual   : none
   --

If I run bats bats/tests/registry/creds.bats in isolation I don't get the error -- for some reason the credsStore field gets set to osxkeychain at the start of the run

I don't see this breakage when running in main.

I've run both sequences three times in both branches and this happens consistently.

[ericpromislow]

See note on breakage in #4573 (comment)

[jandubois]

See note on breakage in #4573 (comment)

Please provide details on how to reproduce breakage; it works for me.

Also, if this is using RD_LOCATION=npm, then it should be fixed in a follow-up PR; this PR is about the structural changes needed so you can specify RD_LOCATION.

[ericpromislow]

Because the breakage I saw on this PR is also visible in main, there's no reason to hold this PR back.