Meson functional tests

[Ericson2314]

Motivation

Run the functional test suite with Meson

Context

• Rewrite build system in Meson #2503

• Depends on Put flake functional tests in their own group #11176

• Depends on Change skipped test error code from 99 to 77 #11177

• Depends on Move NIX_BIN_DIR and all logic using it to the Nix executable itself #11178

• Depends on Misc fixes #11179

Priorities and Process

Add 👍 to pull requests you find important.

The Nix maintainer team uses a GitHub project board to schedule and track reviews.

[fricklerhandwerk]

Triaged in Nix team meeting:

• @roberth tried to work around some failures and also had design questions
  • it should probably follow the convention we already have for the unit tests, even if we're not building much
  • @Ericson2314: instead of copying the sources to the build directory we could actually "install" them
• @roberth: this also needs to work in the VM test
  • can definitely be done
  • would be fantastic if we could just package the tests, but maybe it's just not how it works
  • @Ericson2314: we could absolutely do both, but there's still a lot of reworking to get the functional tests how we want them

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2024-07-10-nix-team-meeting-minutes-160/49101/1

[Ericson2314]

@GoldsteinE might you be interested in helping out with this? I think I just (hackily) fixed the CI eval issue so the actual interesting failures will show up.

The differences between the Meson and Make bash environments for the functional tests is...very subtle! :( It makes debugging these failures hard.

[Ericson2314]

Fishing out some non-test-suite files like this is no good with the broken-up build. If we want to test .. we should do it a different way (e.g. make some dirs in $TEST_ROOT and cd inside there).

[GoldsteinE]

I’ll take a look, but I don’t really know Meson.

Unrelated, but I think one of the end goals here should be rewriting functional tests to something that is not bash. Writing logic in bash is extremely error-prone and I think replacing it with e.g. Python would make all of this logic much less brittle.

[Ericson2314]

@GoldsteinE Thank you!!

Unrelated, but I think one of the end goals here should be rewriting functional tests to something that is not bash.

I agree! I wrote up #10823 for this.

[Ericson2314]

The PRs that this depends on should be reviewed separately, but then it's good to go!

[dpulls]

🎉 All dependencies have been resolved !

[Ericson2314]

It seems that the macOS failure is due to libcurl is failing to initialize:

2024-08-13T14:04:56.2756290Z nix-functional-tests> building '/private/tmp/nix-build-nix-functional-tests-2.25.0pre20240813_1df9fb0.drv-0/nix-test/check/store/98i8amk2i6hj4h9w9fk6hcnmbkcspa3y-eval-okay-xml.exp.xml.drv'...
2024-08-13T14:04:56.2757530Z nix-functional-tests> objc[25124]: +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called.
2024-08-13T14:04:56.2759080Z nix-functional-tests> objc[25124]: +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
2024-08-13T14:04:56.2761140Z nix-functional-tests> error: builder for '/private/tmp/nix-build-nix-functional-tests-2.25.0pre20240813_1df9fb0.drv-0/nix-test/check/store/98i8amk2i6hj4h9w9fk6hcnmbkcspa3y-eval-okay-xml.exp.xml.drv' failed due to signal 6 (Abort trap: 6)

from the last run. Restarting to see if at least Linux will finish now.

Unfortunately, I could not reproduce these failures in a macOS dev shell: those tests just passed normally.

[roberth]

That is a known issue. You can apply #10164 to work around it.

[Ericson2314]

@roberth Look good now?

[roberth]

NixOS VM test is yet to be ported, but looking good so far.
2 suggestions.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2024-08-14-nix-team-meeting-minutes-169/50633/1