macOS support for NixOS tests #282401

Gabriella439 · 2024-01-20T19:31:09Z

Closes #193336
Closes #261694
Related to #108984

This pull request is a composite of the changes from #193336 and also #261694, with some changes of my own that I made.

The goal here was to get the following flake to build and run on aarch64-darwin:

{ inputs.nixpkgs.url = "github:Gabriella439/nixpkgs/gabriella/macOS_NixOS_test";

  outputs = { nixpkgs, ... }: {
    checks.aarch64-darwin.default =
      nixpkgs.legacyPackages.aarch64-darwin.nixosTest {
        name = "test";

        nodes.machine = { };

        testScript = ''
          print(machine.succeed('echo hello'));
        '';
      };
  };
}

… and after this change it does. There's no longer a need for the user to set nodes.*.nixpkgs.pkgs or
nodes.*.virtualisation.host.pkgs as the correct values are inferred from the host system.

This change is spiritually much closer to the approach in #261694 than the approach in #193336. However, I still made a few changes compared to #261694:

I didn't include the change to increase ulimit

I think this change was questionable because I feel like a script provided by Nixpkgs shouldn't be tinkering with the user's ambient ulimit settings.
I named the required system feature hvf instead of apple-virt

I preferred the hvf system feature name chosen by Run nixosTests on darwin #193336.
I didn't use the node.pkgs setting to set nodes.*.nixpkgs.pkgs

That does not work, based on my testing, but setting nixpkgs.system in nixos/lib/testing/nodes.nix does work with some supporting changes to pkgs/build-support/testers/default.nix.

Description of changes

Things done

Built on platform(s)
- aarch64-darwin
Tested, as applicable:
- NixOS test(s) (look inside nixos/tests)
- and/or package tests
- or, for functions and "core" functionality, tests in lib/tests or pkgs/test
- made sure NixOS tests are linked to the relevant packages
Fits CONTRIBUTING.md.

Add a 👍 reaction to pull requests you find important.

@roberth

Closes NixOS#193336 Closes NixOS#261694 Related to 108984 This pull request is a composite of the changes from NixOS#193336 and also NixOS#261694, with some changes of my own that I made. The goal here was to get the following flake to build and run on `aarch64-darwin`: ```nix { inputs.nixpkgs.url = <this branch>; outputs = { nixpkgs, ... }: { checks.aarch64-darwin.default = nixpkgs.legacyPackages.aarch64-darwin.nixosTest { name = "test"; nodes.machine = { }; testScript = ""; }; }; } ``` … and after this change it does. There's no longer a need for the user to set `nodes.*.nixpkgs.pkgs` or `nodes.*.virtualisation.host.pkgs` as the correct values are inferred from the host system. This change is spiritually much closer to the approach in NixOS#261694 than the approach in NixOS#193336. However, I still made a few changes compared to NixOS#261694: - I didn't include the change to increase `ulimit` I think this change was questionable because I feel like a script provided by Nixpkgs shouldn't be tinkering with the user's ambient `ulimit` settings. - I named the required system feature `hvf` instead of `apple-virt` I preferred the `hvf` system feature name chosen by NixOS#193336. - I didn't use the `node.pkgs` setting to set `nodes.*.nixpkgs.pkgs` That does not work, based on my testing, but setting the same thing in `nixos/lib/testing/nodes.nix` does work. - I created a `pkgsLocal` helper This is based on the feedback from @roberth here: https://github.com/NixOS/nixpkgs/pull/261694/files#r1365443955 … and is similar in spirit to `pkgsCross`.

cole-h · 2024-01-20T21:59:27Z

@ofborg eval

pkgs/top-level/stage.nix

nixos/lib/testing/macos-host.nix

nixos/lib/testing/nodes.nix

roberth · 2024-01-21T20:10:45Z

I named the required system feature hvf instead of apple-virt

I don't see this in the code.

apple-virt is autodetected by Nix 2.19 and should have been backported to 2.18, but that hasn't been released yet.

I've chosen this name because the detection appears to be avf/hvf agnostic.

See libstore: Add apple-virt to system features when available nix#9187 (comment)

nixos-discourse · 2024-02-08T22:11:44Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/getting-nixos-to-work-macos-m1-2-3-aarch64-darwin/39510/1

nixos-discourse · 2024-02-15T20:10:18Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/getting-nixos-to-work-macos-m1-2-3-aarch64-darwin/39510/2

@roberth

… as suggested by @roberth Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>

@roberth

… as suggested by @roberth Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>

@roberth

… as suggsted by @roberth

@cole-h

… as caught by @cole-h

pkgs/top-level/stage.nix

nixos/lib/testing/nodes.nix

Gabriella439 · 2024-02-16T23:11:00Z

Yeah, I'm planning on getting to fixing the nixpkgs.pkgs = guestPkgs;. I was just puzzling through the best way to do it.

@robert

… as caught by @robert Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>

@roberth

… based on feedback from @roberth This plays nicer with other changes the user might want to make and also works in conjunction with the `pkgsReadOnly`. Note that this required deleting the `nixpkgs.pkgs` default because that's not compatible with setting `nixpkgs.system`. If `nixpkgs.pkgs` is defined (even as a default-valued option) then `nixpkgs.system` is ignored.

This is no longer necessary now that we're overriding `nixpkgs.system` instead of `nixpkgs.pkgs`

Gabriella439 · 2024-02-21T23:21:18Z

@roberth: Alright, I think I finally understood what you were proposing and implemented that. The main caveat is that I had to get rid of the nixpkgs.pkgs = lib.mkDefault pkgs; in the nixosTest entry point because that breaks this (and some other stuff, too) since defining nixpkgs.pkgs (even just as a default) causes all other nixpkgs.* options to be ignored. I don't know if making that change to the nixosTest entry point will break other stuff, though.

Gabriella439 · 2024-03-01T17:27:05Z

@roberth: Thanks! Your changes look good to me

tfc · 2024-03-01T19:43:37Z

Works great!

Gabriella439 · 2024-03-01T20:30:39Z

I can go ahead and merge this once CI passes if nobody objects since I think all of @roberth and @tfc and I are satisfied with the final set of changes

Gabriella439 · 2024-03-02T00:44:59Z

Is it normal for ofborg-eval to take this long? I'm not sure how I can monitor its progress

Gabriella439 · 2024-03-02T01:14:15Z

@ofborg eval

cole-h · 2024-03-02T03:10:32Z

pkgs/top-level/stage.nix

+    # If already linux: the same package set unaltered
+    # Otherwise, return a natively built linux package set for the current cpu architecture string.
+    # (ABI and other details will be set to the default for the cpu/os pair)
+    pkgsLinux =


I think this might be running into the same issue as when you were using pkgsLocal (#282401 (comment)). ofborg might be happier if you exclude pkgsLinux from https://github.com/NixOS/nixpkgs/blob/1b439a273dadd4c675e81a0e05abbc9258889c02/pkgs/top-level/release-attrpaths-superset.nix

(if the label comes back, that'll be confirmation, but I don't think it'd be harmful to add it to that list before that happens)

@cole-h

… as suggested by @cole-h

Gabriella439 · 2024-03-02T05:41:18Z

Thanks to everybody for y'all's help!

414owen · 2024-03-04T11:30:54Z

Hi @Gabriella439 ,

Thanks for working on this.
I've put your flake from the PR description into an empty git repo, and run nix flake check, making sure I'm on nix 2.19.2.

Doing so gives me:

$ nix flake check
warning: Git tree '/Users/owen/src/nixos-test-darwin' is dirty
warning: Ignoring setting 'auto-allocate-uids' because experimental feature 'auto-allocate-uids' is not enabled
warning: Ignoring setting 'impure-env' because experimental feature 'configurable-impure-env' is not enabled
error: a 'aarch64-linux' with features {} is required to build '/nix/store/nv3ziyvxfnyl4z0ayg9j25kvgp27ziyr-append-initrd-secrets.drv', but I am a 'aarch64-darwin' with features {apple-virt, benchmark, big-parallel, nixos-test}

I'm not sure how to go about debugging this. I can list the referrers, but it just tells me that there's a nixos-system-machine-test that needs it.

Is there a step I've missed? I got the impression that I don't need to specify pkgs in the machine configuration anymore? Is it something to do with hvf/apple-virt?

tfc · 2024-03-04T11:40:50Z

@414owen your mac seems to be all set to execute NixOS integration tests. It still needs a linux builder to build the NixOS guest systems. See https://nixcademy.com/2024/02/12/macos-linux-builder/

Closes NixOS#193336 Closes NixOS#261694 Related to NixOS#108984 The goal here was to get the following flake to build and run on `aarch64-darwin`: ```nix { inputs.nixpkgs.url = <this branch>; outputs = { nixpkgs, ... }: { checks.aarch64-darwin.default = nixpkgs.legacyPackages.aarch64-darwin.nixosTest { name = "test"; nodes.machine = { }; testScript = ""; }; }; } ``` … and after this change it does. There's no longer a need for the user to set `nodes.*.nixpkgs.pkgs` or `nodes.*.virtualisation.host.pkgs` as the correct values are inferred from the host system.

nixos-discourse · 2024-03-04T22:35:09Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/macos-support-for-running-nixos-tests/40801/1

roberth · 2024-03-05T19:53:10Z

Doc update, please review:

nixos/doc: Describe NixOS test system requirements #293573

nixos-discourse · 2024-03-06T19:30:44Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nixostest-vs-runtest/26419/4

Gabriella439 · 2024-03-07T02:51:03Z

@tfc: I was able to reproduce the file limit bug that you were originally trying to address and root cause the problem!

It turns out that the problem is caused by an old (pre-2.5) installation of multi-user Nix. In Nix 2.5 the file limit was raised for the nix-daemon to 4096 by NixOS/nix#5726 (and then later further raised to 1048576 in Nix 2.13 by NixOS/nix#6645).

The easiest way to verify the presence or absence of the problem is to check the /Library/LaunchDaemons/org.nixos.nix-daemon.plist file and to see if there is something increasing the file limit like this:

    <key>SoftResourceLimits</key>
    <dict>
      <key>NumberOfFiles</key>
      <integer>4096</integer>
    </dict>

If that's present then it means you're likely unaffected by this bug. However, if that's absent (typically because Nix was originally installed at version older than 2.5) then you're probably going to be affected by that bug.

If anyone is reading this and wondering how to fix the problem, the solution is to edit /Library/LaunchDaemons/org.nixos.nix-daemon.plist to match https://github.com/NixOS/nix/blob/master/misc/launchd/org.nixos.nix-daemon.plist.in and then restart the nix-daemon:

$ sudo launchctl kickstart -k system/org.nixos.nix-daemon

Gabriella439 requested review from tfc and roberth January 20, 2024 19:31

Gabriella439 requested a review from Ericson2314 as a code owner January 20, 2024 19:31

ofborg bot added the 6.topic: darwin Running or building packages on Darwin label Jan 20, 2024

github-actions bot added the 6.topic: nixos label Jan 20, 2024

cole-h previously requested changes Jan 21, 2024

View reviewed changes

pkgs/top-level/stage.nix Outdated Show resolved Hide resolved

roberth requested changes Jan 21, 2024

View reviewed changes

roberth mentioned this pull request Jan 21, 2024

2.18.2 release NixOS/nix#9826

Closed

ofborg bot added the ofborg-internal-error label Jan 25, 2024

Gabriella439 and others added 4 commits February 16, 2024 13:55

Improve error message

6c013e3

… as suggested by @roberth Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>

Remove unused pkgs parameter

9aa8994

… as suggested by @roberth Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>

s/pkgsLocal/pkgsNative/

ccbfcca

… as suggsted by @roberth

Exclude pkgsNative from build

2bc4a79

… as caught by @cole-h

cole-h removed the ofborg-internal-error label Feb 16, 2024

ofborg bot added 8.has: clean-up 10.rebuild-darwin: 0 10.rebuild-linux: 1-10 labels Feb 16, 2024

roberth reviewed Feb 16, 2024

View reviewed changes

pkgs/top-level/stage.nix Outdated Show resolved Hide resolved

nixos/lib/testing/nodes.nix Outdated Show resolved Hide resolved

nixos/lib/testing/nodes.nix Outdated Show resolved Hide resolved

Fix typo

21f534a

… as caught by @robert Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>

roberth mentioned this pull request Feb 18, 2024

Make dockerTools.buildImageWithNixDb reproducible #289840

Merged

13 tasks

ofborg bot added the ofborg-internal-error label Feb 21, 2024

Gabriella439 added 2 commits February 21, 2024 15:14

Revert the addition of pkgsNative

03f2f87

This is no longer necessary now that we're overriding `nixpkgs.system` instead of `nixpkgs.pkgs`

roberth added 3 commits March 1, 2024 18:00

treewide: Make some NixOS tests darwin-host-proof

b915914

nixos/all-tests: Make runTest work on darwin

ddab359

nixos/testing/pkgs: Warn darwin users about ambiguous pkgs

3ef5de9

roberth self-requested a review March 1, 2024 17:21

roberth approved these changes Mar 1, 2024

View reviewed changes

cole-h reviewed Mar 2, 2024

View reviewed changes

cole-h removed the ofborg-internal-error label Mar 2, 2024

Gabriella439 added 2 commits March 1, 2024 20:42

Exclude pkgsLinux

c73be48

… as suggested by @cole-h

Merge branch 'master' into gabriella/macOS_NixOS_test

39b7626

ofborg bot added 2.status: merge conflict and removed 2.status: merge conflict labels Mar 2, 2024

Gabriella439 merged commit b8698cd into NixOS:master Mar 2, 2024
21 checks passed

Gabriella439 mentioned this pull request Mar 2, 2024

Allowing NixOS VM's to be run on macOS #108984

Closed

ShamrockLee mentioned this pull request Mar 2, 2024

tests.references: use runNixOSTest instead of nixosTest #292759

Merged

13 tasks

roberth mentioned this pull request Mar 5, 2024

nixos/doc: Describe NixOS test system requirements #293573

Merged

13 tasks

ShamrockLee mentioned this pull request Mar 10, 2024

runNixOSTest fails on Darwin ( tests.testers.runNixOSTest-example fails to build) #294725

Open

tie mentioned this pull request Jun 6, 2024

pkgsLinux: set crossSystem instead of localSystem #317651

Draft

13 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

macOS support for NixOS tests #282401

macOS support for NixOS tests #282401

Gabriella439 commented Jan 20, 2024 •

edited

Loading

cole-h commented Jan 20, 2024

roberth commented Jan 21, 2024

nixos-discourse commented Feb 8, 2024

nixos-discourse commented Feb 15, 2024

Gabriella439 commented Feb 16, 2024

Gabriella439 commented Feb 21, 2024

Gabriella439 commented Mar 1, 2024

tfc commented Mar 1, 2024

Gabriella439 commented Mar 1, 2024 •

edited

Loading

Gabriella439 commented Mar 2, 2024

Gabriella439 commented Mar 2, 2024

cole-h Mar 2, 2024 •

edited

Loading

Gabriella439 commented Mar 2, 2024

414owen commented Mar 4, 2024

tfc commented Mar 4, 2024 •

edited

Loading

nixos-discourse commented Mar 4, 2024

roberth commented Mar 5, 2024

nixos-discourse commented Mar 6, 2024

Gabriella439 commented Mar 7, 2024

macOS support for NixOS tests #282401

macOS support for NixOS tests #282401

Conversation

Gabriella439 commented Jan 20, 2024 • edited Loading

Description of changes

Things done

cole-h commented Jan 20, 2024

roberth commented Jan 21, 2024

nixos-discourse commented Feb 8, 2024

nixos-discourse commented Feb 15, 2024

Gabriella439 commented Feb 16, 2024

Gabriella439 commented Feb 21, 2024

Gabriella439 commented Mar 1, 2024

tfc commented Mar 1, 2024

Gabriella439 commented Mar 1, 2024 • edited Loading

Gabriella439 commented Mar 2, 2024

Gabriella439 commented Mar 2, 2024

cole-h Mar 2, 2024 • edited Loading

Choose a reason for hiding this comment

Gabriella439 commented Mar 2, 2024

414owen commented Mar 4, 2024

tfc commented Mar 4, 2024 • edited Loading

nixos-discourse commented Mar 4, 2024

roberth commented Mar 5, 2024

nixos-discourse commented Mar 6, 2024

Gabriella439 commented Mar 7, 2024

Gabriella439 commented Jan 20, 2024 •

edited

Loading

Gabriella439 commented Mar 1, 2024 •

edited

Loading

cole-h Mar 2, 2024 •

edited

Loading

tfc commented Mar 4, 2024 •

edited

Loading