Support for Apple Silicon (aarch64-darwin)

[arianvp]

Apple will be transitioning to ARM in the near future, starting with their Big Sur release.

It would be great if we can get nixpkgs support for compiling ARM packages on Darwin

The NixOS Foundation has access to a Developer Transition Kit for testing, thanks to the friendly folks at Apple.

For now @edolstra and @rbvermaa have access to this DTK.

Note that we can get additional DTKs approved for anyone interested working on this ticket. We have a contact at Developer Relations who can help us with this. Please contact me if you would like to order a DTK to work on Nixpkgs support. (Cost is 500 US Dollars)

We also could get access to cloud machines through MacStadium. This might be interesting at a later stage for Hydra?

The original email that I got from Apple DevRel

Firstly, the most expedient way of obtaining a DTK would be for companies who have Nix maintainers and are registered as Apple developers can learn about the Universal Quick Start Program and apply for a DTK here:

https://developer.apple.com/programs/universal/

If this is the way your maintainers want to go, I would be happy to approve the applications to purchase the program. Someone noted on the thread they had applied and not been approved. I have approved then. Please let me know if there are any others.

Secondly, we have a relationship with MacStadium where qualified open source projects can simply get access to a DTK in the cloud. The maintainers of Nix should email opensource@macstadium.com.

The process for qualified OSS developers will be:

Developer signs up for a general MacStadium account so the MSA can be accepted.
MacStadium will manually add the DTK asset to their account at no cost.
Developer will have access to screen sharing on a DTK and can log in right away

[dhess]

My company has a DTK, so I'm planning to test Nixpkgs on it as soon as Nixpkgs is ready :)

There is already support for a aarch64-darwin platform, but it looks like that was added awhile ago for iOS support. Do we need a new platform tuple?

[matthewbauer]

Yeah - we will probably have to assume aarch64-darwin means Mac instead of iOS now. With the LLVM triple we do have a way to disambiguate:

aarch64-apple-ios

vs.

aarch64-apple-darwin

Which is even more confusing, since iOS is technically still using the Darwin kernel! But nevermind that, I suspect this distinction will get even more blurred with Mac Catalyst. I'd be interested if you can just execute an iOS binary from CLI on the DTK.

Unfortunately, pure-darwin stdenv is left to wait for update of https://github.com/tpoechtrager/cctools-port. We can probably either use the native stdenv or useiOSPrebuilt stuff to get something working though.

[thefloweringash]

Unfortunately, pure-darwin stdenv is left to wait for update of https://github.com/tpoechtrager/cctools-port. We can probably either use the native stdenv or useiOSPrebuilt stuff to get something working though.

What's required from cctools-port for a pure darwin stdenv?

[abathur]

https://eclecticlight.co/2020/08/22/apple-silicon-macs-will-require-signed-code/

[edolstra]

The code signing requirement sounds like the end of the line for macOS support in Nix - or at the very least, for having prebuilt binaries in the binary cache. Users could still build stuff locally using their local signing key (through some impure mechanism, I guess), but they wouldn't be able to share the result with other users and the binaries wouldn't be reproducible.

[arianvp]

There's a similar thread in homebrew about this Homebrew/brew#7857 as they're gonna face similar issues with the code-signing requirement.

From what I understand; signing is done ad-hocly and needs no user interaction and thus does not pose any issues for the availability of a binary cache.

Quoting:

Yep I can confirm signing is all done automatically. It's not associated with any particular identity. The code signing seems to be purely for checksumming and not identity verification.
No further action should be needed here. Anyone here updating from beta 1-3 should note however that you may need to rebuild anything already installed. Also, as always, ensure you are running the latest version of Xcode/CLT.

So it sounds like we're good?

[arianvp]

Or is homebrew not backed by a binary cache? i.e. always built from source?

[edolstra]

Yep I can confirm signing is all done automatically. It's not associated with any particular identity.

Then it's not clear to me what the signing key is (or what the point of this kind of signing is...). The release notes mention "a simple ad-hoc signature issued locally" but if that's tied to the machine then you wouldn't be able to share binaries.

[shlevy]

@arianvp There's a purity/reproducibility problem still, but I guess that's not the worst thing.

Absent proper support for detached signatures the closest thing to a "right" way to do this is to teach Nix's hashing to ignore the signature field 😭

[claui]

Or is homebrew not backed by a binary cache? i.e. always built from source?

brew install downloads and installs binary bottles whenever it can unless you tell it not to.

[thefloweringash]

As far as I can tell adhoc signatures are a hash of the contents of the binary and do not involve an identity, private key or any certificate chain. If so, there's no problem with reproducibility or distribution. My understanding is mostly based of a description of adhoc signing on stack overflow but I haven't found a more authoritative source.

I did a quick test of linking on a DTK. If another DTK user is able to produce the same binary from the same sources it'd be good evidence that there's no machine local key.

[thefloweringash]

I've now thoroughly investigated ad-hoc signatures and implemented a minimal generator: thefloweringash/sigtool. The short version is that it is a list of sha256 hashes of the binary contents up to the start of the embedded signature. I have a work in progress branch of nixpkgs that is able to cross compile and run a basic set of utilities on a dtk. I'm currently using this tool in a setup hook (adding to fixupOutputHooks) that signs every Mach-O file in the output.

[shepting]

@thefloweringash Do you have a link to your branch, or instructions for other folks with a DTK to try out the currently-partially-working version of Nix?

[dhess]

Now that it appears Big Sur is sorted, what’s the status on this one?

[domenkozar]

Note that the foundation has one developer toolkit machine and can set it up for access if that helps anyone work on this. You can request access here :)

Once stdenv compiles, foundation will order a few machines from https://opencollective.com/nixos donations.

[thefloweringash]

@thefloweringash Do you have a link to your branch, or instructions for other folks with a DTK to try out the currently-partially-working version of Nix?

I've tagged something that works as apple-silicon-1. It's incredibly messy, the commit history is equally messy, and still needs a ton of work. However, it can cross-compile bootstrap tools to aarch64-darwin, and then natively build things like hello and nix. My development branch is parrot-principles, but I make no guarantees about anything working there and will rebase and rewrite it frequently.

My cross compilation source machine is x86_64-darwin running 10.15. It should be possible to "cross compile" on aarch64 Mac via rosetta, but I haven't tried it.

---

To natively compile, install regular x86_64-darwin nix and update nix.conf to include

extra-platforms = aarch64-darwin

Natively building hello

$ nix-build --argstr localSystem aarch64-darwin -A hello

---

Cross compiling hello:

$ nix-build --argstr crossSystem aarch64-darwin -A hello

---

Cross compiling bootstrap tools:

$ nix-build pkgs/stdenv/darwin/make-bootstrap-tools.nix --argstr crossSystem aarch64-darwin -A build

[danieldk]

I am not sure if this is the right issue. But would this a good moment to move away from /nix/store? We probably won't have another good opportunity like this on Darwin.

This would mean that the whole dance around creating a Nix store volume would not be necessary on Apple Silicon Macs.

[kloenk]

As long as mac uses something like apfs, and the nix installer has a option to encrypt the partition I would like to keep a partition for the nix store. And keeping it under /nix makes the differences between oses smaller.

[emilazy]

Another advantage of APFS is being able to opt-in to case-sensitivity, which is required for cross-compiling Linux, some nixpkgs ecosystem tools like hackage2nix, and probably other things.

[danieldk]

Another advantage of APFS is being able to opt-in to case-sensitivity, which is required for cross-compiling Linux, some nixpkgs ecosystem tools like hackage2nix, and probably other things.

But that's regardless of whether the store lives in /nix or somewhere else. I think case-sensitive APFS works with most programs nowadays. Also, even if the Nix store is in some other location, you could still use a separate APFS volume if that's a requirement.

[emilazy]

Sure, but I don’t think ”reinstall your entire system on a new case‐sensitive APFS volume” is a reasonable thing for us to suggest users do, and if we’re going to support creating an APFS Nix volume for that reason then we might as well also continue to use it to keep the store prefix the same across platforms. I think that if we can get an automatically‐encrypted‐on‐FileVault, case‐sensitive APFS /nix volume setup in the installer then standardizing on that for all users would be the best approach. (This is kind of off‐topic for this issue, though… AFAIK this has mostly been discussed in #nix-darwin but maybe it would be good to solicit opinions on the Discourse too?)

[uri-canva]

I am not sure if this is the right issue. But would this a good moment to move away from /nix/store?

I think it's unrelated to apple silicon support, so this isn't the right issue.

We probably won't have another good opportunity like this on Darwin.

I'd say it's the opposite: if we were to change the path of the nix store I would prefer if we'd do it either completely before or completely after the apple silicon support: I would not want x86 macOS installations to be different from aarch64 macOS installations in ways that are not strictly necessary, and aarch64 support is likely to be buggy at the start, and I would not want to make that coincide with the change of the path of the nix store, a change that is itself very likely to introduce bugs.

[mull]

I have tested python3, node, and postgres today on a Mac with the M1 chip and my system set to aarch64-darwin. Has worked great so far :-)

[vic]

🎉 Same here! finally set system = aarch64-darwin and extra-platforms = x86_64-darwin aarch64-darwin as pointed by others on my nix.conf, re-installed nixFlakes (i'm using it plus nix-darwin and home-manager on my setup) and most packages work, just did a tiny overlay for few things complaining: haskell packages (till #126195 gets merged). Thanks so much to all people involved on this :)

edit: extracted my mkDarwinSystem function as a flake from my nix configuration if hopes it can help anyone.

[arianvp]

So I suppose we can close this then? Things that don't work yet probably warrant new issues.

[msfjarvis]

So I suppose we can close this then? Things that don't work yet probably warrant new issues.

Has the Nix release mentioned here been made? Though I guess it's not a direct blocker for a Nixpkgs issue.

[abathur]

Has the Nix release mentioned here been made?

https://github.com/NixOS/nix/releases/tag/2.3.14

[kloenk]

What about the ghc. I don't follow that. I would like to keep this open, until that is working

[ilyakooo0]

GHC 8.10.5 with Apple Silicon support has already been released. (Although it has other issues)

[BaoPham92]

Following previous comments of attempts at a successful build, this is my attempt trying the latest release.

MacOs M1 BigSur 11.4 running latest Nix 2.3.14 nix-shell so I can setup a cabal build. The process has not introduced any of the previous mentioned errors.

I'm able to do a proper sh <(curl -L https://nixos.org/nix/install) --daemon.

Only persistent error I tend to run into even with different build attempts previously is:

error: cannot coerce null to a string, at /nix/store/sdjxjsd5phr225rs2qzklj2xci0c9gr0-source/pkgs/stdenv/generic/make-derivation.nix:192:19

Which I had attempted to resolve here with listed attempts & clean installs. I'm pretty new to Nix, its a bit difficult to understand if this would be the OS compatibility being a factor for producing the error.

[ryanbooker]

So far in my setup, these don't work with aarch64-darwin:

• vscode
• comma (Shopify, update: lzma is actually what's failing)
• hub
• neovim

[angerman]

On the GHC side:

• 8.10.5 has been released with sufficient plumbing to have the LLVM backend support aarch64-darwin.
• 8.10.6 should be released soon with a bunch of fixes.

However, a few bugs were found (some stemming from GHC's CI using nix to build GHC, and thus having had some unwanted nix-paths leak into the build product), a crash in the aarch64-darwin linker.
In general apples aarch64 procedure calling convention is rather strict (also the reason or clang to require header files!). And there still seem to be quite a few Foreign Interface Calls, that have the wrong FFI decorations in Haskell, and thus you might see odd behaviour (or even crashes) where GHC ends up calling C functions incorrectly.

The last part is the hardest to rectify quickly, as it essentially entails auditing all Haskell code for proper FFI declarations. We are trying to come up with some automation for this.

[domenkozar]

I've opened #129427 to make sure evaluation errors don't slip into master.

The last bit missing is documentation and user testing.

I'd encourage M1 users to install Nix via:

$ sh <(curl -L https://nixos.org/nix/install) --darwin-use-unencrypted-nix-store-volume --daemon

[arianvp]

vscode
comma (Spotify)
hub
neovim

I would like to avoid this ticket staying open forever with reports for packages not working; and instead mark Apple silicon as "Initital basic support done" and open new tickets for any packages that don't work yet. I'm sure there are hundreds if not thousands of packages that do not compile on M1 and we can all track them in new tickets invdividually.

What about the ghc. I don't follow that. I would like to keep this open, until that is working

Similarly it seems that upstream doesn't even have the M1 story figured out completely yet. it doesn't make sense to have that as a blocker for M1 support in nixpkgs. We can open a new issue for that too.

The last bit missing is documentation and user testing.

Yes documenting M1 is supported in the manual or the website sounds like something we should at least address before closing this.

[callahad]

I apologize for not trying 2.3.14, but 2.4pre20210705_1f93084 from https://hydra.nixos.org/eval/1684067 installed cleanly in multi-user mode and appears to be working perfectly on my M1 MBP.

I've also had great luck with using the Flakes and the new nix commands for mixing Intel and Arm packages, e.g., nix profile install nixpkgs#ripgrep defaults to aarch64, while nix profile install nixpkgs#legacyPackages.x86_64-darwin.neovim pulls in x86_64 binaries.

[ryanbooker]

I would like to avoid this ticket staying open forever with reports for packages not working

Sounds fair. :) The installation itself was flawless.

[bphenriques]

✅ TL; DR: Issue with nix-darwin which is not in the scope of this ticket.

👋 For some reason my nix is considering x86_64 whenever I apply my nix-flake and not arm64 as expected 🤔 Did anyone stumble upon this odd issue? (edit: see bottom of post, think I found the culprit...)

Checked the architecture using uname -m on my terminal (plain Terminal App):

$ uname -m
arm64

Added a custom system.activationScripts.postUserActivation.text to echo uname -m:

{
  system.activationScripts.postUserActivation.text = ''
    echo "What is my CPU?"
    uname -m
  '';
}

Build and apply my flake:

$ nix --version
nix (Nix) 2.4pre20210601_5985b8b
$ nix build .#darwinConfigurations.personal-macos.system
$ ./result/sw/bin/darwin-rebuild switch --flake .#work-macos
...
What is my CPU?
x86_64
...

The impact: I have some issues using Homebrew integration with nix-darwin and my binaries are not in the correct architecture (they are in Mach-O 64-bit executable x86_64).

As suggested I also set the system and extra-platforms:

$ cat /etc/nix/nix.conf
# WARNING: this file is generated from the nix.* options in
# your NixOS configuration, typically
# /etc/nixos/configuration.nix.  Do not edit it!

max-jobs = auto
cores = 0
sandbox = false

substituters = https://cache.nixos.org/
trusted-substituters = 
trusted-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
require-sigs = true
trusted-users = root
allowed-users = *
experimental-features = nix-command flakes
system = aarch64-darwin
extra-platforms = aarch64-darwin x86_64-darwin 

Edit: arrrghhh:

$ file /Users/brunohenriques/.nix-profile/bin/nix
/Users/brunohenriques/.nix-profile/bin/nix: Mach-O 64-bit executable x86_64

So.. my nix executable is running on Rosetta, so this means I need to reinstall Nix to ensure that runs on Apple Native... 😢

Update2:

Reinstalled Nix and renders the same result:

$ uname -m      
arm64
$ where nix
/Users/brunohenriques/.nix-profile/bin/nix
$ file /Users/brunohenriques/.nix-profile/bin/nix
/Users/brunohenriques/.nix-profile/bin/nix: Mach-O 64-bit executable arm64

Any ideas?

Last Update

Issue with nix-darwin which is out-of-scope of this ticket.

[fr34k8]

I've opened #129427 to make sure evaluation errors don't slip into master.

The last bit missing is documentation and user testing.

I'd encourage M1 users to install Nix via:

$ sh <(curl -L https://nixos.org/nix/install) --darwin-use-unencrypted-nix-store-volume --daemon

Installation works flawlessly! Thanks ! any plans on updating the documentation also ?

[domenkozar]

@fr34k8 I've updated https://nix.dev/tutorials/install-nix to have instructions per OS.

Also planning to look at fixing GHC in the following days.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nixos-emulation-on-mac-mini-m1/14659/1

[vamega]

I ended up on this page after failing to follow the instructions that @callahad provided in a thread on hacker news.

It seems that in version 2.4pre20210705_1f93084 as he mentions in his comment on this thread works perfectly.
I had tried two more recent versions, which failed to install with this error message:

nix-store cannot be opened because the developer cannot be verifier

I saw this error message for both of these versions off of Hydra.

1. nix-2.4pre20210823_af94b54-aarch64-darwin.tar.xz
2. nix-2.4pre20210820_7a54b10-aarch64-darwin.tar.xz

I think there has been a regression in between these versions, in case anyone else is interested in pursuing this.

[domenkozar]

GHC now builds on aarch64-darwin as of just merged #135345

Note that for many Haskell packages to build (I hope to fix cachix today) we need to wait for haskell-updates branch to be merged (ETA ~week).

@vamega I haven't seen that before and our installer tests work well. Is your setup somehow special?

[domenkozar]

TL; DR: Issue with nix-darwin which is not in the scope of this ticket.

For some reason my nix is considering x86_64 whenever I apply my nix-flake and not arm64 as expected Did anyone stumble upon this odd issue? (edit: see bottom of post, think I found the culprit...)

Checked the architecture using uname -m on my terminal (plain Terminal App):

$ uname -m
arm64

Added a custom system.activationScripts.postUserActivation.text to echo uname -m:

{
  system.activationScripts.postUserActivation.text = ''
    echo "What is my CPU?"
    uname -m
  '';
}

Build and apply my flake:

$ nix --version
nix (Nix) 2.4pre20210601_5985b8b
$ nix build .#darwinConfigurations.personal-macos.system
$ ./result/sw/bin/darwin-rebuild switch --flake .#work-macos
...
What is my CPU?
x86_64
...

The impact: I have some issues using Homebrew integration with nix-darwin and my binaries are not in the correct architecture (they are in Mach-O 64-bit executable x86_64).

As suggested I also set the system and extra-platforms:

$ cat /etc/nix/nix.conf
# WARNING: this file is generated from the nix.* options in
# your NixOS configuration, typically
# /etc/nixos/configuration.nix.  Do not edit it!

max-jobs = auto
cores = 0
sandbox = false

substituters = https://cache.nixos.org/
trusted-substituters = 
trusted-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
require-sigs = true
trusted-users = root
allowed-users = *
experimental-features = nix-command flakes
system = aarch64-darwin
extra-platforms = aarch64-darwin x86_64-darwin 

Edit: arrrghhh:

$ file /Users/brunohenriques/.nix-profile/bin/nix
/Users/brunohenriques/.nix-profile/bin/nix: Mach-O 64-bit executable x86_64

So.. my nix executable is running on Rosetta, so this means I need to reinstall Nix to ensure that runs on Apple Native...

Update2:

Reinstalled Nix and renders the same result:

$ uname -m      
arm64
$ where nix
/Users/brunohenriques/.nix-profile/bin/nix
$ file /Users/brunohenriques/.nix-profile/bin/nix
/Users/brunohenriques/.nix-profile/bin/nix: Mach-O 64-bit executable arm64

Any ideas?

Last Update

Issue with nix-darwin which is out-of-scope of this ticket.

I'd suggest posting that to LnL7/nix-darwin#334

[starcraft66]

@bphenriques I was having the same issue as you this week (somehow got an x86_64-darwin installation of nix and was unable to reinstall it as an aarch64-darwin installation) and came up with a simple fix.

In my flake, I use specialArgs to override pkgs to point to a special nixpkgs with system explicitly set to aarch64-darwin and with an overlay to install x86 packages which do net yet compile on aarch64. After switching to the flake with nix-darwin, all of my executables were switched to arm. You can take a look at my flake here: https://github.com/starcraft66/os-config/blob/7758b954ddaf09316d627584ff64e4d09867e0b7/flake.nix#L127

[vamega]

@domenkozar I'm not sure how my setup could be special.
I installed this with the command. Going back to the earlier version fixed my issues.

./install --daemon --no-channel-add

[domenkozar]

I'm closing this issue as the original goal of supporting Apple Silicon is achieved!

Our build farm is running 6 M1 macOS machines and https://www.macstadium.com/ has donated us another 5 machines that @grahamc is setting up for use in ofborg/hydra.

Nix natively installs on aarch64-darwin and we have a hydra jobset that builds 21.05 and unstable channels.

GHC has been fixed just last week, with now over 25000 packages building on aarch64-darwin.

Documentation on how to install Nix on macOS

Future work:

• nix-darwin needs to properly support aarch64-darwin
• Nix 2.4 release will bring a number of installer improvements done by @abathur
• aarch64-darwin support in ofborg as mentioned above
• a guide on mixing nixpkgs to use x86_64-darwin as a fallback for unsupported packages
• a tutorial for getting started with Nix on macOS
• further improvements to macOS ecosystem (SDK/LLVM updates, etc) funded by the community
• RFC for official tier support

For any issues please open bugs either here or in Nix repository.