Problems with multi-user (re)installation of nix on macOS

[janvogt]

Preface

I do not know if this is a documentation issue, bug, or something different. Nevertheless it took me close to a day to get my nix working again - and I do not know if I solved the problem correctly.

Scenario

I'm running macOS Mojave 10.14.6 (18G1012) with a working single-user nix install and using darwin-nix. Before updating to macOS Catalina I wanted to switch to multi-user nix for all it's advantages.

Steps

I did not figure out how to convert from single to multi-user so the only viable solution I found was to remove my old installation and reinstall with bash <(curl https://nixos.org/nix/install) --daemon - after all my whole configuration is declaratively stored in my darwin-configuration.nix file.

I first tried the recommendations from that script if you run it with an existing install. After removing these nix related files, the installation went through and suggested I could run nix-shell -p nix-info --run "nix-info -m" in a new shell. But that failed with varying results between:

$ nix-shell -p nix-info --run "nix-info -m"
error (ignored): unable to download [...]: Problem with the SSL CA cert (path? access rights?) (77)
$ nix-shell -p nix-info --run "nix-info -m"
error: unexpected end-of-file

Now I tried several things and found many different potential reasons - not all documented. But to name a few:

• clean macOS user account
• trying to remove all traces of nix with a script expelledboy wrote
• as root lead to another error:

$ sudo nix-shell -p nix-info --run "nix-info -m"
Password:
error: file 'nixpkgs' was not found in the Nix search path (add it using $NIX_PATH or -I), at (string):1:13

• multiple variations of reboots and uninstall and reinstalls.
• Trying to set NIX_SSL_CERT_FILE despite the file did not exist.
• Finding someone with a virtually identical problem, who did not share his solution, because it was unrelated to the OPs question.

all without success.

Solution (working, but is it correct?)

The break through was, when I found expipiplus1's tutorial how to convert a nix single-user install into multi-user one:

There he linked NIX_SSL_CERT_FILE to /nix/var/nix/profiles/default/etc/ssl/certs/ca-bundle.crt for the deamon on linux. This file did exist and so I tried:

$ sudo launchctl setenv NIX_SSL_CERT_FILE /nix/var/nix/profiles/default/etc/ssl/certs/ca-bundle.crt
$ sudo launchctl kickstart -k system/org.nixos.nix-daemon
$ nix-shell -p nix-info --run "nix-info -m"
[...]
 - system: `"x86_64-darwin"`
 - host os: `Darwin 18.7.0, macOS `
 - multi-user?: `yes`
 - sandbox: `no`
 - version: `nix-env (Nix) 2.3.1`
 - channels(root): `"nixpkgs-20.03pre204216.cc6cf0a96a6"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixpkgs`

and it worked. But I have my doubts that this is the correct way, since I found this comment on discourse:

Another option to consider is to change the Nix single-user install script to set up the default profile to include the cacert package. The downside here is it would likely never get updated again unless the user intentionally mucks with the default profile.

Context

$  nix --version
nix (Nix) 2.3.1

[jkachmar]

I've also just run into this and was able to figure out that one of my installation runs left a dangling symlink or malformed file at /Library/LaunchDaemons/org.nixos.nix-daemon.plist.

This prevented one of the final steps of the multi-user installer from being run, and it would consistently error out with what you've been describing.

Once I ran my cleanup scripts, deleted that file, and then reran the multi-user installer everything seemed to work.

[jkachmar]

Also of note is now my user-defined Nix channels aren’t being picked up by default.

I’m assuming that there’s some stale bashrc config somewhere that’s loading up a global list of channels and ignoring them. If I manually add $HOME/.nix-defexpr/channels (I think? I’m going off of memory) to my NIX_PATH and then install everything I had setup with nix-darwin and home-manager then it all seems to work out.

When I load the shell installed and configured by home-manager, all the paths are set appropriately again and my channels are locked up as expected.

[mpfaff]

I had the same issue and your solution worked for me.

[mpfaff]

I spoke too soon. I just tried to install nix-darwin and the error came back.

$ nix-build https://github.com/LnL7/nix-darwin/archive/master.tar.gz -A installer
error: unable to download 'https://github.com/LnL7/nix-darwin/archive/master.tar.gz': Problem with the SSL CA cert (path? access rights?) (77)

[mrdomino]

I just had the same issue, and removing the old plist and reinstalling appears to have resolved it.

Seems like there are two problems. One, the installer's uninstall instructions should mention to remove /Library/LaunchDaemons/org.nixos.nix-daemon.plist (not just stop and unload it), and two, the installer should do something reasonable if it sees a non-matching one.

[CorbanR]

I ran into a very similar issue. I reran the multi-user OSX install script in an effort to upgrade nix from 2.3.3 to 2.3.4. The issue was caused (at least for me) by /etc/ssl/certs/ca-certificates.crt being a dead link. It was previously linked to /etc/static/...... Im not sure which project added /etc/static, nix-darwin or nix-home (I use both..and love both ;)). Fixing the symlink and running

sudo launchctl setenv NIX_SSL_CERT_FILE /etc/ssl/certs/ca-certificates.crt
sudo launchctl kickstart -k system/org.nixos.nix-daemon

fixed the issue for me. I initially got the same errors described in #2794 as well.

[domenkozar]

cc @LnL7

[wav]

... the installer's uninstall instructions should mention to remove /Library/LaunchDaemons/org.nixos.nix-daemon.plist

In my case (nix 2.3.5) there was a /Library/LaunchDaemons/org.nixos.activate-system.plist. I unloaded/removed it and reinstalled; afterwards the error Problem with the SSL CA cert was gone upon re-installation.

[maljub01]

I believe this was fixed in #4023. Perhaps someone here can confirm this?

[shlevy]

I'm having a different but seemingly related error... After installing Catalina, I reinstalled Nix with multiuser and added the line to the daemon conf that @maljub01 added. Now I'm getting SSL peer certificate or SSH remote key was not OK (60) when I try to hit the cache.

[shlevy]

This also seems to happen without the daemon... And occurs during installation when the channel is initially updated, without any failure message from the installer?

[shlevy]

Never mind, this is a problem with my work computer's environment, using their provided CA bundle fixed it.

[jdheyburn]

For those still with the issue after trying the above, check to see if this PR helps: #4171

[paulyoung]

I also had the dead symlink mentioned in #3261 (comment). This worked for me:

sudo ln -s /nix/var/nix/profiles/default/etc/ssl/certs/ca-bundle.crt /etc/ssl/certs/ca-certificates.crt

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/anyone-up-for-picking-at-some-nix-onboarding-improvements/13152/6

[stale]

I marked this as stale due to inactivity. → More info

[janvogt]

I didn't come accross this problem anymore. There are also aparent fixes and an unchecked solution. Since this issue lacks a clear definition of done, I am taking the freedom as OP to close it now. If someone objects, it can be reopened.

[jeff-hykin]

I also had the dead symlink mentioned in #3261 (comment). This worked for me:

sudo ln -s /nix/var/nix/profiles/default/etc/ssl/certs/ca-bundle.crt /etc/ssl/certs/ca-certificates.crt

I had to do the opposite 🙃 (MacOS Monterey, Nix 2.11.1, multi-user install (reinstall))

• After creating/confirming a /etc/ssl/certs/ca-certificates.crt
  and having export NIX_SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt didn't work/help
• the launchctl setenv NIX_SSL_CERT_FILE methods didn't work
• the /nix/var/nix/profiles/default/etc/ssl/certs/ folder didn't even exist

This was my solution

#
# create an up-to-date cert with keychain
#
sudo mkdir -p /etc/ssl/certs/
sudo rm -f /etc/ssl/certs/ca-certificates.crt
sudo security export -t certs -f pemseq -k /System/Library/Keychains/SystemRootCertificates.keychain -o /etc/ssl/certs/ca-certificates.crt
# force/ensure correct permissions on folders/files
sudo chown root /etc /etc/ssl /etc/ssl/certs/ /etc/ssl/certs/ca-certificates.crt
sudo chmod u=rwx,g=rx,o=rx /etc
sudo chmod u=rwx,g=rx,o=rx /etc/ssl
sudo chmod u=rwx,g=rx,o=rx /etc/ssl/certs/
sudo chmod u=rw,g=r,o=r  /etc/ssl/certs/ca-certificates.crt

#
# link cert into nix
#
target="/nix/var/nix/profiles/default/etc/ssl/certs/ca-bundle.crt" # not sure if different for single-user install
export NIX_SSL_CERT_FILE="/etc/ssl/certs/ca-certificates.crt"
sudo mkdir -p "$(dirname "$target")"
# if one exists back it up
[ -f "$target" ] && { [ -f "$target.old" ] || sudo mv "$target" "$target.old" 2>/dev/null; }
# link into nix profile
sudo ln -s "$NIX_SSL_CERT_FILE" "$target"

[mrtolkien]

Replying here to say I ran into the same issue today and the messages helped me fix it.

I don't feel like the issue should be closed until a good solution is found. Re-installing nix on a machine should not result in cryptic errors like an SSL certificate error.

[gshpychka]

I ran into a very similar issue. I reran the multi-user OSX install script in an effort to upgrade nix from 2.3.3 to 2.3.4. The issue was caused (at least for me) by /etc/ssl/certs/ca-certificates.crt being a dead link. It was previously linked to /etc/static/...... Im not sure which project added /etc/static, nix-darwin or nix-home (I use both..and love both ;)). Fixing the symlink and running

sudo launchctl setenv NIX_SSL_CERT_FILE /etc/ssl/certs/ca-certificates.crt
sudo launchctl kickstart -k system/org.nixos.nix-daemon

fixed the issue for me. I initially got the same errors described in #2794 as well.

This is still happening. Fixing the symlink was enough to fix the issue for me, though.

[bestlem]

I got this problem as well.
I did two fixes either which should work. The issue is to set NIX_SSL_CERT_FILE the launchctl setenv failed on my Ventura 13.4 as macOS blocked it due to SIP (or I made an error)

1. ZSH

I fixed it by changing the shell in the org.nixos.nix-daemon plist to /bin/zsh

Then moved the loading of /nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh from /etc/zshrc to /etc/zshenv This means that zsh will load the environment variables for non interactive shells.

I suspect this change should be made for other uses of zsh - zshenv is the correct place for environment variables - the only issue is altering the path in macOS as Apple messes it up.

So I think the full zsh solution is to do this move to /etc/zshenv and then also do a correction to the $PATH in /etc/zshrc after Apple's path_helper is run.

2. Just alter the environment variable

I edited the org.nixos.nix-daemon plist adding to the EnvironmentVariables tag already there. End result is

  <key>EnvironmentVariables</key>
   <dict>
   	<key>NIX_SSL_CERT_FILE</key>
   	<string>/nix/var/nix/profiles/default/etc/ssl/certs/ca-bundle.crt</string>
   	<key>OBJC_DISABLE_INITIALIZE_FORK_SAFETY</key>
   	<string>YES</string>
   </dict>

I think the latter is now the best solution

[caheredia]

@bestlem thank you. Your second solution is the only thing that worked for me. I've been experimenting with nix for three days now and it's been a wild ride thus far.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/ssl-ca-cert-error-on-macos/31171/5

[cvanlabe]

Same issue today. After following the uninstall instructions on MacOS (I was uninstalling as my system wasn't reflecting anything the documentation says), I couldn't get it to run at the nix-shell -p nix-info --run "nix-info -m" step mentioned by the installer:

...
error: unable to download 'https://cache.nixos.org/nar/1alkkz4r2l711j9wj9br4c3fbvf7mmvj7gn60gpdnbvvzmiqf8xs.nar.xz': Problem with the SSL CA cert (path? access rights?) (77)
copying path '/nix/store/v5ssy4lwf6jmffv1nqikbg8c4qhqym6n-libffi-3.4.6' from 'https://cache.nixos.org'...
copying path '/nix/store/mbs1jjyzy83vw85q8qiqj4r07z6ig4kw-openssl-3.0.13' from 'https://cache.nixos.org'...
error: some substitutes for the outputs of derivation '/nix/store/6y3knnwq4587k9ss025ir77d6ias1anm-bash-5.2p26.drv' failed (usually happens due to networking issues); try '--fallback' to build derivation from source
error: build of '/nix/store/3ys407788hnq08244fjijwix35za2mb7-stdenv-darwin.drv', '/nix/store/6y3knnwq4587k9ss025ir77d6ias1anm-bash-5.2p26.drv', '/nix/store/mqx8z4a2bhm8jq3sxilbrn0nq43ck1fa-bash-interactive-5.2p26.drv', '/nix/store/y8wb73lv7j1ryml818wvz86x9n8xycdw-nix-info.drv' failed
...

Steps above helped:

sudo rm /etc/ssl/certs/ca-certificates.crt
sudo ln -s /nix/var/nix/profiles/default/etc/ssl/certs/ca-bundle.crt /etc/ssl/certs/ca-certificates.crt

The "problem" now though is that if we uninstall nix (and /nix gets removed), /etc/ssl/certs/ca-certificates.crt is a dead symlink. I need to check where the original file came from. As per the above discussion, it seemed to have linked to /etc/static/.... On my system now /etc/static is a broken symlink too:

$ file /etc/static
/etc/static: broken symbolic link to /nix/store/l6h2gq5xjpc5vb832pqcz4iazcn98n14-etc/etc

I'm not immediately able with my still inexperienced nix knowledge to find the right folder in the store.

=> It seems the uninstall instructions are in need for a review.

[jeff-hykin]

I'm not immediately able with my still inexperienced nix knowledge to find the right folder in the store.

Try doing an ls -l on the file to see where the symlink is pointing to. I'd recommend deleting broken symlinks to just clear everything out and then do a fresh install

[cvanlabe]

Try doing an ls -l on the file to see where the symlink is pointing to. I'd recommend deleting broken symlinks to just clear everything out and then do a fresh install

Yeah.. figured that out after I had already deleted the broken symlink :-) I couldn't find any <...>-etc/ folder in the nix store. I wiped it all out, and then was able to find the <...>-etc/ directory on the nix store and how it all was symlinked.

The uninstall instructions should contain some verbiage on deleting the files in /etc too. I'll see if I can find some time and make a PR for this in the next couple of days.

[janvogt]

This seems to be an issue at least for some users. So it should be reopened.

[tomberek]

Occasionally running this works as well.

$ security find-certificate -a -p /System/Library/Keychains/SystemRootCertificates.keychain | sudo tee /etc/ssl/certs/ca-certificates.crt
$ security find-certificate -a -p /System/Library/Keychains/System.keychain | sudo tee -a /etc/ssl/certs/ca-certificates.crt
$ sudo chmod uog+r /etc/ssl/certs/ca-certificates.crt

Add something like this to the common error outputs? Or have nix doctor check for this problem and offer it?

[mnowotnik]

Had to run the same command as @tomberek , but also bundle the cert with /Library/Keychains/System.keychain. Got info from this post

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nix-darwin-installation-question/54502/3