installer: simplify the per-build installation

[zimbatm]

The goal is to allow the installation and testing of arbitrary Nix
versions. Extend the base installer to accept a --tarball-url-prefix <url> to
change where the Nix tarball is getting downloaded from.

Once this is merged it should allow to:

1. Pick an evaluation at https://hydra.nixos.org/jobset/nix/master that
   looks healthy
2. Select the installedScript build and find the store path.

Now equipped with all of this, use an instance of nar-serve to fetch the
install script and release tarballs:

curl -sfL https://nar-serve.numtide.com/nix/store/rkv4yh7pym941bhj0849zqdkg2546bdv-installer-script/install \
  | sh --tarball-url-prefix https://nar-serve.numtide.com/nix/store

Or with cachix, strip the /nix/store and derivation name and then:

curl -sfL https://mycache.cachix.org/serve/rkv4yh7pym941bhj0849zqdkg2546bdv/install \
  | sh --tarball-url-prefix https://mycache.cachix.org/serve

Fixes #4047

[zimbatm]

This should work in theory. It's a little bit hard for me to test this as it requires all the architectures to produce the installer script.

[zimbatm]

If that approach is not satisfactory, my other proposal would be to generate a second installer script that downloads the tarballs from https://nar-serve.numtide.com/

[matthewbauer]

So having to set NIX_RELEASE_URL is annoying, but this is definitely an improvement over not being able to easily install Nix via Hydra builds since the script hardcodes releases.nixos.org.

The main concern I have is that the hash is only going to match if NIX_RELEASE_URL is the same as the install script's Nix. Perhaps it shouldn't error if a custom NIX_RELEASE_URL is provided but the hash doesn't match.

I was brainstorming some alternatives that don't require users to set NIX_RELEASE_URL. It's kind of tricky - we know the hash of the tarball, so we could get the NAR of the Nix tarball from hydra.nixos.org, but that's in NAR, so we can't really extract it without a working Nix. Alternatively, we could upload all of the Hydra to tarballs.nixos.org, and just use that, but it only works if you are the official Hydra, so not so great.

A long term goal, I think, would be to bootstrap Nix with a statically built Nix binary. This installer would do the same stuff as the existing shell script, but also be embedded with a Nix expression to build itself dynamically. If that Nix expression is cached, it would be downloaded, otherwise Nix can build itself. This avoids the issue of relying on being in some official Nix release. But this is of course out of scope for this PR.

[edolstra]

Some issues:

• Environment variables like NIX_RELEASE_URL are not discoverable.

• It's not very user-friendly that you have to figure out two URLs and they have to correspond (e.g. https://hydra.nixos.org/build/1234/download/1/install and https://hydra.nixos.org/build/5678/download/1/XXX).

• Hydra is not very suitable as a release server. For instance, builds could be deleted from the database, store paths could be garbage collected. Also Hydra is slow.

Maybe a better approach is to install a stable Nix and then do nix-env -i /nix/store/... to install an unstable version from the binary cache.

[zimbatm]

There are some overlapping but distinct ideas:
a. Make it easier to test the install script. I'm thinking in particular the macOS one. In that case, I think it's okay to get temporary URLs that might break, even if it's a bit tricky.
b. Bootstrap Nix to any version. In that case, Eelco's approach of installing the stable Nix and then realizing the unstable one is okay. It would be nice if there was a one-shot approach available.

For NIX_RELEASE_URL, it could be replaced by a command-line argument.

What other ways forward is there?

[zimbatm]

Here is another proposal. It allows us to use nar-serve (cf commit message).

[zimbatm]

installer: allow nar-serve installations

The goal is to allow the installation and testing of arbitrary Nix
versions. Extend the base installer to accept a --tarball-url <url> to
change where the Nix tarball is getting downloaded from.

Once this is merged it should allow to:

1. Pick an evaluation at https://hydra.nixos.org/jobset/nix/master that
   looks healthy
2. Select the installedScript build and find the store path.

Now equipped with all of this, use an instance of nar-serve to fetch the
install script and release tarballs:

curl -sfL https://nar-serve.numtide.com/nix/store/rkv4yh7pym941bhj0849zqdkg2546bdv-installer-script/install \
  | sh --tarball-url https://nar-serve.numtide.com/nix/store

Or with cachix:

curl -sfL https://mycache.cachix.org/serve/rkv4yh7pym941bhj0849zqdkg2546bdv/install \
  | sh --tarball-url https://mycache.cachix.org/serve

[zimbatm]

fixed the merge conflict with the addition of Darwin.arm64

[zimbatm]

@edolstra is it still good like this? the PR is ready on my side.

[andersk]

This should have been updated to @tarballHash_x86_64-darwin@: #4769.

[abathur]

Good catch.

FWIW, Darwin.arm64 was added in #4243, which was opened after this PR and merged shortly before it--I'm pretty sure this was just a slip-up during merge-conflict resolution.

[abathur]

I suspect it won't happen since it hasn't already come up, but:

On the off chance anyone tries to backport this to 2.3-maintenance, the fix in #4769 needs to come along for the ride.