minimal-bootstrap: use recursive FOD to make nix unpack bootstrap sources

[emilytrau]

Description of changes

This is an addendum to work done in #238357. Investigating it seems like the contents of the narball wasn't being unpacked, causing hex0 to not find sources for hex1 and hang. Switching to a recursive FOD seems like a solution?

Related #227914

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 23.11 Release Notes (or backporting 23.05 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

[Ericson2314]

It builds again, but unfortunately I get a segfault building glibc.

[zimbatm]

I'm not sure we fully understand the implications of this change. It's effectively changing the position of cache.nixos.org from being a cache to being a source for bootstrapping nixpkgs.

The advantage is pretty obvious, but I think there is a downside where bootstrapping becomes harder if cache.nixos.org were to disappear or lost those files.

[Ericson2314]

I would say it's probably better to worry about such things when we're ready to switch over to this.

tarballs.nixos.org can be a substituter for fixed output things with a little bit of work, native git hashing support should hopefully make fetching these sources easier, etc.

[ghost]

The advantage is pretty obvious, but I think there is a downside where bootstrapping becomes harder if cache.nixos.org were to disappear or lost those files.

If that happens, the build will fail with an error message which tells the user that they can resume the build without cache.nixos.org by running nix-build '<nixpkgs>' -A make-minimal-bootstrap-sources, which will be byte-for-byte identical to the file expected from cache.nixos.org (it's a nar archive containing only source code).

It's impossible to give instructions like this for our current binary-bootstrap scheme. That's one of the major benefits of the minimal-bootstrap work.

Ultimately, this will allow tarballs.nixos.org to be reduced to nothing more than a cache. We might someday be able to retire it and have cache.nixos.org take over all of its duties!

[zimbatm]

Isn't nix-build '<nixpkgs>' -A make-minimal-bootstrap-sources relying on the previous tarball to build itself?

[ghost]

Isn't nix-build '<nixpkgs>' -A make-minimal-bootstrap-sources relying on the previous tarball to build itself?

Keep reading....

nixpkgs/pkgs/os-specific/linux/minimal-bootstrap/stage0-posix/bootstrap-sources.nix

Lines 95 to 97 in 4179372

	# Or, if you prefer, you can create this file using only `git`,
	# `nix`, and `xz`. For the commands needed in order to do this,
	# see `make-bootstrap-sources.nix`. Once you have the manual

Of course, it would be better if we actually printed those lines, so the user could copy-paste them into the terminal:

  curl -L https://github.com/oriansj/stage0-posix/archive/3189b5f325b7ef8b88e3edec7c1cde4fce73c76c.tar.gz | tar -xvzf -
  mv stage0-posix-3189b5f325b7ef8b88e3edec7c1cde4fce73c76c stage0-posix-3189b5f325b7ef8b88e3edec7c1cde4fce73c76c-unstable-2023-05-02-source
  nix-store --add-fixed --recursive sha256 stage0-posix-3189b5f325b7ef8b88e3edec7c1cde4fce73c76c-unstable-2023-05-02-source

... but we would need a way to make sure these instructions stay "in sync" with the fetchFromGitHub expression used in make-minimal-bootstrap-sources.