pkgs/stdenv: normalize attributes where setup.sh makes no distinction

[ghost]

This should wait until after staging for 22.11 has branched-off.

Description of changes

This PR normalizes the final attrset immediately before it is passed to builtins.derivation, deleting attributes whose value is [], "", or false if setup.sh treats that particular value for that particular attribute indentically to the attribute being absent.

As a result of this PR, it is possible to add, to an existing derivation, an attribute with the form:

  hardeningFlags = lib.optionals stdenv.hostPlatform.isFrobnicator [
    ...

without triggering an unnecessary mass-rebuild unless Frobnicator is one of the Hydra supportedPlatforms. Previously it was necessary to either use the non-idiomatic:

  if !stdenv.hostPlatform.isFrobnicator then null else [

or to trigger a mass-rebuild and suffer OfBorg's "red ovals of doom" on the PR. The latter case tends to lead to stale and forgotten PRs.

Recent example: #199978 (comment)

Things done

• Built on platform(s)
  • x86_64-linux (in progress)
  • powerpc64le-linux (in progress)
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 22.11 Release Notes (or backporting 22.05 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
  • (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes
• Fits CONTRIBUTING.md.

[roberth]

This will have a significant evaluation performance cost. It also increases the size of .drv files, making them less readable, but that might be a minor problem.

Possible optimizations:

• move the attrset of normalized attrs to a let binding on line 2, so that it is only constructed once
• apply the normalized attrs as late as possible, and don't return them in the package attribute set:

- derivationArgBeforeNormalization =
+ derivationArg =
    (removeAttrs attrs
      ["meta" "passthru" "pos"

-  (derivation derivationArg);
+  (legacyDerivationAttrs // derivationStrict (normalizedAttrs derivationArg))

legacyDerivationAttrs = ..... derivationArg;

legacyDerivationAttrs will need to be written to restore the (sub-optimal) functionality of derivation, that it returns its arguments alongside the proper derivation output attributes. We can't just break that, but we can elide the normalized attrs from it, so that the normalized attrs can be garbage collected, saving peak memory usage.

The implementation details won't be too complicated, but has to match what Nix does internally.

Also we're going to need non-automated benchmarks for this PR, because ofborg won't be providing its eval performance report. This one is a mass rebuild after all.

[roberth]

I can't promise that the performance cost can be overcome.

Perhaps at some point a fresh and more modular replacement for mkDerivation may fix this problem out of the box, but that's highly speculative and years from now.

[ghost]

@roberth, thank you so much for looking at this!

As you surely noticed, it currently does not work. As you probably also noticed, it turns out to be much more difficult than I initially expected.

I have to bump it down my priority list a few notches -- so it may be a month or two before I can revisit it.