lib.extendMkDerivation, lib.adaptMkDerivation: init

[ShamrockLee]

Description of changes

This PR introduces a unified approach to implement build helpers that support fixed-point arguments (lib.extendMkDerivation) and bring such support to existing build helpers (lib.adaptMkDerivation).

The fixed-point arguments support in stdenv.mkDerivation is introduced in #119942, and there are ongoing attempts to make other build helpers support such syntax (buildRustPackage refactor #194475, buildGoModule refactor #225051). The overlay style overrideAttrs brought by the stdenv.mkDerivation change can be used to implement the functionality, which is adopted by the buildRustPackage refactor and the previous version of the buildGoModule refactor. The challenge of such an approach is that the whole set pattern matching the input arguments are degenerated into a single identifier, making it hard to see from the source code which attributes to the build helper accepts.

The new Nixpkgs Library function, lib.extendMkDerivation accepts a base build helper and an attribute overlay (an overlay in the form finalAttrs: args: <attrsToUpdate>), and returns a new build helper by extending the base build helper with the attribute overlay via its <pkg>.overrideAttrs.

The following is the definition of an example build helper, mkLocalDerivation:

lib.extendMkDerivation { } stdenv.mkDerivation (
  finalAttrs:
  {
    preferLocalBuild ? true,
    allowSubstitute ? false,
    ...
  }@args:

  # No need of `// args` here.
  # The whole set of input arguments is passed in advance.
  {
    # Attributes to update
    inherit preferLocalBuild allowSubstitute;
  }
)

For existing build helpers with arguments that cannot be passed to the base build helper, another Nixpkgs library function, lib.adaptMkDerivation, is provided. Instead of an attribute overlay, it takes a function that turns the arguments of the result build helper to those of the base build helper, known as an argument set adapter. This way, developers can decide not to pass some arguments down to the base build helper.

lib.adaptMkDerivation { } stdenv.mkDerivation (
  finalAttrs:
  {
    preferLocalBuild ? true,
    allowSubstitute ? false,
    specialArg ? (_: false),
    ...
  }@args:

  removeAttrs [
    # Don't pass specialArg into mkDerivation
    "specialArg"
  ] args
  // {
    # Arguments to pass
    inherit preferLocalBuild allowSubstitute;
    # Some expressions involving specialArg
    greeting = if specialArg "hi" then "hi" else "hello";
  }
)

The reason to have two functions instead of just lib.adaptMkDerivation, is that we want to encourage (new) build helpers to pass all their input arguments properly down to stdenv.mkDerivation, and provide override functionality through the standardized <pkg>.overrideAttrs instead of custom overriders such as overridePythonAttrs, overrideNimAttrs, etc, by encouraging the use of lib.extendMkDerivation. In the meantime, providing lib.extendMkDerivation-incompatible build helpers with fixed-point arguments support (by lib.adaptMkDerivation) enables a smoother transition to a lib.extendMkDerivation-compatible style.

Fewer expression changes is another plus to convert existing build helpers with lib.adaptMkDerivation to take fixed-point arguments. Working build helper to support recursive attributes. A subsequent PR #271387 enables the fixed-point arguments support in buildPythonPackage and buildPythonApplication with slight modification against mk-python-derivation.nix with no rebuilds.

Both lib.extendMkDerivation and lib.adaptMkDerivation take a set of optional arguments to manipulate their behaviors. For example lib.adaptMkDerivation { modify = drv: toPythonModule drv; } applies toPythonModule to the derivation produced by the derived build helper.

Cc:
Python maintainers: @FRidh @mweinelt @jonringer
Author of the buildRustPackage refactor PR @amesgen
Reviewer of the buildGoModule refactor PR @zowoq
Author of the merged recursive stdenv.mkDerivation PR @roberth
People who mention 119942 in Python application definition: @LunNova

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 23.05 Release Notes (or backporting 22.11 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

[ShamrockLee]

I should have checked more carefully.

There's a function called makeOverride that deprives the input builder of their recursive attributes support, and buildPythonPackage and buildPackageApplication happens to be makeOverrided. While the change keeps the current python packages unchanged, it still doesn't bring the recursive attributes support to buildPythonModule and buildPythonApplication unless we fix makeOverridable.

Updat: fixed, see below.

[ShamrockLee]

Fix lib.makeOverridable and makeOverridablePythonPackage. Now buildPython* accepts recursive attributes.
Turn a Python application pyspread into the recursive attributes style.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/proposal-move-values-from-derivation-attributes-to-function-arguments/20697/21

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/avoid-rec-expresions-in-nixpkgs/8293/18

[roberth]

A mix of incremental and fundamental suggestions, because I can't decide for the nixpkgs-python maintainers.

I'm concerned about the increasing complexity, while the goal can be achieved through simplification instead: removing the python-specific level of overriding and turning it into an "overlay" on the mkDerivation arguments instead.

The alternative strategy is to

1. Provide this python layer, which contains the relevant mkPyton* logic in a way that works with overlay-style overriding. This can be done by reading the existing code, attribute for attribute, and adding the logic to the python layer.
2. Change the python packages to use that layer in combination with mkDerivation, instead of the current mkPython* functions.
3. Perhaps make the mkPython functions reuse the overlay so that they don't literally reimplement the same logic. I don't know if this would be worthwhile.
4. Eventually deprecate the mkPython* functions.

I have played around with step 1 of the alternative strategy. It is feasible, but it requires a bit of migration for each package. I'm not a nixpkgs-python maintainer, so I don't feel like I should be the one to make the decision whether to accept the complexity of this PR, or refactor to actually simplify the python logic by fitting it into a mkDerivation layer.

[roberth]

It's worth noting that finalAttrs contains the final arguments to mkDerivation attrs, and not the final arguments to the function you're constructing.

[ShamrockLee]

Now the arguments after applying the modifier function is also considered.

IMO, the finalAttrs is meant to be the final state of the attributes passing to the base builder (mkDerivation). That's how user could access finalAttrs.finalPackage and other goodies. If we just want to get the input argument set manually, the user could just lib.fix the function themselves.

Input arguments that doesn't mean to be passed to the base builder are subject to special care, and should never enter the recursion. Those not-to-pass arguments are also the reason why builder overlays are not drop-in replacement of current builders.

[ShamrockLee]

I tried to add those special arguments to finalAttrs, but that greatly increase the complexity of the code and makes behavior surprising (the specified result will be different from what is got through finalAttrs by design).

A better way would be to encourage passing all the arguments. When all the arguments are passed, they will all be available inside finalAttrs, and we could then switch to the overlay-based workflow specification (from the current, build-helper-based one).

[roberth]

This feels too custom or repeated (not sure yet which). Does this add mkDerivation-like fixpoint logic at the python attrs level?

I believe we should merge the python attrs level into the mkDerivation attrs, so that the interface and implementation become simpler. Having multiple levels of overriding has a huge complexity cost, so getting rid of an unnecessary level would be a huge win. We'd get rid of overridePythonAttrs and all the user facing complexity, implementation complexity and bugs that come with it.

The python-specific attrs can almost be implemented as an "overlay" on the mkDerivation arguments. When I tried this, I think only like 3 attributes had the same name but a slightly different meaning. That made it a breaking change, but migrating those attributes is feasible and would vastly simplify the python/mkDerivation wiring.

[FRidh]

With same names that is generally speaking because the Python builder will extend the lists with some "defaults". That's really the only value of the custom builder over just plain hooks.

[ShamrockLee]

This feels too custom or repeated (not sure yet which). Does this add mkDerivation-like fixpoint logic at the python attrs level?

I personally prefer passing most of the attributes into mkDerivation, encourage the use of overrideAttrs and gradually deprecates overridePythonAttrs as well as other builder-specific override methods.

That will be a mass rebuild, so I just work around the makeOverridablePythonPackage obstacle in order to demonstrate the possibility to add the recursive attributes support without rebuild.

[infinisil]

Recently the Packages Modules Working Group started investigating whether something like the module system could be used to evaluate packages. We're tracking all work in this repository, meeting weekly, but working mostly asynchronously. It would be great if you could join the Matrix room of the WG and chat with us, or even join the team yourself to work on such issues!

[ShamrockLee]

Thank you for taking time reviewing this!

I'm concerned about the increasing complexity, while the goal can be achieved through simplification instead: removing the python-specific level of overriding and turning it into an "overlay" on the mkDerivation arguments instead.

The idea to shift workflow-specific overlays sounds neat, and that could also be friendlier when packaging multi-language projects. Nevertheless, there are some issue on the way to the switch:

1. Current builders rely more or less on arguments they won't pass into mkDerivation attribute set.
   Developers would need to change the way they handle the arguments / attributes, either by passing them directly or through passthru. The impl input argument of the proposed function is essentially an overlay that allows passing not-to-pass-to-mkDerivation arguments through the previous set.

2. Current attempt to try to implement a builder that supports the (finalAttrs: { }) through an overlay passing to .overrideAttrs, such as rustPlatform.buildRustPackage: support finalAttrs style #194475, discards the set pattern of input arguments ({ a, b ? "some default", ... }@args) completely and degenerate it into previousAttrs.
   It is a pity, since the set patterns itself serves as an interface of the builder, providing information about the expected input arguments to both and the Nix interpreter and people reading the Nix expression.
   The solution would be rather simple -- replace the previousAttrs with the set pattern, and then the remaining issue will be point 1.

Overall, the goal of the proposed function is to add (finalAttrs: { }) input support with minimum changes to the existing expressions, and raise the discussion about general approaches to bring such supports to builders.

[fricklerhandwerk]

Suggested change

	A Builder, in the context of Nixpkgs, are functions that produces derivations. (Don't confuse it with the `builder` input argument of function `derivation`, which refers to the path to the executable that builds the derivation.) Such function is usually designed to follow a specific workflow, so that one can produce a package by specifying a few options instead of messing up with the `derivation` function.
	A *builder*, in the context of Nixpkgs, is a function that produces derivations.
	:::{.warning}
	This is not to be confused with the [`builder` argument to `derivation`](https://nixos.org/manual/nix/unstable/language/derivations.html), which refers to the executable that produces the build result.
	:::
	Such a builder is usually designed to follow a specific workflow, so that one can produce a package by setting a limited set of options relevant to the particular use case instead of using the `derivation` function directly.

This is terrible naming. The least we can do is highlight it with red alarms (not sure about admonition syntax, it's all different everywhere...) Since you're going at great lengths to improve documentation (which is absolutely awesome!) we may as well change that while at it. I'd certainly support it by bouncing around proposals. But of course feel free to ignore requests to widen scope.

[ShamrockLee]

Possible candidates:

• Build function

  • Existing use referring to the same thing

    • androidenv.buildApp doc/languages-frameworks/android.section.md ## Building an Android application {#building-an-android-application}

      This build function is particularly useful when it is desired to use
      Hydra: the Nix-based continuous integration solution
      to build Android apps.

    • xcodeenv.composeXcodeWrapper doc/languages-frameworks/ios.section.md ## Deploying a proxy component wrapper exposing Xcode {#deploying-a-proxy-component-wrapper-exposing-xcode}

      The first use case is deploying a Nix package that provides symlinks to the Xcode
      installation on the host system. This package can be used as a build input to
      any build function implemented in the Nix expression language that requires
      Xcode.

  • Existing use referring to something else

    • Inside the build process of pkgs/build-support/rust/build-rust-crate/build-crate.nix

      # configure & source common build functions

• Build helper

  • Existing use referring to the same thing

    • buildIntegration pkgs/tools/networking/dd-agent/integrations-core.nix

      # Build helper to build a single datadog integration package.

  • Existing use referring to something else

    • Inside the meta.description of the Haskell Hackage package hein pkgs/development/haskell-modules/hackage-packages.nix

      An extensible build helper for haskell, in the vein of leiningen

The different use of both candidates doesn't seem to cause confusion. We could just choose from one of them, or come up with something else.

[fricklerhandwerk]

Great research. I like "build helper" more, because its closest to what the thing is about, and also "build function" only contains one word that adds information (almost everything is a function).

When we change section titles we should make sure URLs stay stable. Either keep the anchors or add redirects (we discussed this with @pennae somewhat recently but I can't remember why the script used in the Nix manual was not included).

[pennae]

we discussed this with @pennae somewhat recently but I can't remember why the script used in the Nix manual was not included

there's not technical reason it couldn't be done, it's just that nobody has done the work yet. adding first-class support for fragment redirects to nixos-render-docs would be nicer and less error-prone, but we don't have to do that as a first step.

[FRidh]

I disagree that we should rename this from builder to something else and that "build helper" is better. Quite the contrary, the function when called actually builds something, which I'd say is exactly what a builder is. This naming has been around for a very long time.

The Nix attribute builder is a detail that no typical user would ever encounter. Those interested going through say the Pills or in detail through the Nix manual will see it, but otherwise unless you're interested in writing a custom stdenv.mkDerivation you won't do anything with it. Hence, I don't think that part should even be mentioned here.

[fricklerhandwerk]

Nix language functions don't build anything, not even the derivation primitive. What we typically mean by "build" is to run some process to produce files from other files.

[FRidh]

Following that line of reasoning "build helper" is also incorrect, because we're solely preprocessing before deriving. So in that case "deriver" or "deriver helper" or "derivation preprocessor" or "build planner" is more correct. Though actually they invoke derivation as well so they're not solely peprocessors either.

[fricklerhandwerk]

Clearly this needs discussion, but I don't want to block this PR on a single term. I'm happy with "build helper" because it reduces ambiguity and is not obviously wrong, and I'm not opposed to further improving the terminology in a separate PR.

[ShamrockLee]

Maybe we could move the builder naming to another issue or PR.

BTW, "builders" also refers to "build machines" in the context of "remote builders", which is probably why the documentation of darwin.builder is misplaced under the "Builders" part in the Nixpkgs manual. It would be great to find a more suitable place for it. (#235858)

[infinisil]

Did we ever have a user be confused about these being called builders? I at least haven't, and I don't think we should be even spending time discussing this unless there are users confused about it.

[fricklerhandwerk]

Where do we find information about extendDerivation?

[ShamrockLee]

lib.customisation is currently not presented in the Nixpkgs manual. We'll need to add the documentation for those functions before referring to them.

[ShamrockLee]

Besides, the Nixpkgs Library Functions documentation automatically generated from the comment don't have anchors. So there's no way to link against them so far.

[ShamrockLee]

My fault. The anchor in the form function-library-<long attribute path> is also automatically generated. E. g. function-library-lib.customisation.extendMkDerivation.

This usage is very hard to discover, as the documentation is missing, and the manual provides no hyperlink in the title of each function document.

[ShamrockLee]

I addressed this review in the changes to add a new section about using lib.extendMkDerivatien to define build helpers. Please take a look.

[LunNova]

Just reviewing your change to input-remapper, I like how much this cleaned up its definition. :)

[ShamrockLee]

extendMkDerivation and adaptMkDerivation now take an extra argument set to configure their behaviors. One can leave them empty (adaptMkDerivation { } stdenv.mkDerivation (finalAttrs: args: { ... })) for the default behavior. This change allows customization like adaptMkDerivation { modify = drv: ...; } stdenv.mkDerivation (finalAttrs: args: { ... }) without adding another variant like adaptMkDerivationModified.

I also adopted the relevant format for writing comment block documentation for lib.extendMkDerivation and lib.adaptMkDerivation and formatted all the examples inside fixed-point-arguments.chapter.md with nixfmt.

[jian-lin]

I do not know the motivation and use case of exposing attrsOverlay. Considering composability only, should the exposed overlay take mkDerivationBase into consideration? Something like

Suggested change

	inherit attrsOverlay;
	attrsOverlay = lib.composeExtensions mkDerivationBase.attrsOverlay or { } attrsOverlay;

[ShamrockLee]

I exposed attrsOverlay so that people could experiment with the idea of applying attribute overlays to overrideAttrs proposed by Robert in the review comment:

#234651 (review)

Considering composability, maybe we could change it to attrsOverlays?

{
  attrsOverlays = mkDerivationBase.attrsOverlays or [ ] ++ [ attrsOverlay ];
}

[ShamrockLee]

Addressed.

[ShamrockLee]

After more consideration, I think it would be easier to expose only attrsOverlay and mkDerivationBase. We could add more later if need be.

[jian-lin]

Suggested change

	(
	# Inherit the __functionArgs from the base build helper
	optionalAttrs inheritFunctionArgs (functionArgs mkDerivationBase)
	# Recover the __functionArgs from the derived build helper
	// functionArgs (attrsOverlay { })
	)
	(
	# Recover the __functionArgs from the derived build helper
	functionArgs (attrsOverlay { })
	# Inherit the __functionArgs from the base build helper
	// optionalAttrs inheritFunctionArgs (functionArgs mkDerivationBase)
	)

Note that overlay-a is before overlay-b (the reverse order of adaptMkDerivation).

extendMkDerivation { } (extendMkDerivation { } stdenv.mkDerivation overlay-a) overlay-b

---

Another question: If overlay-a sets one argument foo of functionArgs (overlay-b { }) and foo is not in functionArgs (overlay-a { }), should that argument foo be removed from the result __functionArgs? A similar question also applies to adaptMkDerivation.

[ShamrockLee]

should that argument foo be removed from the result __functionArgs?

Ideally, yes. However, there's no programmatic way to achieve such removal.

[ShamrockLee]

I rebased the branch to resolve merge conflicts, restructured the test.overriding test cases according to (#330434), and improved the documentation.

Outdated

[nyabinary]

Should probably target this for 25.05 (there a merge conflict anyways)

[ShamrockLee]

I'll rebase it before weekend.