Legacy stdenv.mkDerivation

[infinisil]

Issue description

The current base builder of nixpkgs is stdenv.mkDerivation, which has some weird legacy features:

• It is focused on C and Makefiles
• Derivation attributes are stringified and turned into environment variables
• Build phases are finnicky to override
• Source and patches are directly coupled with the builder
• Multitude of ways to specify build inputs
• Probably more

Goal

This issue is done when a better and more consistent stdenv.mkDerivation exists and is used as the base for all builders.

Project Board

[roberth]

Big project. Can add a bunch of context.

It is focused on C and Makefiles

It seems that a more minimal "stdenv", say unixenv, could loosen this. stdenv would become unixenv + some C/UNIX tools.

A minimal alternative to stdenv would satisfy the needs for minimal shell derivations as well.

• Brief mention: mkShellMinimal: Create an ultra minimal nix-shell #132617 (comment)

Derivation attributes are stringified and turned into environment variables

Mentioned this in structuredAttrs PR: to solve not just the stringification (goal of structuredAttrs) but tackle as well the design problem that unrecognized attrs are turned into environment variables.

• Brief mention: stdenv: support opt-in __structuredAttrs #175649 (comment)

Build phases are finnicky to override

User interface can be simplified:

• Fix the problem of forgotten runHook {pre,post}* calls #175605

Source and patches are directly coupled with the builder

Also there's a divide between sources+patches and sources+filters.

CA-derivations blurs the line between eval-time source filtering and build-time source filtering, though we'll want to keep source filtering in order not to be strict in huge subdirectories that are ignored. Ideally we'll unify these concepts.

Source filtering itself should be improved as well

• source combinators: extend, trace, cutAt, focusAt and more #112083
  • new operations
  • redefinition of source to include "focus" (what to cd into)

Multitude of ways to specify build inputs

Referring to the input roles required for cross compilation?
Seems to be more an implementation problem.

Solving "Derivation attributes are stringified and turned into environment variables" somewhat reduces this as well.

Probably more

RFC92 fundamentally decouples package-ness from derivation-ness: only the first derivation exists at eval time, whereas the outputs come from the produces derivation.

• Illustration: [RFC 0092] Computed derivations rfcs#92 (comment)

Defining "package" (as an attrset that goes beyond the derivation itself)

• Proposed definition: Redefine outputs in terms of language-level "package", not necessarily store-level derivation (RFC-92, and multi-drv packages, docs) nix#6507

[FRidh]

Brief overview of issues I encounter with stdenv.mkDerivation. https://gist.github.com/FRidh/be0065018726af6651fe5f435d9569b6

[roberth]

Module style composition would be nice.
It allows the mkDerivation logic (huge) to be split up into module, allow certain bits to be reused between simple packages and RFC-92 packages and improve a number of small annoyances like #190577

The module system we have is a bit too complicated and possibly a little slow for the purpose of packaging. A module system with only the features we need would be simpler; for example: https://gist.github.com/roberth/940dff88ca5f5f95949dc309dbe60a65

[aakropotkin]

A possible example module. I use Autoconf style : "${FOO:=foo}"; declarations out of habit; but in the case of PATCH_SHEBANGS I do actually use it to conditionally use a standalone patch-shebangs.sh script.

Adding JQ = "${pkgs.jq}/bin/jq"; to a derivation satisfies these references with absolute paths. If undeclared they use the raw tool name.

https://github.com/aameen-tulip/at-node-nix/blob/main/pkgs/build-support/setup-hooks/pjs-util.sh

[danielbarter]

As an aside, guix handles the second point really nicely. The store derivations are generated by scheme, and the builders are scheme interpreters (rather than bash in the nixpkgs case). The attributes are just s-expressions, so they can be passed to and handled by the builders naturally.

[roberth]

A modular mkDerivation would only provide the defaults it needs, but provide them nonetheless, making it a potentially simpler and more efficient solution for the problem to be solved in #201131 which can be summarized by the requirement:

Setting a "well-known" mkDerivation argument to its empty and default value should have no effect on the derivation hash.

[aakropotkin]

Just to clarify, do you mean "it does not change the hash with existing implementations", or "after we close this issue we will ensure that empty/default args will no longer change the hash"?

…

On Thu, Nov 17, 2022, 9:33 AM Robert Hensing ***@***.***> wrote: A modular mkDerivation would only provide the defaults it needs, but provide them nonetheless, making it a potentially simpler and more efficient solution for the problem to be solved in NixOS/nixpkgs#201131 <#201131> which can be summarized by the requirement: Setting a mkDerivation argument to its empty and default value should have no effect on the derivation hash. — Reply to this email directly, view it on GitHub <https://github.com/nixpkgs-architecture/issues/issues/3#issuecomment-1318812550>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA3ORYR2QQHMCH4PIUM6RBTWIZF5RANCNFSM535TQOKA> . You are receiving this because you commented.Message ID: ***@***.***>

[roberth]

"it does not change the hash with existing implementations"

This is not the case yet, as it depends on the fate of #201131

"after we close this issue we will ensure that empty/default args will no longer change the hash"

I would say yes, but I don't think this issue is well categorized, as a modular mkDerivation is quite breaking, has a lot of contact area with #295039 and doesn't necessarily require stdenv changes like some of the other suggestions here. I think these broad issues are somewhat suitable for gathering requirements and ideas, but not necessarily as a project management tool.

[fgaz]

Multitude of ways to specify build inputs

Though it's more of an implementation detail, I'll link this comment of mine since it's related to that problem: #204303 (comment)

[roberth]

The lack of package-local variables came up again here when supporting overlay-style mkDerivation in a language integration.

Whatever we use to construct packages should have a namespace where generic code can put or look for information that doesn't automatically end up in the build sandbox and affecting hashes.