-
-
Notifications
You must be signed in to change notification settings - Fork 14.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix splicing in overrideAttrs #201734
base: master
Are you sure you want to change the base?
fix splicing in overrideAttrs #201734
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The root cause of the problem is doing //
on an attrset that is supposed to be modified with .overrideAttrs
instead.
This happens here:
nixpkgs/pkgs/top-level/splice.nix
Lines 50 to 52 in cf01bae
// (lib.optionalAttrs (pkgsBuildHost ? ${name}) { nativeDrv = valueBuildHost; }) | |
// (lib.optionalAttrs (pkgsHostTarget ? ${name}) { crossDrv = valueHostTarget; }) | |
// { |
What this code should do instead, is overrideAttrs
with passthru.__spliced
. That way the splicing is preserved, but broken, because overrideAttrs
doesn't implement your logic yet. For this, we need to make overrideAttrs
overridable. You may cherry-pick hercules-ci@43c8b43 for that. Here's a crude example showing that splicing can be made to work:
{ pkgs ? import ./. { }, ... }:
rec {
inherit pkgs;
inherit (pkgs) lib;
addEntangled = origOverrideAttrs: f:
origOverrideAttrs (
lib.composeExtensions f (self: super: {
passthru = super.passthru // {
entangled = super.passthru.entangled.overrideAttrs f;
overrideAttrs = addEntangled self.overrideAttrs;
};
})
);
entangle = pkg1: pkg2: pkg1.overrideAttrs (self: super: {
passthru = super.passthru // {
entangled = pkg2;
overrideAttrs = addEntangled self.overrideAttrs;
};
});
example = entangle pkgs.hello pkgs.figlet;
overrides1 = example.overrideAttrs (self: super: { pname = "a-better-${super.pname}"; });
repeatedOverrides = overrides1.overrideAttrs (self: super: { pname = "${super.pname}-with-blackjack"; });
test =
assert repeatedOverrides.pname == "a-better-hello-with-blackjack";
assert repeatedOverrides.entangled.pname == "a-better-figlet-with-blackjack";
"ok";
}
ea481e8
to
1a44bd8
Compare
I'll add that it's very crude. I've untangled the entangle function a bit by floating out |
1a44bd8
to
3b188e9
Compare
Thank you @roberth!! This test passes let
pkgs = ((builtins.getFlake (toString ./.)).legacyPackages.${builtins.currentSystem});
pkgsCross = pkgs.pkgsCross.aarch64-multiplatform.__splicedPackages;
lib = pkgs.lib;
test = let
pythonInNativeBuildInputs = lib.elemAt pkgsCross.python3Packages.xpybutil.nativeBuildInputs 0;
overridenAttr = pkgsCross.hello.overrideAttrs (_: { something = "hello"; });
#overridenAttr = (pkgsCross.hello.overrideAttrs (previousAttrs: { something = "hello"; })) // { __spliced = { buildHost = { something = "hello"; }; };};
in {
shouldBeNative = pythonInNativeBuildInputs.stdenv.hostPlatform.system == "x86_64-linux";
notCrossOverriden = pkgs.hello.overrideAttrs (_: _: { passthru = { o = "works"; }; });
inherit overridenAttr;
overridenAttrShouldHaveSpliced = overridenAttr ? __spliced;
overridenAttrShouldHaveSplicedAndSomething = if overridenAttr ? __spliced then overridenAttr.something == overridenAttr.__spliced.buildHost.something else false;
};
in
assert test.shouldBeNative == true;
assert test.overridenAttrShouldHaveSpliced == true;
assert test.overridenAttrShouldHaveSplicedAndSomething == true;
assert test.notCrossOverriden.o == "works";
test |
output getting broken now
after
hmm
|
Pushed a fix.
The TODOs in it are out of scope for this PR. |
Awesome, thanks. |
@@ -19,9 +19,9 @@ let | |||
ff = f origArgs; | |||
overrideWith = newArgs: origArgs // (if pkgs.lib.isFunction newArgs then newArgs origArgs else newArgs); | |||
in | |||
if builtins.isAttrs ff then (ff // { | |||
if builtins.isAttrs ff then ff.overrideAttrs (previousAttrs: { passthru = (previousAttrs.passthru or {}) // { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For performance, it'd be better to composeExtensions
(or similar) this to the origArgs
. That way we don't evaluate all python packages twice.
figured out that this is caused by these nixpkgs/pkgs/stdenv/generic/make-derivation.nix Lines 209 to 240 in dbc68fa
before we just chose the overriden drv because there were no __spliced after |
1c057d2
to
5b9bf35
Compare
args = lib.fix (lib.extends | ||
(_: previousAttrs: { | ||
passthru = (previousAttrs.passthru or { }) // { | ||
overridePythonAttrs = newArgs: makeOverridablePythonPackage f (overrideWith newArgs); | ||
}; | ||
}) | ||
(_: origArgs)); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
is this correct @roberth ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think the ordering of the extends
arguments is ok. "Self" is ignored, so I think the types line up (but I didn't prove it).
Looks about right.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Performance has improved by 2-3%, judging by memory stats before and after "use extends instead of overrideAttrs".
Not enough to make python changes as cheap as the other changes though.
I wonder how much change/noise is caused by master
, which I believe is merged into the feature branch before ofborg evaluates.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
rebased on master again
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should probably bench this manually anyway, since cross is not part of ofborg's eval I think.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
b5d2d22
to
8dcf2d3
Compare
2b58588
to
30a0f5b
Compare
53f01cc
to
a2cee1a
Compare
lib/customisation.nix
Outdated
# is this even necessary? | ||
# nix-diff shows no difference in cross/non | ||
# nix-diff $(nix eval --raw "nixpkgs/$(git merge-base upstream/master HEAD)#nix.drvPath") $(nix eval --raw ".#nix.drvPath") | ||
# nix-diff $(nix eval --raw "nixpkgs/$(git merge-base upstream/master HEAD)#pkgsCross.aarch64-multiplatform.nix.drvPath") $(nix eval --raw ".#pkgsCross.aarch64-multiplatform.nix.drvPath") | ||
#overrideAttrs = fdrv: | ||
# overrideResult (x: x.overrideAttrs fdrv); | ||
#} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the overrideAttrs attr was added in commit 39b64b5
the commit message says
Also update
makeOverridable
so that uses ofoverrideAttrs
can be
followed byoverride
andoverrideDerivation
, i.e. they can be
mix-and-matched.
but it seems not necessary anymore
nix-repl> (bash.overrideAttrs(_: { x = "y"; }))
«derivation /nix/store/vk0b51d4bq0ciyrr83nvgrbf51lp5f7s-bash-5.2-p15.drv»
nix-repl> (bash.overrideAttrs(_: { x = "y"; })).override { withDocs = true; }
«derivation /nix/store/y11vah7v46mah4yjj3yj7161fcaq7lkq-bash-5.2-p15.drv»
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
no overrideAttrs after .override so something like this is necessary
nix-repl> p = (pkgsCross.aarch64-multiplatform.__splicedPackages.bash.overrideAttrs(_: { x = "y"; })).over
ride { withDocs = true; }
nix-repl> p.passthru
{ override = { ... }; overrideDerivation = «lambda @ /home/hm7/nixgits/my-nixpkgs/lib/customisation.nix:96:34»; shellPath = "/bin/bash"; }
just uncommenting it gives
$ nix build ".#sway" --show-trace
error:
… while calling the 'derivationStrict' builtin
at <nix/derivation-internal.nix>:9:12:
8|
9| strict = derivationStrict drvAttrs;
| ^
10|
… while evaluating the attribute 'paths' of the derivation 'sway-1.8'
… while evaluating the attribute 'text' of the derivation 'sway'
… while evaluating the attribute 'buildInputs' of the derivation 'sway-unwrapped-1.8'
… while evaluating the attribute 'buildInputs' of the derivation 'pango-1.50.12'
… while evaluating the attribute 'buildCommand' of the derivation 'gobject-introspection-wrapped-1.74.0'
… from call site
at /home/artturin/nixgits/my-nixpkgs/pkgs/development/libraries/gobject-introspection/wrapper.nix:100:11:
99| eval fixupPhase
100| ${lib.concatMapStrings (output: ''
| ^
101| mkdir -p ${"$" + "${output}"}
… while calling 'concatMapStrings'
at /home/artturin/nixgits/my-nixpkgs/lib/strings.nix:55:25:
54| */
55| concatMapStrings = f: list: concatStrings (map f list);
| ^
56|
… while calling anonymous lambda
at /home/artturin/nixgits/my-nixpkgs/pkgs/development/libraries/gobject-introspection/wrapper.nix:100:33:
99| eval fixupPhase
100| ${lib.concatMapStrings (output: ''
| ^
101| mkdir -p ${"$" + "${output}"}
… while evaluating the attribute 'buildInputs' of the derivation 'gobject-introspection-1.74.0'
… while calling 'getOutput'
at /home/artturin/nixgits/my-nixpkgs/lib/attrsets.nix:827:23:
826| */
827| getOutput = output: pkg:
| ^
828| if ! pkg ? outputSpecified || ! pkg.outputSpecified
… while calling anonymous lambda
at /home/artturin/nixgits/my-nixpkgs/pkgs/stdenv/generic/make-derivation.nix:228:13:
227| (map (drv: drv.__spliced.hostHost or drv) (checkDependencyList "depsHostHost" depsHostHost))
228| (map (drv: drv.__spliced.hostTarget or drv) (checkDependencyList "buildInputs" buildInputs))
| ^
229| ]
… while calling anonymous lambda
at /home/artturin/nixgits/my-nixpkgs/lib/lists.nix:117:29:
116| */
117| imap1 = f: list: genList (n: f (n + 1) (elemAt list n)) (length list);
| ^
118|
… from call site
at /home/artturin/nixgits/my-nixpkgs/lib/lists.nix:117:32:
116| */
117| imap1 = f: list: genList (n: f (n + 1) (elemAt list n)) (length list);
| ^
118|
… while calling anonymous lambda
at /home/artturin/nixgits/my-nixpkgs/pkgs/stdenv/generic/make-derivation.nix:199:81:
198| checkDependencyList = checkDependencyList' [];
199| checkDependencyList' = positions: name: deps: lib.flip lib.imap1 deps (index: dep:
| ^
200| if lib.isDerivation dep || isNull dep || builtins.typeOf dep == "string" || builtins.typeOf dep == "path" then dep
… from call site
at /home/artturin/nixgits/my-nixpkgs/pkgs/stdenv/generic/make-derivation.nix:200:8:
199| checkDependencyList' = positions: name: deps: lib.flip lib.imap1 deps (index: dep:
200| if lib.isDerivation dep || isNull dep || builtins.typeOf dep == "string" || builtins.typeOf dep == "path" then dep
| ^
201| else if lib.isList dep then checkDependencyList' ([index] ++ positions) name dep
… while calling 'isDerivation'
at /home/artturin/nixgits/my-nixpkgs/lib/attrsets.nix:571:5:
570| # Value to check.
571| value: value.type or null == "derivation";
| ^
572|
… from call site
at /home/artturin/nixgits/my-nixpkgs/pkgs/development/libraries/gobject-introspection/default.nix:84:6:
83| buildInputs = [
84| (python3.withPackages pythonModules)
| ^
85| ];
… while calling anonymous lambda
at /home/artturin/nixgits/my-nixpkgs/pkgs/development/interpreters/python/with-packages.nix:3:1:
2|
3| f: let packages = f pythonPackages; in buildEnv.override { extraLibs = packages; }
| ^
4|
… from call site
at /home/artturin/nixgits/my-nixpkgs/lib/trivial.nix:430:7:
429| { # TODO: Should we add call-time "type" checking like built in?
430| __functor = self: f;
| ^
431| __functionArgs = args;
… while calling anonymous lambda
at /home/artturin/nixgits/my-nixpkgs/lib/customisation.nix:88:32:
87| # Re-call the function but with different arguments
88| overrideArgs = copyArgs (newArgs: makeOverridable f (overrideWith newArgs));
| ^
89| # Change the result of the function call by applying g to it
… from call site
at /home/artturin/nixgits/my-nixpkgs/lib/customisation.nix:88:41:
87| # Re-call the function but with different arguments
88| overrideArgs = copyArgs (newArgs: makeOverridable f (overrideWith newArgs));
| ^
89| # Change the result of the function call by applying g to it
… while calling 'makeOverridable'
at /home/artturin/nixgits/my-nixpkgs/lib/customisation.nix:78:24:
77| */
78| makeOverridable = f: origArgs:
| ^
79| let
… from call site
at /home/artturin/nixgits/my-nixpkgs/lib/trivial.nix:430:7:
429| { # TODO: Should we add call-time "type" checking like built in?
430| __functor = self: f;
| ^
431| __functionArgs = args;
… while calling anonymous lambda
at /home/artturin/nixgits/my-nixpkgs/lib/customisation.nix:90:54:
89| # Change the result of the function call by applying g to it
90| overrideResult = g: makeOverridable (copyArgs (args: g (f args))) origArgs;
| ^
91|
… from call site
at /home/artturin/nixgits/my-nixpkgs/lib/customisation.nix:90:60:
89| # Change the result of the function call by applying g to it
90| overrideResult = g: makeOverridable (copyArgs (args: g (f args))) origArgs;
| ^
91|
… while calling anonymous lambda
at /home/artturin/nixgits/my-nixpkgs/lib/customisation.nix:111:29:
110| overrideAttrs = fdrv:
111| overrideResult (x: x.overrideAttrs fdrv);
| ^
112| }
error: function 'anonymous lambda' called with unexpected argument 'extraLibs'
at /home/artturin/nixgits/my-nixpkgs/pkgs/build-support/buildenv/default.nix:8:2:
7| lib.makeOverridable
8| ({ name
| ^
9|
This reverts commit a2cee1acda1f6a07ba2092147075f8812292acf5.
see commit messages
Things done
sandbox = true
set innix.conf
? (See Nix manual)nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD"
. Note: all changes have to be committed, also see nixpkgs-review usage./result/bin/
)nixos/doc/manual/md-to-db.sh
to update generated release notes