Lazy `fetchTree` `outPath` path values #10252

roberth · 2024-03-15T19:02:11Z

Motivation

Improve performance, and make the fetchTree interface more capable while keeping it clean.

Description

This makes fetchTree return lazy InputAccessor-based SourcePaths instead of "cowardly" fetching them to the store and returning absolute "system" paths.
It stays close to existing path semantics, including support for readFile "${toString p}/..", which some expressions rely on.
It does not go as far as lazy-trees, but judging from the amount of change I could reuse, and how little of my own I had to add, lazy-trees will be a natural extension of this PR.

Done:

Packages and NixOS evaluate as usual
- no hash changes, based on my limited testing
Flake is not added to store unless e.g.
- "${flake.outPath}"
- uses module system (needs clever lazy source strings, or a change to the module system)
Note that the above is already an improvement over the status quo - always fetching to the store
iirc 0.1s reduction on nixpkgs#hello, and 1s reduction on nixosTests.simple

Conclusion so far:
Viable

TODO:

Check the TODO and FIXMEs
Update the test suite
- probably some intended behavior change, such as removal of narHash in some observable places
- possibly finds a bug
Check performance again
Cherry-pick the toString path behavior to lazy-trees

Context

Priorities and Process

Add 👍 to pull requests you find important.

The Nix maintainer team uses a GitHub project board to schedule and track reviews.

roberth · 2024-03-15T19:03:00Z

src/libcmd/common-eval-args.cc

These changes are close to lazy-trees.

roberth · 2024-03-15T19:04:33Z

src/libexpr/eval.cc

+void EvalState::registerAccessor(const ref<InputAccessor> accessor)
+{
+    inputAccessors.push_back(accessor);
+}


This is like lazy-trees, except we only do it so that we don't destroy them when we put a non-smart pointer in Value, which has no finalizer because of GC and performance.

roberth · 2024-03-15T19:05:13Z

src/libexpr/eval.cc

@@ -1973,6 +1978,17 @@ void EvalState::concatLists(Value & v, size_t nrLists, Value * * lists, const Po
    }
 }

+// FIXME limit recursion


Known issue, solve later with #10240

roberth · 2024-03-15T19:07:23Z

src/libexpr/eval.cc

+        Value & vTmp0 = *vTmpP++;
+        i->eval(state, env, vTmp0);
+        Value & vTmp = *resolveOutPath(state, &vTmp0, i_pos);

        /* If the first element is a path, then the result will also
           be a path, we don't copy anything (yet - that's done later,
           since paths are copied when they are used in a derivation),
           and none of the strings are allowed to have contexts. */
        if (first) {
            firstType = vTmp.type();
+            if (firstType == nPath) {
+                accessor = vTmp.path().accessor;
+            }


Similar but not equal to lazy-trees.

I've kept the diff smaller by keeping vTmp as a reference

Not adding the first name to the path, because it is added again later. Probably a bug in lazy-trees.

roberth · 2024-03-15T19:09:22Z

src/libexpr/eval.cc

+            : v.path().accessor->toStringReturnsStorePath()
+            ? store->printStorePath(copyPathToStore(context, SourcePath(v.path().accessor, CanonPath::root))) + v.path().path.absOrEmpty()


This is new, in order for readFile "${toString ./.}/.." to work, just as it did before.
Would not recommend to write that, but similar usages of paths exist in the wild.

roberth · 2024-03-15T19:10:58Z

src/libexpr/eval.cc

+        auto i = v.attrs->find(sOutPath);
+        if (i != v.attrs->end()) {
+            return coerceToPath(pos, *i->value, context, errorCtx);
+        }


This rule is not new. Previously, it would have worked by falling through to the coerceToString + rootPath code down below.

roberth · 2024-03-15T19:11:36Z

src/libexpr/nixexpr.hh

+        v.mkPath(
+            &*path.accessor,
+            // TODO: GC_STRDUP
+            strdup(path.path.abs().c_str()));


Like lazy-trees, but I had to add strdup to avoid corruption.

roberth · 2024-03-15T19:12:45Z

src/libexpr/primops/fetchTree.cc

@@ -201,18 +201,31 @@ static void fetchTree(

    state.checkURI(input.toURLString());

-    auto [storePath, input2] = input.fetchToStore(state.store);
+    if (params.returnPath) {


Like #10225, but clang was lagging behind GCC's C++20.

roberth · 2024-03-15T19:14:20Z

src/libutil/source-accessor.hh

+     * In both cases, the returned string functionally identifies the path,
+     * and can still be read.
+     */
+    virtual bool toStringReturnsStorePath() const;


The only truly new semantics.

Either way, both kinds of paths are virtual in the sense that they haven't been copied yet.
If you have copied it, it'd be a string. (Not if and only if, although Nix does discourage that.)

In lazy-trees, this could be changed to a virtual store path string without a problem.

TODO: forward port this rule to prove it.

#10511 (comment) I think rather contain information with which to construct the path, if I understand what is going on here correctly.

@Ericson2314 these are solutions to different problems.

Here I am adding a property to distinguish the behavior between "system" path values and virtual path values in the language.

Linked comment seems to be about optimizing away a copy operation, which iiuc is internal to a fetcher and not perceptible by users.

Both solutions (to different problems!) are trying to solve concerns in the upper layers though.

I'll check if this one can be moved into the evaluator. We could probably just special case the system (ie posix accessor) paths there.

Btw, here I've picked the least breaking semantics:

The return value still uniquely identifies the source

Still deterministic

Still readable when converted to a system path with /. + x or /${x}

Performance is at least as good as the status quo, but not as good as lazy trees, because for toString x to be instant, you need to sacrifice one of the above, or come up with a clever scheme, like opaque placeholders in strings or something.

ConnorBaker · 2024-04-04T18:55:11Z

@roberth is there anything I (or someone in general) could do to try to help move this forward? Thank you for working on this :)

This picks a number of changes from the lazy-trees branch. As it is hand picked, and does not include some other necessary changes, it does not build. Subsequent commits will fix that. I have added a couple of comments of my own as well. Co-authored-by: Eelco Dolstra <edolstra@gmail.com>

This fixes the double copy problem and improves performance for expressions that don't force the whole source to be added to the store. Rules for fast expressions: - Use path literals where possible - import ./foo.nix - Use + operator with slash in string - src = fetchTree foo + "/src"; - Use source filtering, lib.fileset - AVOID toString - If possible, AVOID interpolations ("${./.}") - If possible, move slashes into the interpolation to add less to the store - "${./src}/foo" -> "${./src/foo}" toString may be improved later as part of lazy-trees, so these recommendations are a snapshot. Path values are quite nice though.

This allows clever editors/IDEs to discern the path more easily for Ctrl+Click navigate to functionality, e.g. when building .?ref=HEAD

This showPath is getting a little too ad hoc, but it works for now.

roberth · 2024-04-16T16:21:29Z

@roberth is there anything I (or someone in general) could do to try to help move this forward? Thank you for working on this :)

Hi @ConnorBaker,

Sorry for the late response; I had to take another look at this first, and it took a while to get around to it.
I've rebased the branch, but I seem to have broken the search path somehow.

I see two ways forward, either

re-do the PR with reduced scope
- don't remove the narHash, if possible; should keep more flakes code the same
- carefully look whether other changes were absolutely necessary
keep going; just fix the remaining test failures, and also remove the commit delay the flake outPath semantics change for now

nixos-discourse · 2024-05-07T13:06:04Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2024-04-29-nix-team-meeting-minutes-142/45020/1

nixos-discourse · 2024-07-17T21:10:34Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2024-07-15-nix-team-meeting-minutes-161/49228/1

nixos-discourse · 2024-08-07T22:53:41Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2024-08-07-nix-team-meeting-minutes-167/50287/1

tomberek · 2024-08-08T02:41:44Z

Reverting the last commit (4332b9a) gives us the following comparison with vanilla Nix:

Before

$ nix eval .#data --no-eval-cache | nixfmt
warning: Git tree '/home/tom/nix/t' is dirty
{
  fetchTree = "/nix/store/3a07hs5zz57xf76gqq53i9jkmi3mhyp6-source";
  fetchTreePath = "/nix/store/3a07hs5zz57xf76gqq53i9jkmi3mhyp6-source/ci";
  fetchTreePathStr = "/nix/store/3a07hs5zz57xf76gqq53i9jkmi3mhyp6-source/ci";
  originalStr = "/nix/store/3a07hs5zz57xf76gqq53i9jkmi3mhyp6-source";
  outPathRaw = "/nix/store/3a07hs5zz57xf76gqq53i9jkmi3mhyp6-source";
  outPathStr = "/nix/store/3a07hs5zz57xf76gqq53i9jkmi3mhyp6-source";
  pathRaw = /nix/store/3a07hs5zz57xf76gqq53i9jkmi3mhyp6-source;
  pathRawAdd = /nix/store/3a07hs5zz57xf76gqq53i9jkmi3mhyp6-source/ci;
  pathRawAddStr = "/nix/store/z0qs96vamg1r2ch0rml9pmsn8f002hvw-ci";
  pathStr = "/nix/store/rslrjkrdgd2ggxmlyckc53nv0pxjq5qj-3a07hs5zz57xf76gqq53i9jkmi3mhyp6-source";
}
$ nix eval .#data --no-eval-cache -vvv |& grep copying.*-source
...
copying '/nix/store/3a07hs5zz57xf76gqq53i9jkmi3mhyp6-source/pkgs/development/libraries/glibc/nix-nss-open-files.patch' to the store...
copying '/nix/store/3a07hs5zz57xf76gqq53i9jkmi3mhyp6-source/pkgs/development/libraries/glibc/0001-Revert-Remove-all-usage-of-BASH-or-BASH-in-installed.patch' to the store...
copying '/nix/store/3a07hs5zz57xf76gqq53i9jkmi3mhyp6-source/pkgs/development/libraries/glibc/reenable_DT_HASH.patch' to the store...
copying '/nix/store/3a07hs5zz57xf76gqq53i9jkmi3mhyp6-source/ci' to the store...
copying '/nix/store/3a07hs5zz57xf76gqq53i9jkmi3mhyp6-source' to the store...

After (only the differences)

This has fewer copies to the store, going from 6s to 0.6s!

$ nix eval .#data --no-eval-cache | tr -cd '[[:print:]]' |nixfmt      # because of "»" characters
{
  fetchTree = "github:NixOS/nixpkgs/086a5ea5b3acc4c512f9ec154bfefba55efba4f3?narHash=sha256-LyZtQZiq2v2We5ODev6s9s2iUHNu/ZC8rHIYRh1BIzg%3D:";
  fetchTreePath = "github:NixOS/nixpkgs/086a5ea5b3acc4c512f9ec154bfefba55efba4f3?narHash=sha256-LyZtQZiq2v2We5ODev6s9s2iUHNu/ZC8rHIYRh1BIzg%3D:ci";
  fetchTreePathStr = "/nix/store/z0qs96vamg1r2ch0rml9pmsn8f002hvw-ci";
  originalStr = ...
  outPathRaw = "github:NixOS/nixpkgs/086a5ea5b3acc4c512f9ec154bfefba55efba4f3?narHash=sha256-LyZtQZiq2v2We5ODev6s9s2iUHNu/ZC8rHIYRh1BIzg%3D:";
  outPathStr = ...
  pathRaw = "github:NixOS/nixpkgs/086a5ea5b3acc4c512f9ec154bfefba55efba4f3?narHash=sha256-LyZtQZiq2v2We5ODev6s9s2iUHNu/ZC8rHIYRh1BIzg%3D:";
  pathRawAdd = "github:NixOS/nixpkgs/086a5ea5b3acc4c512f9ec154bfefba55efba4f3?narHash=sha256-LyZtQZiq2v2We5ODev6s9s2iUHNu/ZC8rHIYRh1BIzg%3D:ci";
  pathRawAddStr = ...
  pathStr = "/nix/store/3a07hs5zz57xf76gqq53i9jkmi3mhyp6-source";
}

Source

{
  inputs.nixpkgs.url = "github:NixOS/nixpkgs";
  outputs = _: {
    data = rec {
      fetchTree = (builtins.fetchTree (builtins.fromJSON (builtins.readFile ./flake.lock)).nodes.nixpkgs.locked).outPath;
      fetchTreePath = fetchTree + "/ci";

      # original = _.nixpkgs;
      originalStr = "${_.nixpkgs}";
      outPathRaw = _.nixpkgs.outPath;
      outPathStr = "${_.nixpkgs.outPath}";
      pathRaw = _.nixpkgs.legacyPackages.x86_64-linux.path;
      pathStr = "${_.nixpkgs.legacyPackages.x86_64-linux.path}";
      pathRawAdd = _.nixpkgs.legacyPackages.x86_64-linux.path + "/ci";
      pathRawAddStr = "${_.nixpkgs.legacyPackages.x86_64-linux.path + "/ci"}";
    };
  };
}

tomberek · 2024-08-08T06:17:45Z

Noticed an eval failure when using lib.fileset compared to normal. This is because the value retains a context that it otherwise loses.

A more critical location this happens is in lib.fileset (@infinisil ): A dynamic attribute name: https://github.com/NixOS/nixpkgs/blob/master/lib/fileset/internal.nix#L249

looks like a path is implicitly converted to a string here, but retains a context

A minimal repro:

[tom@tframe:~/nix/t2]$ ~/.nix-profile-new/bin/nix eval .#value
{ gv00g760bh9xa2kp42z1c2wcv91p7yhy-source = 1; }

[tom@tframe:~/nix/t2]$ ../outputs/out/bin/nix eval .#value
error:
       … while evaluating the name of a dynamic attribute
         at flake.nix:4:7:
            3|     value = {
            4|       "${builtins.baseNameOf ./.}" = 1;
             |       ^
            5|     };

       error: the string 'gv00g760bh9xa2kp42z1c2wcv91p7yhy-source' is not allowed to refer to a store path (such as 'gv00g760bh9xa2kp42z1c2wcv91p7yhy-source')

[tom@tframe:~/nix/t2]$ cat flake.nix
{
  outputs = {...}: {
    value = {
      "${builtins.baseNameOf ./.}" = 1;
    };
  };
}

Perhaps this does the wrong thing when coerceToString happens regarding copyToStore or perhaps we need special handling the lazy path type? (https://github.com/NixOS/nix/blob/cfe66dbec325d5dcb601b642bd9c149ae1353147/src/libexpr-c/nix_api_external.cc#L108C48-L108C59)

NixOS#10252 (comment)

roberth · 2024-08-08T10:51:14Z

       … while evaluating the name of a dynamic attribute
         at flake.nix:4:7:
            3|     value = {
            4|       "${builtins.baseNameOf ./.}" = 1;
             |       ^
            5|     };

Yeah that looks like baseNameOf is performing a conversion to string, which is unnecessary. I've pushed a fix.
dirOf seems to be ok, but readFile has logic like state.store->isInStore(path.path.abs()) which does not account for the possibly non-system accessor. realisePath returns path values unchanged, so that is a thing now.
It looks like the primops need to be audited for how they handle the .path field where they do.

nixos-discourse · 2024-08-21T20:14:53Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2024-08-21-nix-team-meeting-minutes-171/50950/1

nyabinary · 2024-08-22T02:20:53Z

The performance label should probably be added to this, right?

NixOS#10252 (comment)

rebase of NixOS#10252

NixOS#10252 (comment)

roberth · 2024-08-25T08:29:12Z

We're turning this into a collaborative effort at #11367 (backed by an upstream branch for easy pushing)

Co-authored-by: Eelco Dolstra <edolstra@gmail.com> Co-authored-by: Robert Hensing <robert@roberthensing.nl> fixup: remove FlakeCache Make path values lazy This fixes the double copy problem and improves performance for expressions that don't force the whole source to be added to the store. Rules for fast expressions: - Use path literals where possible - import ./foo.nix - Use + operator with slash in string - src = fetchTree foo + "/src"; - Use source filtering, lib.fileset - AVOID toString - If possible, AVOID interpolations ("${./.}") - If possible, move slashes into the interpolation to add less to the store - "${./src}/foo" -> "${./src/foo}" toString may be improved later as part of lazy-trees, so these recommendations are a snapshot. Path values are quite nice though. SourceAccessor: insert colon after prefix This allows clever editors/IDEs to discern the path more easily for Ctrl+Click navigate to functionality, e.g. when building .?ref=HEAD Fix evalState::rootFS paths' to_string() This showPath is getting a little too ad hoc, but it works for now. Fix nix flake init eval for path value WIP fix baseNameOf (needs test maybe) NixOS#10252 (comment) fix findFile assertion failure A string is only allowed to create one path component; containing no slashes. tests: nix:derivation-internal.nix renders with a scheme now fixup: Re-enable import .drv code

github-actions bot added new-cli Relating to the "nix" command with-tests Issues related to testing. PRs with tests have some priority fetching Networking with the outside (non-Nix) world, input locking labels Mar 15, 2024

roberth commented Mar 15, 2024

View reviewed changes

thufschmitt added this to the fetch-tree stabilisation milestone Mar 19, 2024

roberth mentioned this pull request Mar 22, 2024

nix flake {update,metadata,...}: shows input's narHash #10249

Open

tomberek added idea approved The given proposal has been discussed and approved by the Nix team. An implementation is welcome. and removed idea approved The given proposal has been discussed and approved by the Nix team. An implementation is welcome. labels Apr 8, 2024

roberth mentioned this pull request Apr 14, 2024

Add outputHashMode argument attribute to bulitins that could support it #10480

Open

4 tasks

roberth force-pushed the lazy-fetchTree branch 2 times, most recently from a29ade5 to 8cb0fd4 Compare April 16, 2024 13:54

edolstra and others added 8 commits April 16, 2024 15:55

fetchTree: Return a path instead of a store path

202b188

SourceAccessor: insert colon after prefix

9707d2f

This allows clever editors/IDEs to discern the path more easily for Ctrl+Click navigate to functionality, e.g. when building .?ref=HEAD

Fix evalState::rootFS paths' to_string()

bde36d8

This showPath is getting a little too ad hoc, but it works for now.

Fix nix flake init eval for path value

6f01e4b

WIP please revert

446d59b

WIP: delay the flake outPath semantics change for now

4332b9a

roberth force-pushed the lazy-fetchTree branch from 8cb0fd4 to 4332b9a Compare April 16, 2024 13:58

roberth mentioned this pull request May 13, 2024

Virtual path semantics that are deterministic, lazy and suitable for performing deduplication of values by a path value key. #10689

Open

roberth mentioned this pull request May 20, 2024

Path coercion in Flakes causes files to be added to the store twice #9428

Open

roberth mentioned this pull request Jun 22, 2024

git-lfs support #10153

Open

roberth mentioned this pull request Jul 5, 2024

Source roots result in /nix/store/<hash>-<hash>-source #10627

Open

roberth mentioned this pull request Jul 23, 2024

Lazy, async instantiation and fetching (tracking) #11161

Open

9 tasks

roberth self-assigned this Aug 7, 2024

WIP fix baseNameOf (needs test maybe)

e21e54e

NixOS#10252 (comment)

WIP: opt-out, inputs.self.outPathIsString = true;

23d8c06

roberth mentioned this pull request Aug 10, 2024

Update lookup-path.md #11276

Open

tomberek pushed a commit to tomberek/nix that referenced this pull request Aug 23, 2024

WIP fix baseNameOf (needs test maybe)

ed16517

NixOS#10252 (comment)

tomberek pushed a commit to tomberek/nix that referenced this pull request Aug 24, 2024

fetchTree: Return a path instead of a store path

549fab1

rebase of NixOS#10252

tomberek pushed a commit to tomberek/nix that referenced this pull request Aug 24, 2024

fetchTree: Return a path instead of a store path

9c01a74

rebase of NixOS#10252

tomberek pushed a commit to tomberek/nix that referenced this pull request Aug 24, 2024

WIP fix baseNameOf (needs test maybe)

52c0ef2

NixOS#10252 (comment)

roberth mentioned this pull request Aug 25, 2024

Lazy paths #11367

Draft

6 tasks

roberth added the performance label Aug 25, 2024

roberth closed this Aug 25, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Lazy `fetchTree` `outPath` path values #10252

Lazy `fetchTree` `outPath` path values #10252

roberth commented Mar 15, 2024 •

edited

Loading

roberth Mar 15, 2024

roberth Mar 15, 2024

roberth Mar 15, 2024

roberth Mar 15, 2024

roberth Mar 15, 2024

roberth Mar 15, 2024

roberth Mar 15, 2024

roberth Mar 15, 2024

roberth Mar 15, 2024

roberth Mar 15, 2024

Ericson2314 Apr 15, 2024 •

edited

Loading

roberth Apr 16, 2024

roberth Apr 16, 2024

ConnorBaker commented Apr 4, 2024

roberth commented Apr 16, 2024

nixos-discourse commented May 7, 2024

nixos-discourse commented Jul 17, 2024

nixos-discourse commented Aug 7, 2024

tomberek commented Aug 8, 2024 •

edited

Loading

tomberek commented Aug 8, 2024 •

edited

Loading

roberth commented Aug 8, 2024

nixos-discourse commented Aug 21, 2024

nyabinary commented Aug 22, 2024

roberth commented Aug 25, 2024

		: v.path().accessor->toStringReturnsStorePath()
		? store->printStorePath(copyPathToStore(context, SourcePath(v.path().accessor, CanonPath::root))) + v.path().path.absOrEmpty()

Lazy fetchTree outPath path values #10252

Lazy fetchTree outPath path values #10252

Conversation

roberth commented Mar 15, 2024 • edited Loading

Motivation

Description

Context

Priorities and Process

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Ericson2314 Apr 15, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ConnorBaker commented Apr 4, 2024

roberth commented Apr 16, 2024

nixos-discourse commented May 7, 2024

nixos-discourse commented Jul 17, 2024

nixos-discourse commented Aug 7, 2024

tomberek commented Aug 8, 2024 • edited Loading

Before

After (only the differences)

Source

tomberek commented Aug 8, 2024 • edited Loading

roberth commented Aug 8, 2024

nixos-discourse commented Aug 21, 2024

nyabinary commented Aug 22, 2024

roberth commented Aug 25, 2024

Lazy `fetchTree` `outPath` path values #10252

Lazy `fetchTree` `outPath` path values #10252

roberth commented Mar 15, 2024 •

edited

Loading

Ericson2314 Apr 15, 2024 •

edited

Loading

tomberek commented Aug 8, 2024 •

edited

Loading

tomberek commented Aug 8, 2024 •

edited

Loading