Flakes: Allow computation of flake meta-data

[kalhauge]

Is your feature request related to a problem? Please describe.

Let me start by saying that flakes are a great extentions to Nix, and even in
the current state is much better than nothing!

In #3075, the content of flakes were limited to only being attribute sets. This
greatly limits the configuration, and usability of flakes. The reason this was
done is to allow quick meta-data extraction of flake.nix files.

There are many use-cases for using computation in meta-data. For example (from
#3966 (comment)), a haskell
program might be very picky about the dependencies it needs; and therefore
fetch all of them from the hackage server:

{
  description = "My Picky Haskell Program";
  inputs = let
    hackage = name: {
      url = "https://hackage.haskell.org/package/${name}/${name}.tar.gz";
      flake = false;
    };
  in {
   package1 = hackage "package1-1.2.3";
   package2 = hackage "package2-3.2.1";
   ...
  };
  outputs = { self, package1, package2, ...}: 
    #...
  ;
}

Another example from (#4753), explains that flattening depenencies is hard to
manage.

Describe the solution you'd like

First of all, I hope to start a discussion of the trade-offs of making flakes
computeable. If nothing else to settle the question.

Here is my idea on how to get get the best of both
worlds: Instead of flakes.nix is interpreted by the nix command, it is
instead "evaluated" by the developer of the repository into meta-data.
Specifically, the 'inputs' and 'description' attributes are fully evaluated
(while not allowing underspecified inputs) and added to the flakes.lock file.
Furthermore, a hash of the flakes.nix file is added to verify the origin of
the flakes.lock file.

The meta-data is always scraped from the flakes.lock file, and evaluation of
the flakes.nix file is only needed if the outputs is evaluated.
This might even improve meta-data reading times.

Downsides

• flakes.lock will be a required part of a flake.

Describe alternatives you've considered

• Not doing anything. Flakes will still remain a very usefull tool, but not
  as configurable.

• Finding a limited version of nix that allows some computation, but not all.
  This will allways lead to friction between how much power should be allowed.

• Redesigning the flakes format: Change the flake.nix format to be a nix expression
  that evaluates to the outputs attrset with a description field. Then allowing the
  user to access and define inputs using a builtin input { name = <name>; url = <flake-address>; flake = <bool>; follows = { name -> flake }; }. Every
  input is then stored and maintained in the flake.lock file, and is filled
  in during the evaluation. This allows inputs to depend on eachother, and gets around
  some of the akward syntax related to double defintion of inputs in inputs and outputs. This
  would be a mayor change, so I'll maybe include it in another proposal if this
  proposal gains traction.

Additional context
This is a duplicate of multiple issues. I wanted to create a feature request to
have a single point to discuss the problem. I apologize if this is already a
settled debate, but at least the record will then show this.

• Introduced by Require that flake attributes are not thunks #3075
• error: '/nix/store/.../flake.nix' must be an attribute set  #3966
• Can't eta-reduce with flakes: expected a function but got a thunk #4384
• Flakes: builtin function to flatten inputs using follows #4753

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/improving-flakes/12831/53

[thufschmitt]

An interesting use-case that can require computation in a metadata is the post-build-hook (or any other hook actually) nix config.

This can currently be set to point to a file in the flake directory (setting nixConfig.post-build-hook = ./build-hook/main.sh), but it will then be copied as its own store path − because that’s the normal semantics for Nix-language paths − meaning that it can’t refer to other files by their relative path (say the build hook starts by source ./lib.sh, that would break).

A nice way around that would be to allow something like nixConfig.post-build-hook = "${./build-hook}/main.sh", which would cause the whole build-hook folder to be copied to the store, so that main.sh can properly import lib.sh

[nrdxp]

Something I mused about recently was possibly extending the 'flake.lock' format to, perhaps optionally, cache the contents of a flake. This is largely motivated by the fact that simply to run nix flake show to visualize the contents of a flake introduces arbitrary computation. It'd be nice to see what I'm working with without potentially waiting forever.

This may well illustrate the situation here; we already have arbitrary computation to simply view the meta-data of a flake, so why not allow it in inputs? In fact, as is, it would be better than the situation for outputs since the 'flake.lock' could serve as an effective "eval cache" to avoid computing the values more than once.

On a related note, I've heard it suggested by @Ericson2314 and others that perhaps we could benefit from a separate tool, completely independant of Nix, to parse flake information. Perhaps something like that, which could also write and parse a valid 'flake.lock' is what is truly needed?

[tejing1]

There are so many legitimate uses for computation in flake inputs, for example, if allowed, I would be doing this:

{
  inputs = let
    nixosRelease = "21.11";
  in {
    nixpkgs.url      = "github:NixOS/nixpkgs/nixos-${nixosRelease}";
    home-manager.url = "github:nix-community/home-manager/release-${nixosRelease}";
    home-manager.inputs.nixpkgs.follows = "nixpkgs";
  };
}

It encodes a basic invariant that I want my system to have, and stops me having to think about the invariant whenever I change the code. This kind of thing is a major part of the value proposition of nixos and nixops.

Disallowing this in order to prevent long computations being invoked by commands that use flake metadata kinda feels like nix telling me "what you're doing is probably fine, but it might be a bad idea, so I'm just gonna disallow it". I've never been fond of software that assumes I'm an idiot. This is also quite confusing for people new to flakes, due to the intuition than nix code should be nix code, and here, it isn't.

It isn't quite that simple, since the implications of interacting with flakes written by others need to be considered, but we definitely should be able to do something better with this tradeoff than what we do now. At the very least, we should allow users to disable the restriction if they want to. Being flat out denied the ability to do something that's perfectly reasonable by your software is never fun, and different people will surely see the tradeoff differently here.

Regarding @kalhauge's proposal, I would note that hashing flake.nix isn't enough, unless you disallow the use of functions which reference other files, which brings us right back to the "nix code but not really" situation. Instead, it should be based on a hash of the entire flake except for flake.lock. This also ties in nicely with plans to make flake.lock inaccessible to the nix code in the flake.

This brings to the foreground the other issue that needs to be addressed with this proposal: what happens when the hash doesn't match? It will happen, and pretty often. Sure, experienced nix developers will probably put in a hook to make sure it's recalculated on each commit, but most every new user of flakes will run into this situation a number of times. Also, what about building from dirty worktrees? I think we all want to keep being able to do that easily. However it's handled needs to be very ergonomic.

I'm thinking some sort of error message explaining the need to recalculate, and a simple switch you can add to the command to make it "shut up and do what I say". In addition to this, an option in nix.conf that tells it to always recalculate when necessary and just print a warning about the mismatch would be a good idea. Some people just won't want to be bothered about this issue and will be fine with metadata-related commands sometimes just taking a long time.

This also ties in with another existing problem. Nix is quite picky and honestly rather stupid about incomplete lockfiles. For example, it tries to create/update lockfiles for read-only flakes, which obviously fails, making the flake completely unusable without --no-write-lock-file. --no-write-lock-file isn't even always an option: if I use builtins.getFlake to grab home-manager (which intentionally avoids locking its nixpkgs input) in the repl, then try to tab-complete its attributes, the REPL completely quits... ewww.

These kinds of situations would need to be considered carefully, and probably a few options to change nix's behavior in regards to them are needed.

Doing a similar thing for nix flake show as suggested by @nrdxp sounds good to me as well, though it would require standardizing a little more carefully how deeply nix flake show traverses the outputs and what info it needs. The current implementation of nix flake show still seems a bit like a rough draft to me.

Still, though, overall, I like this plan. Caching the results of these long-running computations should allow the best of both worlds, and I don't think we need to go so far as to require the ability to calculate inputs in terms of other inputs, at least so long as we're willing to keep expanding the builtins into a fairly decent basic library for pure nix coding. We just have to have a solid plan for how to keep the invalid cache issue from becoming too much of an annoyance.

[tejing1]

It occurs to me that making inputs able to be computed in terms of other inputs could actually be done fairly easily: allow inputs to optionally take the same argument as outputs does, taking advantage of the lazy loading of flake inputs. It would be up to the programmer to avoid infinite recursion. Stricter defaults could be applied to flakes that include the argument, in terms of automatically recomputing the inputs, given that that code would now effectively have network access, but it would then be pretty much maximally flexible for those that wanted/needed that.

[tejing1]

It occurred to me recently that we could thread the needle on the cache recomputation significantly better than hashing the whole flake besides flake.lock, without adversely affecting the validity.

We'd just need to track what files are involved in evaluating inputs and hash only them. In the vast majority of cases, this would be only flake.nix, but in other cases it would probably be 1 or 2 more files. We could even go yet more granular and hash the relevant portion of the syntax tree.

This would reduce the frequency of recomputations greatly, and thus also reduce the frequency of situations where the cache is out of date, probably to levels where the additional annoyance would be tolerable.

[cyntheticfox]

We'd just need to track what files are involved in evaluating inputs and hash only them. In the vast majority of cases, this would be only flake.nix, but in other cases it would probably be 1 or 2 more files. We could even go yet more granular and hash the relevant portion of the syntax tree.

I mean, is this significantly different from how a derivation works? It'd probably be a little unwieldy to use something that flexible for inputs, but that sounds like a decent amount of overlap