support arbitrary files in flakes or filter them out

[jorsn]

Is your feature request related to a problem? Please describe.

A local flake which is not a git repo gets fetched as an ordinary path by nix. Therefore, in a flake repo you can only have file types which are supported in the nix store. This is a painful restriction: For example I created a fifo in the flake directory to quickly try something out, and suddenly the flake was completely broken because fifos aren't supported in the store. Another problem are are large files irrelevant to nix but present in the flake tree. They always get copied to the store.

It doesn't help if I make a subdirectory for the flake and import the main directory, because the inputs are also copied to the store.

Describe the solution you'd like

I'd like to have a means to filter which files belong to a flake, because adding arbitrary files to the store doesn't seem to be right.

I'm not sure what the best implementation would be. See alternatives.

Describe alternatives you've considered

1. Have an optional top-level attribute src or self and filterSource support (I heard this was disabled in restricted eval mode). This attribute could be a function of the inputs, like outputs. Problem: Nix must execute arbitrary code every time it accesses a flake.
2. Use a dummy git repo and add all files that should be considered part of the flake. This is really ugly.
3. Implement a .flakeignore. Maybe: Try to make sense of any .*ignore. Yet another special file. But not turing complete.
4. Use (1.) together with a non-recursive dialect of Nix… See error: '/nix/store/.../flake.nix' must be an attribute set  #3966.
5. Make the flake source an input and add a filteredPath fetcher.

I don't see an elegant solution which is not turing complete or (4.).

[edolstra]

Option 2 is generally the best way. Otherwise you'll always end up with files being copied that you didn't intend, even if they're supported.

We could add a filter to the path fetcher though. However the filter should be a regexp, not a function like with builtins.path, since we don't want arbitrary code execution.

[maisiliym]

Flakes would gain tremendous utility if they could subdivide their content into smaller flakes. It would offer the utility of filterSource without the overhead of copying files in the store, and allow this particular use case.
Related: #3732

[RealityAnomaly]

Filtering out paths would be very useful. I don't want my git-crypt secrets copied to the store, for example.

[colinxs]

Just ran into #3732 after banging my head against the wall. I would love for builtins.path/builtins.filterSource to usable from flakes. Building on what @edolstra suggested, would something more powerful than a single top-level regexp (but less than arbitrary code) be possible? Perhaps something similar to nixpkgs.sourcesByRegex, which recursively applies the regexp and takes a list of regexp's, but with something like includeRegex and excludeRegex (since negating a regex can be difficult).

[Ma27]

Filtering out paths would be very useful. I don't want my git-crypt secrets copied to the store, for example.

And I'm not even sure if everyone who does that is aware of it. When I look at https://github.com/search?q=flake.nix+git-crypt&type=code, there are quite a bunch of repositories that use git-crypt within a directory with a flake.nix, so I think we should come up with a solution in 2.4 (cc @edolstra @regnat)

I personally think that either a flakeignore (I don't think it's that special, a lot of tools have this, e.g. git, docker or helm, perhaps even more) or a regex is a solution, however I'd prefer the first one, otherwise you'd have to nixos-rebuild switch --flake /etc/nixos?includeRegex=...#cfg each time.

[rnhmjoj]

And I'm not even sure if everyone who does that is aware of it. When I look at https://github.com/search?q=flake.nix+git-crypt&type=code, there are quite a bunch of repositories that use git-crypt within a directory with a flake.nix, so I think we should come up with a solution in 2.4 (cc @edolstra @regnat)

I've just realised this after converting my configurations into a flake and noting that toString ./. was being turned into a store path. I'll have to postpone that since this design is completely incompatible with the way I manage secrets.

I guess this is one of those things that looks absolutely obvious if you're being working with flakes for years, but I only read a man page, a couple of blog posts and they don't mention of this. The manual should have a big warning that everything in the flake directory is going to be copied into the Nix store by default.

[Ma27]

I agree, it should be warned about that. Also, I still think that a .flakeignore might be useful for a fair share of use-cases.

In fact, I started hacking something together a few weeks ago, but didn't have time to finish. @edolstra WDYT? If other people agree, I'd finish the patch :)

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/is-it-possible-to-make-a-flake-that-has-no-source-tree/16037/6

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/tweag-nix-dev-update-24/17230/1

[bbbbbailey]

Big 👍 on .flakeignore. Just ran into this problem with building cloud images from flakes via Terraform. If any file changes in the repository, the flake source changes, causing the store path of the image to change, finally resulting in the image needing to be redeployed according to Terraform.

Being able to ignore paths would be excellent. The current workaround is to write a .flakeignore file, hash files that make it through the filter, and then use that as the image hash.

[bergkvist]

I also think a .flakeignore would be amazing. This is also very intuitive to people familiar with git (.gitignore) and docker (.dockerignore).

[roberth]

When lazy-trees, #6530 makes its way into Nix, the whole flake directory won't be copied into the store, so the only purpose of a .flakeignore would be to filter the sources that go into derivations. This is better achieved with the existing source filtering functions, because a per-derivation level of granularity is needed to avoid rebuilds effectively. So between lazy-trees and source filters, not much of a gap will remain; just the blanket ignores you may want for security.

[iwanb]

In case that would be useful to others, my workaround it to copy the wanted files using git archive to a new repo and then call nix build on that, it's a whitelist instead of blacklist:

flake_tmpdir="$(mktemp -d)"
git archive --format=tar HEAD \
'*.nix' \
flake.lock \
# This could come from a file, e.g.
# $(cat .flakeinclude)
'needed_file' \
'*.glob' \
| (cd "$flake_tmpdir"; tar x) > /dev/null 2>/dev/null
git init "$flake_tmpdir" > /dev/null 2>/dev/null
git -C "$flake_tmpdir" add . > /dev/null 2>/dev/null
GIT_AUTHOR_NAME="nobody" \
GIT_AUTHOR_EMAIL="nobody@nobody.com" \
GIT_AUTHOR_DATE="1970-01-01T00:00:00.000Z" \
GIT_COMMITTER_NAME="nobody" \
GIT_COMMITTER_EMAIL="nobody@nobody.com" \
GIT_COMMITTER_DATE="1970-01-01T00:00:00.000Z" \
git -C "$flake_tmpdir" commit -m "Flake snapshot" > /dev/null 2>/dev/null
echo "$flake_tmpdir"

Then you call it e.g. like:

tmp_dir="$(./mktemp_flake_repo.sh)"
nix build "${tmp_dir}#package-name"
rm -rf "$tmp_dir"

The main drawback (on top of the unnecessary copy/delete) is that this depends on bash, git and tar.

The commit is essentially content addressed so it's cached on the second run.

[Nemo157]

Another place this causes issues is when using alternative source control systems, I just tried switching to using jujutsu for my local clone of a flake, but that results in the .jj directory being copied into the source.

[bbigras]

I just tried switching to using jujutsu for my local clone of a flake, but that results in the .jj directory being copied into the source.

For rust projects (with jj), it also copies the whole target directory.

[thecaralice]

👍 on .flakeignore, since "I want X to be tracked by VCS" and "I need X for Nix" are not necessarily equivalent. For example, it is a common practice for me to work on git projects which do not include a flake.nix or anything similar, so I would normally write a local flake and place it in .git/info/exclude so that it is not tracked. However, that would prevent the flake from actually working, so I have to keep flake.nix and flake.lock tracked while keeping my eyes on them so I don't accidentally stage them for a commit.

[GoldsteinE]

One workaround for having a devshell in a Git project that doesn’t use flakes is doing something like

mkdir flake/
echo /flake/ >> .git/info/exclude
# place flake.nix into flake/flake.nix
nix develop path:./flake