Submodules of flakes are not working

[faulesocke]

Describe the bug
Having git submodules in a git repository I cannot access the submodules in my flake.nix. This is especially annoying when I want to provide a flake.nix for software that is split across multiple git repositories that are all included as submodules in one main repo.

Steps To Reproduce

git init top
git init sub

cd sub/
echo "foobar" > foobar
git add foobar && git commit -m foobar
cd ..

cd top/
git submodule add ../sub sub
git commit -m submodules
git add flake.nix

And top/flake.nix is

{
  outputs = { self, nixpkgs }:
  with import nixpkgs { system = "x86_64-linux"; };
  {
    defaultPackage.x86_64-linux = stdenv.mkDerivation {
      name = "foobar";
      src = self;
      installPhase = ''
        cat sub/foobar > $out
      '';
    };
  };
}

Then run nix build.

Expected behavior
I expect result to contain the string foobar.

Actual behaviour

% nix build
warning: Git tree '/tmp/top' is dirty
error: builder for '/nix/store/wlxmp43q9q13477yjc2hvzqja8ng3sgf-foobar.drv' failed with exit code 1;
       last 10 log lines:
       > unpacking sources
       > unpacking source archive /nix/store/kzsvnqmnjrpyvma444jaz3zv3kgry8az-source
       > source root is source
       > patching sources
       > configuring
       > no configure script, doing nothing
       > building
       > no Makefile, doing nothing
       > installing
       > cat: sub/foobar: No such file or directory
       For full logs, run 'nix log /nix/store/wlxmp43q9q13477yjc2hvzqja8ng3sgf-foobar.drv'.

nix-env --version output
nix-env (Nix) 2.8.1
I also tried 2.8.0 and 2.9.1. The submodule is just missing and I see no way to access it.

Additional context
Also manually adding the submodule as an input like so

{
  inputs.sub = {
    url = "./sub";
    flake = false;
  };
 ...
}

and then changing the src = self to src = sub in the outputs, just results in a weird error message instead of the submodule just working as it should:

% nix build
warning: Git tree '/tmp/top' is dirty
error (ignored): error: end of string reached
error: getting status of '/nix/store/ms70asysgv7alri3972wpzpd4r6ivbiz-source/sub': No such file or directory
(use '--show-trace' to show detailed location information)

Especially the error (ignored): error: end of string reached line is worrying me and probably show that this is actually a bug and should work. Of course this could also just be my own stupidity and I'm doing something completely wrong.

[n8henrie]

Working for me if you include the submodules=1 flag.

$ nix build .?submodules=1
$ cat result
foobar

[FlorianFranzen]

Even with submodules=1 is still very weird to use submodules efficiently. I started using submodules as separate input to improve cache invalidation, but both ways I found have major caveats:

Either you just use the submodule path:

inputs = {
    submodule = {
      url = "./path/to/submodule";
      flake = false;
    };
}

However, this updates the lockfile on each invocation and using the flake in the repl is cumbersome.

Or you use it as a git file input. That solves the repeated locking but just ignores it the referenced submodule is dirty:

inputs = {
    submodule = {
      url = "git+file:./path/to/submodule";
      flake = false;
    };
  };

[n8henrie]

Huh, I'm not referencing the submodule (directly) as an input at all and it's been working well for my use case (maintaining my nix-darwin flake-based config containing a vim submodule that references my "pre-nix" vim config). Haven't messed around with seeing what happens if vim is dirty though.

[AshtonKem]

Ahhhh! This just got me too. nix build .?submodules=1 worked for me, is there any way to configure this permanently in flake.nix?

[n8henrie]

• nix flakes: add support for git submodules #4423
• feat: support managing submodules via inputs.self #7862

[AshtonKem]

Phenomenal, thank you!

[doronbehar]

Does anyone else here experiences that for a nix build that requires submodules, if your tree is dirty the submodules are not checked out in the builder's src = self?

NOTE: I am using nix build -L .?submodule=1\#$PKG

[doronbehar]

Answering my own question: If the tree is dirty, and at least 1 git submodule is not checked out, the tree has to be clean for Nix to pick up the missing submodule using the ?submodule=1 flag. Makes sense sort of, but indeed it's a confusing. hopefully things will improve with #7862.

[ghost]

Worth noting for people reading to the end in future: it's ?submodules=1, not submodule in the singular.

[haskelious]

Is it possible to also apply the ?submodules=1ˋ to a ˋnixos-rebuild command that makes use of the flake with submodules for the whole system configuration?

[haskelious]

I found the answer to my question. One should use: nixos-rebuild switch --flake .? submodules=1

[imadnyc]

I found the answer to my question. One should use: nixos-rebuild switch --flake .? submodules=1

When I do this command nixos-rebuild dry-build --impure --flake .#nix-deck .\?submodules=1 (slash cuz zsh), I get

/run/current-system/sw/bin/nixos-rebuild: unknown option `.?submodules=1'. Do you know anything about this?

[ghost]

Try this:

nixos-rebuild dry-build  --impure --flake .\?submodules=1#nix-deck

The ?… bit is part of the flake URI.

[imadnyc]

Thanks! That makes a lot more sense.

[hamishmack]

For some reason on MacOS I've not been able to get . on its own to work. Instead I have to put the full URL in like this:

nix develop 'git+file:///Users/hamish/ghcjs-project?submodules=1#'

[imadnyc]

Are you in the directory when you call develop? It uses the current dir's flake unless specified like you did.

[x-mass]

In 2.20.1 version, nix build '.?submodules=1' did not work for me, but nix build '.?submodules=1#' did.

PS: to reproduce the miminal example from the issue, you should run git -c protocol.file.allow=always submodule add ../sub sub to supress git security fatal: transport 'file' not allowed error

[yshui]

Having the same problem, '.?submodules=1' doesn't work, '.?submodules=1#' and 'git+file:///path/to/flake?submodules=1' works

[thufschmitt]

It looks like there's a bug in the flakeref parser that causes .?something to be interpreted like . and ignore the ?something.

That's likely my fault, I'll look into it

[thufschmitt]

I've opened #10125 to fix the parsing issue (was hairier than expected, because there were actually two bugs stacked on top of each other 😒 )

[Rubikoid]

/path/to/folder/with/flake?submodules=1#hostname for me also was broken on 2.20 and 2.19