nix flakes: add support for git submodules

[DanilaFe]

I recently learned about Nix Flakes, and I tried out making my blog a perfectly reproducible build. The trouble is, my blog uses a Git submodule to reference some code that I then present in my blog posts. This is fine and all, but I run into an issue - although I can fetch my blog source using Flakes' input attribute, it doesn't fetch submodules! This means that the code that I reference in my build process is simply not there.

This seems to be almost possible; from a brief glance around Nix's source code, I discovered that a flake input gets turned into a "regular" input, which can point to a git repo and have a submodules attribute that, I think, would do what I want. It feels as though it was intended that this would work, but it doesn't.

It doesn't work because, for a reason unknown to me, a flake input's attributes must all be strings (except for flake). Here's the code that rejects non-string arguments:

nix/src/libexpr/flake/flake.cc

Lines 123 to 127 in 724b7f4

	if (attr.value->type() == nString)
	attrs.emplace(attr.name, attr.value->string.s);
	else
	throw TypeError("flake input attribute '%s' is %s while a string is expected",
	attr.name, showType(*attr.value));

If non-string arguments were not rejected, a submodules field could be forwarded to the Git input scheme, which I suspect would make this work. Is there a reason that Flakes rejects all non-string "third party" attributes? It puts them into an attribute set anyway. Right now, I'm falling into a rather amusing loop:

If submodules is a string, it's rejected by the Git input scheme. But if it's a bool, it's rejected by the Flake code.

[knedlsepp]

I think this is a duplicate of #4357

[blaggacao]

Apparently closed by #4435

[knedlsepp]

I think there is still one piece missing: the flake itself is not being fetched using submodules.

[DanilaFe]

Yes, perhaps it's worth closing this issue, and opening the original one (since this is a duplicate)?

Also, I am not sure how to go about requiring a flake itself to pull its own submodules. If a flake only has inputs and outputs, and an external definition determines the method for downloading it (a flake can be a local folder, too, or a tarball, can't it?), then where's the compromise?

A flake is assumed to be a git repo, so it would make sense for a universal "fetch submodules" option, perhaps as an output attribute. This would be orthogonal to the type = "git"; submodules = "true" attributes inside an input, since it wouldn't work for non-flake inputs.

[blaggacao]

What would be a good & concise way to present the design decision abstract primitives still in question to @edolstra / the maintainers? (possibly on another issue for proper framing)

[L-as]

@knedlsepp

I think there is still one piece missing: the flake itself is not being fetched using submodules.

I made an ugly patch where submodules are always fetched: L-as@ce0a547

The idea is that if a flake has submodules, they should probably always be fetched (why would you add them otherwise?).
My C++ skills are subpar though, and my patch may just make it fetch submodules for all git flake inputs (haven't tested), but it works for my case.

[ruler501]

@L-as yeah that's just turning it on for everything. If you look at the constructor for one of the classes for the nix develop command it has a default url of . making that instead be .?submodules=1 or whatever the correct format for that url is should cause it to only affect the flake in the repo the command is running in.

[L-as]

It has to work with all nix commands though, and I can't see the use in being forced to manually write ?submodules=1 every time.

[ncfavier]

The idea is that if a flake has submodules, they should probably always be fetched (why would you add them otherwise?).

Consider a public NixOS config repo with a private submodule. One may want to be able to build the config even without access to the private parts.

[L-as]

You could make it so that failure to clone the submodules is a non-fatal error.

[bqv]

The idea is that if a flake has submodules, they should probably always be fetched (why would you add them otherwise?).

Consider a public NixOS config repo with a private submodule. One may want to be able to build the config even without access to the private parts.

I'm not sure that's desirable behaviour. It's nondeterministic, then

[edolstra]

We should enable submodules by default and make sure they work for top-level flakes.

[ony]

Are there any work-arounds for top flake?

nix build  '.?submodules=1#nixosConfigurations.myhost.config.system.build.toplevel'

Doesn't work. Inside of /nix/store/*-source/ I see .gitmodules, but no corresponding folders.

[FlorianFranzen]

I currently just use fetchGit to re-clone the flake input with submodules like this:

flake_input_fixed = builtins.fetchGit {
      url = "<your-url>";
      inherit (flake_input) rev;
      ref = "<branch>"; # optional if not default
      submodules = true;
};

@ony: You might be able to do the same with self and then import any nix expressions that required the submodule from there.

EDIT: It should be noted that self.rev is only available when the repo is clean, so this approach is probably quite limiting when used with self. For now I just have a separate flake repo that I manually keep in sync, which works but is rather annoying.

[DieracDelta]

I'm having trouble fitting flakes to the use case of developing a git repo (e.g. self) that has multiple submodules.

My use case is as follows: I have a top level flake (and git repo) with a bunch of git submodules. I want to change these submodules and have those changes show up when I build outputs of the top level flake. Maybe I'm going about this wrong? Is there an easy way to get this sort of workflow?

I see the listing all submodule directories as url = path:$SUBMODULE_PATH as a possible work around, but this is impure. Perhaps something akin to url = path+git:$SUBMODULE_PATH that uses git to maintain purity.

Another solution could be a flag that specifies whether or not to fetch self submodules from git or to use the locally checked out versions. Though, this would require gathering the local submodule path information from somewhere...

[matu3ba]

The only usable way I see is to get all the submodule hashes and print them in nix-usable format with a nix-specific forward like nix git submodule init
and nix git submodule status --recursive (printing non-recursively is a footgun).

Ideally one can then extract already existing pixpkgs or flakes to use instead of the very brittle to use hashes.

However I am not sure, what guarantees git makes about the output format as the status has no --porcelain. git submodules without guarantees how to machine-readable the package status sounds more like a hackish thing, so I am not entirely sure if nixOS should endorse that.
I would favor a solution that descibes how to get away from git submodules until git gives better guarantees.

[ztlevi]

Is top-level flake supported?

I'm using a submodule inside my flake repo, in nixos, I can use nixos-install/rebuild to build it without any issue. But for the github workflow, it's using ubuntu and only has nix binary.

I'm build the system with this [command])https://github.com/ztlevi/dotty-nix/blob/main/.github/workflows/build.yml#L45). But it pops submodule error. The submodule here is call config.

See my build failure here

Since the issue is closed, @edolstra to add some visibility.

[DieracDelta]

Strawman question: have you tried a more recent version of nix @ztlevi ? It looks like the #4922 was merged 3 days ago. But the nix compiler version is from 2021-08-23 (af94b54) which is several weeks ago. You might try bumping it to a more recent version of master.

Just to add in my 2 cents: I've built flake attributes that use git submodules on yesterday's nix master on ubuntu machines with nix installed and have had no issues. However, this has not been for nixos configurations.

[ztlevi]

@DieracDelta How am I able to test the latest nix build in the github workflow? ttps://github.com/ztlevi/dotty-nix/blob/main/.github/workflows/build.yml#L45

The nix release is too old

• https://github.com/NixOS/nix/releases
• https://github.com/numtide/nix-unstable-installer/releases/

[DieracDelta]

Don't have an "elegant" answer. My solution was to use old nix to build new nix (it is a flake after all and the cli is great for this). Then I would just call new nix.

EDIT ncfaviers solution is better.

[cxandru]

@jakubgs I don't even know how to get it to work in CLI. How do you do that?

[jakubgs]

Exactly as shown in #4423 (comment).

[cxandru]

@jakubgs but how do we set type="git" to self? inputs.self.type = "git" didn't work for me. Also, does the .gitmodules file need to be committed or is it fine if it's only stashed?

[hraban]

In case anyone is interested in using submodules on flake inputs , not just the flake directory itself, you have to specifically use =1, not =true, and you have to use a "git" type input, notably not github:user/repo syntax because that's just a HTTP download of an archive from github's "download as .tar.gz" endpoint.

Example:

  inputs = {
    nyxt = {
      flake = false;
      url = "git+https://github.com/atlas-engineer/nyxt?submodules=1";
    };

C.f.

nix/src/libfetchers/git.cc

Lines 270 to 271 in 31ffd0c

	else if (name == "shallow" || name == "submodules" || name == "allRefs")
	attrs.emplace(name, Explicit<bool> { value == "1" });

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/get-nix-flake-to-include-git-submodule/30324/1

[hraban]

In case anyone is interested in using submodules on flake inputs , not just the flake directory itself, you have to specifically use =1, not =true, and you have to use a "git" type input, notably not github:user/repo syntax because that's just a HTTP download of an archive from github's "download as .tar.gz" endpoint.

Example:

  inputs = {
    nyxt = {
      flake = false;
      url = "git+https://github.com/atlas-engineer/nyxt?submodules=1";
    };

C.f.

nix/src/libfetchers/git.cc

Lines 270 to 271 in 31ffd0c

	else if (name == "shallow" || name == "submodules" || name == "allRefs")
	attrs.emplace(name, Explicit<bool> { value == "1" });

Update: Be careful, fetching from a git with submodules is not deterministic and you will get narhash clashes. See NixOS/nixpkgs#100498 for workarounds using e.g. pkgs.fetchgit.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/handling-git-submodules-in-flakes-from-nix-2-18-to-2-22-nar-hash-mismatch-issues/45118/1