builtins.fetchGit doesn't fetch submodules

[ElvishJerricco]

No description provided.

[ElvishJerricco]

This is a pretty big blocker for me. I can't reasonably use builtins.fetchGit without submodule support.

[langston-barrett]

At least when building with nixpkgs, one can use

  src = fetchgit {
    url = "https://github.com/GaloisInc/some-repo.git";
    deepClone = true;
    sha256 = ...;
    rev = ...;
  };

[NorfairKing]

This is also a blocker for me. Specifically when it comes to private git repositories that have submodules.
I looked around in the code to see what would have to be changed to allow for deepClone = true in builtins.fetchGit:

nix/src/libexpr/primops/fetchGit.cc

Line 125 in 18b4c53

runProgram("git", true, { "-C", cacheDir, "fetch", "--quiet", "--force", "--", uri, *ref + ":" + localRef });

It looks like the current approach with a bare repository and git fetch instead of git clone will not work because submodules require a working tree and cannot work with a bare repository.
Either we fix up git, or we do a full git clone --recursive and remove the .git directories recursively inside that.

cc @nh2, Thanks for the help!

[tbenst]

This issue causes difficult to debug error messages when trying to package PyToch 1.1 (NixOS/nixpkgs#63073), and probably elsewhere, too. I took a clumsy stab at getting nested level of 2 to work with no luck: NixOS/nixpkgs@master...tbenst:pytorch11#diff-8c857b370607b23d073067f7df717593

[qolii]

Any progress/interest on this one?

[tbenst]

@qolii might want to check out NixOS/nixpkgs#61820 (comment). I'm no longer sure if this issue is valid?

[evanjs]

@qolii might want to check out NixOS/nixpkgs#61820 (comment). I'm no longer sure if this issue is valid?

I think fetchGit is still somewhat preferred if private repos are used. Somebody please correct me if I’m wrong.

[qolii]

I think fetchGit is still somewhat preferred if private repos are used.

Exactly. From my reading, NixOS/nixpkgs#61820 is using pkgs.fetchgit. But that seems to be very difficult to use with private repos (my use case), whereas builtins.fetchGit is much more amenable.

[blitz]

I also ended up here, because I wanted to clone a private repository with submodules.

@tfc

[qolii]

Looking quickly at fetchgit.cc, this appears like it would be trivial?

• Add another input argument to builtins.fetchGit, along the lines of a boolean fetchSubmodules or so)
• Then based on it, add a conditional extra call to runProgram, to perform a git submodule init.

I am sure I am missing something... Please let me know. If it is this easy, I can put a patch together myself.

Ok, so, I should have actually paid attention to @NorfairKing's comment earlier. Sorry!

Did you make any progress with your git clone --recursive approach?

[NorfairKing]

@qolii I made no attempt to 'fix' this issue. I'm working around it by making my repos public :)

[blitz]

I've given this a shot after a discussion with @edolstra and @tfc at NixCon. Comments welcome.

[knedlsepp]

Finally it's happening. Can't wait for the next release. ❤️

[jakubgs]

Any news on when this will be included in a release?