fetchFromGitHub: add fetchSubmodules option

[rasendubi]

Motivation for this change

Things done

• Tested using sandboxing
  (nix.useSandbox on NixOS,
  or option build-use-sandbox in nix.conf
  on non-NixOS)
• Built on platform(s)
  • NixOS
  • macOS
  • Linux
• Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
• Tested execution of all binary files (usually in ./result/bin/)
• Fits CONTRIBUTING.md.

---

This commit extends fetchFromGitHub with ability to fetch GitHub
repositories with submodules, so we can use the function consistently
with all GitHub repositories.

Note it doesn't change the previous behavior.

[mention-bot]

@rasendubi, thanks for your PR! By analyzing the history of the files in this pull request, we identified @MarcWeber, @shlevy and @rbvermaa to be potential reviewers.

[rasendubi]

rasen@ashmalko ~/nixpkgs> nix-shell -p nox --run 'nox-review -k pr 20518'
=== Reviewing PR 20518 : fetchFromGitHub: add fetchSubmodules option
==> We're in a git repo, trying to fetch it
==> Fetching base (master)
==> Fetching PR
==> Fetching extra history for merging
==> Merging PR into base
... md5 warnings
Nothing changed

[rasendubi]

@Profpatsch, you might like it ;)

[copumpkin]

Awesome! That fixes #15559

[copumpkin]

Actually, no, not this implementation. We can do fetchzip recursively to do this without using fetchgit. fetchgit feels like a regression in performance, as it can take arbitrarily long. In big-O notation, fetchgit takes O(size of repo), whereas fetchFromGitHub nowadays takes O(size of tree you request).

[copumpkin]

Anyway, I'm not super opposed to this version, but there are a few large git repos that I currently fetch via fetchFromGitHub and then manually plug submodules into. I could keep doing that with fetchSubmodules = false of course, but I'd prefer the recursive fetchzip implementation if not too much work.

[rasendubi]

@copumpkin I have no idea how can I do that in general fashion now. Could you point me to the mentioned packages?

[copumpkin]

Sorry, they're mostly in non-public projects, so not really. I just use fetchFromGitHub all over the place, basically, so I'm sensitive to it not getting slower 😄

[rasendubi]

Could someone approve this? I'm starting to think the real reason is not hash stability but the speed and saving network usage.

[rasendubi]

It won't get slower, as the default value for fetchSubmodules is false. So basically, this patch makes it convenient to use fetchFromGitHub all over the place.

[copumpkin]

I have no idea how can I do that in general fashion now

I go into that a bit in the other ticket, fwiw.

Either way this improves things, thanks.