Use shallow clones for submodules

[jyn514]

This reduces the amount of git history downloaded for submodules from ~67M to ~11M. For comparison, a shallow clone of rust-lang/rust is 103M and a deep clone is 740M, so this almost halves the amount of history necessary if you made a shallow clone to start, and it's a significant reduction even if not.

Closes #63978. r? @Mark-Simulacrum

[Mark-Simulacrum]

I've tried to tweak the lines in the above bootstrap.py script to limit the depth of the submodule update. Unfortunately, using --depth=1 doesn't work, because the submodule commit referenced here may not be the latest in the submodule repo, and apparently git cannot clone at a specific commit.

Is this fixed by something? IIRC, this was (one of) the reasons we did not do this historically. I also have some recollection that fetching new commits in a shallow repository is more work for GitHub and the client, but I could be wrong about that...

[jyn514]

using --depth=1 doesn't work, because the submodule commit referenced here may not be the latest in the submodule repo, and apparently git cannot clone at a specific commit.

Is this fixed by something?

I'm not sure what error they ran into originally, but I tested cloning rust-lang/rust from scratch and having x.py do a shallow clone for the first time, and it worked fine.

I also have some recollection that fetching new commits in a shallow repository is more work for GitHub and the client, but I could be wrong about that...

Hmm, so in CocoaPods/CocoaPods#4989 (comment) Github said shallow clones were more expensive over time if you don't subsequently use --depth=1. But we do use --depth=1 in further updates; e.g. here's a change from nightly to 1.55, which has to update most submodules:

$ git checkout 1.55
M	library/backtrace
M	library/stdarch
M	src/tools/cargo
M	src/tools/miri
M	src/tools/rls
$ x check
Updating only changed submodules
Updating submodule src/tools/cargo
Submodule path 'src/tools/cargo': checked out '32da73ab19417aa89686e1d85c1440b72fdf877d'
Updating submodule src/tools/rls
Submodule path 'src/tools/rls': checked out 'a82a0527057eab59c24a0452add35ccb39584e80'
Updating submodule src/tools/miri
Submodule path 'src/tools/miri': checked out 'e2872a3f2a26154b91a6a6085d56016509803c61'
Updating submodule library/backtrace
Submodule path 'library/backtrace': checked out '221483ebaf45df5c956adffee2d4024e7c3b96b8'
Submodule 'backtrace-sys/src/libbacktrace' (https://github.com/rust-lang/libbacktrace) registered for path 'library/backtrace/crates/backtrace-sys/src/libbacktrace'
Cloning into '/home/joshua/src/rust/rust-shallow/library/backtrace/crates/backtrace-sys/src/libbacktrace'...
remote: Enumerating objects: 363, done.        
remote: Total 363 (delta 0), reused 0 (delta 0), pack-reused 363        
Receiving objects: 100% (363/363), 846.61 KiB | 2.20 MiB/s, done.
Resolving deltas: 100% (262/262), done.
Submodule path 'library/backtrace/crates/backtrace-sys/src/libbacktrace': checked out '5c88e094a691bb803d4bba342403a10459abad9e'
Updating submodule library/stdarch
Submodule path 'library/stdarch': checked out '3001c75a1d2a81d2a76bef139c69387cb2ebb820'

So I'm not sure that problem is relevant here. (We also don't have many thousands of people running fetch every second, I would expect most contributors to only update submodules maybe once or twice a week depending how often they contribute.)

[Mark-Simulacrum]

I'm not sure what error they ran into originally, but I tested cloning rust-lang/rust from scratch and having x.py do a shallow clone for the first time, and it worked fine.

Great! This seems fine to try at least then, and we can always revert easily if folks run into problems.

It may also be that we can use this (presumably new) support to replace our current github archive fetching on CI (https://github.com/rust-lang/rust/blob/master/src/ci/init_repo.sh) which seems like a nice cleanup, but that doesn't need to happen in this PR.

@bors r+

[bors]

📌 Commit 9177fa3 has been approved by Mark-Simulacrum