Cargo downloads git dependencies very slowly

[WaffleLapkin]

Problem

When compiling a project that uses a lot of git dependencies it takes a lot of time for cargo to download them.

Example that I just seen:

• It took minutes to download 50 git dependencies
• It then took 7.23 seconds to download 300 crates.io dependencies

Seems like there is a big potential for optimizations here.

Steps

1. Clone https://github.com/zed-industries/zed/tree/main
2. Run git submodule update --init --recursive
3. Run cargo build
4. Observe the slow downloading of git dependencies

Possible Solution(s)

• Using a shallow clone, instead of clonning the full repo
• Downloading multiple repositories in parallel

Notes

You may need to delete some caches to be able to repeat this test. I think you need to delete ~/.cargo/git/.

Version

cargo 1.76.0 (c84b36747 2024-01-18)
release: 1.76.0
commit-hash: c84b367471a2db61d2c2c6aab605b14130b8a31b
commit-date: 2024-01-18
host: x86_64-unknown-linux-gnu
libgit2: 1.7.1 (sys:0.18.1 vendored)
libcurl: 8.5.0-DEV (sys:0.4.70+curl-8.5.0 vendored ssl:OpenSSL/1.1.1w)
ssl: OpenSSL 1.1.1w  11 Sep 2023
os: Arch Linux Rolling Release [64-bit]

[epage]

For initial downloads, one big slow down compared to crates.io is that we download the entire history. We have unstable support for shallow clone and would recommend trying that to see how much it improves the problem. See #13285

For updates, there seem to be performance issues in libgit2 (#9167) and we are working on transitioning to gitoxide. Maybe give that a try as well. See #1181.3

Now, you may not want to rely on a nightly toolchain for development. Doing the above will help us narrow down root cause and determine if this is a duplicate. Once that is done, you could see if net.git-fetch-with-cli works around the problem for you.