cargo install crate1 crate2 .. updates index repeatedly for every crate in list.

[nain-F49FF806]

Problem

Cargo updates the index repeatedly even though we are installing all the binaries at once.
This seems inefficient and overkill, as it's unlikely that the index would change much between downloading one crate and moving to the next.

See steps section for example.

Steps

Install multiple cargo binary crates at once, like so

cargo install eza bat ripgrep

Example docker run

[1/2] STEP 4/5: RUN cargo install $(cat useful-crates)
    Updating crates.io index
 Downloading crates ...
  Downloaded eza v0.18.3
    Updating crates.io index
 Downloading crates ...
  Downloaded bat v0.24.0
    Updating crates.io index
 Downloading crates ...
  Downloaded zoxide v0.9.3
    Updating crates.io index
 Downloading crates ...
  Downloaded bottom v0.9.6
    Updating crates.io index
 Downloading crates ...
  Downloaded ripgrep v14.1.0
    Updating crates.io index
 Downloading crates ...
  Downloaded xsv v0.13.0
    Updating crates.io index
 Downloading crates ...
  Downloaded fd-find v9.0.0
    Updating crates.io index
 Downloading crates ...
  Downloaded yazi-fm v0.2.3
    Updating crates.io index
 Downloading crates ...
  Downloaded du-dust v0.9.0
    Updating crates.io index
 Downloading crates ...
  Downloaded hexyl v0.14.0
    Updating crates.io index
 Downloading crates ...
  Downloaded bingrep v0.11.0
    Updating crates.io index
 Downloading crates ...
  Downloaded git-delta v0.16.5
    Updating crates.io index
 Downloading crates ...
  Downloaded gitui v0.24.3
    Updating crates.io index
 Downloading crates ...
  Downloaded ouch v0.5.1
    Updating crates.io index
 Downloading crates ...
  Downloaded nu v0.90.1
    Updating crates.io index
 Downloading crates ...
  Downloaded starship v1.17.1
    Updating crates.io index
 Downloading crates ...
  Downloaded watchexec-cli v1.25.1
  Installing eza v0.18.3
    Updating crates.io index
 Downloading crates ...
  Downloaded palette_derive v0.7.4
  Downloaded idna v0.5.0
  Downloaded either v1.10.0
  Downloaded base64 v0.21.7
  .
  .
  .

Possible Solution(s)

Update index only once per command invocation?

Notes

No response

Version

cargo 1.76.0 (c84b36747 2024-01-18)
release: 1.76.0
commit-hash: c84b367471a2db61d2c2c6aab605b14130b8a31b
commit-date: 2024-01-18
host: x86_64-unknown-linux-gnu

# The one on latest docker image https://github.com/rust-lang/docker-rust/blob/e5921dbae8fe56407a992307d1e2c2716ba8202d/1.76.0/bookworm/Dockerfile

[weihanglo]

Are you using sparse registry protocol?

I believe the Updating crates.io index fetches index files on demand for each crate, so it is necessary for sparse index protocol. And in fact it is more efficient. Have you spotted anything that is really redundant?

If you switch to git index protocol, that index update could be down to one pass. Instead you'll need to download the entire crates.io index upfront, and that will kill your docker container.

[nain-F49FF806]

@weihanglo Thanks. TIL about the sparse registry protocol. I haven't changed the defaults, so according to this page, I suppose sparse is indeed being used. That is a relief to know.

Seeing the results of benchmarking the protocol implementation here, I am satisfied and shall be staying with sparse.

Out of curiosity, though, I cloned the git index manually on my machine (with --depth=1). And while the download was small enough ~100M, I didn't realise it'd be so huge checked out ~1.9G ! That indicates to me that this route might be optimised though (by shallow cloning the git index and sparse checking out only the necessary files). But this may be out of scope for the issue. So please feel free to close. :)

[weihanglo]

For shallow clone, we have -Zgitoxide and -Zgit currently experimenting that. For sparse checkouts, see #11165.

Thanks for the suggestion anyway :)