Sparse-registry fetching total size keeps increasing

[pickfire]

Problem

From original thread https://internals.rust-lang.org/t/call-for-testing-cargo-sparse-registry/16862/20?u=pickfire

One thing I noticed is that the fetching need to go through several loops now, previously it just fetch until 100% then done, but now when it reached 100%, then it had to fetch another time again, and the total stuff needed to fetch which keeps increasing (1 -> 10 -> 50 -> 100 -> 130) resulting in the progress decreasing, feels weird.

I feel like the total should be - until there is a definite answer to the total number that needs to fetch then only show it rather than keep increasing it.

Steps

1. rustup update nightly
2. cargo +nightly -Z sparse-registry update

Possible Solution(s)

A way to improve that in the registry would be to on publish, provide an optional expected_dep_forest_size from the number of transitive dependencies used at publish time. This number could be used to provide a reasonable progress estimate quickly.

Notes

No response

Version

No response

[arlosi]

This is expected behavior, but I understand it's confusing to have progress go backwards.

@rustbot modify labels: -C-bug +C-enhancement

[rustbot]

Error: The feature relabel is not enabled in this repository.
To enable it add its section in the triagebot.toml in the root of the repository.

Please file an issue on GitHub at triagebot if there's a problem with this bot, or reach out on #t-infra on Zulip.

[arlosi]

Some additional ideas on what we could do.

Remove the progress bar

• But keep the X/Y status. Y will still change as deps are discovered.

Fetching (35/38)

Include an estimate of the dependency graph size when publishing

• Dependency graph size can change depending on features, platform, etc.
• Would require running a batch job to generate for all existing crate versions
• Increases metadata size

Attempt to run the resolver using cached metadata to get an estimate

• Only works if the local cache is fresh
• Wastes CPU time

Use the lockfile to get an estimate of the total number of dependencies

• Only works if a lockfile is present.

[pickfire]

But keep the X/Y status. Y will still change as deps are discovered.

That sounds pretty much the same issue to me, it's basically a progress bar that never ends, I rather it be X/- or X/∞ if we don't know it yet.

[arlosi]

Maybe instead of 35/38 something like:

Fetching (35 completed; 3 downloading)

[pickfire]

Or maybe X/N+3 then until it is confirmed, X/Y

[arlosi]

What do you mean by confirmed? Each dependency that's fetched could potentially require additional dependencies that aren't fetched yet.

[pickfire]

Confirmed as in cargo knows that there won't be any additional dependencies.