Cache downloaded wheel when range requests aren't supported

[charliermarsh]

Summary

When range requests aren't supported, we fall back to streaming the wheel, stopping as soon as we hit a METADATA file. This is a small optimization, but the downside is that we don't get to cache the resulting wheel...

We don't know whether METADATA will be at the beginning or end of the wheel, but it seems like a better tradeoff to download and cache the entire wheel?

Closes: #5088.

Sort of a revert of: #1792.

[charliermarsh]

I'm sort of torn on this.

[zanieb]

Would it be a pain to use a heuristic like... only finish downloading if it's greater than some percentage into the file? If we find a metadata entry 10% into the file it seems excessive to unconditionally download the whole wheel when we're trying many versions.

[morotti]

We don't know whether METADATA will be at the beginning or end of the wheel, but it seems like a better tradeoff to download and cache the entire wheel?

METADATA are at end of the wheel, as per PEP 407. https://peps.python.org/pep-0427/#recommended-archiver-features

I vaguely recall there was a similar discussion in pip to try to only download metadata (I can't find the thread anymore). They checked many packages and only found one exception to confirm the rule, some google packages had the metadata at the start (around tensorflow if I recall well), because google used their own build system that did its own thing. I think they fixed it a while back.

[charliermarsh]

@zanieb - Perhaps... It would be harder to implement for sure. Given that this is not the happy path and PEP 407 recommends putting the archive at the end anyway, I'm inclined to just move forward with it.