extend fast-deps to sdists (and allow downloading foreign sdists) (working prototype!)

[cosmicexplorer]

What's the problem this feature will solve?

1. --use-feature=fast-deps is awesome. After Download wheels in batch at the end of .prepare_linked_requirements_more() #8896, the stage is set to try downloading all the lazy deps in parallel at once! However, we still run into a mild slowdown by downloading, unpacking, then doing some further processing on every non-wheel dependency! It turns out this isn't necessary.
2. It is not possible to invoke pip to resolve transitive dependencies for pip download or for a --dry-run resolve as per Add pip install --dry-run or similar, to get resolution result #53 if any of those dependencies are only provided as sdists.

Describe the solution you'd like
Extend fast-deps to sdists by extracting the single PKG-INFO file, without any further processing. This file existed in every sdist download I was able to quickly check on pypi (~5000 projects).

• This should improve the parallelism available when building sdists in pip wheel, and downloading dists in pip download, once that feature is implemented.
• As described in Add pip install --dry-run or similar, to get resolution result #53 (comment), it would be wonderful to perform a "dry run" resolve without having to incur the full download costs at all. The use case described in that comment is a PEX file which resolves its dependencies when first executed, allowing the deployed PEX to be much much smaller. Right now, it would still be impossible to deploy such an "ipex" file to a foreign platform if there were any sdist-only dependencies, even if Add pip install --dry-run or similar, to get resolution result #53 was otherwise fully implemented.

Alternative Solutions
Pex invokes pip for all download, wheel-building, and installation tasks, so its API for resolving wheels at least is predicated upon any limitations in pip. In this case, it would be lovely to improve download performance as well as enable dry-run resolves in Pex, but it's not clear to me whether there is any workaround without upstream pip changes.

Additional context
This is prototyped in the branch https://github.com/cosmicexplorer/pip/tree/fast-deps-sdists (on top of #8896) and appears to work really smoothly.

[pfmoore]

Unfortunately, PKG-INFO isn't reliable for sdists - see this discussion.

Doing something like this is probably blocked on standardising sdist metadata...

[cosmicexplorer]

This is fantastically helpful, thank you!

[cosmicexplorer]

At the very least, this branch can remain as an easily-patchable testament to the benefit of following through on that standardization, if we can demonstrate the performance improvements I've claimed from parallelism using this technique. The second idea of crawling foreign sdists is probably a little more specialized, but also quite exciting too.

[cosmicexplorer]

Closing this issue as I cannot see how it could ever work reliably given the discussion linked in #8929 (comment), where requests-2.22.0.tar.gz's PKG-INFO file does not contain any Requires-Dist entries. I will be replacing this issue with a couple alternative ideas for different use cases.

[pradyunsg]

We do have PEP 643, but that should cover sdists via the PyPI implementation of exposing metadata files.