[FR] Deduplicate vendored dependencies

[mgorny]

What's the problem this feature will solve?

Currently, pkg_resources, setuptools and setuptools._distutils all have their own _vendor directories. In many cases, the vendored packages overlap — especially packaging is now vendored 3 times. Besides potential version mismatches such as #4323, it also increases the disk space use and wheel size. From a quick and dirty deduplication attempt, compressed wheel goes down from 920 KiB to 710 KiB, and the install from 3.5 MiB to 2.7 MiB.

Describe the solution you'd like

I think it would make sense to establish some kind of reuse hierarchy, and e.g. given that setuptools imports stuff from pkg_resources anyway, make them reuse the vendored packages from pkg_resources as well, and only vendor these that aren't common to both.

Alternative Solutions

No response

Additional context

No response

Code of Conduct

• I agree to follow the PSF Code of Conduct

[jaraco]

Thanks for the suggestion.

Since pkg_resources and distutils is deprecated, I'd like to keep these concerns separate.

Distutils needs its own vendored dependencies because it's still tested independently in pypa/distutils.

I'm trying to make setuptools not depend at all on pkg_resources. The long-term strategy is to stop vendoring packages altogether, or at least declare the dependencies normally and only provide vendored copies as a fallback. Here's how I'm thinking that would work:

• vendored dependencies would only be supported when installed on the file system (limited or no support for zip-based or other alternative installs).
• vendored dependencies would be stored unaltered to a directory or set of directories that would be added to the end of sys.path when necessary (bootstrapping), but naturally installed dependencies would be preferred.
• These vendored dependencies might be provided in a top-level package that could be shared between pkg_resources and setuptools (_setuptools_vendored?).

This approach would make setuptools largely behave as a naturally-installed package (no vendoring), but provide the dependencies for bootstrapping purposes (maybe only opt-in).

Let's focus on those efforts and set aside 1MiB of overhead until we've tackled these other concerns.

[mgorny]

Sure, that would work as well. In fact, I think it would make devendoring much easier than it is now.