feat: use external pip if available #736

henryiii · 2024-02-19T22:26:50Z

Trying #733 out. Pip is used if it is in the same environment as build and new enough. Saves a bit of time setting up the venvs. Haven't worked on tests yet.

From quick testing building both an sdist and wheel for itself with virtualenv installed this goes from about 4.7 seconds to 3.2 seconds. With venv only, it goes from 10.5 seconds to 4.4 seconds.

FYI, with uv (hacked in) it took 2.36 secs.

(This is a very simple build, of course, with only a small handful of deps)

uv's experimental native build support took 571ms, but that's only building the wheel (above is SDist and wheel).

Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>

layday · 2024-02-25T19:32:31Z

src/build/env.py

+
+    try:
+        return _has_valid_pip()
+    except ModuleNotFoundError:  # Python 3.7 only


Why does this happen?

I would prefer if we always threw this from _has_valid_pip instead of the cryptic StopIteration, which arguably you should never have to handle, especially at a distance. It would also match importlib's implementation of distribution (singular):

try: return next(cls.discover(name=name)) except StopIteration: raise PackageNotFoundError(name)

(PackageNotFoundError being a subclass of ModuleNotFoundError.)

Okay. And the error was if importlib_metadata was missing, though we have it as a dep, so that would only be for bootstrapping. This makes it have the same signature as pip missing, though, so no need to special case.

layday · 2024-02-25T19:39:52Z

src/build/env.py

@@ -83,13 +83,29 @@ def _has_valid_pip(purelib: str) -> bool:
    else:
        from importlib import metadata


Import this from _importlib.

Forgot about that. I always call it _compat/importlib (and have a _compat/typing, etc).

+1 to rename it to _compat/importlib.

layday · 2024-02-25T19:41:40Z

src/build/env.py

@@ -70,7 +70,7 @@ def _minimum_pip_version() -> str:
    return '19.1.0'


-def _has_valid_pip(purelib: str) -> bool:
+def _has_valid_pip(**distargs: object) -> bool:


I'd rather we didn't lose the typing here, but Python doesn't make it easy:

if typing.TYPE_CHECKING: from typing_extensions import NotRequired, TypedDict, Unpack class _Distargs(TypedDict): path: NotRequired[list[str]] ... def _has_valid_pip(**distargs: Unpack[_Distargs]) -> bool:

I left it along for now, would prefer a _compat/typing module if we add it.

I can do that in a separate PR. I also usually have _compat/tomllib (https://github.com/scikit-build/scikit-build-core/tree/main/src/scikit_build_core/_compat for example).

layday · 2024-02-25T19:48:03Z

src/build/env.py

@@ -139,6 +155,12 @@ def python_executable(self) -> str:
        """The python executable of the isolated build environment."""
        return self._python_executable

+    def _pip_args(self, *, isolate: bool = False) -> list[str]:


This doesn't appear to ever be called without isolate being true.

Okay, dropped the arg.

layday · 2024-02-25T19:49:39Z

src/build/env.py

@@ -201,7 +221,11 @@ def _create_isolated_env_virtualenv(path: str) -> tuple[str, str]:
    """
    import virtualenv

-    cmd = [str(path), '--no-setuptools', '--no-wheel', '--activators', '']
+    if _valid_global_pip():
+        cmd = [str(path), '--no-seed', '--activators', '']


The path is already a string.

Just inherited from previous code, but okay, changed.

layday · 2024-02-25T19:53:47Z

src/build/env.py

        _subprocess([executable, '-m', 'pip', 'install', f'pip>={_minimum_pip_version()}'])

    # Avoid the setuptools from ensurepip to break the isolation
-    _subprocess([executable, '-m', 'pip', 'uninstall', 'setuptools', '-y'])
+    if _valid_global_pip():


Redundant, setuptools is not installed if with_pip is false. (It is also not installed on Python 3.12+.)

Ah, okay, yes. Removed this branch.

Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>

layday

Thanks!

henryiii added 2 commits February 19, 2024 17:25

feat: use external pip if available

a68a36f

Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>

fix(types): make the passthrough general

e83bfa6

Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>

henryiii marked this pull request as ready for review February 23, 2024 00:29

henryiii requested review from FFY00, gaborbernat and layday as code owners February 23, 2024 00:29

layday reviewed Feb 25, 2024

View reviewed changes

henryiii force-pushed the henryiii/feat/externalpip branch 2 times, most recently from c5ac342 to afe6f02 Compare February 26, 2024 17:22

chore: address feedback

04dd0d1

Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>

henryiii force-pushed the henryiii/feat/externalpip branch from afe6f02 to 04dd0d1 Compare February 26, 2024 17:43

layday approved these changes Feb 26, 2024

View reviewed changes

layday merged commit 894998a into pypa:main Feb 26, 2024
61 checks passed

henryiii deleted the henryiii/feat/externalpip branch February 26, 2024 20:41

cr1901 mentioned this pull request Feb 29, 2024

no such option: --python #745

Closed

zvecr mentioned this pull request Jun 24, 2024

Bump minimum python version to 3.9 qmk/qmk_cli#178

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: use external pip if available #736

feat: use external pip if available #736

henryiii commented Feb 19, 2024 •

edited

Loading

layday Feb 25, 2024

layday Feb 25, 2024 •

edited

Loading

henryiii Feb 26, 2024

layday Feb 25, 2024

henryiii Feb 26, 2024

layday Feb 26, 2024

layday Feb 25, 2024

henryiii Feb 26, 2024

henryiii Feb 26, 2024

layday Feb 25, 2024 •

edited

Loading

henryiii Feb 26, 2024

layday Feb 25, 2024

henryiii Feb 26, 2024

layday Feb 25, 2024

henryiii Feb 26, 2024

layday left a comment

		@@ -83,13 +83,29 @@ def _has_valid_pip(purelib: str) -> bool:
		else:
		from importlib import metadata

feat: use external pip if available #736

feat: use external pip if available #736

Conversation

henryiii commented Feb 19, 2024 • edited Loading

Choose a reason for hiding this comment

layday Feb 25, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

layday Feb 25, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

layday left a comment

Choose a reason for hiding this comment

henryiii commented Feb 19, 2024 •

edited

Loading

layday Feb 25, 2024 •

edited

Loading

layday Feb 25, 2024 •

edited

Loading