Switch to `uv pip compile` #2958

CoolCat467 · 2024-02-16T03:57:09Z

This pull request follows up on #2956 and is attempting to switch the continuous integration system to use astral.sh' new tool uv.
This particular pull request is a subset of #2957 and is in charge of only replacing pip-compile.

This seems nontrivial, can we break this into 2 prs?
1. replace pip-compile

2. replace pip install
Personally I see (1) as much more valuable-- (2) doesn't change much. Personally I'm more excited by uv's ability to get the lowest compatible version so we can add a new CI run than by it replacing pip install.

Originally posted by @A5rocks in #2957 (comment)

codecov · 2024-02-16T03:58:57Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 99.63%. Comparing base (17b3644) to head (96446a5).
Report is 89 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #2958   +/-   ##
=======================================
  Coverage   99.63%   99.63%           
=======================================
  Files         120      120           
  Lines       17830    17830           
  Branches     3204     3204           
=======================================
  Hits        17765    17765           
  Misses         46       46           
  Partials       19       19

A5rocks

Could you update autodeps too?

test-requirements.txt

A5rocks · 2024-02-16T04:43:37Z

test-requirements.txt

-    #   -r test-requirements.in
-    #   pytest
+exceptiongroup==1.2.0
+    # via pytest


This change too seems wrong

I think it's intended, as we specify --python-version=3.8. If we generate separate test-requirements-pyXX.txt files for each python version we test we'll only get exceptiongroup in the ones that are <3.11

So I think that means we should have a requirements directory wherein we generate txt files for all versions we test.

test-requirements.txt

I don't have 3.8 on my system you see

docs-requirements.txt

See python-trio#2958 (comment)

This reverts commit 383729f. It did not work.

A5rocks · 2024-06-26T00:42:08Z

uv seems to support keeping keeping things like implementation_name now: astral-sh/uv#4503

I think once this uses that flag, this is good for me

CoolCat467 · 2024-06-26T01:59:17Z

So, added --no-strip-markers, but since there hasn't been a proper release with these changes yet, using a git commit dependency for uv at the moment, which works, but CI fails because full cargo isn't installed on the windows or alpine runners.

I think we just need to wait for a proper release with these changes to come out, but otherwise this pull request should be almost done!

charliermarsh · 2024-06-26T12:54:04Z

I think we just need to wait for a proper release with these changes to come out, but otherwise this pull request should be almost done!

Heads up we plan to cut a release today.

CoolCat467 · 2024-06-27T06:00:19Z

Ok so it looks like everything is working except for an issue that happens when importing cryptography on the 3.13 runner, with this error:
E ImportError: /opt/hostedtoolcache/Python/3.13.0-beta.2/x64/lib/python3.13/site-packages/_cffi_backend.cpython-313-x86_64-linux-gnu.so: undefined symbol: _PyErr_WriteUnraisableMsg

I don't think this is caused by uv, probably an issue with cryptography.
Version that 3.13 runner used to have this result: cryptography-42.0.8

A5rocks · 2024-06-27T06:51:10Z

I'm surprised pulling in master didn't work. I manually edited the test requirements to use a pre-release of cffi (IIRC? Or maybe cryptography) to fix that.

test-requirements.txt

T-256

Since I already subscribed to this PR, here is my 2cents

test-requirements.in

test-requirements.txt

A5rocks

We might want to try out this new --universal flag (I'm not sure when it was added? but seems useful)

... but that can be a followup. This is good as a drop in replacement.

Co-authored-by: T-256 <132141463+T-256@users.noreply.github.com>

CoolCat467 · 2024-07-08T00:33:10Z

Weird SSL issue on MacOS 3.12 runner:

  ________________________ test_ssl_client_basics[tls13] _________________________
    + Exception Group Traceback (most recent call last):
    |   File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/_pytest/runner.py", line 341, in from_call
    |     result: Optional[TResult] = func()
    |                                 ^^^^^^
    |   File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/_pytest/runner.py", line 241, in <lambda>
    |     lambda: runtest_hook(item=item, **kwds), when=when, reraise=reraise
    |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pluggy/_hooks.py", line 513, in __call__
    |     return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pluggy/_manager.py", line 120, in _hookexec
    |     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pluggy/_callers.py", line 139, in _multicall
    |     raise exception.with_traceback(exception.__traceback__)
    |   File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pluggy/_callers.py", line 122, in _multicall
    |     teardown.throw(exception)  # type: ignore[union-attr]
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/_pytest/threadexception.py", line 87, in pytest_runtest_call
    |     yield from thread_exception_runtest_hook()
    |   File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/_pytest/threadexception.py", line 63, in thread_exception_runtest_hook
    |     yield
    |   File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pluggy/_callers.py", line 122, in _multicall
    |     teardown.throw(exception)  # type: ignore[union-attr]
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/_pytest/unraisableexception.py", line 90, in pytest_runtest_call
    |     yield from unraisable_exception_runtest_hook()
    |   File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/_pytest/unraisableexception.py", line 65, in unraisable_exception_runtest_hook
    |     yield
    |   File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pluggy/_callers.py", line 122, in _multicall
    |     teardown.throw(exception)  # type: ignore[union-attr]
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/_pytest/logging.py", line 850, in pytest_runtest_call
    |     yield from self._runtest_for(item, "call")
    |   File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/_pytest/logging.py", line 833, in _runtest_for
    |     yield
    |   File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pluggy/_callers.py", line 122, in _multicall
    |     teardown.throw(exception)  # type: ignore[union-attr]
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/_pytest/capture.py", line 878, in pytest_runtest_call
    |     return (yield)
    |             ^^^^^
    |   File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pluggy/_callers.py", line 122, in _multicall
    |     teardown.throw(exception)  # type: ignore[union-attr]
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/_pytest/skipping.py", line 257, in pytest_runtest_call
    |     return (yield)
    |             ^^^^^
    |   File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pluggy/_callers.py", line 103, in _multicall
    |     res = hook_impl.function(*args)
    |           ^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/_pytest/runner.py", line 183, in pytest_runtest_call
    |     raise e
    |   File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/_pytest/runner.py", line 173, in pytest_runtest_call
    |     item.runtest()
    |   File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/_pytest/python.py", line 1632, in runtest
    |     self.ihook.pytest_pyfunc_call(pyfuncitem=self)
    |   File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pluggy/_hooks.py", line 513, in __call__
    |     return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pluggy/_manager.py", line 120, in _hookexec
    |     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pluggy/_callers.py", line 139, in _multicall
    |     raise exception.with_traceback(exception.__traceback__)
    |   File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pluggy/_callers.py", line 103, in _multicall
    |     res = hook_impl.function(*args)
    |           ^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/_pytest/python.py", line 162, in pytest_pyfunc_call
    |     result = testfunction(**testargs)
    |              ^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/trio/testing/_trio_test.py", line 44, in wrapper
    |     return _core.run(
    |            ^^^^^^^^^^
    |   File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/trio/_core/_run.py", line 2305, in run
    |     raise runner.main_task_outcome.error
    |   File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/trio/_tests/test_ssl.py", line 436, in test_ssl_client_basics
    |     async with ssl_echo_server(client_ctx) as s:
    |   File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/contextlib.py", line 217, in __aexit__
    |     await anext(self.gen)
    |   File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/trio/_tests/test_ssl.py", line 190, in ssl_echo_server
    |     async with ssl_echo_server_raw(**kwargs) as sock:
    |   File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/contextlib.py", line 217, in __aexit__
    |     await anext(self.gen)
    |   File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/trio/_tests/test_ssl.py", line 172, in ssl_echo_server_raw
    |     async with trio.open_nursery() as nursery:
    |   File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/trio/_core/_run.py", line 959, in __aexit__
    |     raise combined_error_from_nursery
    | ExceptionGroup: Exceptions from Trio nursery (1 sub-exception)
    +-+---------------- 1 ----------------
      | Traceback (most recent call last):
      |   File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/trio/_threads.py", line 437, in to_thread_run_sync
      |     return msg_from_thread.unwrap()
      |            ^^^^^^^^^^^^^^^^^^^^^^^^
      |   File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/outcome/_impl.py", line 213, in unwrap
      |     raise captured_error
      |   File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/trio/_threads.py", line 363, in do_release_then_return_result
      |     return result.unwrap()
      |            ^^^^^^^^^^^^^^^
      |   File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/outcome/_impl.py", line 213, in unwrap
      |     raise captured_error
      |   File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/trio/_threads.py", line 382, in worker_fn
      |     ret = context.run(sync_fn, *args)
      |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
      |   File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/trio/_tests/test_ssl.py", line 134, in ssl_echo_serve_sync
      |     wrapped.unwrap()
      |   File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/ssl.py", line 1296, in unwrap
      |     s = self._sslobj.shutdown()
      |         ^^^^^^^^^^^^^^^^^^^^^^^
      | ssl.SSLWantWriteError: The operation did not complete (write) (_ssl.c:2685)
      +------------------------------------
  ----------------------------- Captured stdout call -----------------------------
  ssl_echo_serve_sync got unexpected error: The operation did not complete (write) (_ssl.c:2685)

Pretty sure it isn't related to this pull request's changes though, going to try to get CI to run again. Might be an instance of test flakiness.

Switch to uv pip compile

f71fe81

CoolCat467 added the dependencies Pull requests that update a dependency file label Feb 16, 2024

CoolCat467 requested a review from A5rocks February 16, 2024 03:58

CoolCat467 added 2 commits February 15, 2024 22:11

Apply uv's changes

6738ce4

Fix two lines I missed

76d8326

A5rocks requested changes Feb 16, 2024

View reviewed changes

Use uv pip compile for autodeps as well

9f597e0

CoolCat467 requested a review from A5rocks February 16, 2024 04:28

A5rocks reviewed Feb 16, 2024

View reviewed changes

CoolCat467 added 4 commits February 15, 2024 23:01

Move uv to test-requirements and fix autodeps updateing

a20b4ce

Bootstrap uv install

6c8d62b

I don't have 3.8 on my system you see

Update test-requrements.txt

876d6a5

Remove manual pip install uv from autodeps as well

69b5868

T-256 reviewed Feb 16, 2024

View reviewed changes

docs-requirements.txt Show resolved Hide resolved

CoolCat467 added 8 commits February 17, 2024 12:28

Update uv version to 0.1.3

7f211e5

Merge branch 'master' into pip-compile-switch-uv

a0a8b5a

Update uv to 0.1.6

4225702

Apply uv's changes

6747417

Attempt not including version specifier

383729f

See python-trio#2958 (comment)

Merge branch 'master' into pip-compile-switch-uv

72e3fbd

Revert "Attempt not including version specifier"

e5277f0

This reverts commit 383729f. It did not work.

Update uv to 0.1.11

d75ef85

CoolCat467 mentioned this pull request Feb 27, 2024

uv pip compile Does not retain PEP-508 environment markers in output astral-sh/uv#1429

Closed

CoolCat467 added 6 commits February 29, 2024 16:56

Merge branch 'master' into pip-compile-switch-uv

283e27b

Merge branch 'master' into pip-compile-switch-uv

148a995

Update to uv 0.1.13

1b95b71

Update to uv to 0.1.17

7759de8

Merge branch 'python-trio:master' into pip-compile-switch-uv

199160e

Merge branch 'master' into pip-compile-switch-uv

bb73318

CoolCat467 added 3 commits March 28, 2024 20:34

Update to uv 0.1.26

42eceae

Merge branch 'master' into pip-compile-switch-uv

4f1438d

Fix requirements files

1748ca9

CoolCat467 added 3 commits June 25, 2024 20:12

Merge branch 'master' into pip-compile-switch-uv

d1939e4

Update dependencies with new version of uv

db7a7fe

Use latest commit of uv with --no-strip-markers support

2c15691

Require future version of uv that does not currently exist

3a3778c

CoolCat467 added 2 commits June 27, 2024 00:46

Update to uv 0.2.17

7255bd8

Merge branch 'master' into pip-compile-switch-uv

1b51930

A5rocks reviewed Jun 27, 2024

View reviewed changes

test-requirements.txt Show resolved Hide resolved

Fix cffi

d51aae3

CoolCat467 requested review from A5rocks and jakkdl June 27, 2024 07:04

jakkdl approved these changes Jun 28, 2024

View reviewed changes

Merge branch 'main' into pip-compile-switch-uv

572891a

T-256 approved these changes Jul 7, 2024

View reviewed changes

test-requirements.in Outdated Show resolved Hide resolved

test-requirements.txt Show resolved Hide resolved

test-requirements.txt Show resolved Hide resolved

T-256 mentioned this pull request Jul 7, 2024

Filter and flatten markers astral-sh/uv#4639

Merged

A5rocks approved these changes Jul 7, 2024

View reviewed changes

Apply suggestions from code review

dd5aa1a

Co-authored-by: T-256 <132141463+T-256@users.noreply.github.com>

Trigger CI

96446a5

CoolCat467 enabled auto-merge (squash) July 8, 2024 00:38

CoolCat467 merged commit 72f5931 into python-trio:main Jul 8, 2024
35 checks passed

CoolCat467 deleted the pip-compile-switch-uv branch July 8, 2024 00:49

CoolCat467 mentioned this pull request Jul 8, 2024

Use --universal flag with uv pip compile #3032

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Switch to `uv pip compile` #2958

Switch to `uv pip compile` #2958

CoolCat467 commented Feb 16, 2024 •

edited

Loading

codecov bot commented Feb 16, 2024 •

edited

Loading

A5rocks left a comment

A5rocks Feb 16, 2024

jakkdl Feb 16, 2024

jakkdl Feb 16, 2024

A5rocks commented Jun 26, 2024

CoolCat467 commented Jun 26, 2024

charliermarsh commented Jun 26, 2024

CoolCat467 commented Jun 27, 2024

A5rocks commented Jun 27, 2024

T-256 left a comment

A5rocks left a comment

CoolCat467 commented Jul 8, 2024

Switch to uv pip compile #2958

Switch to uv pip compile #2958

Conversation

CoolCat467 commented Feb 16, 2024 • edited Loading

codecov bot commented Feb 16, 2024 • edited Loading

Codecov Report

A5rocks left a comment

Choose a reason for hiding this comment

A5rocks Feb 16, 2024

Choose a reason for hiding this comment

jakkdl Feb 16, 2024

Choose a reason for hiding this comment

jakkdl Feb 16, 2024

Choose a reason for hiding this comment

A5rocks commented Jun 26, 2024

CoolCat467 commented Jun 26, 2024

charliermarsh commented Jun 26, 2024

CoolCat467 commented Jun 27, 2024

A5rocks commented Jun 27, 2024

T-256 left a comment

Choose a reason for hiding this comment

A5rocks left a comment

Choose a reason for hiding this comment

CoolCat467 commented Jul 8, 2024

Switch to `uv pip compile` #2958

Switch to `uv pip compile` #2958

CoolCat467 commented Feb 16, 2024 •

edited

Loading

codecov bot commented Feb 16, 2024 •

edited

Loading