[core] [2/N] Implement uv processor #48486

dentiny · 2024-11-01T04:14:58Z

This PR does two things:

Implement the uv and package installation
Extract common utils between uv and pip into multiple util files

Signed-off-by: hjiang <hjiang@anyscale.com>

python/ray/_private/runtime_env/uv_test.py

Signed-off-by: hjiang <hjiang@anyscale.com>

dentiny · 2024-11-05T01:28:18Z

.buildkite/data.rayci.yml

@@ -29,28 +29,28 @@ steps:

  # tests
  - label: ":database: data: arrow v9 tests"
-    tags: 
+    tags:


linter makes all these changes, otherwise I cannot push

could you split the unrelated lint changes to another PR?

That's a good suggestion and I thought about it also, my confusion is, when I tried to push on the master branch, prepush hook doesn't seem to work :(

you can just checkout the changes of these files.

I think the hook is configured to fix changed files only.

the people who added the pre-commit's should have fixed all the lint errors across the entire repo first.. and then run it on CI with -a flag..

dentiny · 2024-11-05T01:28:56Z

rllib/algorithms/dreamerv3/README.md

@@ -49,13 +49,13 @@ in combination with the following scripts and command lines in order to run RLli
 ### [Atari100k](../../tuned_examples/dreamerv3/atari_100k.py)
 ```shell
 $ cd ray/rllib/tuned_examples/dreamerv3/
-$ python atari_100k.py --env ale_py:ALE/Pong-v5 
+$ python atari_100k.py --env ale_py:ALE/Pong-v5


Prepush linter hook change, otherwise I cannot push

what prepush linter are you using? the pre-commit one?

Yes. I add it to my git pre-push hook, which automatically fixes linting issues before push.

PR: #48557, I guess it needs to be merged before this PR, otherwise cannot update and push still. :(

release/ray_release/byod/requirements_byod_3.9.txt

jjyao

LG.

Also rebase with master to for [1/N]

python/ray/_private/runtime_env/env_utils.py

python/ray/_private/runtime_env/path_utils.py

python/ray/_private/runtime_env/uv.py

Signed-off-by: hjiang <hjiang@anyscale.com>

dentiny · 2024-11-05T05:51:23Z

Also rebase with master to for [1/N]

This PR has already rebased upon first PR, which gets merged earlier today :)

python/ray/_private/runtime_env/validation_test.py

Signed-off-by: hjiang <hjiang@anyscale.com>

python/ray/tests/unit/test_uv.py

python/ray/_private/runtime_env/uv.py

jjyao · 2024-11-05T06:20:48Z

python/ray/runtime_env/runtime_env.py

@@ -310,6 +311,7 @@ def __init__(
        _validate: bool = True,
        mpi: Optional[Dict] = None,
        image_uri: Optional[str] = None,
+        uv: Optional[str] = None,


Is this type annotation right?

Just revisited, I think both pip and uv type annotation is incorrect, fixed.

jjyao · 2024-11-05T06:25:16Z

python/ray/tests/unit/test_uv.py

+    runtime_env = TestRuntimeEnv()
+
+    uv_processor = uv.UvProcessor(target_dir=target_dir, runtime_env=runtime_env)
+    await uv_processor._run()


What we are testing here?

The whole uv process only does download work, which is not expected in unit test.

For here, I simply goes though the initialization and mocks all the IO operations out.
It might sound weird, but I did uncover a few bugs during UT (i.e. pip not correctly replaced with uv).

Signed-off-by: hjiang <hjiang@anyscale.com>

python/ray/_private/runtime_env/uv.py

jjyao · 2024-11-05T15:09:41Z

python/ray/runtime_env/runtime_env.py

+                "The 'pip' field, 'conda' field and 'uv' field of "
                "runtime_env cannot both be specified.\n"


Only one of 'pip' field, 'conda' field and 'uv' field of runtime_env can be specified.

Follow with another comment, revert all exposed public interfaces.

jjyao · 2024-11-05T15:11:27Z

python/ray/runtime_env/runtime_env.py

+        uv: Either a list of pip packages, or a python dictionary that has one fields:
+            packages (required, List[str]): a lists of uv packages.


We should add the doc when 'uv' is usable. With this PR, people cannot use 'uv' plugin yet.

I think changes to this file should be in the next PR when we actually implement the uv plugin.

Makes sense! Updated.

Signed-off-by: dentiny <dentinyhao@gmail.com>

python/ray/_private/runtime_env/uv.py

Signed-off-by: dentiny <dentinyhao@gmail.com>

dentiny · 2024-11-06T08:36:02Z

@jjyao Finally I cleaned up all the stale usage after the refactor, could you please help me merge the PR? Thank you!

Signed-off-by: hjiang <hjiang@anyscale.com>

Signed-off-by: hjiang <hjiang@anyscale.com> Signed-off-by: mohitjain2504 <mohit.jain@dream11.com>

dentiny added 2 commits October 31, 2024 23:45

validate uv options

97bdec9

Signed-off-by: hjiang <hjiang@anyscale.com>

[core] uv processor to install packages

4c4bf37

Signed-off-by: hjiang <hjiang@anyscale.com>

dentiny requested review from pcmoritz, jjyao and rynewang November 1, 2024 04:14

jjyao reviewed Nov 1, 2024

View reviewed changes

python/ray/_private/runtime_env/uv_test.py Outdated Show resolved Hide resolved

python/ray/_private/runtime_env/uv_test.py Outdated Show resolved Hide resolved

dentiny mentioned this pull request Nov 1, 2024

[core] [1/N] Validate uv options #48479

Merged

3 tasks

dentiny requested a review from jjyao November 1, 2024 20:25

dentiny added 2 commits November 5, 2024 00:53

Merge branch 'master' into hjiang/implement-uv-processor

ab2edc7

use pytest for unit tests

dd857ec

Signed-off-by: hjiang <hjiang@anyscale.com>

dentiny requested review from a team, sven1977, simonsays1980, scottjlee, bveeramani, raulchen, stephanie-wang, omatthew98 and alexeykudinkin as code owners November 5, 2024 01:27

dentiny commented Nov 5, 2024

View reviewed changes

dentiny added the go add ONLY when ready to merge, run all tests label Nov 5, 2024

khluu reviewed Nov 5, 2024

View reviewed changes

release/ray_release/byod/requirements_byod_3.9.txt Outdated Show resolved Hide resolved

dentiny requested a review from khluu November 5, 2024 01:43

jjyao reviewed Nov 5, 2024

View reviewed changes

dentiny added 2 commits November 5, 2024 05:43

fix logging typo

310ed00

Signed-off-by: hjiang <hjiang@anyscale.com>

rename to virtual env

930a14a

Signed-off-by: hjiang <hjiang@anyscale.com>

jjyao reviewed Nov 5, 2024

View reviewed changes

python/ray/_private/runtime_env/validation_test.py Outdated Show resolved Hide resolved

cleanup git conflict

ab31734

Signed-off-by: hjiang <hjiang@anyscale.com>

dentiny requested a review from jjyao November 5, 2024 06:04

jjyao reviewed Nov 5, 2024

View reviewed changes

dentiny added 6 commits November 5, 2024 06:36

rename env uv test

0946c8a

Signed-off-by: hjiang <hjiang@anyscale.com>

Add back missing import

f82016c

Signed-off-by: hjiang <hjiang@anyscale.com>

comment for --disable-pip-version-check

18e121e

Signed-off-by: hjiang <hjiang@anyscale.com>

rename dump deps to gen deps

8a0d275

Signed-off-by: hjiang <hjiang@anyscale.com>

fix uv and pip annotation

e385e54

Signed-off-by: hjiang <hjiang@anyscale.com>

add sanity check for uv pip and conda

df4b43f

Signed-off-by: hjiang <hjiang@anyscale.com>

dentiny requested a review from jjyao November 5, 2024 06:56

Merge branch 'master' into hjiang/implement-uv-processor

149dd02

jjyao reviewed Nov 5, 2024

View reviewed changes

dentiny added 4 commits November 5, 2024 18:14

fix typo

c9a93f8

Signed-off-by: dentiny <dentinyhao@gmail.com>

remove unnecessaery pip arg

d5dafad

Signed-off-by: dentiny <dentinyhao@gmail.com>

revert uv exposure to interfaces

0a92a11

Signed-off-by: dentiny <dentinyhao@gmail.com>

fix uv install param

606e1c9

Signed-off-by: dentiny <dentinyhao@gmail.com>

dentiny requested a review from jjyao November 5, 2024 18:27

dentiny added 2 commits November 5, 2024 18:28

Merge branch 'master' into hjiang/implement-uv-processor

13f3e25

anothert typo -> uv

33a1df8

Signed-off-by: dentiny <dentinyhao@gmail.com>

jjyao approved these changes Nov 5, 2024

View reviewed changes

python/ray/_private/runtime_env/uv.py Show resolved Hide resolved

python/ray/_private/runtime_env/uv.py Show resolved Hide resolved

dentiny added 6 commits November 5, 2024 19:25

move documentations around

58d48b0

Signed-off-by: dentiny <dentinyhao@gmail.com>

fix package

9374ad0

Signed-off-by: dentiny <dentinyhao@gmail.com>

fix package name

7f165c3

Signed-off-by: dentiny <dentinyhao@gmail.com>

fix another package import

0d4e6e2

Signed-off-by: dentiny <dentinyhao@gmail.com>

fix function name...

327f148

Signed-off-by: dentiny <dentinyhao@gmail.com>

fix bash test

8d33d5d

Signed-off-by: dentiny <dentinyhao@gmail.com>

jjyao merged commit a49978f into ray-project:master Nov 6, 2024
5 checks passed

JP-sDEV pushed a commit to JP-sDEV/ray that referenced this pull request Nov 14, 2024

[core] [2/N] Implement uv processor (ray-project#48486)

1a921d4

Signed-off-by: hjiang <hjiang@anyscale.com>

mohitjain2504 pushed a commit to mohitjain2504/ray that referenced this pull request Nov 15, 2024

[core] [2/N] Implement uv processor (ray-project#48486)

d1ffe48

Signed-off-by: hjiang <hjiang@anyscale.com> Signed-off-by: mohitjain2504 <mohit.jain@dream11.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[core] [2/N] Implement uv processor #48486

[core] [2/N] Implement uv processor #48486

dentiny commented Nov 1, 2024

dentiny Nov 5, 2024

aslonnie Nov 5, 2024

dentiny Nov 5, 2024

aslonnie Nov 5, 2024

dentiny Nov 5, 2024

aslonnie Nov 5, 2024

dentiny Nov 5, 2024

dentiny Nov 5, 2024 •

edited

Loading

jjyao left a comment •

edited

Loading

dentiny commented Nov 5, 2024

jjyao Nov 5, 2024

dentiny Nov 5, 2024

jjyao Nov 5, 2024

dentiny Nov 5, 2024 •

edited

Loading

jjyao Nov 5, 2024

dentiny Nov 5, 2024

jjyao Nov 5, 2024

dentiny Nov 5, 2024

dentiny commented Nov 6, 2024

		"The 'pip' field, 'conda' field and 'uv' field of "
		"runtime_env cannot both be specified.\n"

		uv: Either a list of pip packages, or a python dictionary that has one fields:
		packages (required, List[str]): a lists of uv packages.

[core] [2/N] Implement uv processor #48486

[core] [2/N] Implement uv processor #48486

Conversation

dentiny commented Nov 1, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dentiny Nov 5, 2024 • edited Loading

Choose a reason for hiding this comment

jjyao left a comment • edited Loading

Choose a reason for hiding this comment

dentiny commented Nov 5, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dentiny Nov 5, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dentiny commented Nov 6, 2024

dentiny Nov 5, 2024 •

edited

Loading

jjyao left a comment •

edited

Loading

dentiny Nov 5, 2024 •

edited

Loading