-
Notifications
You must be signed in to change notification settings - Fork 5.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[core] [1/N] Validate uv options #48479
Changes from 1 commit
97bdec9
b8f757a
dfd13b3
612aa8b
9dc3b77
60c0000
54cf250
6d4087f
d08af64
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,17 @@ | ||
load("@rules_python//python:defs.bzl", "py_library", "py_test") | ||
|
||
package(default_visibility = ["//visibility:public"]) | ||
|
||
py_library( | ||
name = "validation", | ||
srcs = ["validation.py"], | ||
) | ||
|
||
py_test( | ||
name = "validation_test", | ||
srcs = ["validation_test.py"], | ||
tags = ["team:core"], | ||
deps = [ | ||
":validation", | ||
], | ||
) |
Original file line number | Diff line number | Diff line change | ||||||||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
@@ -107,6 +107,66 @@ def parse_and_validate_conda(conda: Union[str, dict]) -> Union[str, dict]: | |||||||||||||||||||||||||||||||||||||
return result | ||||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||||
# TODO(hjiang): More package installation options to implement: | ||||||||||||||||||||||||||||||||||||||
# 1. Allow users to pass in a local requirements.txt file, which relates to all | ||||||||||||||||||||||||||||||||||||||
# packages to install; | ||||||||||||||||||||||||||||||||||||||
# 2. Allow specific version of `uv` to use; as of now we only use latest version. | ||||||||||||||||||||||||||||||||||||||
def parse_and_validate_uv(uv: Union[str, List[str], Dict]) -> Optional[Dict]: | ||||||||||||||||||||||||||||||||||||||
"""Parses and validates a user-provided 'uv' option. | ||||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||||
The value of the input 'uv' field can be one of two cases: | ||||||||||||||||||||||||||||||||||||||
1) A List[str] describing the requirements. This is passed through. | ||||||||||||||||||||||||||||||||||||||
Example usage: "packages":["tensorflow", "requests"] | ||||||||||||||||||||||||||||||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. There're three options supported in runtime env now:
I feel remove the key is better, just to reduce confusion on list vs dict, updated. |
||||||||||||||||||||||||||||||||||||||
2) A python dictionary that has three fields: | ||||||||||||||||||||||||||||||||||||||
a) packages (required, List[str]): a list of uv packages, it same as 1). | ||||||||||||||||||||||||||||||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. what are the other 2 fields? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Sorry I don't understand, I'm trying to mimic the documentation for ray/python/ray/_private/runtime_env/validation.py Lines 111 to 128 in 056d596
Could you please tell me which part am I missing here? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I mean we should support pip_check field as well and possibly uv_version field. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. About About One concrete example might be, uv's |
||||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||||
The returned parsed value will be a list of packages. If a Ray library | ||||||||||||||||||||||||||||||||||||||
(e.g. "ray[serve]") is specified, it will be deleted and replaced by its | ||||||||||||||||||||||||||||||||||||||
dependencies (e.g. "uvicorn", "requests"). | ||||||||||||||||||||||||||||||||||||||
""" | ||||||||||||||||||||||||||||||||||||||
assert uv is not None | ||||||||||||||||||||||||||||||||||||||
if sys.platform == "win32": | ||||||||||||||||||||||||||||||||||||||
logger.warning( | ||||||||||||||||||||||||||||||||||||||
"runtime environment support is experimental on Windows. " | ||||||||||||||||||||||||||||||||||||||
"If you run into issues please file a report at " | ||||||||||||||||||||||||||||||||||||||
"https://github.com/ray-project/ray/issues." | ||||||||||||||||||||||||||||||||||||||
) | ||||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||||
result: str = "" | ||||||||||||||||||||||||||||||||||||||
if isinstance(uv, list) and all(isinstance(dep, str) for dep in uv): | ||||||||||||||||||||||||||||||||||||||
result = dict(packages=uv) | ||||||||||||||||||||||||||||||||||||||
elif isinstance(uv, dict): | ||||||||||||||||||||||||||||||||||||||
if set(uv.keys()) - {"packages"}: | ||||||||||||||||||||||||||||||||||||||
raise ValueError( | ||||||||||||||||||||||||||||||||||||||
"runtime_env['uv'] can only have these fields: " | ||||||||||||||||||||||||||||||||||||||
"packages, but got: " | ||||||||||||||||||||||||||||||||||||||
f"{list(uv.keys())}" | ||||||||||||||||||||||||||||||||||||||
) | ||||||||||||||||||||||||||||||||||||||
if "packages" not in uv: | ||||||||||||||||||||||||||||||||||||||
raise ValueError( | ||||||||||||||||||||||||||||||||||||||
f"runtime_env['uv'] must include field 'packages', but got {uv}" | ||||||||||||||||||||||||||||||||||||||
) | ||||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||||
result = uv.copy() | ||||||||||||||||||||||||||||||||||||||
if not isinstance(uv["packages"], list): | ||||||||||||||||||||||||||||||||||||||
raise ValueError( | ||||||||||||||||||||||||||||||||||||||
"runtime_env['uv']['packages'] must be of type list, " | ||||||||||||||||||||||||||||||||||||||
f"got: {type(uv['packages'])}" | ||||||||||||||||||||||||||||||||||||||
) | ||||||||||||||||||||||||||||||||||||||
else: | ||||||||||||||||||||||||||||||||||||||
raise TypeError( | ||||||||||||||||||||||||||||||||||||||
"runtime_env['uv'] must be of type " f"List[str], got {type(uv)}" | ||||||||||||||||||||||||||||||||||||||
dentiny marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||||||||||||||||||||||||||||||||||
) | ||||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||||
# Deduplicate packages for package lists. | ||||||||||||||||||||||||||||||||||||||
result["packages"] = list(OrderedDict.fromkeys(result["packages"])) | ||||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||||
if len(result["packages"]) == 0: | ||||||||||||||||||||||||||||||||||||||
result = None | ||||||||||||||||||||||||||||||||||||||
logger.debug(f"Rewrote runtime_env `uv` field from {uv} to {result}.") | ||||||||||||||||||||||||||||||||||||||
return result | ||||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||||
def parse_and_validate_pip(pip: Union[str, List[str], Dict]) -> Optional[Dict]: | ||||||||||||||||||||||||||||||||||||||
"""Parses and validates a user-provided 'pip' option. | ||||||||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||||||||
|
@@ -280,6 +340,7 @@ def parse_and_validate_env_vars(env_vars: Dict[str, str]) -> Optional[Dict[str, | |||||||||||||||||||||||||||||||||||||
"excludes": parse_and_validate_excludes, | ||||||||||||||||||||||||||||||||||||||
"conda": parse_and_validate_conda, | ||||||||||||||||||||||||||||||||||||||
"pip": parse_and_validate_pip, | ||||||||||||||||||||||||||||||||||||||
"uv": parse_and_validate_uv, | ||||||||||||||||||||||||||||||||||||||
"env_vars": parse_and_validate_env_vars, | ||||||||||||||||||||||||||||||||||||||
"container": parse_and_validate_container, | ||||||||||||||||||||||||||||||||||||||
} |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,36 @@ | ||
from python.ray._private.runtime_env import validation | ||
|
||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We already have There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. No plan at the moment, but I could put them in the same folder, if you want. Let me explain my thoughts:
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. If you check There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes, but the test suite and test file overall is an integration test, which we import latest version of whole If you're fine about it, may I move all unit test into a separate unit test file? I guess what you care about is, we should place related test in one single place? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes. Having all unit tests in one place. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Sounds good! I move a few test cases into my unit test file, also moved it under |
||
import unittest | ||
|
||
|
||
class TestVaidation(unittest.TestCase): | ||
def test_parse_and_validate_uv(self): | ||
# Valid case w/o duplication. | ||
result = validation.parse_and_validate_uv({"packages": ["tensorflow"]}) | ||
self.assertEqual(result, {"packages": ["tensorflow"]}) | ||
|
||
# Valid case w/ duplication. | ||
result = validation.parse_and_validate_uv( | ||
{"packages": ["tensorflow", "tensorflow"]} | ||
) | ||
self.assertEqual(result, {"packages": ["tensorflow"]}) | ||
|
||
# Valid case, use `list` to represent necessary packages. | ||
result = validation.parse_and_validate_uv( | ||
["requests==1.0.0", "aiohttp", "ray[serve]"] | ||
) | ||
self.assertEqual( | ||
result, {"packages": ["requests==1.0.0", "aiohttp", "ray[serve]"]} | ||
) | ||
|
||
# Invalid case, `str` is not supported for now. | ||
with self.assertRaises(TypeError) as _: | ||
result = validation.parse_and_validate_uv("./requirements.txt") | ||
|
||
# Invalid case, unsupport keys. | ||
with self.assertRaises(ValueError) as _: | ||
result = validation.parse_and_validate_uv({"random_key": "random_value"}) | ||
|
||
|
||
if __name__ == "__main__": | ||
unittest.main() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Default, we should use whatever version that's currently installed?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There're two possibilities:
uv
installed in the env already, we should use it without installation; this hasn't been implemented in [core] [2/N] Implement uv processor #48486, but I left a commentuv
found in the env, we should install the default version, as you mentionedIf user specify a particular version to use, then that's another story.