Make Ray compatible with `pydantic>=2.5.0` #40451

edoakes · 2023-10-18T16:30:29Z

Why are these changes needed?

This change makes Ray compatible with pydantic>=2.5.0. It makes all Ray unit tests use pydantic==2.5.0. It also adds a test suite to check that Ray Serve works with pydantic<2.

Additional Changes:

This PR also updates the following packages:
- mypy: 0.982 -> 1.7.0
- deepspeed: 0.8.3 -> 0.12.3

Follow-up:

We need to make a follow-up PR that makes Ray release tests run with pydantic>=2.5.0.

Related issue number

Addresses:

Checks

I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
I've run scripts/format.sh to lint the changes in this PR.
I've included any doc changes needed for https://docs.ray.io/en/master/.
- I've added any new APIs to the API Reference. For example, if I added a
  method in Tune, I've added it in doc/source/tune/api/ under the
  corresponding .rst file.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
Testing Strategy
- Unit tests
  - This change updates Ray unit tests.

Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com>

doc/requirements-doc.txt

Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com>

…n-pydantic

Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com>

…n-pydantic

Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com>

python/ray/tests/test_pydantic_serialization.py

aslonnie

(seems @can-anyscale is going to review this one)

edoakes · 2023-10-19T16:46:11Z

FYI to reviewers: this is a WIP, I won't merge it until after the next pydantic release that includes changes to make models cloudpickle-serializable.

ddelange · 2023-10-23T10:24:27Z

FYI to reviewers: this is a WIP, I won't merge it until after the next pydantic release that includes changes to make models cloudpickle-serializable.

do you have a link for this? will that mean that this PR will have to require pydantic>=2.5?

edoakes · 2023-10-23T15:37:14Z

@ddelange here's the link to the Pydantic issue: pydantic/pydantic#6763

After that is merged and released, we will require either < 2.0 or >= 2.5 (assuming 2.5 is the the release containing the fixes).

Signed-off-by: Shreyas Krishnaswamy <shrekris@anyscale.com>

ericl · 2023-11-17T19:57:18Z

python/requirements.txt

@@ -58,6 +58,6 @@ starlette
 typer
 fsspec
 pandas>=1.3
-pydantic<2
+pydantic!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,<3


=2.5.0,<3 doesn't work?

That works, but we want to allow users to use pydantic<2 for Serve, so no need to pin it to 2.5.0 explicitly for now.

Can you add a comment to this effect?

Yeah, I added one.

ericl

Stamping for setup.py changes

Signed-off-by: Shreyas Krishnaswamy <shrekris@anyscale.com>

shrekris-anyscale · 2023-11-17T21:11:57Z

@architkulkarni This change is ready to merge.

architkulkarni · 2023-11-17T21:45:31Z

@shrekris-anyscale Looks like premerge is still running, please let me know again when it's ready

shrekris-anyscale · 2023-11-18T00:15:25Z

@architkulkarni Premerge passed. This PR is ready to merge.

This change makes Ray compatible with pydantic>=2.5.0. It makes all Ray unit tests use pydantic==2.5.0. It also adds a test suite to check that Ray Serve works with pydantic<2. Additional Changes: This PR also updates the following packages: mypy: 0.982 -> 1.7.0 deepspeed: 0.8.3 -> 0.12.3 Follow-up: We need to make a follow-up PR that makes Ray release tests run with pydantic>=2.5.0. --------- Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com> Signed-off-by: Shreyas Krishnaswamy <shrekris@anyscale.com> Co-authored-by: Shreyas Krishnaswamy <shrekris@anyscale.com> Signed-off-by: Xiaoshuang Liu <lxsbit@126.com>

As a follow-up to #40451, this change makes all the Ray release tests use pydantic>=2.5.0. Additional changes: The linter now uses mypy==1.7.0. All Ray release tests, including workspace templates, now use deepspeed>=0.12.3. This change fix the serve_resnet_benchmark.py release test, which is broken on master. --------- Signed-off-by: can <can@anyscale.com> Signed-off-by: Shreyas Krishnaswamy <shrekris@anyscale.com> Co-authored-by: Shreyas Krishnaswamy <shrekris@anyscale.com>

edoakes · 2023-11-23T06:07:41Z

Happy to see this merged, thanks for pushing it forward @shrekris-anyscale

This change makes Ray compatible with pydantic>=2.5.0. It makes all Ray unit tests use pydantic==2.5.0. It also adds a test suite to check that Ray Serve works with pydantic<2. Additional Changes: This PR also updates the following packages: mypy: 0.982 -> 1.7.0 deepspeed: 0.8.3 -> 0.12.3 Follow-up: We need to make a follow-up PR that makes Ray release tests run with pydantic>=2.5.0. --------- Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com> Signed-off-by: Shreyas Krishnaswamy <shrekris@anyscale.com> Co-authored-by: Shreyas Krishnaswamy <shrekris@anyscale.com>

As a follow-up to ray-project#40451, this change makes all the Ray release tests use pydantic>=2.5.0. Additional changes: The linter now uses mypy==1.7.0. All Ray release tests, including workspace templates, now use deepspeed>=0.12.3. This change fix the serve_resnet_benchmark.py release test, which is broken on master. --------- Signed-off-by: can <can@anyscale.com> Signed-off-by: Shreyas Krishnaswamy <shrekris@anyscale.com> Co-authored-by: Shreyas Krishnaswamy <shrekris@anyscale.com>

This is a follow-up to where the Pydantic v1 shim was added [1]. It's inspired by Ray's approach [2]. This locks the version in the CI and I think that's okay since we don't need to keep on updating them, as they're intended as smoke tests. [1] #882 (comment) [2] ray-project/ray#40451

* test: Add Pydantic version to matrix variable This is a follow-up to where the Pydantic v1 shim was added [1]. It's inspired by Ray's approach [2]. This locks the version in the CI and I think that's okay since we don't need to keep on updating them, as they're intended as smoke tests. [1] #882 (comment) [2] ray-project/ray#40451 * build: Formatting

edoakes added 2 commits October 18, 2023 11:31

fix

3c5d560

Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com>

fix

7d33244

Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com>

edoakes requested review from richardliaw and ericl as code owners October 18, 2023 16:54

fix

00acef1

Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com>

edoakes requested review from aslonnie, can-anyscale and krfricke as code owners October 18, 2023 16:55

fix

35e4e00

Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com>

edoakes requested a review from a team as a code owner October 18, 2023 17:18

fix

f161050

Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com>

can-anyscale reviewed Oct 18, 2023

View reviewed changes

doc/requirements-doc.txt Outdated Show resolved Hide resolved

edoakes added 6 commits October 18, 2023 12:58

fix

dc4a19e

Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com>

requirements_compiled

889a7c8

Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com>

Merge branch 'master' of https://github.com/ray-project/ray into unpi…

c630c4e

…n-pydantic

fix docs build pls

cbb2176

Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com>

Merge branch 'master' of https://github.com/ray-project/ray into unpi…

603aa06

…n-pydantic

fix docs plssss

8b6064d

Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com>

edoakes requested review from shrekris-anyscale, sihanwang41, zcin and architkulkarni as code owners October 18, 2023 20:05

edoakes added 2 commits October 18, 2023 15:07

fix

2547194

Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com>

maybe fix docs this time

5dc8642

Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com>

shrekris-anyscale approved these changes Oct 18, 2023

View reviewed changes

python/ray/tests/test_pydantic_serialization.py Outdated Show resolved Hide resolved

aslonnie reviewed Oct 19, 2023

View reviewed changes

edoakes self-assigned this Oct 24, 2023

shrekris-anyscale added 3 commits November 16, 2023 19:19

Remove comment

9ad6b8d

Signed-off-by: Shreyas Krishnaswamy <shrekris@anyscale.com>

Merge branch 'master' of github.com:ray-project/ray into unpin-pydantic

545bc6b

Use version range in setup.py

545c739

Signed-off-by: Shreyas Krishnaswamy <shrekris@anyscale.com>

shrekris-anyscale requested a review from rkooo567 November 17, 2023 03:27

pcmoritz approved these changes Nov 17, 2023

View reviewed changes

Add upperbound

f8835f5

Signed-off-by: Shreyas Krishnaswamy <shrekris@anyscale.com>

matthewdeng approved these changes Nov 17, 2023

View reviewed changes

aslonnie approved these changes Nov 17, 2023

View reviewed changes

ericl reviewed Nov 17, 2023

View reviewed changes

ericl approved these changes Nov 17, 2023

View reviewed changes

Added comment

adc9a5c

Signed-off-by: Shreyas Krishnaswamy <shrekris@anyscale.com>

Merge branch 'master' of github.com:ray-project/ray into unpin-pydantic

b441202

architkulkarni merged commit 31c6631 into ray-project:master Nov 18, 2023
15 of 18 checks passed

shrekris-anyscale mentioned this pull request Nov 18, 2023

Make Ray releases tests use pydantic>=2.5.0 #41244

Merged

6 tasks

rickyyx mentioned this pull request Nov 22, 2023

[ci][core] Perf regression on tasks_per_second, pgs_per_second #41338

Closed

ddelange mentioned this pull request Dec 6, 2023

Bump ray to 2.10 autogluon/autogluon#3774

Merged

This was referenced Apr 23, 2024

fix: Add Pydantic v1 shim SeldonIO/alibi-detect#882

Merged

test: Add Pydantic version to matrix variable SeldonIO/alibi-detect#890

Merged

peytondmurray mentioned this pull request May 2, 2024

[Doc] Fix broken references in serve documentation #45113

Merged

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make Ray compatible with `pydantic>=2.5.0` #40451

Make Ray compatible with `pydantic>=2.5.0` #40451

edoakes commented Oct 18, 2023 •

edited by shrekris-anyscale

Loading

aslonnie left a comment

edoakes commented Oct 19, 2023 •

edited

Loading

ddelange commented Oct 23, 2023

edoakes commented Oct 23, 2023

ericl Nov 17, 2023

shrekris-anyscale Nov 17, 2023

ericl Nov 17, 2023

shrekris-anyscale Nov 17, 2023

ericl left a comment

shrekris-anyscale commented Nov 17, 2023

architkulkarni commented Nov 17, 2023

shrekris-anyscale commented Nov 18, 2023

edoakes commented Nov 23, 2023 •

edited

Loading

Make Ray compatible with pydantic>=2.5.0 #40451

Make Ray compatible with pydantic>=2.5.0 #40451

Conversation

edoakes commented Oct 18, 2023 • edited by shrekris-anyscale Loading

Why are these changes needed?

Related issue number

Checks

aslonnie left a comment

Choose a reason for hiding this comment

edoakes commented Oct 19, 2023 • edited Loading

ddelange commented Oct 23, 2023

edoakes commented Oct 23, 2023

ericl Nov 17, 2023

Choose a reason for hiding this comment

shrekris-anyscale Nov 17, 2023

Choose a reason for hiding this comment

ericl Nov 17, 2023

Choose a reason for hiding this comment

shrekris-anyscale Nov 17, 2023

Choose a reason for hiding this comment

ericl left a comment

Choose a reason for hiding this comment

shrekris-anyscale commented Nov 17, 2023

architkulkarni commented Nov 17, 2023

shrekris-anyscale commented Nov 18, 2023

edoakes commented Nov 23, 2023 • edited Loading

Make Ray compatible with `pydantic>=2.5.0` #40451

Make Ray compatible with `pydantic>=2.5.0` #40451

edoakes commented Oct 18, 2023 •

edited by shrekris-anyscale

Loading

edoakes commented Oct 19, 2023 •

edited

Loading

edoakes commented Nov 23, 2023 •

edited

Loading