[Core] Ray 2.9.0 with grcpio1.58 from conda-forge does not work

[mattip]

What happened + What you expected to happen

Maybe related to #35383, which was closed when ray refactored its grpcio handling. Reported in conda-forge/ray-packages-feedstock#136. Can be worked around by preferring grpcio-1.59 over grpcio-1.58 from conda-forge. The issue does not occur when using grpcio from PyPI (via pip install).

Even though there is a work-around, it would be good to understand what is going on: is conda-forge's build broken or is it the bazel-based build of ray?

cc @anyscalesam, originally reported by @apmorton

One thing that came up in other discussions around this issue is that conda-forge's approach to packaging is to re-use installable packages as build dependencies. That is, conda-forge alread builds many of the third-party build dependencies of ray, and it would simplify packaging (less patching and building) if the bazel-based ray build would use them. That would require a way to tell bazel "please use these system libraries", something that bazel verison 5 cannot easily do. But bazel version 6 does have a way to specify this. Something to think about.

Versions / Dependencies

linux x86_64
ray 2.9.0 (from conda-forge)
grpcio (from conda-forge)
1.59.3 - working
1.58.2 - broken
1.57.0 - working
1.56.0 - working
1.55.1 - broken
1.54.3 - broken

Reproduction script

$ env/bin/ray start --head
$ env/bin/python -c 'import ray; import logging; ray.init(address="ray://127.0.0.1:10001", logging_level=logging.DEBUG); print("hello")'

Issue Severity

Medium: It is a significant difficulty but I can work around it.

[rynewang]

Hi @mattip after some discussions we decide to not update on our end because in pip it works fine. Would you mind continuing the path in conda-forge/conda-forge-repodata-patches-feedstock#638 and we can call it fixed?

[rynewang]

cc @rkooo567

[mattip]

OK

[anyscalesam]

@rynewang can we think of a systematic fix for this; is there nothing we can do so that dependency crawls between pip and conda are aligned so that if it works in pip it'll also work in condaforge?

[rynewang]

I don't have extensive conda knowledge but based on the issue I guess grpcio compiled by conda is broken, while the one compiled by pip is good, so Ray don't have a great way to auto-fix this. @mattip do you have any insights on this?

[mattip]

I tried but could not make progress in figuring out what is wrong with that mix of package versions. I would be more motivated if we had a use case that requires grpcio-1.58, and cannot resolve without it.

[anyscalesam]

grabbed time to brainstorm about this in a couple of weeks

[h-vetinari]

In conda-forge, we bring our own grpc and grpcio. We need to be able to build ray with the same version, which means unvendoring grpc. This is unfortunately extremely difficult with bazel, which is why it's been an open issue for a long time: conda-forge/ray-packages-feedstock#90.

[anyscalesam]

@aslonnie @mattip chatting about this on monday

[anyscalesam]

closing as we've updated our bazel version at this point.