resync grpcio requirements

[mattip]

Checklist

• Used a personal fork of the feedstock to propose changes
• Bumped the build number (if the version is unchanged)
• Reset the build number to 0 (if the version changed)
• Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
• Ensured the license file is being packaged.

Addresses #90 to update grpcio version on non-darwin like upstream

[mattip]

@conda-forge-admin, please rerender

[mattip]

the comment just above was wrong, the difference is due to compile flags

[mattip]

There is in addition to the grpcio package a libgrpc package. Where do these come from? conda list says they are being pulled from the conda-forge channel, but I cannot find the feedstocks for them. The download page for libgrpc does not say how it is built, there is no libgrpc-feedstock repo, and the grpcio-feedstock is marked "archived" without a redirect.

[mattip]

If I remove the conda-supplied grpc and install the pip 1.51.1 version, ray works well.

[cread]

There is in addition to the grpcio package a libgrpc package. Where do these come from? conda list says they are being pulled from the conda-forge channel, but I cannot find the feedstocks for them. The download page for libgrpc does not say how it is built, there is no libgrpc-feedstock repo, and the grpcio-feedstock is marked "archived" without a redirect.

At the top of the README in the grpcio-feedstock is a link to https://github.com/conda-forge/grpc-cpp-feedstock which appears to be the new source.

[h-vetinari]

At the top of the README in the grpcio-feedstock is a link to https://github.com/conda-forge/grpc-cpp-feedstock which appears to be the new source.

Yes. grpcio hasn't changed, and libgrpc is the new name for grpc-cpp (though we've had a couple of versions where we kept the latter as a compatibility wrapper)

[h-vetinari]

If I remove the conda-supplied grpc and install the pip 1.51.1 version, ray works well.

If someone is able to point me to anything we're doing wrong on the grpc-cpp feedstock, I'm happy to fix it - when the first issues in ray appeared, I checked but couldn't find anything. We're building grpcio directly on top of libgrpc and testing it (perhaps not extensively enough).

The larger issue AFAICT is that ray still uses it's internal grpc version, and we need to unvendor that (xref #90). It's possible that the conda-forge builds are "just" exposing that (e.g. by pointing grpcio to our symbols rather than what the pip-installed one may find within ray; just speculation though). The main reason I haven't tackled this more seriously yet (aside from lack of time) is because injecting external artefacts into bazel is such a pain, and I don't have much experience with that.

[mattip]

The larger issue AFAICT is that ray still uses it's internal grpc version, and we need to unvendor that (xref #90).

The two environments can be set up by

$ mamba create -n throw-away1 python=3.9 ray-serve=2.4.0
$ conda activate throw-away1
$ python -c "import ray; ray.init()"
# succeeds
$ mamba install --no-deps grpcio=1.51.1 libgrpc=1.51.1 grpc-cpp=1.51.1
$ python -c "import ray; ray.init()"
# fails

$ mamba create -n throw-away1 python=3.9 ray-serve=2.4.0
$ conda activate throw-away1
$ python -c "import ray; ray.init()"
# succeeds
$ pip install grpcio==1.51.1
$ python -c "import ray; ray.init()"
# succeeds

So I don't think there is something intrinsically wrong with the 2.4.0 recipe the way it is, more like something in the conda-forge 1.51.1 build is missing. Is there a known way to debug grpc transport issues?

Another approach would be to rip out the grpc code embedded in ray. But I think there still needs to be some compilation of the protobuf sources. Is there another client package with a recipe that uses grpc that I can compare to?

[krfricke]

Ray has recently removed the grpcio dependency from ray core: ray-project/ray#38243

It has been added to ray[default] instead and a new extras called ray[serve-grpc] (ray-project/ray#38386).

I don't think we necessarily need to add ray[serve-grpc] here, but moving grpcio to ray-default probably makes sense.

[mattip]

Closing, the way grpcio and ray interact changed for 2.7.0