[Bazel] Build fails with Bazel 6.0.0: Constraints from "@bazel_tools" removed

[peytondmurray]

What happened + What you expected to happen

When building ray from source with cd python && pip install -ve ., bazel throws a build error and the build fails during the build_ext step:

...
    ERROR: /home/pdmurray/.cache/bazel/_bazel_pdmurray/fd1a49e9b70d1b9b9f23ee71905db765/external/bazel_tools/platforms/BUILD:89:6: in alias rule @bazel_tools//platforms:windows: Constraints from @bazel_tools//platforms have been removed. Please use constraints from @platforms repository embedded in Bazel, or preferably declare dependency on https://github.com/bazelbuild/platforms. See https://github.com/bazelbuild/bazel/issues/8622 for details.
    ERROR: /home/pdmurray/.cache/bazel/_bazel_pdmurray/fd1a49e9b70d1b9b9f23ee71905db765/external/bazel_tools/platforms/BUILD:89:6: Analysis of target '@bazel_tools//platforms:windows' failed
    ERROR: /home/pdmurray/.cache/bazel/_bazel_pdmurray/fd1a49e9b70d1b9b9f23ee71905db765/external/com_google_protobuf/BUILD:513:10: While resolving toolchains for target @com_google_protobuf//:protoc: invalid registered toolchain '@bazel_skylib//toolchains/unittest:cmd_toolchain':
    ERROR: Analysis of target '//:ray_pkg' failed; build aborted:
...

I've tried doing bazel clean --expunge and rm -r ~/.cache/bazel/ but neither solved the issue. From the error message it looks like this is coming from a bazel toolchain dependency.

Note: Okay, when using Bazel 5.4.0 it looks like Ray is able to build without issue, but with 6.0.0 it fails. Here's the release for 6.0.0, it looks like a lot changed but I haven't gone through it thoroughly enough to isolate the issue. As a temporary workaround, installing bazelisk and creating a .bazeliskrc with USE_BAZEL_VERSION=5.x at the root of the Ray repo will fix the issue.

Versions / Dependencies

Ray: 5c7826f
Python: 3.10.8
OS: Linux 6.1.3-arch1-1 x86_64
Java: java-11-openjdk
Bazel: 6.0.0

Reproduction script

git checkout 5c7826f55
cd python
pip install -ve .

Issue Severity

High: It blocks me from completing my task.

[cadedaniel]

FYI you can use ./ci/env/install-bazel.sh to install a working version of bazel (currently 4.2)

[mbelalsh]

Downgrading to 4.2.0 worked for me. I was using the latest 6.0.0 before, and the build was constantly failing.

• Use this command to uninstall: sudo apt-get --purge remove bazel.
• Use this command to install: sudo apt install bazel-4.2.0. Don't forget to run the next line.
• sudo ln -s /usr/bin/bazel-4.2.0 /usr/bin/bazel

[lolitsgab]

That is so weird... I wonder why 6.0+ is broken. I downgraded to 5.1.0 and it works now. Thanks for the tip, @mbelalsh !

Edit: you can set default bazel version if you are using bazelisk. In the dir where your WORKSPACE file is, make a .bazelversion and put the version you want, like 5.1.0 example

[stale]

Hi, I'm a bot from the Ray team :)

To help human contributors to focus on more relevant issues, I will automatically add the stale label to issues that have had no activity for more than 4 months.

If there is no further activity in the 14 days, the issue will be closed!

• If you'd like to keep the issue open, just leave any comment, and the stale label will be removed!
• If you'd like to get more attention to the issue, please tag one of Ray's contributors.

You can always ask for help on our discussion forum or Ray's public slack channel.

[mattip]

Is upgrading bazel on the ray roadmap?

[h-vetinari]

Is upgrading bazel on the ray roadmap?

Respectfully, it must be. It's not reasonable to keep a dependency pinned forever. And if that's not enough of an argument, then bazel 6 specificaly would give some potentially very large advantages on the packaging side due to bzlmod, which should ideally unblock things like osx-support in conda-forge, improving the grpc-situation, etc. etc.

[aslonnie]

as @cadedaniel said, to build ray, it needs to run ci/env/install-bazel.sh to install bazel, or at least needs to respect the bazel version in .bazeliskrc (like just use bazelisk with bazel symlinked to bazelisk).

https://docs.ray.io/en/latest/ray-contribute/development.html#preparing-to-build-ray-on-linux

I agree that the build toolchain setup is a bit convoluted right now. we are working on simplifying it.

we do not have plans to upgrade bazel in the near term. to upgrade bazel, a big cleanup needs to be performed first, on the (weird) use of bazel rules in the ray repo today, and discarding old, legacy stuff from the repo that is not worth upgrading. there is no planned timeline for now.

although upgrading bazel is a nice to have, the current version of bazel serves okay for our purpose of building/testing/releasing ray. and upgrading it does not seem to give us any immediate benefit.

fwiw, we will probably upgrade bazel some time in the future when the repo structure is cleaner, and hence upgrading can be easier.

specifically for the grpc related issues, it is unclear about the urgency, or even if upgrading grpc is the right way to go at the moment. in the worst case, the grpc stubs can be generated in other ways. upgrading bazel is not a must.

unfortunately, conda-forge support is not really in our scope today afaik. (sorry, conda-forge friends..) if you are looking for smoother integration in the long run, maybe consider reaching out to @zhe-thoughts or Ray leadership for a more formal collaboration deal or something. :)

---

I am closing this issue as the original issue (cannot build with 6.0.0) is "intended behavior".

for "ray's build tool chain needs to upgrade to bazel 6", the current answer is "won't fix". sorry.

[mattip]

It is not only conda-forge support that is broken. Windows support is also broken, see #42098. I fear the CI builds are succeeding due to a very fragile setup that happens to work, and could break at any tool update.

[aslonnie]

I fear the CI builds are succeeding due to a very fragile setup that happens to work, and could break at any tool update.

fwiw, this statement is also kind of true for Linux and OSX. It is just that Windows's toolchain config steps are inherently more complex than Linux and OSX (and foreign to many developers), so when something is broken, it is harder to fix.

Ray is not really using bazel in bazel's idiomatic/recommended way. It is not using bazel to define everything in the build toolchain, but using the native ones from the system all the time. Using an old version of bazel is not really the root cause here.