Darwin ARM64 prebuilt binaries for java_tools are not compatible (ijar bad CPU type)

[devversion]

Description of the problem / feature request:

Currently, when using Darwin ARM64 with a remote JDK for example, the use of ijar in a Java project seems to fail. e.g.

ERROR: /private/var/tmp/_bazel_paul/4828bcaab66ebffc539d5f0797b06c19/external/io_bazel/third_party/BUILD:453:20: Extracting interface @io_bazel//third_party:guava_checked_in failed: (Exit 1): 

ijar failed: error executing command external/remote_java_tools_darwin/java_tools/ijar/ijar external/io_bazel/third_party/guava/failureaccess-1.0.1.jar ... (remaining 3 arguments skipped)

Use --sandbox_debug to see verbose messages from the sandbox
src/main/tools/process-wrapper-legacy.cc:80: "execvp(external/remote_java_tools_darwin/java_tools/ijar/ijar, ...)":
    Bad CPU type in executable
Target //java/com/google/copybara:copybara failed to build

I assume this happens because the prebuilt binaries are only built for Windows, Linux and macOS but there is no separation of Darwin ARM64 or x86_64. Currently only x64 binaries are being built & shipped:

bazel/tools/jdk/BUILD.tools

Lines 153 to 157 in e1c404e

	for OS in [
	"linux",
	"darwin",
	"windows",
	]

How to reproduce

1. Have a M1-based Mac
2. Clone the copybara repository from Google: https://github.com/google/copybara
3. Run: bazel build //java/com/google/copybara
   ^{Note: I have been using --java_runtime_version=remotejdk_11. Down the line, the Maven runner failed with Java not being found. I have java not setup at all; only the embedded version. I set --action_env=JAVA_HOME=<bazel-output_base>/external/remotejdk11_macos_aarch64 to workaround the Maven issue}

What operating system are you running Bazel on?

macOS Big Sur M1 Arm64

What's the output of bazel info release?

release 5.0.0-pre.20210826.1

(also confirmed with 4.2.0 stable)

[comius]

There is non-prebuilt toolchain that support such use cases. Can you try with

load(
  "@bazel_tools//tools/jdk:default_java_toolchain.bzl",
  "default_java_toolchain", "NONPREBUILT_TOOLCHAIN_CONFIGURATION"
)

default_java_toolchain(
  name = "nonprebuilt_toolchain",
  configuration = NONPREBUILT_TOOLCHAIN_CONFIGURATION, 
)

and

register_toolchain("nonprebuilt_toolchain") in the WORKSPACE file.

More info on https://docs.bazel.build/versions/main/bazel-and-java.html

I'm reluctant to start compiling java_tools for one more platfort. @philwo - but if we really want to, we'd need CI for it I guess.

There are two things that could be done:

• Correct prebuilt toolchain CPU matching - if that was fixed, I think toolchain from sources would get selected automatically.
• Add an unregistered definition of default_java_toolchain with NONPREBUILT_TOOLCHAIN_CONFIGURATION to @bazel_tools//tools/jdk, so that the instructions above reduce to a single line.

[comius]

I triaged to P2, but any help with fixes I mentioned above is welcome.

[jlaxson]

I'm reluctant to start compiling java_tools for one more platfort. @philwo - but if we really want to, we'd need CI for it I guess.

Can remote_java_tools_darwin be compiled fat to obviate the need for extra set of plumbing?

Add an unregistered definition of default_java_toolchain with NONPREBUILT_TOOLCHAIN_CONFIGURATION to @bazel_tools//tools/jdk, so that the instructions above reduce to a single line.

It seems unfortunate that every java project using Bazel would have to make this change to successfully build on arm.

[philwo]

I think if we publish remote_java_tools_darwin, then it would be great to do so in a way that contains support for both x86_64 and arm64.

@comius Is this something the community could help with and contribute a PR that does it? It's easily possible to cross-compile on macOS (we currently build Bazel arm64 binaries on our Intel machines), so maybe it's not much work. Building macOS universal binaries I think is a matter of building them once for each platform and then merging them into a universal binary via lipo -create -output universalbinary intelbinary armbinary.

[comius]

A PR updating https://github.com/bazelbuild/bazel/blob/master/src/create_java_tools_release.sh ,
https://github.com/bazelbuild/bazel/blob/master/src/upload_all_java_tools.sh
https://github.com/bazelbuild/continuous-integration/blob/master/pipelines/java_tools-binaries.yml

would help. Except that there is probably limited options for testing the CI.

cc @hvadehra

[ledyba-z]

Any updates?

I can't build bazel itself on mac M1 by this issue! (It looks CI uses x86_64 mac.)
I would like to debug bazel to solve the build issue on our project, but I can't run the debugger 😭

% bazel build //src:bazel
Starting local Bazel server and connecting to it...
DEBUG: /private/var/tmp/_bazel_ryoh/22d224d1147394a69d84b1d91a7e07d8/external/build_bazel_rules_nodejs/index.bzl:122:10: WARNING: check_rules_nodejs_version has been removed. This is a no-op, please remove the call.
INFO: Analyzed target //src:bazel (365 packages loaded, 10113 targets configured).
INFO: Found 1 target...
ERROR: /Users/ryoh/umi/src/github.com/bazelbuild/bazel/third_party/BUILD:440:20: Extracting interface //third_party:guava_checked_in failed: (Exit 1): ijar failed: error executing command external/remote_java_tools_darwin/java_tools/ijar/ijar third_party/guava/failureaccess-1.0.1.jar ... (remaining 3 arguments skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
src/main/tools/process-wrapper-legacy.cc:80: "execvp(external/remote_java_tools_darwin/java_tools/ijar/ijar, ...)": Bad CPU type in executable
Target //src:bazel failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 73.726s, Critical Path: 5.76s
INFO: 1022 processes: 185 internal, 837 darwin-sandbox.
FAILED: Build did NOT complete successfully

[fmeum]

@comius I am using NONPREBUILT_TOOLCHAIN_CONFIGURATION as a workaround, but am seeing the following error on macOS with --features=layering_check enabled:

ERROR: /private/var/tmp/_bazel_runner/5b536b8aa56f54cacdf2506d7dc3d463/external/remote_java_tools/BUILD:367:10: Compiling java_tools/src/tools/singlejar/singlejar_main.cc [for tool] failed: (Exit 1): cc_wrapper.sh failed: error executing command (from target @remote_java_tools//:singlejar_cc_bin) external/local_config_cc/cc_wrapper.sh -U_FORTIFY_SOURCE -fstack-protector -Wall -Wthread-safety -Wself-assign -Wunused-but-set-parameter -Wno-free-nonheap-object -fcolor-diagnostics ... (remaining 61 arguments skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
external/remote_java_tools/java_tools/src/tools/singlejar/singlejar_main.cc:15:10: error: module @remote_java_tools//:singlejar_cc_bin does not depend on a module exporting 'src/tools/singlejar/combiners.h'
#include "src/tools/singlejar/combiners.h"
         ^
external/remote_java_tools/java_tools/src/tools/singlejar/singlejar_main.cc:16:10: error: module @remote_java_tools//:singlejar_cc_bin does not depend on a module exporting 'src/tools/singlejar/diag.h'
#include "src/tools/singlejar/diag.h"
         ^
2 errors generated.
ERROR: /Users/runner/work/jazzer/jazzer/agent/src/jmh/java/com/code_intelligence/jazzer/instrumentor/BUILD.bazel:27:13 Building agent/src/jmh/java/com/code_intelligence/jazzer/instrumentor/libcoverage_instrumentation_benchmark.jar (1 source file) and running annotation processors (BenchmarkProcessor) failed: (Exit 1): cc_wrapper.sh failed: error executing command (from target @remote_java_tools//:singlejar_cc_bin) external/local_config_cc/cc_wrapper.sh -U_FORTIFY_SOURCE -fstack-protector -Wall -Wthread-safety -Wself-assign -Wunused-but-set-parameter -Wno-free-nonheap-object -fcolor-diagnostics ... (remaining 61 arguments skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging

[fmeum]

@hvadehra Now that M1 Macs are available in Bazel CI, would it be possible to get a java_tools release for them?

[fmeum]

@fweikert Looks like this issue is what is currently breaking the arm64 postsubmit introduced in e008462.

[hvadehra]

This is fixed after #17780 . java_tools for darwin are now separately available for x86_64 and arm64 after v12.0 : https://github.com/bazelbuild/java_tools/releases/tag/java_v12.0