building bazel libcpu_profiler fails to find jni_md.h (ppc64le)

[clnperez]

Description of the problem / feature request:

compiling from master on ppc64le fails:

bazel-out/ppc-fastbuild/bin/external/bazel_tools/tools/jdk/include/jni.h:45:10: fatal error: jni_md.h: No such file or directory

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

bazel build --sandbox_debug -s --verbose_failures --host_javabase=@local_jdk//:jdk //src:bazel

What operating system are you running Bazel on?

Ubuntu 18.04.3

What's the output of bazel info release?

building with:

release 1.1.0- (@non-git)

If bazel info release returns "development version" or "(@non-git)", tell us how you built Bazel.

https://github.com/Unicamp-OpenPower/bazel-build/blob/master/.travis.yml

What's the output of git remote get-url origin ; git rev-parse master ; git rev-parse HEAD ?

root@ede9b339f10f:/bazel# git remote get-url origin ; git rev-parse master ; git rev-parse HEAD
https://github.com/bazelbuild/bazel.git
1ae1c16
314cd33

Have you found anything relevant by searching the web?

no

Any other information, logs, or outputs that you want to share?

This commit seems to have broken us: 314cd33

More or the error here: https://gist.github.com/clnperez/781149e8c6882ea668d06483325e1151

[clnperez]

@alandonovan -- i've tried to figure out how to get this to build and can't. i thought maybe it was because we build using the local jdk, but i think that's the default now. i also tried that on x86 and this works fine. so now i'm guessing there's some place i can't find with an linux_x86_64 option and a blank default for this header file. Any suggestions?

[alandonovan]

Take a look at the BUILD.oss file in the linked commit:
314cd33#diff-96bd71d4e041b52d389d631025d2c09fR167
It clearly doesn't have a case for linux on non-x86 platforms, so this error is expected, at least for now. You could fix it by adding a case for ppc to that select, but really there needs to be a cc_library rule for the JNI package so that Java clients don't need to think about such details.

[clnperez]

@alandonovan I had tried some things there, but guess I missed the obvious one of just copying the linux_x86_64 case to ppc. PR submitted. Thanks!

[vielmetti]

I just ran into the same problem on aarch64 (arm64).

[moderation]

I compile Bazel on aarch64 from source and have been able to successfully do so up until 2.2.0. I compiled 2.1.1 last week successfully with env EXTRA_BAZEL_ARGS="--host_javabase=@local_jdk//:jdk" bash ./compile.sh.

I tried 2.2.0 this morning and get:

ERROR: /home/<username>/Library/bazel/src/main/java/com/google/devtools/build/lib/syntax/BUILD:150:1: C++ compilation of rule '//src/main/java/com/google/devtools/build/lib/syntax:libcpu_profiler.so' failed (Exit 1): clang failed: error executing command
  (cd /tmp/bazel_IGlvQbWR/out/execroot/io_bazel && \
  exec env - \
    PATH=/home/<username>/.cargo/bin:/home/<username>/.local/bin:/home/<username>/Library/go/bin:/home/<username>/Library/code/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games \
    PWD=/proc/self/cwd \
  /usr/local/bin/clang -U_FORTIFY_SOURCE -fstack-protector -Wall -Wthread-safety -Wself-assign -fcolor-diagnostics -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections -fdata-sections '-std=c++0x' -MD -MF bazel-out/aarch64-opt/bin/src/main/java/com/google/devtools/build/lib/syntax/_objs/libcpu_profiler.so/cpu_profiler_unimpl.pic.d '-frandom-seed=bazel-out/aarch64-opt/bin/src/main/java/com/google/devtools/build/lib/syntax/_objs/libcpu_profiler.so/cpu_profiler_unimpl.pic.o' -fPIC -iquote . -iquote bazel-out/aarch64-opt/bin -iquote external/bazel_tools -iquote bazel-out/aarch64-opt/bin/external/bazel_tools -isystem external/bazel_tools/tools/jdk/include -isystem bazel-out/aarch64-opt/bin/external/bazel_tools/tools/jdk/include -no-canonical-prefixes -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c src/main/java/com/google/devtools/build/lib/syntax/cpu_profiler_unimpl.cc -o bazel-out/aarch64-opt/bin/src/main/java/com/google/devtools/build/lib/syntax/_objs/libcpu_profiler.so/cpu_profiler_unimpl.pic.o)
Execution platform: //:default_host_platform
In file included from src/main/java/com/google/devtools/build/lib/syntax/cpu_profiler_unimpl.cc:17:
bazel-out/aarch64-opt/bin/external/bazel_tools/tools/jdk/include/jni.h:45:10: fatal error: 'jni_md.h' file not found
#include "jni_md.h"
         ^~~~~~~~~~
1 error generated.
Target //src:bazel_nojdk failed to build

On further inspection the following additions for aarch64 didn't make it into 2.2.0 - https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/syntax/BUILD#L166-L176

[alandonovan]

Yes, you'll need to build a version of Bazel more recent than 2.2.0 (20 days old) as the fix was only added in 2402c8f (15 days ago).

[JulianGerhard21]

Hi @alandonovan ,

I am unable to find a more recent bazel version than 2.2.0 - can you please give me a hint where I can download specifically the dist, because I need to bootstrap it?

Kind regards
Julian

[laurentlb]

We have now a release candidate for Bazel 3.0, see https://releases.bazel.build/3.0.0/rc1/index.html
We'll have a proper release in a few weeks.

The alternative is to build Bazel yourself.

[codeNick7]

I am using the latest release dist package (3.3.1) and I see this issue on Raspbian (Linux raspberrypi 4.19.118-v7l+ #1311 SMP Mon Apr 27 14:26:42 BST 2020 armv7l GNU/Linux):

ERROR: /home/pi/workspace/mediapipe/bazel/src/main/java/com/google/devtools/build/lib/syntax/BUILD:139:10: C++ compilation of rule '//src/main/java/com/google/devtools/build/lib/syntax:libcpu_profiler.so' failed (Exit 1): gcc failed: error executing command 
  (cd /tmp/bazel_jLqe6uBU/out/execroot/io_bazel && \
  exec env - \
    PATH=/home/pi/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/games:/usr/games \
    PWD=/proc/self/cwd \
  /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections -fdata-sections '-std=c++0x' -MD -MF bazel-out/arm-opt/bin/src/main/java/com/google/devtools/build/lib/syntax/_objs/libcpu_profiler.so/cpu_profiler_unimpl.pic.d '-frandom-seed=bazel-out/arm-opt/bin/src/main/java/com/google/devtools/build/lib/syntax/_objs/libcpu_profiler.so/cpu_profiler_unimpl.pic.o' -fPIC -iquote . -iquote bazel-out/arm-opt/bin -iquote external/bazel_tools -iquote bazel-out/arm-opt/bin/external/bazel_tools -isystem external/bazel_tools/tools/jdk/include -isystem bazel-out/arm-opt/bin/external/bazel_tools/tools/jdk/include -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c src/main/java/com/google/devtools/build/lib/syntax/cpu_profiler_unimpl.cc -o bazel-out/arm-opt/bin/src/main/java/com/google/devtools/build/lib/syntax/_objs/libcpu_profiler.so/cpu_profiler_unimpl.pic.o)
Execution platform: //:default_host_platform
In file included from src/main/java/com/google/devtools/build/lib/syntax/cpu_profiler_unimpl.cc:17:
bazel-out/arm-opt/bin/external/bazel_tools/tools/jdk/include/jni.h:45:10: fatal error: jni_md.h: No such file or directory
 #include "jni_md.h"
          ^~~~~~~~~~
compilation terminated.
Target //src:bazel_nojdk failed to build
INFO: Elapsed time: 754.797s, Critical Path: 78.28s
INFO: 805 processes: 805 local.
FAILED: Build did NOT complete successfully

Could someone help?

[alandonovan]

The fact that Bazel is compiling cpu_profiler_unimpl.cc means that the config_setting //src/conditions:linux_arm is false on your platform, which is surprising. This condition is determined by the flag --cpu=arm. Did you set that flag?

You should open a new issue for this; although it is similar to the ppc64le problem, it clearly requires a different fix.

[codeNick7]

Hi,
I have not changed any build settings other than JVM heap settings. I will open a new issue to this and link back here.
Thanks for your response and consideration!

[alandonovan]

To be clear, I meant: you will need to set --cpu=arm.

[codeNick7]

Ahh.. let me try that. I have created this new issue: #11709, I will try with that flag and update. Thanks!

[codeNick7]

Hi,
I am not able to set that flag. What have found so far is : https://github.com/samjabrahams/tensorflow-on-raspberry-pi/blob/master/GUIDE.md and as per this, I have added a 'return "arm" in get_cpu_value function inside tools/cpp/lib_cc/configure.bzl. This did not change anything and I hit the same error. Could you help?

[rashmi-ibm]

@here I am running into the same error building bazel using the zip from 3.3.0 distro https://github.com/bazelbuild/bazel/releases/download/3.3.0/bazel-3.3.0-dist.zip.

bazel-out/ppc-opt/bin/external/bazel_tools/tools/jdk/include/jni.h:45:10: fatal error: jni_md.h: No such file or directory
#include "jni_md.h"
^~~~~~~~~~
compilation terminated.
Target //src:bazel_nojdk failed to build

Any pointers to resolve this.

[SimonStHilaire]

@here I am running into the same error building bazel using the zip from 3.3.0 distro https://github.com/bazelbuild/bazel/releases/download/3.3.0/bazel-3.3.0-dist.zip.

bazel-out/ppc-opt/bin/external/bazel_tools/tools/jdk/include/jni.h:45:10: fatal error: jni_md.h: No such file or directory
#include "jni_md.h"
^~~~~~~~~~
compilation terminated.
Target //src:bazel_nojdk failed to build

Any pointers to resolve this.

Same here.

[moderation]

I'd try Bazel 3.4.1 or 3.5.0

[SimonStHilaire]

Hi, just did and got the same error.

Cheers,

Simon

[alandonovan]

@aiuto Tony, should I be able to reproduce this in a cross build? I tried bazel build --cpu=ppc src/bazel (with 3.5.0) and I got this error:

ERROR: [...]external/local_config_cc/BUILD:47:19: in cc_toolchain_suite rule @local_config_cc//:toolchain: cc_toolchain_suite '@local_config_cc//:toolchain' does not contain a toolchain for cpu 'ppc'

Any help much appreciated.

[aiuto]

Perhaps 38c85a4 should have added ppc64le instead of replacing it?
It is also unclear why the default condition is empty rather than ":jni_md_header-linux".
Having to enumerate every platform while one is a good default is brittle.

[alandonovan]

The two conditions have equal definitions, so I don't see why that should matter.
https://github.com/bazelbuild/bazel/blob/master/src/conditions/BUILD#L37

Linux is only a good default if you're actually using Linux. I'd rather it fail fast than use the wrong header.

[xloem]

I'm on i686 debian bullseye and running into this issue trying to build bazel 3.7.2 from the dist archive. It's unclear how to make it work, generally, on an unsupported or failing system.

EDIT: This fails because bazel specifically does not provide support or configurations for 32 bit systems. There are instructions elsewhere for patching hazel for 32 bit systems. Further pursuit of this on 32 bit would likely involve an issue or pull request to notify the user that their CPU architecture is not accepted to build with.

This particular issue is triggered similarly as other answers in this thread: there is no condition for the architecture, so the file is not copied in. The jni_md.h file is os-dependent, held in an os-specific subfolder.