Bazel binary for aarch64 architecture

[ossdev07]

Description of the problem / feature request:

Bazel binary is not present for the aarch64 architecture.

Feature requests: what underlying problem are you trying to solve with this feature?

To provide multi-architecture support in bazel

What operating system are you running Bazel on?

Linux

Any other information, logs, or outputs that you want to share?

I have tried to build bazel on my local aarch64 machine, by following the instruction mentioned at https://docs.bazel.build/versions/master/install-compile-source.html#bootstrap-bazel

To build the bazel from source code I have used bazel-0.27.1-dist.zip. I was successfully able to build and generate the binary

root@ps-docker-images-do-not-delete:/vol2/aarch64/qwert# env EXTRA_BAZEL_ARGS="--host_javabase=@local_jdk//:jdk" bash ./compile.sh .
🍃  Building Bazel from scratch..find: ‘src/tools/xcode-common/java/com/google/devtools/build/xcode/common’: No such file or directory
find: ‘src/tools/xcode-common/java/com/google/devtools/build/xcode/util’: No such file or directory
....
🍃  Building Bazel with Bazel.
.WARNING: --batch mode is deprecated. Please instead explicitly shut down your Bazel server using the command "bazel shutdown".
DEBUG: /tmp/bazel_jm8BGrdX/out/external/build_bazel_rules_nodejs/internal/common/check_bazel_version.bzl:49:5:
Current Bazel is not a release version, cannot check for compatibility.
DEBUG: /tmp/bazel_jm8BGrdX/out/external/build_bazel_rules_nodejs/internal/common/check_bazel_version.bzl:51:5: Make sure that you are running at least Bazel 0.17.1.
INFO: Analyzed target //src:bazel_nojdk (214 packages loaded, 10406 targets configured).
INFO: Found 1 target...
INFO: From Compiling src/main/tools/daemonize.c:
src/main/tools/daemonize.c: In function 'WritePidFile':
src/main/tools/daemonize.c:95:3: warning: ignoring return value of 'write', declared with attribute warn_unused_result [-Wunused-result]
   write(pid_done_fd, &dummy, sizeof(dummy));
   ^
INFO: From Generating Java (Immutable) proto_library //src/main/protobuf:execution_statistics_proto:
[libprotobuf WARNING external/com_google_protobuf/src/google/protobuf/compiler/java/java_file.cc:228] The optimize_for = LITE_RUNTIME option is no longer supported by protobuf Java code generator and may generate broken code. It will be ignored by protoc in the future and protoc will always generate full runtime code for Java. To use Java Lite runtime, users should use the Java Lite plugin instead. See:
  https://github.com/google/protobuf/blob/master/java/lite.md
INFO: From JavacBootstrap src/main/java/com/google/devtools/build/lib/shell/libshell-skylark.jar [for host]:
warning: Implicitly compiled files were not subject to annotation processing.
  Use -proc:none to disable annotation processing or -implicit to specify a policy for implicit compilation.
1 warning
INFO: From Compiling src/main/cpp/blaze_util_posix.cc:
src/main/cpp/blaze_util_posix.cc: In function 'uint64_t blaze::AcquireLock(const string&, bool, bool, blaze::BlazeLock*)':
src/main/cpp/blaze_util_posix.cc:650:30: warning: ignoring return value of 'int ftruncate(int, __off_t)', declared with attribute warn_unused_result [-Wunused-result]
   (void) ftruncate(lockfd, 0);
                              ^
INFO: From JavacBootstrap src/java_tools/singlejar/java/com/google/devtools/build/singlejar/libbootstrap.jar [for host]:
warning: Implicitly compiled files were not subject to annotation processing.
  Use -proc:none to disable annotation processing or -implicit to specify a policy for implicit compilation.
1 warning
INFO: From JavacBootstrap src/java_tools/buildjar/java/com/google/devtools/build/buildjar/libskylark-deps.jar [for host]:
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
INFO: From JavacBootstrap src/java_tools/buildjar/java/com/google/devtools/build/buildjar/libbootstrap_VanillaJavaBuilder.jar [for host]:
warning: Implicitly compiled files were not subject to annotation processing.
  Use -proc:none to disable annotation processing or -implicit to specify a policy for implicit compilation.
Note: src/java_tools/buildjar/java/com/google/devtools/build/buildjar/VanillaJavaBuilder.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
1 warning
Target //src:bazel_nojdk up-to-date:
  bazel-bin/src/bazel_nojdk
INFO: Elapsed time: 346.215s, Critical Path: 271.47s
INFO: 1747 processes: 1511 local, 236 worker.
INFO: Build completed successfully, 1791 total actions
WARNING: --batch mode is deprecated. Please instead explicitly shut down your Bazel server using the command "bazel shutdown".
DEBUG: /tmp/bazel_jm8BGrdX/out/external/build_bazel_rules_nodejs/internal/common/check_bazel_version.bzl:49:5:
Current Bazel is not a release version, cannot check for compatibility.
DEBUG: /tmp/bazel_jm8BGrdX/out/external/build_bazel_rules_nodejs/internal/common/check_bazel_version.bzl:51:5: Make sure that you are running at least Bazel 0.17.1.

Build successful! Binary is here: /vol2/aarch64/qwert/output/bazel

I am curious to know if you will be interested in providing the binary for aarch64 architecture.

[aiuto]

I'm glad to see you got it to build for aarch64. Doing a distribution for that architecture is probably not going to be a priority right now. We would need test infrastructure and it would multiply the number of packages to publish.

If this were to be done as part of an overall Debian packaging project (rather than a standalone binary with the Bazel releases), would that be of interest?

[vielmetti]

I'm interested in helping provide test infrastructure to make this happen, as part of the Works on Arm project. There has been work going on for about a year now to support this, cf https://github.com/WorksOnArm/cluster/issues/102 with the help of @ArielleA .

One of the motivating factors for this is arm64 support for Tensorflow, which depends on Bazel.

[philwo]

@ossdev07 Thanks for trying Bazel on ARM!

@aiuto I don't see how it would "multiply" our packages if we provided additional Linux binaries built on ARM64? That's just one binary and the installer, right? The lack of machines is an issue, though...

@vielmetti Thanks again for sponsoring that machine 😊 I've been working on it and am currently integrating it into our Buildkite CI so that we can run postsubmit tests on it and keep an eye on the platform so that we don't accidentally break it.

I also fixed the remaining issues with bootstrapping, so that beginning from Bazel 0.28.0 you can bootstrap it via ./compile.sh from the -dist.zip without any flags or patches on ARM64.

In order to build official binaries though, we'd need more ARM64 machines, so that we can run presubmit tests without slowing developers down and one additional machine that we can use as a "trusted" machine that has limited login access and is only used to build release binaries on.

I'll ask some of the TensorFlow people about their plans for the platform and will get back to this.

[powderluv]

you may still hit this issue: #7135 but workaround suggested there seems to work

[powderluv]

Here is a 0.29 binary built with ubuntu 18 aarch64 and JDK8
https://github.com/powderluv/bazel-bin/blob/master/bazel-aarch64-29

[ogahozy]

Here is a 0.29 binary built with ubuntu 18 aarch64 and JDK8
https://github.com/powderluv/bazel-bin/blob/master/bazel-aarch64-29

How can I use this on tab s5e runing ubuntu under termux

[vielmetti]

The Bazel 1.0.0 release is out; if anyone has a 1.0.0 version for arm64, that would be useful news.

[powderluv]

https://github.com/powderluv/bazel-bin/blob/master/bazel-aarch64-1.0.0

[pisymbol]

What about just arm32/armv7? i.e. "runs on Raspberry Pi"

One of the problems with getting Envoy (k8s network proxy) to work on Pi is its bazel based build.

[powderluv]

I only have a raspberrry pi4. I you can build on arm32 happy to merge it in

[vielmetti]

@pisymbol I'd suggest a separate issue for the arm32/armv7 "Raspberry Pi" port for Bazel. That's a 32-bit architecture (aarch64 is 64-bit) and may have a complete other set of technical considerations.

The Pi 4 will run 64-bit operating systems, though stock Raspbian is 32-bits only.

(And yes, building Envoy is one of my targets for this.)

[neuralmimicry]

@philwo
@vielmetti

At Linaro we currently host a number of Aarch64 CI environments and have taken on-board the need to include Bazel due to Tensorflow requirements. Our initial work, some time ago, gave rise to this solution - https://collaborate.linaro.org/display/BDTS/Building+and+Installing+Tensorflow+on+AArch64

Our CI for Tensorflow is: https://github.com/Linaro/hpc_tensorflowci

Our teams have a high interest in ensuring that dependencies for builds are available and work for Aarch64.

Current AI/ML interests are here: https://www.linaro.org/engineering/artificial-intelligence/
With the latest talk: https://www.linaro.org/blog/can-we-make-ai-super/

What we would like to propose is being invited to submit results of builds to a Community Supported Builds page. (And any offer of additional hardware to support the CI infrastructure is greatly appreciated!). Naturally, in the longterm we would hope that full support for the overall package on Aarch64 would be included by default alongside other architectures.

I look forward to receiving your feedback.

Paul.
(Linaro HPC Tech Lead)

[philwo]

Thank you all for the feedback here!

I'm happy to say that we have been sponsored a shiny new and powerful ARM64 server hosted by Packet thanks to @vielmetti :) I'm currently setting it up and will get us some semi-official ARM64 builds for Bazel using that machine. I'll then look into hooking it up to Bazel's CI to ensure that we keep Bazel on ARM in reasonable shape.

I'll keep you updated here how it goes.

@neuralmimicry I'll add "Community Supported Builds" to our website. Do you have a link that I should add there where you host your builds?

[philwo]

Building Bazel 3.1.0 on the new ARM64 server worked out of the box just like on a x86_64 machine, no flags or anything special required:

$ uname -a
Linux arm1.bazel.build 5.3.0-51-generic #44~18.04.2-Ubuntu SMP Thu Apr 23 14:19:48 UTC 2020 aarch64 aarch64 aarch64 GNU/Linux

$ cat /etc/lsb-release 
[...]
DISTRIB_DESCRIPTION="Ubuntu 18.04.4 LTS"

$ sudo apt install openjdk-11-jdk-headless build-essential zip unzip git
$ mkdir bazel-3.1.0-dist
$ cd bazel-3.1.0-dist
$ wget https://github.com/bazelbuild/bazel/releases/download/3.1.0/bazel-3.1.0-dist.zip
$ unzip bazel-3.1.0-dist.zip

$ ./compile.sh
🍃  Building Bazel from scratch......
🍃  Building Bazel with Bazel.
Build successful! Binary is here: /home/philwo/src/bazel-3.1.0-dist/output/bazel

I was then able to use that Bazel binary as usual to build Bazel from a normal git clone of its master branch. 🎉

[mumbleskates]

My process:

1. Bootstrap bazel from the downloaded archive
2. Clone the repo at the desired commit (for me, current release)
3. Using the bootstrapped binary, bazel build -c opt //src:bazel //scripts:bazel-complete.bash in the bazel repo root
4. Create symlinks to the bazel binary and bazel-complete.bash files from bazel-out in /usr/bin/ and /etc/bash_completion.d/ respectively (like: sudo ln -s /home/widders/sourceprograms/bazel/bazel-out/aarch64-opt/bin/src/bazel /usr/bin/bazel)
5. Delete the bootstrap archive folder and back up a copy of either binary somewhere in case the bazel cache for the workspace gets deleted
6. To update bazel, update the repository and just run the same build command again

[settle]

@philwo I was unable to reproduce what you said worked in your last post (below is my Dockerfile based on your post). Would you provide more details, perhaps a Dockerfile?

ARG UBUNTU_VERSION=18.04

FROM arm64v8/ubuntu:${UBUNTU_VERSION} AS base

RUN uname -a

RUN cat /etc/lsb-release

RUN apt-get update && \
    DEBIAN_FRONTEND=noninteractive TZ=America/Los_Angeles apt-get install -y \
        openjdk-11-jdk-headless \
        build-essential \
        curl \
        zip \
        unzip \
        git

# Build and install bazel
ENV BAZEL_VERSION 3.1.0
WORKDIR /
RUN mkdir /bazel && \
    cd /bazel && \
    curl -fSsL -O https://github.com/bazelbuild/bazel/releases/download/$BAZEL_VERSION/bazel-$BAZEL_VERSION-dist.zip && \
    unzip bazel-$BAZEL_VERSION-dist.zip
RUN cd /bazel && \
    bash ./compile.sh

[philwo]

@settle I think you're only missing python and python3 in your apt-get install line, then it should work. This Dockerfile worked for me just now: https://gist.github.com/philwo/f2b01692cb4b46af79a8372082af1416

FWIW, technically Bazel would be fine with python3 only, but some scripts still use #!/usr/bin/env python in their shebang line. Ubuntu only provides the python executable when you install Python 2. :/

[settle]

@philwo Thanks for that missing piece! I'm now able to successfully build bazel on my Raspberry Pi 4 running Ubuntu 18.04.4 aarch64.

@powderluv Thanks for that tip! I also noticed in TensorFlow's devel-cpu-ppc64le.Dockerfile they used the following hack so they don't need to install python2. I tested this hack while building bazel (staying as close to the TF way above) and it worked.

ARG UBUNTU_VERSION=18.04

FROM arm64v8/ubuntu:${UBUNTU_VERSION} AS base

RUN apt-get update && \
    DEBIAN_FRONTEND=noninteractive TZ=America/Los_Angeles apt-get install -y \
        build-essential \
        curl \
        git \
        openjdk-8-jdk \
        zip \
        unzip \
        python3

# Some TF tools expect a "python" binary
RUN ln -s $(which python3) /usr/local/bin/python

# Build and install bazel
ENV BAZEL_VERSION 3.1.0
WORKDIR /
RUN mkdir /bazel && \
    cd /bazel && \
    curl -fSsL -O https://github.com/bazelbuild/bazel/releases/download/$BAZEL_VERSION/bazel-$BAZEL_VERSION-dist.zip && \
    unzip bazel-$BAZEL_VERSION-dist.zip && \
    bash ./compile.sh && \
    cp output/bazel /usr/local/bin/ && \
    rm -rf /bazel && \
    cd -

[jin]

@philwo #11201 is an outstanding PR that migrates all our scripts to py3.

[vielmetti]

Relating a recent request from a member of the Envoy team (Harvey) to get a supported binary release of Bazel for arm64.

[philwo]

The chances are very good that Bazel 3.4 will be the first release that comes with official arm64 binaries built on our packet.net machine integrated into our Bazel CI. 🎉

[tinatsou]

The chances are very good that Bazel 3.4 will be the first release that comes with official arm64 binaries built on our packet.net machine integrated into our Bazel CI. 🎉

Look forward to that. When will be Bazel 3.4 out?

[philwo]

Look forward to that. When will be Bazel 3.4 out?

The release baseline was picked today, so if all goes well it should be out in ~2 weeks.

[tinatsou]

Look forward to that. When will be Bazel 3.4 out?

The release baseline was picked today, so if all goes well it should be out in ~2 weeks.

Thank you @philwo ^ ^

[mixalako]

Hello , i want to build tensorflow with bazel for aarch64 but i cannot find the binary installer for aarch64 and i cannot docker buildx build wihtout it for aarch64.

[nSircombe]

This blog post: https://community.arm.com/developer/tools-software/tools/b/tools-software-ides-blog/posts/aarch64-docker-images-for-pytorch-and-tensorflow
...and the accompanying Docker build scripts here: https://github.com/ARM-software/Tool-Solutions/tree/master/docker/tensorflow-aarch64 might be of some assistance. The scripts build the TF 1.15 or 2.2 stack on AArch64, including Bazel.

[philwo]

Official Bazel 3.4.0 binaries for arm64 Linux have been released.

Direct download links:

• https://github.com/bazelbuild/bazel/releases/download/3.4.0/bazel-3.4.0-linux-arm64
• https://releases.bazel.build/3.4.0/release/bazel-3.4.0-linux-arm64

Please let me know if you encounter any issues with these. Next, I'll add support for arm64 to Bazelisk.

[advancedwebdeveloper]

@philwo , so it is able to execute on smartphones and tablets?

[tinatsou]

Bazel 3.4.0 has Arm64 release, however, didn't see 3.4.1 has Arm version. Why?
In addition, it seems that Bazel 3.4.0 has some problem on x86, but 3.4.1 is OK.
Thoughts?

[philwo]

@tinatsou Sorry for the delay. Bazel 3.4.1 binaries are now available on our GitHub release page and on https://releases.bazel.build/3.4.1/release/.

The release process is still partly manual for arm64, I'm working to automate and integrate it fully with the other platforms for future releases.

[philwo]

@advancedwebdeveloper If you have a smartphone or tablet that runs Linux, it should work :) Let us know if you get it working!

[advancedwebdeveloper]

@philwo , is it possible to that working during remote demos and workshops?
Anything possible for demonstrating on a remotely operated 64bit ARM board/gadget?

[tinatsou]

@tinatsou Sorry for the delay. Bazel 3.4.1 binaries are now available on our GitHub release page and on https://releases.bazel.build/3.4.1/release/.

The release process is still partly manual for arm64, I'm working to automate and integrate it fully with the other platforms for future releases.

Thank you @philwo

[kushalkoolwal]

I believe this issue is resolved, should we close this?

[herrahmo]

Hi everyone, I am a bit of a new programmer, I am trying to Install Bazel on my Ubuntu ARM64 20.04

Official Bazel 3.4.0 binaries for arm64 Linux have been released.

Direct download links:

• https://github.com/bazelbuild/bazel/releases/download/3.4.0/bazel-3.4.0-linux-arm64
• https://releases.bazel.build/3.4.0/release/bazel-3.4.0-linux-arm64

Please let me know if you encounter any issues with these. Next, I'll add support for arm64 to Bazelisk.

Sorry for my ignorance, but how do install using this file?

[philwo]

@herrahmo You can just download the Bazel file, put it somewhere and run it. I usually move it to /home/philwo/bin and put that directory on my $PATH so that I can just run Bazel by typing bazel. There is no need to install it or run a setup step - Bazel is a self-contained binary and will just work. :)

[mihaigalos]

Is there a roadmap for support for bazel on aarch64 linux nojdk?.. Asking for a friend.. :D

[vielmetti]

@mihaigalos Can you open a separate issue for tracking purposes? This issue has been closed.