[native-image] Cross compilation support? #407

sureshg · 2018-05-06T19:38:00Z

Any plans to add cross-compilation feature to native-image? I know it's too early to ask for these kind of requirements since we don't even have a working Windows build. IMHO, this would be a nice feature (golang has nice support for it) to have especially if we want to build binaries for all platforms from a CI (Jenkins) machine.

https://medium.com/@chrisgseaton/i-dont-think-we-have-any-immediate-plans-for-cross-compilation-but-it-s-not-a-major-problem-to-bf789384beaa

The text was updated successfully, but these errors were encountered:

plombardi89 · 2018-06-07T16:45:41Z

Definitely would love to see this feature! This would make it so much nicer to use Java for CLI apps and servers that need to be distributed to users on Linux, Mac and Windows.

alza-bitz · 2018-11-26T11:36:36Z

It would be amazing to have cross-compilation support. Is this still not planned for yet, or is it dependent on the Windows build? It would still be useful to have cross-compilation available to create Mac binaries on a (Linux) CI server, and then add support for Windows cross-compilation later?

alza-bitz · 2019-01-11T16:06:17Z

@thomaswue @chrisseaton I was just wondering if this is on the roadmap for 2019, or has any ETA etc? I think it would be a game-changer!

alza-bitz · 2019-07-09T16:59:37Z

@thomaswue @chrisseaton I was just wondering if this is on the roadmap for 2019, or has any ETA etc? I think it would be a game-changer!

Hi @thomaswue @chrisseaton since we're halfway through the year, I just thought I'd check in on this one..

Has there been any development on this issue, or any news/progress to report?

Thanks in advance! 🙂

- Support building via `native-image` - Support pushing to container repo for native image - No cross-platform build support yet (oracle/graal#407)

* Feature: Packaging This changeset introduces initial packaging for Gust, and for apps that use Gust. Changes so far: - [x] Fixup package for eventual publish to `npm` - [x] Add support for GitHub packages - [x] Add support for `rules_docker` targets - [x] Add support for tarball targets * Update Micronaut -> 1.3.0 * Upgrade Yarn dependencies * Upgrade Bazel dependencies * Fixes and functionality for crosslib builds - Fix defs when used in a cross-lib context - Fix DOM test - Tighten up DualStackTest * Initial support for container targets - Add Java container target - Add example push command (currently blocked on bazelbuild/rules_docker#1413) * Re-enable container tag via BUILD_SCM_VERSION * Fix workspace status command file * Working container build-and-run flow - Ability to build Java app containers, and push them to GCR - Tested containers locally (manually for now) * Update fork of rules_closure * Remove forced JDK8 build flags * Update J2CL with static bootclasspath * Update rules_graal with new attrs for debugging * Initial build flow for Graal native binary targets - Setup build lib - Attach to Graal image - Override default compiler path (workaround for Bazel bug) * Re-enable disk caching in CI * Native binary support, with container image - Support building via `native-image` - Support pushing to container repo for native image - No cross-platform build support yet (oracle/graal#407) * Fix injected testdeps * Don't enable native targets without a flag * Add CI routine for baking samples * Cleaner CI flow * Attempt GCR auth fix, flush keys * Another credentials fix * Fix Bazel label in regular CI job * Try auth via gcloud * Explicitly activate gcloud key * Apply new auth to sample image build * Fixup key paths in CI * Move GCP install step up in CI flow * Cleanup CI configs * Use relative access to service key from env * Unify image bake, restore Docker index pull * Add base container build file * Use new base container

sureshg · 2020-02-28T22:16:42Z

@thomaswue Now we have java 11 and windows support ready, are there any plans to consider cross-compilation support in the near future?

thomaswue · 2020-02-28T22:28:01Z

Adding the support is quite tricky. The issue is that there are parts of the JDK initialized during the native image generation process that could be platform dependent and then end up in the image. One of the alternatives could be to offer a service for creating native images. Also, maybe one could use maybe a system like GitHub actions to build images for different platforms without the need to maintain/setup those machines. Would this help for your use cases or is true cross compilation a requirement?

xiaodong-xie · 2020-02-28T22:48:37Z

I do think people should have mentioned this workaround elsewhere.

I am using this https://hub.docker.com/r/oracle/graalvm-ce/ docker image, to build the native-image running on Linux, on my MacBook laptop.

sureshg · 2020-02-29T04:23:46Z

Would this help for your use cases

@thomaswue Thanks. Yeah, GitHub action would work for OSS projects. The issue is, our build system is mostly on Linux and wants to build native-images for windows and mac (mostly for CLI apps). So the cross-compilation would have definitely helped in this scenario.

nhoughto · 2020-02-29T06:24:24Z

Yeah it’s not really cross compilation if you need the target platform to build it, that’s just regular old compilation. If Cross compilation is a feature being targeted all the jvm weirdness will need to be solved, I guess the question is, is/when is it a priority?

sureshg · 2020-03-01T02:13:58Z

Here is some discussion on the current limitations - https://graalvm.slack.com/archives/CN9KSFB40/p1582755160011700

truh · 2020-03-03T14:15:46Z

Here is some discussion on the current limitations - https://graalvm.slack.com/archives/CN9KSFB40/p1582755160011700

Would it be possible to share a summary of the discussion? The slack archive can only be accessed by people that are member to the workspace.

sureshg · 2020-03-08T09:53:15Z

@truh

Christian Wimmer : 
No, cross compilation between different OS is more or less impossible in
 the current approach because OS specific JDK classes need to be loaded 
in the image generator

The biggest problem is, e.g., the file system and network stack. The JDK code
 for that is very different for Linux, MacOS, and Windows. You cannot load the 
code from two platforms at the same time, but you need to load the code from 
the platform that the image generator runs on.

CircuitRCAY · 2020-07-24T01:28:24Z

Anything on cross compiling between architectures? (x86_64 -> arm64, for example)

8Keep · 2020-11-05T03:04:15Z

This is pretty huge for me. Is there any future plan at all to find a way to do this?

SaraDark · 2020-11-10T19:12:07Z

+1000 to Cross compilation support

tuhuynh27 · 2020-11-13T09:57:57Z

+10000 to Cross compilation support

The complexity of setting a build pipeline for GraalVM native image and having access to both macOs/linux for both x64 and ARM architecture is too much of an overhead for a small opensource project. This might be reconsidered in the future when availability on those OS/arch will be more widely spread or that GraalVM will offer cross compilation support. See oracle/graal#407

rmannibucau · 2022-11-09T07:28:09Z

Hi,

from Linux/amd64 to windows|arm64 for me.

Any CI is not an option on some env so would be great to have a solution close to c++ solutions.

xlight05 · 2022-11-09T08:08:16Z

What is the primary source/destination architecture you would be interested in to cross-compile, is it Linux/amd64 to Linux/aarch64? Is it possible to solve your use case with the GitHub Action workaround?

@thomaswue

I think the real advantage of cross compilation comes when JVM languages/frameworks adopt GraalVM Ex - Quarkus. Lets assume I wanna deploy an azure function using my framework(which is a perfect usecase where AOT shines). As far as I know, Azure functions only supports executable with linux/amd64. The language/framework can wrap graalvm and generate that executable for the user by hiding that complexity from user.

So the github actions workaround would not work in this case as it needs to happen from the framework's side, not the users side. The closest workaround i have is building it using docker using buildx. The issue with this is the docker dependency. Still, even with docker, its hard to do the cross compilation for different operating systems.

brainchild0 · 2022-11-29T01:04:01Z

For enterprise deployments, it may suffice in many contexts for a vendor to utilize a server for building each new release of a project. However, cross building is extremely standard for deployments of open-source platforms and packages. It is hard to see how a tool would gain much traction for open source projects these days if it constrains the target and build environment to be the same platform or machine family.

thomaswue · 2022-11-29T01:59:18Z

@brainchild0 Agreed. We believe however that many open source (and even commercial) projects can use our GitHub Action for GraalVM to build their releases. Is this not a reasonable workaround? https://github.com/marketplace/actions/github-action-for-graalvm

We could also maybe build an Oracle cloud service for building.

brainchild0 · 2022-11-29T02:42:58Z

Is this not a reasonable workaround?

@thomaswue The question has no general answer. In some cases it may be, others not. My remark directed at cases for which the answer would be negative, while acknowledging others for which it would be more optimistic.

For some open source components or platforms, you build everything in one place from scratch (or take binaries if available, but reserve the capability to build). That's just how it is.

What is worth considering is that the limitation probably will make adoption more reluctant even in border cases, due to concerns about dependence on a tool limited in broader adoptability.

We could also maybe build an Oracle cloud service for building.

Such a process is suited to projects supported by the community but still distributed as prebuilt binaries targeting major platforms. It is obviously a common case, but not the only.

brainchild0 · 2022-11-29T02:45:11Z

Not having any useful background on the subject, I am curious to ask about feasibility for running the tool version corresponding to the target platform inside Qemu.

rmannibucau · 2022-11-29T08:08:14Z

We believe however that many open source (and even commercial) projects can use our GitHub Action for GraalVM to build their releases. Is this not a reasonable workaround?

No because most of this solution wouldn't be free or too expensive, plus not usable in dev - without referencing projects not using Github, including on premise projects using Gitea/Drone or alike solutions. The key point is not NOT depend on a CI to be able to build. There are cases you want the release to be signed by an employe/user and not a technical key so for these cases you need an offline solution and hopefully local only (ie not multiple VM).

brainchild0 · 2022-11-30T12:40:04Z

There can be differences between operating systems how the JDK or third-party library dependencies are initialized and those end up snapshotted into the native image. Therefore, the problem of cross compilation for native image is more difficult to solve than cross compilation for e.g. C programs.

@thomaswue So the NI tool interrogates the memory state of the running JDK, and captures a representation for inclusion into the build target? Is the captured image specific to the application being translated, or only to the combination of kernel and machine types?

anentropic · 2022-12-22T00:16:21Z

What is the primary source/destination architecture you would be interested in to cross-compile, is it Linux/amd64 to Linux/aarch64? Is it possible to solve your use case with the GitHub Action workaround?

For me it would be compiling macos/aarch64 on macos/x86_64

Because GitHub actions does not have aarch64 macos runners, and for an OSS hobby project I do not want to pay for hosted Apple Silicon runners

Since all new macs are aarch64 it no longer seems worth building native images with Graal as I had been, since I can only support obsolete platforms that way

mikehearn · 2022-12-22T09:19:44Z

@rmannibucau

There are cases you want the release to be signed by an employe/user and not a technical key so for these cases you need an offline solution and hopefully local only (ie not multiple VM).

It's worth separating out the build from the release here. Whilst native-image can't cross-build, Conveyor can cross-package and cross-sign and is free for open source projects. So your pipeline could look like this:

Develop on regular JVM.
Compile with native-image using a multi-platform CI system (yes I know this is painful - I built one with home hardware to do this).
Write a Conveyor config that pulls the artifacts from your CI outputs and then produces signed, notarized binaries and packages from whatever local hardware you have your keys on. It supports HSMs.

rmannibucau · 2022-12-22T10:02:20Z

@mikehearn main point is to request a solution not depending on any CI like solution which are workarounds to this issue and do not cover all build/releases cases (not merging both cases but listing both as distinct but valid).

mikehearn · 2022-12-22T10:18:45Z

Yes, but that may not arrive any time soon and wouldn't solve the signing or packaging aspects, you'd still need other tools for that. I mention it here as an answer to your second part, so people who want to distribute CLI or GUI tools to desktops with local signing can know where to look.

thomaswue · 2022-12-25T13:16:07Z

@brainchild0 The state is specific to the application being translated.

In addition to this, we are depending on the local C developer tool chain a lot, so whatever cross-compilation combination we should support also needs to be available from that tool chain.

There is also the option to build within a docker container.

So, when looking at the existing workarounds such as Docker, GitHub Actions, Conveyor and the complexity of supporting all source/target platform combinations (with 5 supported platforms that would be 25), it seems like it would not be the best prioritization of engineering effort for the project at the moment. We will look more into build server setups that we can make available. Such a scenario can have additional advantages such as faster builds.

rmannibucau · 2022-12-25T14:38:25Z

All the mentionned solutions are at least as bad as requiring a local stack and most are more impacting like requiring to oss the project, to pay or just to not be able to run locally. All that are blockers to a solution to this issue so please invest in cross chains more than platforms which are just the current workaround.

brainchild0 · 2022-12-26T00:12:49Z

In addition to this, we are depending on the local C developer tool chain a lot, so whatever cross-compilation combination we should support also needs to be available from that tool chain.

Such a constraint by itself is not particularly limiting. Cross-compilation solutions have evolved quite substantially in the past decade, and the dissociation between build environment and build target is now rather common, and available without penalty.

it seems like it would not be the best prioritization of engineering effort for the project at the moment.

As previously stated, it is a matter of context and perspective . For some projects, remote builds would be a tremendous help. For others, it has no value. I think it is not completely accurate that CI is a reasonable substitute for local availability of cross-building operations, in the general case.

NCLnclNCL · 2023-08-28T09:38:45Z

.

chenzhiguo · 2023-11-06T12:18:47Z

I wonder, is there any support for cross-compilation?

thomaswue added the question label May 6, 2018

thomaswue self-assigned this May 6, 2018

cstancu mentioned this issue Sep 27, 2018

cross compilation #696

Closed

cstancu mentioned this issue Oct 27, 2018

Cross-Platform Support for native-image #765

Closed

cstancu mentioned this issue Jan 11, 2019

Provide option to generate native binaries for foreign processor architecture using native-image. #906

Closed

markelliot mentioned this issue Mar 3, 2019

Choose architecture / Build for all architectures palantir/gradle-graal#62

Closed

ansalond mentioned this issue Jun 4, 2019

Using official docker container for multi-platform builds #1347

Closed

koppor mentioned this issue Aug 19, 2019

Enable native image adoble/adr-j#16

Merged

sgammon added a commit to sgammon/elide-archive that referenced this issue Feb 6, 2020

Native binary support, with container image

f2fa0d0

- Support building via `native-image` - Support pushing to container repo for native image - No cross-platform build support yet (oracle/graal#407)

Anuken mentioned this issue Mar 5, 2021

Support Graal native-image libgdx/libgdx#6436

Closed

rodrigar-mx mentioned this issue Nov 9, 2021

how to use --target #3986

Closed

olyagpl mentioned this issue Dec 30, 2021

[Documentation] Small suggestion to add a sentence or so to "native-image" compiling standalone files #4159

Closed

mkurz mentioned this issue Jan 3, 2023

Build native image for Linux aarch64 sbt/sbtn-dist#1

Merged

vavdoshka mentioned this issue Feb 4, 2023

native image build gets stuck quarkusio/quarkus#16225

Closed

mageddo mentioned this issue Mar 3, 2023

Enable cross-compilation of native libraries #1446

Open

mageddo mentioned this issue Mar 12, 2023

Windows Binary Release mageddo/dns-proxy-server#306

Closed

gzm55 mentioned this issue Mar 30, 2023

Add Apple M1 build. apache/maven-mvnd#337

Closed

fniephaus mentioned this issue May 22, 2023

Unable to use --target option during native-image command #6645

Closed

fniephaus mentioned this issue Jun 29, 2023

native-image crashes, when CCompilerInvoker is not present #6875

Closed

k-wall mentioned this issue Jul 11, 2023

Multi-arch support? ozangunalp/kafka-native#78

Closed

micossow mentioned this issue Aug 11, 2023

add native arm image softwaremill/elasticmq#724

Closed

pvlugter mentioned this issue Oct 11, 2023

Document native image cross build akka/akka-projection#1034

Open

fniephaus mentioned this issue Dec 22, 2023

Time to think about allowing cross compiling in native-image ? #2757

Closed

ChadKillingsworth mentioned this issue Jan 2, 2024

feature request: add support of arm64 architecture on macOS google/closure-compiler-npm#296

Closed

asabil mentioned this issue Feb 5, 2024

ci: add Linux aarch64 build WhatsApp/erlang-language-platform#15

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[native-image] Cross compilation support? #407

[native-image] Cross compilation support? #407

sureshg commented May 6, 2018 •

edited

Loading

plombardi89 commented Jun 7, 2018

alza-bitz commented Nov 26, 2018

alza-bitz commented Jan 11, 2019

alza-bitz commented Jul 9, 2019

sureshg commented Feb 28, 2020

thomaswue commented Feb 28, 2020

xiaodong-xie commented Feb 28, 2020

sureshg commented Feb 29, 2020

nhoughto commented Feb 29, 2020

sureshg commented Mar 1, 2020

truh commented Mar 3, 2020

sureshg commented Mar 8, 2020

CircuitRCAY commented Jul 24, 2020

8Keep commented Nov 5, 2020

SaraDark commented Nov 10, 2020

tuhuynh27 commented Nov 13, 2020

rmannibucau commented Nov 9, 2022

xlight05 commented Nov 9, 2022 •

edited

Loading

brainchild0 commented Nov 29, 2022

thomaswue commented Nov 29, 2022

brainchild0 commented Nov 29, 2022 •

edited

Loading

brainchild0 commented Nov 29, 2022

rmannibucau commented Nov 29, 2022

brainchild0 commented Nov 30, 2022

anentropic commented Dec 22, 2022 •

edited

Loading

mikehearn commented Dec 22, 2022

rmannibucau commented Dec 22, 2022

mikehearn commented Dec 22, 2022

thomaswue commented Dec 25, 2022

rmannibucau commented Dec 25, 2022

brainchild0 commented Dec 26, 2022

NCLnclNCL commented Aug 28, 2023

chenzhiguo commented Nov 6, 2023

[native-image] Cross compilation support? #407

[native-image] Cross compilation support? #407

Comments

sureshg commented May 6, 2018 • edited Loading

plombardi89 commented Jun 7, 2018

alza-bitz commented Nov 26, 2018

alza-bitz commented Jan 11, 2019

alza-bitz commented Jul 9, 2019

sureshg commented Feb 28, 2020

thomaswue commented Feb 28, 2020

xiaodong-xie commented Feb 28, 2020

sureshg commented Feb 29, 2020

nhoughto commented Feb 29, 2020

sureshg commented Mar 1, 2020

truh commented Mar 3, 2020

sureshg commented Mar 8, 2020

CircuitRCAY commented Jul 24, 2020

8Keep commented Nov 5, 2020

SaraDark commented Nov 10, 2020

tuhuynh27 commented Nov 13, 2020

rmannibucau commented Nov 9, 2022

xlight05 commented Nov 9, 2022 • edited Loading

brainchild0 commented Nov 29, 2022

thomaswue commented Nov 29, 2022

brainchild0 commented Nov 29, 2022 • edited Loading

brainchild0 commented Nov 29, 2022

rmannibucau commented Nov 29, 2022

brainchild0 commented Nov 30, 2022

anentropic commented Dec 22, 2022 • edited Loading

mikehearn commented Dec 22, 2022

rmannibucau commented Dec 22, 2022

mikehearn commented Dec 22, 2022

thomaswue commented Dec 25, 2022

rmannibucau commented Dec 25, 2022

brainchild0 commented Dec 26, 2022

NCLnclNCL commented Aug 28, 2023

chenzhiguo commented Nov 6, 2023

sureshg commented May 6, 2018 •

edited

Loading

xlight05 commented Nov 9, 2022 •

edited

Loading

brainchild0 commented Nov 29, 2022 •

edited

Loading

anentropic commented Dec 22, 2022 •

edited

Loading