Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Multiple issues when cross-compiling .NET runtime using official docker images #1002

Closed
lauxjpn opened this issue Apr 12, 2024 · 5 comments
Closed

Multiple issues when cross-compiling .NET runtime using official docker images #1002

lauxjpn opened this issue Apr 12, 2024 · 5 comments
Labels
area-dockerfiles bug untriaged

Comments

@lauxjpn
Copy link

lauxjpn commented Apr 12, 2024 •
edited
Loading

The issue

I am trying to build the .NET runtime for different platforms, by using the Dockerfiles for .NET Core Builds.

There are multiple issues with the cross-compilation process.

Lets assume the following simple dockerfile to cross-compile the dotnet runtime on an x64 machine for arm64:

FROM mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-22.04-cross-arm64
WORKDIR /src/dotnet/runtime
RUN <<EOT
    git clone --branch v8.0.4 --depth 1 https://github.com/dotnet/runtime.git .
    git checkout -b v8.0 v8.0.4
EOT
ENV ROOTFS_DIR=/crossrootfs/arm64
RUN ./build.sh --cross --arch arm64 --os linux --configuration Debug

If I build this dockerfile, I get the following error:

No usable version of clang found.

The reason is that while .../prereqs:ubuntu-22.04-coredeps (the base image used by .../prereqs:ubuntu-22.04-cross-arm64) installs llvm/clang via script, it does not set the installed version as the default:

Symbolic links missing for default llvm/clang usage 
root@e269ef91fec7:/src/dotnet/runtime# ls -lah /usr/bin/*clang*

lrwxrwxrwx 1 root root 26 Mar 17 02:35 /usr/bin/clang++-18 -> ../lib/llvm-18/bin/clang++
lrwxrwxrwx 1 root root 24 Mar 17 02:35 /usr/bin/clang-18 -> ../lib/llvm-18/bin/clang
lrwxrwxrwx 1 root root 28 Mar 17 02:35 /usr/bin/clang-cpp-18 -> ../lib/llvm-18/bin/clang-cpp
lrwxrwxrwx 1 root root 25 Mar 17 02:35 /usr/bin/clangd-18 -> ../lib/llvm-18/bin/clangd

root@e269ef91fec7:/src/dotnet/runtime# ls -lah /usr/bin/*llvm*

lrwxrwxrwx 1 root root 33 Mar 17 02:35 /usr/bin/llvm-addr2line-18 -> ../lib/llvm-18/bin/llvm-addr2line
lrwxrwxrwx 1 root root 26 Mar 17 02:35 /usr/bin/llvm-ar-18 -> ../lib/llvm-18/bin/llvm-ar
lrwxrwxrwx 1 root root 26 Mar 17 02:35 /usr/bin/llvm-as-18 -> ../lib/llvm-18/bin/llvm-as
lrwxrwxrwx 1 root root 34 Mar 17 02:35 /usr/bin/llvm-bcanalyzer-18 -> ../lib/llvm-18/bin/llvm-bcanalyzer
lrwxrwxrwx 1 root root 37 Mar 17 02:35 /usr/bin/llvm-bitcode-strip-18 -> ../lib/llvm-18/bin/llvm-bitcode-strip
lrwxrwxrwx 1 root root 27 Mar 17 02:35 /usr/bin/llvm-cat-18 -> ../lib/llvm-18/bin/llvm-cat
lrwxrwxrwx 1 root root 34 Mar 17 02:35 /usr/bin/llvm-cfi-verify-18 -> ../lib/llvm-18/bin/llvm-cfi-verify
lrwxrwxrwx 1 root root 30 Mar 17 02:35 /usr/bin/llvm-config-18 -> ../lib/llvm-18/bin/llvm-config
lrwxrwxrwx 1 root root 27 Mar 17 02:35 /usr/bin/llvm-cov-18 -> ../lib/llvm-18/bin/llvm-cov
lrwxrwxrwx 1 root root 30 Mar 17 02:35 /usr/bin/llvm-c-test-18 -> ../lib/llvm-18/bin/llvm-c-test
lrwxrwxrwx 1 root root 30 Mar 17 02:35 /usr/bin/llvm-cvtres-18 -> ../lib/llvm-18/bin/llvm-cvtres
lrwxrwxrwx 1 root root 31 Mar 17 02:35 /usr/bin/llvm-cxxdump-18 -> ../lib/llvm-18/bin/llvm-cxxdump
lrwxrwxrwx 1 root root 31 Mar 17 02:35 /usr/bin/llvm-cxxfilt-18 -> ../lib/llvm-18/bin/llvm-cxxfilt
lrwxrwxrwx 1 root root 30 Mar 17 02:35 /usr/bin/llvm-cxxmap-18 -> ../lib/llvm-18/bin/llvm-cxxmap
lrwxrwxrwx 1 root root 42 Mar 17 02:35 /usr/bin/llvm-debuginfo-analyzer-18 -> ../lib/llvm-18/bin/llvm-debuginfo-analyzer
lrwxrwxrwx 1 root root 34 Mar 17 02:35 /usr/bin/llvm-debuginfod-18 -> ../lib/llvm-18/bin/llvm-debuginfod
lrwxrwxrwx 1 root root 39 Mar 17 02:35 /usr/bin/llvm-debuginfod-find-18 -> ../lib/llvm-18/bin/llvm-debuginfod-find
lrwxrwxrwx 1 root root 28 Mar 17 02:35 /usr/bin/llvm-diff-18 -> ../lib/llvm-18/bin/llvm-diff
lrwxrwxrwx 1 root root 27 Mar 17 02:35 /usr/bin/llvm-dis-18 -> ../lib/llvm-18/bin/llvm-dis
lrwxrwxrwx 1 root root 31 Mar 17 02:35 /usr/bin/llvm-dlltool-18 -> ../lib/llvm-18/bin/llvm-dlltool
lrwxrwxrwx 1 root root 33 Mar 17 02:35 /usr/bin/llvm-dwarfdump-18 -> ../lib/llvm-18/bin/llvm-dwarfdump
lrwxrwxrwx 1 root root 33 Mar 17 02:35 /usr/bin/llvm-dwarfutil-18 -> ../lib/llvm-18/bin/llvm-dwarfutil
lrwxrwxrwx 1 root root 27 Mar 17 02:35 /usr/bin/llvm-dwp-18 -> ../lib/llvm-18/bin/llvm-dwp
lrwxrwxrwx 1 root root 32 Mar 17 02:35 /usr/bin/llvm-exegesis-18 -> ../lib/llvm-18/bin/llvm-exegesis
lrwxrwxrwx 1 root root 31 Mar 17 02:35 /usr/bin/llvm-extract-18 -> ../lib/llvm-18/bin/llvm-extract
lrwxrwxrwx 1 root root 32 Mar 17 02:35 /usr/bin/llvm-gsymutil-18 -> ../lib/llvm-18/bin/llvm-gsymutil
lrwxrwxrwx 1 root root 27 Mar 17 02:35 /usr/bin/llvm-ifs-18 -> ../lib/llvm-18/bin/llvm-ifs
lrwxrwxrwx 1 root root 41 Mar 17 02:35 /usr/bin/llvm-install-name-tool-18 -> ../lib/llvm-18/bin/llvm-install-name-tool
lrwxrwxrwx 1 root root 31 Mar 17 02:35 /usr/bin/llvm-jitlink-18 -> ../lib/llvm-18/bin/llvm-jitlink
lrwxrwxrwx 1 root root 40 Mar 17 02:35 /usr/bin/llvm-jitlink-executor-18 -> ../lib/llvm-18/bin/llvm-jitlink-executor
lrwxrwxrwx 1 root root 27 Mar 17 02:35 /usr/bin/llvm-lib-18 -> ../lib/llvm-18/bin/llvm-lib
lrwxrwxrwx 1 root root 38 Mar 17 02:35 /usr/bin/llvm-libtool-darwin-18 -> ../lib/llvm-18/bin/llvm-libtool-darwin
lrwxrwxrwx 1 root root 28 Mar 17 02:35 /usr/bin/llvm-link-18 -> ../lib/llvm-18/bin/llvm-link
lrwxrwxrwx 1 root root 28 Mar 17 02:35 /usr/bin/llvm-lipo-18 -> ../lib/llvm-18/bin/llvm-lipo
lrwxrwxrwx 1 root root 27 Mar 17 02:35 /usr/bin/llvm-lto-18 -> ../lib/llvm-18/bin/llvm-lto
lrwxrwxrwx 1 root root 28 Mar 17 02:35 /usr/bin/llvm-lto2-18 -> ../lib/llvm-18/bin/llvm-lto2
lrwxrwxrwx 1 root root 26 Mar 17 02:35 /usr/bin/llvm-mc-18 -> ../lib/llvm-18/bin/llvm-mc
lrwxrwxrwx 1 root root 27 Mar 17 02:35 /usr/bin/llvm-mca-18 -> ../lib/llvm-18/bin/llvm-mca
lrwxrwxrwx 1 root root 26 Mar 17 02:35 /usr/bin/llvm-ml-18 -> ../lib/llvm-18/bin/llvm-ml
lrwxrwxrwx 1 root root 34 Mar 17 02:35 /usr/bin/llvm-modextract-18 -> ../lib/llvm-18/bin/llvm-modextract
lrwxrwxrwx 1 root root 26 Mar 17 02:35 /usr/bin/llvm-mt-18 -> ../lib/llvm-18/bin/llvm-mt
lrwxrwxrwx 1 root root 26 Mar 17 02:35 /usr/bin/llvm-nm-18 -> ../lib/llvm-18/bin/llvm-nm
lrwxrwxrwx 1 root root 31 Mar 17 02:35 /usr/bin/llvm-objcopy-18 -> ../lib/llvm-18/bin/llvm-objcopy
lrwxrwxrwx 1 root root 31 Mar 17 02:35 /usr/bin/llvm-objdump-18 -> ../lib/llvm-18/bin/llvm-objdump
lrwxrwxrwx 1 root root 34 Mar 17 02:35 /usr/bin/llvm-opt-report-18 -> ../lib/llvm-18/bin/llvm-opt-report
lrwxrwxrwx 1 root root 29 Mar 17 02:35 /usr/bin/llvm-otool-18 -> ../lib/llvm-18/bin/llvm-otool
lrwxrwxrwx 1 root root 31 Mar 17 02:35 /usr/bin/llvm-pdbutil-18 -> ../lib/llvm-18/bin/llvm-pdbutil
lrwxrwxrwx 1 root root 38 Mar 17 02:35 /usr/bin/llvm-PerfectShuffle-18 -> ../lib/llvm-18/bin/llvm-PerfectShuffle
lrwxrwxrwx 1 root root 32 Mar 17 02:35 /usr/bin/llvm-profdata-18 -> ../lib/llvm-18/bin/llvm-profdata
lrwxrwxrwx 1 root root 31 Mar 17 02:35 /usr/bin/llvm-profgen-18 -> ../lib/llvm-18/bin/llvm-profgen
lrwxrwxrwx 1 root root 30 Mar 17 02:35 /usr/bin/llvm-ranlib-18 -> ../lib/llvm-18/bin/llvm-ranlib
lrwxrwxrwx 1 root root 26 Mar 17 02:35 /usr/bin/llvm-rc-18 -> ../lib/llvm-18/bin/llvm-rc
lrwxrwxrwx 1 root root 31 Mar 17 02:35 /usr/bin/llvm-readelf-18 -> ../lib/llvm-18/bin/llvm-readelf
lrwxrwxrwx 1 root root 31 Mar 17 02:35 /usr/bin/llvm-readobj-18 -> ../lib/llvm-18/bin/llvm-readobj
lrwxrwxrwx 1 root root 32 Mar 17 02:35 /usr/bin/llvm-readtapi-18 -> ../lib/llvm-18/bin/llvm-readtapi
lrwxrwxrwx 1 root root 30 Mar 17 02:35 /usr/bin/llvm-reduce-18 -> ../lib/llvm-18/bin/llvm-reduce
lrwxrwxrwx 1 root root 34 Mar 17 02:35 /usr/bin/llvm-remarkutil-18 -> ../lib/llvm-18/bin/llvm-remarkutil
lrwxrwxrwx 1 root root 30 Mar 17 02:35 /usr/bin/llvm-rtdyld-18 -> ../lib/llvm-18/bin/llvm-rtdyld
lrwxrwxrwx 1 root root 27 Mar 17 02:35 /usr/bin/llvm-sim-18 -> ../lib/llvm-18/bin/llvm-sim
lrwxrwxrwx 1 root root 28 Mar 17 02:35 /usr/bin/llvm-size-18 -> ../lib/llvm-18/bin/llvm-size
lrwxrwxrwx 1 root root 29 Mar 17 02:35 /usr/bin/llvm-split-18 -> ../lib/llvm-18/bin/llvm-split
lrwxrwxrwx 1 root root 30 Mar 17 02:35 /usr/bin/llvm-stress-18 -> ../lib/llvm-18/bin/llvm-stress
lrwxrwxrwx 1 root root 31 Mar 17 02:35 /usr/bin/llvm-strings-18 -> ../lib/llvm-18/bin/llvm-strings
lrwxrwxrwx 1 root root 29 Mar 17 02:35 /usr/bin/llvm-strip-18 -> ../lib/llvm-18/bin/llvm-strip
lrwxrwxrwx 1 root root 34 Mar 17 02:35 /usr/bin/llvm-symbolizer-18 -> ../lib/llvm-18/bin/llvm-symbolizer
lrwxrwxrwx 1 root root 30 Mar 17 02:35 /usr/bin/llvm-tblgen-18 -> ../lib/llvm-18/bin/llvm-tblgen
lrwxrwxrwx 1 root root 35 Mar 17 02:35 /usr/bin/llvm-tli-checker-18 -> ../lib/llvm-18/bin/llvm-tli-checker
lrwxrwxrwx 1 root root 31 Mar 17 02:35 /usr/bin/llvm-undname-18 -> ../lib/llvm-18/bin/llvm-undname
lrwxrwxrwx 1 root root 31 Mar 17 02:35 /usr/bin/llvm-windres-18 -> ../lib/llvm-18/bin/llvm-windres
lrwxrwxrwx 1 root root 28 Mar 17 02:35 /usr/bin/llvm-xray-18 -> ../lib/llvm-18/bin/llvm-xray

There are e.g. no symbolic links for /usr/bin/clang, but only for /usr/bin/clang-18. The dotnet runtime build system does not check the installed llvm/clang versions by default, but just tries to call clang, which then fails.

I have to explicitly specifiying the --clang18 parameter to build.sh, which requires me to know that the installed llvm version is 18:

FROM mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-22.04-cross-arm64
WORKDIR /src/dotnet/runtime
RUN <<EOT
    git clone --branch v8.0.4 --depth 1 https://github.com/dotnet/runtime.git .
    git checkout -b v8.0 v8.0.4
EOT
ENV ROOTFS_DIR=/crossrootfs/arm64
RUN ./build.sh --cross --arch arm64 --os linux --configuration Debug --clang18

(It probably would also work to add /usr/lib/llvm-18/bin/ to the PATH environment variable instead.)

This now starts the compilation process, until it hits the following error:

[ 23%] Building CXX object nativeaot/Runtime/eventpipe/CMakeFiles/eventpipe-enabled.dir/src/dotnet/runtime/src/native/containers/dn-umap.c.o
  [ 23%] Built target System.Security.Cryptography.Native.OpenSsl-Static
  /src/dotnet/runtime/src/coreclr/pal/src/cruntime/filecrt.cpp:89:9: error: use of undeclared identifier 'va_start'
     89 |         va_start(ap, nFlags);
        |         ^
  [ 23%] Building CXX object nativeaot/Runtime/eventpipe/CMakeFiles/eventpipe-enabled.dir/src/dotnet/runtime/src/native/containers/dn-vector.c.o
  /src/dotnet/runtime/src/coreclr/pal/src/cruntime/filecrt.cpp:91:9: error: use of undeclared identifier 'va_end'
     91 |         va_end(ap);
        |         ^
  [ 23%] Building CXX object gcinfo/CMakeFiles/gcinfo_win_x86.dir/simplerhash.cpp.o
  /src/dotnet/runtime/src/coreclr/pal/src/cruntime/filecrt.cpp:126:9: error: use of undeclared identifier 'va_start'
  [ 23%] Building CXX object md/compiler/CMakeFiles/mdcompiler_ppdb.dir/regmeta.cpp.o
    126 |         va_start(ap, nFlags);
        |         ^
  /src/dotnet/runtime/src/coreclr/pal/src/cruntime/filecrt.cpp:128:9: error: use of undeclared identifier 'va_end'
    128 |         va_end(ap);
        |         ^
  4 errors generated.
  make[2]: *** [pal/src/CMakeFiles/coreclrpal.dir/build.make:90: pal/src/CMakeFiles/coreclrpal.dir/cruntime/filecrt.cpp.o] Error 1
  make[1]: *** [CMakeFiles/Makefile2:2270: pal/src/CMakeFiles/coreclrpal.dir/all] Error 2
  make[1]: *** Waiting for unfinished jobs....

Looks like the source code is incompatible with llvm/clang 18.

To fix this issue, I install and use llvm/clang 14 instead:

FROM mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-22.04-cross-arm64
# Install llvm-toolchain version 14 using official script.
RUN wget -O- https://apt.llvm.org/llvm.sh | bash -s -- 14 \
        clang \
        clang-tools \
        liblldb-dev \
        lld \
        lldb \
        llvm \
        python3-lldb \
    && rm -rf /var/lib/apt/lists/*
WORKDIR /src/dotnet/runtime
RUN <<EOT
    git clone --branch v8.0.4 --depth 1 https://github.com/dotnet/runtime.git .
    git checkout -b v8.0 v8.0.4
EOT
ENV ROOTFS_DIR=/crossrootfs/arm64
RUN ./build.sh --cross --arch arm64 --os linux --configuration Debug --clang14

If I build this docker file, the runtime build continues further, until I hit:

Generating native code
  objcopy: Unable to recognise the format of the input file `/src/dotnet/runtime/artifacts/bin/coreclr/linux.arm64.Debug/ilc/native/ilc'
/root/.nuget/packages/microsoft.dotnet.ilcompiler/8.0.0-rc.1.23406.6/build/Microsoft.NETCore.Native.targets(376,5): error MSB3073: The command ""objcopy" --only-keep-debug "/src/dotnet/runtime/artifacts/bin/coreclr/linux.arm64.Debug/ilc/native/ilc" "/src/dotnet/runtime/artifacts/bin/coreclr/linux.arm64.Debug/ilc/native/ilc.dbg"" exited with code 1. [/src/dotnet/runtime/src/coreclr/tools/aot/ILCompiler/ILCompiler.csproj]

It appears that the wrong version of objcopy (the not-cross-compile version) is used by the build system.

There seems to be an ObjCopyName msbuild property that can be set to the name of the file that should be used instead, which would be llvm-objcopy-14:

FROM mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-22.04-cross-arm64
# Install llvm-toolchain version 14 using official script.
RUN wget -O- https://apt.llvm.org/llvm.sh | bash -s -- 14 \
        clang \
        clang-tools \
        liblldb-dev \
        lld \
        lldb \
        llvm \
        python3-lldb \
    && rm -rf /var/lib/apt/lists/*
WORKDIR /src/dotnet/runtime
RUN <<EOT
    git clone --branch v8.0.4 --depth 1 https://github.com/dotnet/runtime.git .
    git checkout -b v8.0 v8.0.4
EOT
ENV OBJCOPYNAME=llvm-objcopy-14
ENV ROOTFS_DIR=/crossrootfs/arm64
RUN ./build.sh --cross --arch arm64 --os linux --configuration Debug --clang14

(Appending -p:ObjCopyName=llvm-objcopy-14 to the build.sh parameter list instead would probably work as well.)

This dockerfile is now finally able to finish the build process successfully.

Suggestions

  • Bundle a toolchain version in the base image that actually works for building the runtime. I have only checked that llvm/clang 14 works and that 18 does not. Other versions in-between might work as well.
  • Make the llvm/clang version that comes bundled with the docker image the default, so that users don't have to go down the rabbit hole to find out, why the installed llvm/clang version is not being used, and that they need to explicitly specify the --clang<major_version> parameter (or change the PATH environment variable).
  • Make the llvm/clang objcopy version that comes bundled with the docker image the default for the build process.
  • Update the runtime building docs on GitHub, because they suggest that building the runtime is as simple as selecting one of the predefined docker images and run the build script, which is currently not the case.
  • There should be CI tests that ensure that all intended docker files for all platforms are successfully building the runtime.

It took me quite some time to analyze and fix these issues. Currently, it is unreasonably hard to build a cross-compiled version of the dotnet runtime for someone who has never done it before.
@MichaelSimons MichaelSimons transferred this issue from dotnet/dotnet-docker Apr 12, 2024
@lbussell
Copy link
Contributor

[Triage] @richlander @sbomer do either of you have any input on this issue with cross-compiling the .NET runtime?

@sbomer
Copy link
Member

sbomer commented Apr 18, 2024

@lauxjpn the image you are using (mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-22.04-cross-arm64) was specifically added for our .NET 7 builds, before we transitioned to using CBL-Mariner in official builds in .NET 8 (and Azure Linux 3.0 in .NET 9).

I'd recommend checking https://github.com/dotnet/runtime/blob/main/eng/pipelines/common/templates/pipeline-with-resources.yml which lists the images we use for cross-building in our ci. For 8.0.4, you'll have better luck with mcr.microsoft.com/dotnet-buildtools/prereqs:cbl-mariner-2.0-cross-arm64.

@richlander
Copy link
Member

richlander commented Apr 18, 2024 •
edited
Loading

There are good instructions here: https://github.com/dotnet/runtime/blob/b194416ea68e2d1c93a91fc7abf80eb2607b4831/docs/workflow/building/coreclr/linux-instructions.md

Also checkout https://github.com/dotnet/dotnet

@lauxjpn
Copy link
Author

lauxjpn commented Apr 18, 2024

There are good instructions here: https://github.com/dotnet/runtime/blob/b194416ea68e2d1c93a91fc7abf80eb2607b4831/docs/workflow/building/coreclr/linux-instructions.md

Yes, all those runtime workflow docs are the first ones I read. But since the docs say This table of images might often become stale as we change our images as our requirements change. and because the table mentioned the long outdated Alpine 3.13 and Ubuntu 16.04 and I didn't recognize mariner, so I thought it is probably some very old distribution code name, I directly went to the referenced dotnet-buildtools-prereqs-docker repo instead and then chose a cross-compilation enabled dockerfile of a distribution that I recognized (so that I can easily extend it if necessary) and that seemed current.

Assumption is the mother of all mistakes.

@lauxjpn the image you are using (mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-22.04-cross-arm64) was specifically added for our .NET 7 builds, before we transitioned to using CBL-Mariner in official builds in .NET 8 (and Azure Linux 3.0 in .NET 9).

I'd recommend checking https://github.com/dotnet/runtime/blob/main/eng/pipelines/common/templates/pipeline-with-resources.yml which lists the images we use for cross-building in our ci. For 8.0.4, you'll have better luck with mcr.microsoft.com/dotnet-buildtools/prereqs:cbl-mariner-2.0-cross-arm64.

Thanks, yes, the cbl-mariner-2.0-cross images do just work out-of-the-box.

I guess we can close this one.

@lauxjpn lauxjpn closed this as completed Apr 18, 2024
@github-project-automation github-project-automation bot moved this from Backlog to Done in .NET Docker Apr 18, 2024
@richlander
Copy link
Member

The table is accurate, however, I grasp the assumption.

This issue has relevant context: dotnet/runtime#100942

It is on my list of TODOs to update that table to reflect the state of that merged PR.

We use a cross build approach for everything and use very old targets to ensure that our users can run our binaries (for the most part) everywhere. It enables us to both satisfy secure supply chain requirements and offer binary compat. That's very hard to do well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area-dockerfiles bug untriaged
Projects
.NET Docker
Status: Done
Milestone
No milestone
Development

No branches or pull requests
4 participants
@sbomer @richlander @lauxjpn @lbussell