[VMR] linux/mac-arm64 cross build fails w/ CoreCLR runtime #3698

steveisok · 2023-10-30T16:51:16Z

Describe the Bug

When triggering a building targeting arm64 from a linux or mac x64 host, the VMR build will fail at runtime with the following error:

      [ 29%] Building CXX object nativeaot/Runtime/Full/CMakeFiles/Runtime.ServerGC.dir/__/EHHelpers.cpp.o
      [ 29%] Building CXX object pal/src/CMakeFiles/coreclrpal.dir/thread/threadsusp.cpp.o
      /Users/steve/dev/dotnet-vmr/src/runtime/artifacts/source-build/self/src/src/coreclr/pal/src/synchmgr/synchmanager.cpp:4633:6: error: "Don't know how to get hi-res current time on this platform"
          #error "Don't know how to get hi-res current time on this platform"
           ^
      [ 30%] Building CXX object nativeaot/Runtime/Full/CMakeFiles/Runtime.WorkstationGC.dir/__/MethodTable.cpp.o
      1 error generated.
      make[2]: *** [pal/src/CMakeFiles/coreclrpal.dir/synchmgr/synchmanager.cpp.o] Error 1
      make[2]: *** Waiting for unfinished jobs....

Steps to Reproduce

Apply the patch @directhex made in [VMR] Add cross build support #3671 (comment)
Build the VMR

For Mac:

./prep.sh 
./build.sh --online -- /p:OverrideTargetRid=osx-arm64

For Linux:

docker run -e CROSSCOMPILE=1 -e ROOTFS_DIR=/crossrootfs/arm64 -w pwd--platform linux/amd64 -vpwd:pwd -it mcr.microsoft.com/dotnet-buildtools/prereqs:cbl-mariner-2.0-cross-arm64 
./build.sh --online -- /p:OverrideTargetRid=linux-arm64

Other Information

The text was updated successfully, but these errors were encountered:

dotnet-issue-labeler · 2023-10-30T16:51:21Z

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

steveisok · 2023-10-30T16:52:00Z

/cc @jkoritzinsky

directhex · 2023-11-01T13:38:18Z

This is the error I get with Linux:

      Executing "/home/directhex/Projects/dotnet/src/runtime/artifacts/source-build/self/src/src/coreclr/build-runtime.sh" -cmakeargs "-DCLR_CROSS_COMPONENTS_BUILD=1" -arm64 -release -ci -portablebuild=false -keepnativesymbols -os linux -hostarch x64 -hostos linux -outputrid linux-arm64 -component crosscomponents
      Commencing CoreCLR Repo build
      __OutputRid: linux-arm64
      Setting up directories for build
      Checking prerequisites...
      Commencing build of "  crosscomponents " target in "CoreCLR component" for linux.arm64.Release in /home/directhex/Projects/dotnet/src/runtime/artifacts/source-build/self/src/artifacts/obj/coreclr/linux.arm64.Release/x64
      Invoking "/home/directhex/Projects/dotnet/src/runtime/artifacts/source-build/self/src/eng/native/gen-buildsys.sh" "/home/directhex/Projects/dotnet/src/runtime/artifacts/source-build/self/src/src/coreclr" "/home/directhex/Projects/dotnet/src/runtime/artifacts/source-build/self/src/artifacts/obj/coreclr/linux.arm64.Release/x64" x64 linux clang Release ""  -DCLR_CMAKE_TARGET_ARCH=arm64 -DCLR_CMAKE_PGO_INSTRUMENT=0 -DCLR_CMAKE_OPTDATA_PATH= -DCLR_CMAKE_PGO_OPTIMIZE=0 -DCLI_CMAKE_FALLBACK_OS="linux" -DFEATURE_DISTRO_AGNOSTIC_SSL=1 -DCLR_CROSS_COMPONENTS_BUILD=1  -DCLR_CMAKE_KEEP_NATIVE_SYMBOLS=true
      /home/directhex/Projects/dotnet/src/runtime/artifacts/source-build/self/src/artifacts/obj/coreclr/linux.arm64.Release/x64 /home/directhex/Projects/dotnet/src/runtime/artifacts/source-build/self/src/src/coreclr
      Not searching for unused variables given on the command line.
      loading initial cache file /home/directhex/Projects/dotnet/src/runtime/artifacts/source-build/self/src/eng/native/tryrun.cmake
      -- The C compiler identification is Clang 16.0.0
      -- The CXX compiler identification is Clang 16.0.0
      -- Detecting C compiler ABI info
      -- Detecting C compiler ABI info - failed
      -- Check for working C compiler: /usr/local/bin/clang-16
      -- Check for working C compiler: /usr/local/bin/clang-16 - broken
      CMake Error at /usr/share/cmake-3.21/Modules/CMakeTestCCompiler.cmake:69 (message):
        The C compiler
      
          "/usr/local/bin/clang-16"
      
        is not able to compile a simple test program.
      
        It fails with the following output:
      
          Change Dir: /home/directhex/Projects/dotnet/src/runtime/artifacts/source-build/self/src/artifacts/obj/coreclr/linux.arm64.Release/x64/CMakeFiles/CMakeTmp
          
          Run Build Command(s):/usr/bin/make -f Makefile cmTC_f4a38/fast && /usr/bin/make  -f CMakeFiles/cmTC_f4a38.dir/build.make CMakeFiles/cmTC_f4a38.dir/build
          make[1]: Entering directory '/home/directhex/Projects/dotnet/src/runtime/artifacts/source-build/self/src/artifacts/obj/coreclr/linux.arm64.Release/x64/CMakeFiles/CMakeTmp'
          Building C object CMakeFiles/cmTC_f4a38.dir/testCCompiler.c.o
          /usr/local/bin/clang-16 --target=x86_64-linux-gnu --gcc-toolchain=/crossrootfs/arm64/usr --sysroot=/crossrootfs/arm64    -MD -MT CMakeFiles/cmTC_f4a38.dir/testCCompiler.c.o -MF CMakeFiles/cmTC_f4a38.dir/testCCompiler.c.o.d -o CMakeFiles/cmTC_f4a38.dir/testCCompiler.c.o -c /home/directhex/Projects/dotnet/src/runtime/artifacts/source-build/self/src/artifacts/obj/coreclr/linux.arm64.Release/x64/CMakeFiles/CMakeTmp/testCCompiler.c
          Linking C executable cmTC_f4a38
          /usr/bin/cmake -E cmake_link_script CMakeFiles/cmTC_f4a38.dir/link.txt --verbose=1
          /usr/local/bin/clang-16 --target=x86_64-linux-gnu --gcc-toolchain=/crossrootfs/arm64/usr --sysroot=/crossrootfs/arm64 -Wl,--rpath-link=/crossrootfs/arm64/lib/x86_64-linux-gnu -Wl,--rpath-link=/crossrootfs/arm64/usr/lib/x86_64-linux-gnu -Wl,--rpath-link=/crossrootfs/arm64/lib/x86_64-linux-gnu -Wl,--rpath-link=/crossrootfs/arm64/usr/lib/x86_64-linux-gnu -fuse-ld=lld  CMakeFiles/cmTC_f4a38.dir/testCCompiler.c.o -o cmTC_f4a38 
          ld.lld: error: cannot open Scrt1.o: No such file or directory
          ld.lld: error: cannot open crti.o: No such file or directory
          ld.lld: error: cannot open crtbeginS.o: No such file or directory
          ld.lld: error: unable to find library -lgcc
          ld.lld: error: unable to find library -lgcc_s
          ld.lld: error: unable to find library -lc
          ld.lld: error: unable to find library -lgcc
          ld.lld: error: unable to find library -lgcc_s
          ld.lld: error: cannot open crtendS.o: No such file or directory
          ld.lld: error: cannot open crtn.o: No such file or directory
          clang-16: error: linker command failed with exit code 1 (use -v to see invocation)
          make[1]: *** [CMakeFiles/cmTC_f4a38.dir/build.make:100: cmTC_f4a38] Error 1
          make[1]: Leaving directory '/home/directhex/Projects/dotnet/src/runtime/artifacts/source-build/self/src/artifacts/obj/coreclr/linux.arm64.Release/x64/CMakeFiles/CMakeTmp'
          make: *** [Makefile:127: cmTC_f4a38/fast] Error 2
          
          
      
        
      
        CMake will not be able to correctly generate this project.
      Call Stack (most recent call first):
        CMakeLists.txt:9 (project)
      
      
      -- Configuring incomplete, errors occurred!

I don't know enough about CoreCLR to know what crosscomponents is, but if it's needed, it indicates a need for bi-arch Docker images to satisfy the attempts to compile x64 code for the x64 host on the arm64 build (a valid need in theory, but every instance needs auditing)

steveisok · 2023-11-01T13:54:10Z

@jtschuster @agocke looping you guys in too

jtschuster · 2023-11-21T16:48:29Z

I have a build finished and the arm64 sdk runs on my machine with qemu.
I've made a PR on my fork with the diff for reference: jtschuster/dotnet#1

I have dotnet/dotnet cloned at /src/dotnet in wsl, then ran

> docker run -e ROOTFS_DIR=/crossrootfs/arm64 -w /src/dotnet --platform linux/amd64 -v \\wsl.localhost\Ubuntu\src\dotnet:/src/dotnet -it mcr.microsoft.com/dotnet-buildtools/prereqs:cbl-mariner-2.0-cross-arm64
$ ./prep.sh && ./build.sh --online -- /p:OverrideTargetRid=linux-arm64 /p:OverrideTargetArch=arm64

To make the runtime compatible with cross-arch compilation, we should add --cross to the source-build build, and avoid publishing ReadyToRun or AOT compiled. The compilers don't have a valid runtime pack for the host during the cross-build at this point, so trying to run them fails. Part of dotnet/source-build#3698

jtschuster · 2023-12-08T18:54:09Z

The issue requiring workarounds is that we don't specify the build machine's runtime pack as a prerequisite and runtime is told not to download packages (DotNetBuildOffline = true), even using --online at the root. The in-build crossgen and ilc are published self-contained, but without the runtime pack, they fail when they try to run. I've added workarounds for this so that the build can finish (see mentions above), but it's probably not what we want to ship. I can't figure out how to declare source-build prerequisites to properly fix this, can anyone point me to that?

In the VMR, we don't yet specify the host runtime pack as a prerequisite for cross-arch builds, so the crossgen2 package that would be run on the build machine doesn't work. For now we can work around this by not running crossgen on cross-arch VMR builds. Related to dotnet/source-build#3698

jtschuster · 2023-12-14T21:14:25Z

Made #3793 to track reenabling crossgen and ilc.

…iting in source-build (#96858) In #75597, for source-build we replace all the RIDs with only the `PackageRID` in the `KnownFrameworkReference` for `NetCurrent` when we should have only added an additional RID. This was causing issues in crossbuilds of vertical builds. The crossgen and ilc projects that run within the build require a runtime for the build machine's RID, but it wasn't found in `KnownFrameworkReference` and the build failed during restore. Part of reenabling Crossgen and ILC in arm64 crossbuild verticals dotnet/source-build#3698

…iting in source-build (dotnet#96858) In dotnet#75597, for source-build we replace all the RIDs with only the `PackageRID` in the `KnownFrameworkReference` for `NetCurrent` when we should have only added an additional RID. This was causing issues in crossbuilds of vertical builds. The crossgen and ilc projects that run within the build require a runtime for the build machine's RID, but it wasn't found in `KnownFrameworkReference` and the build failed during restore. Part of reenabling Crossgen and ILC in arm64 crossbuild verticals dotnet/source-build#3698

steveisok added this to .NET Unified Build Oct 30, 2023

github-project-automation bot added this to .NET Source Build Oct 30, 2023

dotnet-issue-labeler bot added the untriaged label Oct 30, 2023

github-project-automation bot moved this to Backlog in .NET Source Build Oct 30, 2023

steveisok removed this from .NET Source Build Oct 30, 2023

steveisok mentioned this issue Nov 1, 2023

[VMR] Snapshot of build status for each configuration #3695

Open

MichaelSimons added area-unified-build and removed untriaged labels Nov 2, 2023

jtschuster mentioned this issue Nov 21, 2023

Don't run crossgen2 or ilc when cross-building in source-build dotnet/runtime#95088

Merged

jtschuster mentioned this issue Nov 29, 2023

Support VMR cross arch build dotnet/runtime#95425

Merged

jtschuster mentioned this issue Dec 8, 2023

Don't run crossgen in cross-arch VMR vertical builds dotnet/installer#17937

Merged

jtschuster mentioned this issue Dec 14, 2023

Pass CrossBuild to inner builds dotnet/installer#17972

Merged

jtschuster mentioned this issue Dec 14, 2023

Disable crossgen for VMR cross builds dotnet/aspnetcore#52828

Merged

mmitche mentioned this issue Jan 5, 2024

Implement and Verify Product Verticals #3902

Closed

66 tasks

jtschuster mentioned this issue Jan 11, 2024

Add non-portable RID to list of known runtime packs instead of overwriting dotnet/runtime#96858

Merged

jtschuster self-assigned this Jan 12, 2024

directhex mentioned this issue Mar 5, 2024

[VMR] Try lighting up ARM32/ARM64 lanes dotnet/installer#18883

Merged

jkoritzinsky self-assigned this Mar 7, 2024

jkoritzinsky mentioned this issue Mar 7, 2024

Set the CrossBuild property instead of the CROSSCOMPILE environment variable dotnet/installer#18915

Merged

jkoritzinsky closed this as completed in dotnet/installer#18915 Mar 11, 2024

github-project-automation bot moved this to Done in .NET Unified Build Mar 11, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[VMR] linux/mac-arm64 cross build fails w/ CoreCLR runtime #3698

[VMR] linux/mac-arm64 cross build fails w/ CoreCLR runtime #3698

steveisok commented Oct 30, 2023

dotnet-issue-labeler bot commented Oct 30, 2023

steveisok commented Oct 30, 2023

directhex commented Nov 1, 2023

steveisok commented Nov 1, 2023

jtschuster commented Nov 21, 2023

jtschuster commented Dec 8, 2023 •

edited

Loading

jtschuster commented Dec 14, 2023

[VMR] linux/mac-arm64 cross build fails w/ CoreCLR runtime #3698

[VMR] linux/mac-arm64 cross build fails w/ CoreCLR runtime #3698

Comments

steveisok commented Oct 30, 2023

Describe the Bug

Steps to Reproduce

Other Information

dotnet-issue-labeler bot commented Oct 30, 2023

steveisok commented Oct 30, 2023

directhex commented Nov 1, 2023

steveisok commented Nov 1, 2023

jtschuster commented Nov 21, 2023

jtschuster commented Dec 8, 2023 • edited Loading

jtschuster commented Dec 14, 2023

jtschuster commented Dec 8, 2023 •

edited

Loading