Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Re-enable arm64 and x64 Mono fullAOT llvm and mini jobs #90427

Open
4 of 7 tasks
SamMonoRT opened this issue Aug 11, 2023 · 5 comments
Open
4 of 7 tasks

Re-enable arm64 and x64 Mono fullAOT llvm and mini jobs #90427

SamMonoRT opened this issue Aug 11, 2023 · 5 comments
Assignees
Labels
area-Infrastructure-mono disabled-test The test is disabled in source code against the issue tracking This issue is tracking the completion of other related issues.
Milestone

Comments

@SamMonoRT
Copy link
Member

SamMonoRT commented Aug 11, 2023

Description

Currently, the Mono fullAOT LLVM jobs are encountering failures and don't provide any test coverage. Specifically, the linux-arm64 job must build the cross-compiler using an image which incorporates both x64 and arm64 rootfs. Additionally, the aot-compiler.c utilizes GNU as which is not available since we don't have GNU binutils on the Mariner.

To resolve this issue, we have to fix the mono-aot-cross build for linux-arm64 to use target rootfs when building the runtime and host rootfs when building the compiler. After that, the aot-compiler.c has to be updated to use llvm-as or clang instead of GNU as.

Additionally, the AOT compiler with llvm 16 toolchain hits OOM issue with CBL_Mariner images.

Tasks

Initial discussion can be found at #86324 (comment).

Disabled tests

Upon the initial setup of the jobs, the following tests require triage.

FullAOT mini

AOT compilation failure:

src/tests/JIT/Methodical/Boxing/boxunbox/BoxPatternMatchAndSideEffects.csproj
src/tests/JIT/Methodical/Methodical_others.csproj
src/tests/JIT/Regression/JitBlue/GitHub_26491/GitHub_26491.ilproj
src/tests/JIT/Regression/Regression_3.csproj
src/tests/Interop/StringMarshalling/AnsiBSTR/AnsiBStrTest.csproj
src/tests/Interop/StringMarshalling/BSTR/BSTRTest.csproj
src/tests/Interop/StringMarshalling/LPTSTR/LPTSTRTest.csproj
src/tests/Interop/StringMarshalling/VBByRefStr/VBByRefStrTest.csproj
src/tests/Interop/Interop.csproj
src/tests/readytorun/coreroot_determinism/readytorun_coreroot_determinism.csproj
src/tests/readytorun/readytorun.csproj

Execution failure:

JIT/Methodical/Invoke/25params/25paramMixed_d/**
JIT/Methodical/Invoke/SEH/catchfault_jmp_d/**
JIT/Methodical/Invoke/SEH/catchfinally_jmpind_d/**
JIT/Methodical/Invoke/SEH/catchfinally_jmp_d/**
JIT/Methodical/tailcall/deep_array_d/**
JIT/Methodical/tailcall/deep_array_nz_d/**
JIT/Methodical/tailcall/deep_gc_d/**
JIT/Methodical/tailcall/deep_inst_d/**
JIT/Methodical/tailcall/deep_value_d/**
JIT/Methodical/tailcall/deep_virt_d/**
JIT/Methodical/VT/callconv/jumps2_d/**
JIT/Directed/zeroinit/tail_zeroinit/**
JIT/Directed/callconv/ThisCall/ThisCallTest/**
JIT/Directed/pinvoke/tail_pinvoke/**
JIT/Methodical/Invoke/25params/25paramMixed_r/**
JIT/Methodical/Invoke/SEH/catchfault_jmp_r/**
JIT/Methodical/Invoke/SEH/catchfinally_jmpind_r/**
JIT/Methodical/Invoke/SEH/catchfinally_jmp_r/**
JIT/Methodical/tailcall/deep_array_nz_r/**
JIT/Methodical/tailcall/deep_array_r/**
JIT/Methodical/tailcall/deep_gc_r/**
JIT/Methodical/tailcall/deep_inst_r/**
JIT/Methodical/tailcall/deep_value_r/**
JIT/Methodical/tailcall/deep_virt_r/**
JIT/Methodical/VT/callconv/jumper4_r/**
JIT/Methodical/VT/callconv/jumper5_r/**
JIT/Methodical/VT/callconv/jumps2_r/**
Regressions/coreclr/16354/notimplemented/**
JIT/Directed/perffix/primitivevt/callconv3_d/**
JIT/Directed/perffix/primitivevt/callconv3_r/**
JIT/SIMD/Vector3Interop_r/Vector3Interop_r.**
JIT/SIMD/Vector3Interop_ro/Vector3Interop_ro.**
JIT/Regression/CLR-x86-JIT/V1.2-M01/b13452/b13452/**
JIT/Regression/CLR-x86-JIT/V2.0-Beta2/b353858/b353858/**
JIT/Methodical/Boxing/misc/tailjump_r/**
JIT/Methodical/Boxing/misc/tailjump_d/**
baseservices/compilerservices/UnsafeAccessors/UnsafeAccessorsTests/**
baseservices/TieredCompilation/TieredVtableMethodTests/**
baseservices/varargs/varargsupport/**
JIT/opt/Devirtualization/Comparer_get_Default/Comparer_get_Default.**
JIT/Regression/JitBlue/Runtime_80731/Runtime_80731/**
Loader/classloader/DictionaryExpansion/DictionaryExpansion/**
Loader/classloader/explicitlayout/NestedStructs/case03/**
Loader/classloader/explicitlayout/NestedStructs/case04/**
Loader/classloader/explicitlayout/NestedStructs/case05/**
Loader/classloader/explicitlayout/Regressions/ASURT/ASURT150271/test13/**
Loader/classloader/MethodImpl/generics_override1/**
Loader/classloader/Statics/Misc/LiteralStatic/**
Loader/CustomAttributes/DynamicObjects/**
JIT/jit64/localloc/call/call05_dynamic/**
JIT/jit64/verif/sniff/fg/ver_fg_13/**
JIT/Directed/coverage/importer/badendfinally/**
JIT/Directed/coverage/importer/Desktop/badendfinally_d/**
JIT/Directed/coverage/importer/Desktop/badendfinally_r/**
JIT/Directed/coverage/importer/ceeillegal/**
JIT/Directed/coverage/importer/Desktop/ceeillegal_d/**
JIT/Directed/coverage/importer/Desktop/ceeillegal_r/**
JIT/SIMD/Vector3Interop_r/**
JIT/SIMD/Vector3Interop_ro/**
JIT/opt/Devirtualization/Comparer_get_Default/**
baseservices/varargs/varargsupport_r/**
Loader/classloader/MethodImpl/CovariantReturns/ReturnTypeValidation/**
Loader/classloader/MethodImpl/CovariantReturns/UnitTest/UnitTest_GVM/**
Interop/StringMarshalling/AnsiBSTR/AnsiBStrTest/**
Interop/StringMarshalling/BSTR/BSTRTest/**
Interop/StringMarshalling/LPTSTR/LPTSTRTest/**
Interop/StringMarshalling/VBByRefStr/VBByRefStrTest/**
JIT/Methodical/Boxing/boxunbox/BoxPatternMatchAndSideEffects/**
JIT/Intrinsics/TypeIntrinsics_r/**
JIT/Intrinsics/TypeIntrinsics_ro/**
ilasm/System/Runtime/CompilerServices/MethodImplOptionsTests/**
ilasm/PortablePdb/IlasmPortablePdbTests/**

FullAOT LLVM

AOT compilation failure:

src/tests/JIT/Methodical/Boxing/boxunbox/BoxPatternMatchAndSideEffects.csproj
src/tests/JIT/Methodical/Methodical_others.csproj
src/tests/JIT/Regression/JitBlue/GitHub_26491/GitHub_26491.ilproj
src/tests/JIT/Regression/Regression_3.csproj
src/tests/Interop/StringMarshalling/AnsiBSTR/AnsiBStrTest.csproj
src/tests/Interop/StringMarshalling/BSTR/BSTRTest.csproj
src/tests/Interop/StringMarshalling/LPTSTR/LPTSTRTest.csproj
src/tests/Interop/StringMarshalling/VBByRefStr/VBByRefStrTest.csproj
src/tests/Interop/Interop.csproj
src/tests/readytorun/coreroot_determinism/readytorun_coreroot_determinism.csproj
src/tests/readytorun/readytorun.csproj

Execution failure:

JIT/Methodical/Invoke/25params/25paramMixed_r
JIT/Methodical/Invoke/SEH/catchfault_jmp_r
JIT/Methodical/Invoke/SEH/catchfinally_jmpind_r
JIT/Methodical/Invoke/SEH/catchfinally_jmp_r
JIT/Methodical/tailcall/deep_array_nz_r
JIT/Methodical/tailcall/deep_array_r
JIT/Methodical/tailcall/deep_gc_r
JIT/Methodical/tailcall/deep_inst_r
JIT/Methodical/tailcall/deep_value_r
JIT/Methodical/tailcall/deep_virt_r
JIT/Methodical/VT/callconv/jumper4_r
JIT/Methodical/VT/callconv/jumper5_r
JIT/Methodical/VT/callconv/jumps2_r
JIT/Directed/perffix/primitivevt/callconv3_d
JIT/Directed/perffix/primitivevt/callconv3_r
JIT/jit64/localloc/call/call05_dynamic
JIT/jit64/verif/sniff/fg/ver_fg_13
JIT/Methodical/Invoke/25params/25paramMixed_d
JIT/Methodical/Invoke/SEH/catchfault_jmp_d
JIT/Methodical/Invoke/SEH/catchfinally_jmpind_d
JIT/Methodical/Invoke/SEH/catchfinally_jmp_d
JIT/Methodical/tailcall/deep_array_d
JIT/Methodical/tailcall/deep_array_nz_d
JIT/Methodical/tailcall/deep_gc_d
JIT/Methodical/tailcall/deep_inst_d
JIT/Methodical/tailcall/deep_value_d
JIT/Methodical/tailcall/deep_virt_d
JIT/Methodical/VT/callconv/jumps2_d
JIT/SIMD/Vector3Interop_r
JIT/SIMD/Vector3Interop_ro
JIT/HardwareIntrinsics/X86/Sse2.X64/StoreNonTemporal_r
JIT/HardwareIntrinsics/X86/Sse2.X64/StoreNonTemporal_ro
JIT/Directed/coverage/importer/badendfinally
JIT/Directed/coverage/importer/Desktop/badendfinally_d
JIT/Directed/coverage/importer/Desktop/badendfinally_r
JIT/Directed/coverage/importer/ceeillegal
JIT/Directed/coverage/importer/Desktop/ceeillegal_d
JIT/Directed/coverage/importer/Desktop/ceeillegal_r
JIT/HardwareIntrinsics/General/Vector128_1
baseservices/varargs/varargsupport_r
JIT/Methodical/Boxing/misc/tailjump_r
Regressions/coreclr/16354/notimplemented
JIT/Regression/CLR-x86-JIT/V1.2-M01/b13452/b13452
JIT/Regression/CLR-x86-JIT/V2.0-Beta2/b353858/b353858
JIT/Methodical/Boxing/misc/tailjump_d
JIT/Directed/zeroinit/tail_zeroinit
JIT/Directed/callconv/ThisCall/ThisCallTest
JIT/Directed/pinvoke/tail_pinvoke
Loader/classloader/generics/ByRefLike/ValidateNegative
JIT/Regression/JitBlue/Runtime_80731/Runtime_80731
JIT/opt/Devirtualization/Comparer_get_Default
Loader/classloader/DictionaryExpansion/DictionaryExpansion
Loader/classloader/explicitlayout/NestedStructs/case03
Loader/classloader/explicitlayout/NestedStructs/case04
Loader/classloader/explicitlayout/NestedStructs/case05
Loader/classloader/explicitlayout/Regressions/ASURT/ASURT150271/test13
Loader/classloader/MethodImpl/CovariantReturns/ReturnTypeValidation/ImplicitOverrideSameSigAsDecl
Loader/classloader/MethodImpl/CovariantReturns/ReturnTypeValidation/OverrideSameSigAsDecl
Loader/classloader/MethodImpl/CovariantReturns/UnitTest/UnitTest_GVM
Loader/classloader/MethodImpl/generics_override1
Loader/classloader/Statics/Misc/LiteralStatic
Loader/CustomAttributes/DynamicObjects
baseservices/compilerservices/UnsafeAccessors/UnsafeAccessorsTests
baseservices/TieredCompilation/TieredVtableMethodTests
baseservices/varargs/varargsupport
@ghost
Copy link

ghost commented Aug 11, 2023

Tagging subscribers to this area: @directhex
See info in area-owners.md if you want to be subscribed.

Issue Details

Refer to #86324 (comment), to re-enable the disabled lane.

Author: SamMonoRT
Assignees: directhex, kotlarmilos
Labels:

area-Infrastructure-mono

Milestone: 9.0.0

@ghost ghost added in-pr There is an active PR which will close this issue when it is merged and removed in-pr There is an active PR which will close this issue when it is merged labels Sep 7, 2023
@kotlarmilos kotlarmilos changed the title Re-enable ARM64 Mono-fullAOT LLVM lane Re-enable arm64 and x64 Mono fullAOT LLVM jobs Sep 7, 2023
@kotlarmilos
Copy link
Member

Mono fullAOT linux runtime tests now utilize the new Mariner image. This image does not include binutils, and as a result, GNU as cannot be used as a native assembler. The AOT compiler utilizes llvm opt and ilc to generate the llvm binary file. Then, a native assembler and linker are required to produce the final binary.

One option is to utilize llvm-mc (clang), which can convert assembly into a binary object file. However, I am not sure if there's a 1:1 mapping between the llvm-mc and GNU as. Another option is to ship binutils with the image.

For linux-arm64, either option would require a cross-compiled toolchain for x64 targeting arm64.

/cc: @directhex @lambdageek

@kotlarmilos
Copy link
Member

After several attempts to enable fullAOT llvm runtime tests on the extra-platforms, the job consistently hits OOM issue after 2h. To simplify the process, the job compiles only the baseservices/mono and System.Private.CoreLib with the maxcpucount set to 1.

The CI is using an azure instance with 16gb, docker on top of that is limited to less, probably 8gb. At peak, the AOT compilation takes about 16gb to AOT compile the System.Private.CoreLib. When I tried to reproduce the issue locally on the cbl-mariner-2.0-cross-amd64, it completed the compilation successfully within 10min. There is a tracking issue in azdo where we have requested a hardware upgrade dotnet/dnceng#1163.

@steveisok @SamMonoRT While we work on resolving the OOM issue, I would appreciate hearing your thoughts on temporarily disabling the affected jobs to ensure a consistently green CI.

@steveisok
Copy link
Member

I'm fine with disabling for the time being.

I wonder if requiring so much memory is a sign we can improve our aot compiler utilization?

@kotlarmilos
Copy link
Member

I wonder if requiring so much memory is a sign we can improve our aot compiler utilization?

Yes, let's create a tracking issue so that we can brainstorm our next steps. Additionally, I'll temporarily disable the failing jobs.

@directhex directhex removed their assignment Feb 5, 2024
@kotlarmilos kotlarmilos changed the title Re-enable arm64 and x64 Mono fullAOT LLVM jobs Re-enable arm64 and x64 Mono fullAOT llvm and mini jobs Mar 8, 2024
@kotlarmilos kotlarmilos added tracking This issue is tracking the completion of other related issues. disabled-test The test is disabled in source code against the issue labels Mar 8, 2024
sbomer added a commit that referenced this issue May 7, 2024
The biarch image is currently unused in our ci builds.

Use of the biarch image was added in
#91019 but the job that
used it was disabled for arm64 in
#92057. #90427
tracks adding back the arm64 jobs.

The arm64 fullaot job was added in
#55362 which explains:

> We don't ship any products using FullAOT on Linux x64/arm64, but this
compilation mode is used for iOS, FullAOT-related issues that affect iOS are
likely to be caught by FullAOT on Linux x64/arm64, and it is a lot easier to
build, develop, and debug on desktop OSes.

If the arm64 job is enabled again in the future, it should be
made to use a new entry in `pipeline-with-resources.yml`, instead
of modifying the image used for all other linux_arm64 jobs.

Fixes #101707
michaelgsharp pushed a commit to michaelgsharp/runtime that referenced this issue May 9, 2024
The biarch image is currently unused in our ci builds.

Use of the biarch image was added in
dotnet#91019 but the job that
used it was disabled for arm64 in
dotnet#92057. dotnet#90427
tracks adding back the arm64 jobs.

The arm64 fullaot job was added in
dotnet#55362 which explains:

> We don't ship any products using FullAOT on Linux x64/arm64, but this
compilation mode is used for iOS, FullAOT-related issues that affect iOS are
likely to be caught by FullAOT on Linux x64/arm64, and it is a lot easier to
build, develop, and debug on desktop OSes.

If the arm64 job is enabled again in the future, it should be
made to use a new entry in `pipeline-with-resources.yml`, instead
of modifying the image used for all other linux_arm64 jobs.

Fixes dotnet#101707
Ruihan-Yin pushed a commit to Ruihan-Yin/runtime that referenced this issue May 30, 2024
The biarch image is currently unused in our ci builds.

Use of the biarch image was added in
dotnet#91019 but the job that
used it was disabled for arm64 in
dotnet#92057. dotnet#90427
tracks adding back the arm64 jobs.

The arm64 fullaot job was added in
dotnet#55362 which explains:

> We don't ship any products using FullAOT on Linux x64/arm64, but this
compilation mode is used for iOS, FullAOT-related issues that affect iOS are
likely to be caught by FullAOT on Linux x64/arm64, and it is a lot easier to
build, develop, and debug on desktop OSes.

If the arm64 job is enabled again in the future, it should be
made to use a new entry in `pipeline-with-resources.yml`, instead
of modifying the image used for all other linux_arm64 jobs.

Fixes dotnet#101707
@kotlarmilos kotlarmilos modified the milestones: 9.0.0, 10.0.0 Jul 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area-Infrastructure-mono disabled-test The test is disabled in source code against the issue tracking This issue is tracking the completion of other related issues.
Projects
None yet
Development

No branches or pull requests

4 participants