Update the CPUID and XSAVE logics for APX #104637

Ruihan-Yin · 2024-07-09T20:03:26Z

Previously #103019

Overview on the changes:

Infrastructural changes on XArchIntrinsicConstants: Compress all the Avx512 related flags into 1 - Avx512f+bw+cd+dq+vl to Avx512, this saves more space in XArchIntrinsicConstants so that we can hold more x86 ISAs, like here, APX.

Edit: remove unnecessary Avx512VL ISA flags. #103144 and Cleanup some handling around Avx10v1 #103241 have resolved this part.

CPUID check updates for APX: CR4[XSAVE](existing) -> XCR0[APX_F] -> CPUID(7,1).EDX[APX_F] - the current status is that due to the missing macro definition for APX on the OS level, the second check will fail anyways, and it may break the build on CI (to be verified).

Edit: made some workaround for this part in the code.

APX introduced 16 EGPRs, which are XSTATE enabled, this PR provides the corresponding changes for XSAVE behaviors.

Ruihan-Yin · 2024-07-16T00:05:47Z

Hi @tannergooding,

I reopened the PR for APX CPUID updates here, I will resolve the conflict on guid, let CI start soon and fix fails popping up.

I understand this PR is targeting on the next release cycle, say .Net 10, so when your schedule allows, I wonder if you could review this PR for the first round, thanks!

Ruihan-Yin · 2024-07-17T23:05:52Z

resolved the conflict.

Ruihan-Yin · 2024-08-16T21:41:34Z

The REX2 enabling PR (#106557) is there, that PR is based on this one, since now main is accepting .net 10 changes, I will rebase the changes to latest main and continue the works.

Ruihan-Yin · 2024-08-28T21:55:26Z

resolved conflicts with main

Ruihan-Yin · 2024-08-29T20:33:34Z

@tannergooding This PR is ready for review.

Build failures are related to the changes in https://github.com/dotnet/runtime/blob/main/src/coreclr/pal/src/arch/amd64/context2.S

the native compiler seems to not recognize EGPRs, e.g. r16 as a legal operand. I'm not sure how we can resolve this or if this part is needed for now. And for the changes to accommodate the XSTATE changes for EGPRs, I would be very willing to taking some suggestions from the high level, there might be some changes missing or not reasonable at all. it will be much appreciated if some advice can be shared.

janvorli

We will also need to update the CONTEXT& CONTEXT::operator=(const CONTEXT& ctx) to properly copy the new registers if the XStateFeaturesMask has a mask for this feature. And also make sure that when it is not set, we copy only the necessary part of the context.

janvorli · 2024-08-29T22:52:12Z

src/coreclr/pal/src/arch/amd64/context2.S

@@ -183,6 +183,29 @@ LOCAL_LABEL(Done_Restore_CONTEXT_FLOATING_POINT):
    kmovq k6, qword ptr [rdi + (CONTEXT_KMask0 + 6 * 8)]
    kmovq k7, qword ptr [rdi + (CONTEXT_KMask0 + 7 * 8)]

+    // TODO-xarch-apx: the definition of XSTATE mask value for APX is now missing on the OS level, 


This is Unix only code, so the XSTATE mask is not coming from the OS headers, but rather pal.h for C/C++ code and src/coreclr/pal/src/arch/amd64/asmconstants.h for asm. This comment can be removed.

janvorli · 2024-08-29T22:55:33Z

src/coreclr/pal/src/arch/amd64/context2.S

+    // TODO-xarch-apx: the definition of XSTATE mask value for APX is now missing on the OS level, 
+    //                 we are currently using bare value to hack it through the build process, and test the implementation through CI.
+    //                 those changes will be removed when we have the OS support for APX.
+    test    BYTE PTR [rdi + CONTEXT_XStateFeaturesMask], 524288


Can you please add symbolic definition for the mask to src/coreclr/pal/src/arch/amd64/asmconstants.h next to the other XSTATE_xxx definitions?

src/coreclr/vm/threadsuspend.cpp

mangod9 · 2024-10-14T15:05:32Z

@tannergooding assume you plan to review this change?

dotnet-policy-service · 2024-10-21T20:04:39Z

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

Ruihan-Yin · 2024-10-23T17:31:23Z

I will start to work on the merge conflicts on this PR and let the CI start.

Ruihan-Yin · 2024-10-23T22:01:09Z

wasm failures looks irrelevant.

linux and arm related build failures originate from the ASM code that uses EGPRs in the MOV instructions, while now the compiler (presumably LLVM) does not accept EGPRs as operand. This might be able to be fixed by updating the dependent version of LLVM.

Ruihan-Yin · 2024-10-30T21:14:29Z

Hi @tannergooding, @BruceForstall

This PR is ready for review, the reason for the failures are specified in #104637 (comment), it would be appreciated it if some inputs can be shared on this issue, thanks!

tannergooding · 2024-10-30T21:17:20Z

We should get #109210 merged before this or the AVX10.2 one goes in, since it fixes an issue with the thunk generator.

The changes in here mostly look good to me, however. I just need to finish a final review pass over it.

BruceForstall · 2024-11-04T23:32:33Z

#109210 has been merged. You'll need to update to fix the merge conflicts.

BruceForstall · 2024-11-04T23:54:35Z

linux and arm related build failures originate from the ASM code that uses EGPRs in the MOV instructions, while now the compiler (presumably LLVM) does not accept EGPRs as operand. This might be able to be fixed by updating the dependent version of LLVM.

Did you fix this issue?

We're using clang 18.1.8 to build (in azure Linux containers): dotnet/dotnet-buildtools-prereqs-docker#1207). What version has the APX features required?

BruceForstall · 2024-11-05T00:06:07Z

src/coreclr/vm/threadsuspend.cpp

@@ -71,6 +71,16 @@ extern "C" void             RedirectedHandledJITCaseForGCStress_Stub(void);
 #define IS_VALID_WRITE_PTR(addr, size)      _ASSERTE((addr) != NULL)
 #define IS_VALID_CODE_PTR(addr)             _ASSERTE((addr) != NULL)

+#if defined(TARGET_AMD64) || defined(TARGET_X86)
+// These values should be picked up from winrt.h, defining them in case they are missing there.


nit

Suggested change

// These values should be picked up from winrt.h, defining them in case they are missing there.

// These values should be picked up from winnt.h, defining them in case they are missing there.

BruceForstall · 2024-11-05T00:07:29Z

src/native/minipal/cpufeatures.c

+static uint32_t apxStateSupport()
+{
+#if defined(HOST_APPLE)
+    return false;


Shouldn't this be zero, or an assert, and not false?

jkotas · 2024-11-05T00:13:37Z

docs/design/features/xarch-apx.md

@@ -0,0 +1,3 @@
+# APX Integration in .NET


This doc can be added only once there is something worth talking about.

Sure, I can remove this for now and may add some specification to the design when REX2 or APX-EVEX changes coming in.

Ruihan-Yin · 2024-11-07T18:30:47Z

linux and arm related build failures originate from the ASM code that uses EGPRs in the MOV instructions, while now the compiler (presumably LLVM) does not accept EGPRs as operand. This might be able to be fixed by updating the dependent version of LLVM.

Did you fix this issue?

We're using clang 18.1.8 to build (in azure Linux containers): dotnet/dotnet-buildtools-prereqs-docker#1207). What version has the APX features required?

We've communicated with our LLVM folks and were told LLVM-19 should have APX support, but eventually we will need LLVM-20 to accommodate the Avx10.2 codegen changes.

Do you have any preference how we can make this through? we are okay with upgrading to LLVM-19 this time and LLVM-20 later, or upgrade once altogether. (LLVM-20 release should be out early next year.)

tannergooding · 2024-11-07T18:45:28Z

src/coreclr/pal/src/arch/amd64/context2.S

+    mov r16, qword ptr [rdi + CONTEXT_Egpr + 0 * 8]
+    mov r17, qword ptr [rdi + CONTEXT_Egpr + 1 * 8]
+    mov r18, qword ptr [rdi + CONTEXT_Egpr + 2 * 8]
+    mov r19, qword ptr [rdi + CONTEXT_Egpr + 3 * 8]
+    mov r20, qword ptr [rdi + CONTEXT_Egpr + 4 * 8]
+    mov r21, qword ptr [rdi + CONTEXT_Egpr + 5 * 8]
+    mov r22, qword ptr [rdi + CONTEXT_Egpr + 6 * 8]
+    mov r23, qword ptr [rdi + CONTEXT_Egpr + 7 * 8]
+    mov r24, qword ptr [rdi + CONTEXT_Egpr + 8 * 8]
+    mov r25, qword ptr [rdi + CONTEXT_Egpr + 9 * 8]
+    mov r26, qword ptr [rdi + CONTEXT_Egpr + 10 * 8]
+    mov r27, qword ptr [rdi + CONTEXT_Egpr + 11 * 8]
+    mov r28, qword ptr [rdi + CONTEXT_Egpr + 12 * 8]
+    mov r29, qword ptr [rdi + CONTEXT_Egpr + 13 * 8]
+    mov r30, qword ptr [rdi + CONTEXT_Egpr + 14 * 8]
+    mov r31, qword ptr [rdi + CONTEXT_Egpr + 15 * 8]


We shouldn't strictly need LLVM 19 here (or LLVM 20 for AVX10.2).

Both MASM and GAS have directives that allow defining macros and directives that allow emitting raw bytes.

Correspondingly, we should be able to just do .byte 0x00, 0x00, 0x00 (using appropriate bytes of course) and add a comment indicating what it translates to

We could theoretically build something more robust with macros as well, allowing something like mov_egpr(r16, ...) but that might be "too much" given what we need here is fairly static

Thanks for the inputs!

So we can replace these asm code with raw hex code for now to get the CI pass, am I understanding it right?

Right, and log a TODO for us to replace it with the proper instructions when we do eventually update to a newer assembler with the APX support.

janvorli · 2024-11-08T19:54:53Z

src/coreclr/pal/src/arch/amd64/asmconstants.h

@@ -91,7 +93,8 @@
 #define CONTEXT_KMask0 CONTEXT_Ymm0H+(16*16)
 #define CONTEXT_Zmm0H CONTEXT_KMask0+(8*8)
 #define CONTEXT_Zmm16 CONTEXT_Zmm0H+(32*16)
-#define CONTEXT_Size CONTEXT_Zmm16+(64*16)
+#define CONTEXT_Egpr  CONTEXT_Zmm16+(16*8)
+#define CONTEXT_Size CONTEXT_Egpr+(64*16)


This doesn't look right. It seems it should be

#define CONTEXT_Egpr CONTEXT_Zmm16+(64*16) #define CONTEXT_Size CONTEXT_Egpr+(16*8)

janvorli · 2024-11-08T19:58:33Z

src/coreclr/tools/Common/JitInterface/ThunkGenerator/InstructionSetDesc.txt

@@ -22,6 +22,8 @@

 ; DO NOT CHANGE R2R NUMERIC VALUES OF THE EXISTING SETS. Changing R2R numeric values definitions would be R2R format breaking change.

+; The ISA definiitons should also be mapped to `hwintrinsicIsaRangeArray` in hwintrinsic.cpp.


A nit:

Suggested change

; The ISA definiitons should also be mapped to `hwintrinsicIsaRangeArray` in hwintrinsic.cpp.

; The ISA definitions should also be mapped to `hwintrinsicIsaRangeArray` in hwintrinsic.cpp.

janvorli · 2024-11-08T20:01:28Z

src/native/minipal/cpufeatures.c

@@ -125,6 +155,19 @@ static uint32_t avx512StateSupport()
    return ((_xgetbv(0) & 0xE6) == 0x0E6) ? 1 : 0;
 }

+#ifndef XSTATE_MASK_APX


This was already defined in this file at line 75

Line 75 is the definition under UNIX (Avx512 seems to adopt the design that have these mask only explicitly defined under UNIX, so I followed it for APX). And under windows, I suppose the definition should be from winnt.h, I'm not quite familiar with how PAL works with each part. If so, we will need this definition here until OS supports APX and defines these masks in winnt?

janvorli · 2024-11-08T20:03:35Z

src/coreclr/pal/src/thread/context.cpp

@@ -313,6 +313,16 @@ typedef int __ptrace_request;
        ASSIGN_CONTROL_REGS \
        ASSIGN_INTEGER_REGS \

+#if defined(HOST_AMD64)  && defined(XSTATE_SUPPORTED)


It was already defined in pal.h, it should be visible here.

dotnet-issue-labeler bot added the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label Jul 9, 2024

dotnet-policy-service bot added the community-contribution Indicates that the PR has been added by a community member label Jul 9, 2024

am11 added area-PAL-coreclr and removed needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners labels Jul 19, 2024

Ruihan-Yin changed the title ~~[NO REVIEW] Update the CPUID and XSAVE logics for APX~~ [NO MERGE] Update the CPUID and XSAVE logics for APX Jul 19, 2024

Ruihan-Yin mentioned this pull request Aug 16, 2024

[JIT] Enable EGPRs in JIT by adding REX2 encoding to the backend. #106557

Draft

Ruihan-Yin added 5 commits August 28, 2024 13:41

Add APX doc.

9f8f67c

script-gen changes.

a68865c

XSTATE changes

6dfb92b

hand-written CPUID check part

b998cdf

fix

3446e28

Ruihan-Yin force-pushed the apx-CPUID-XSAVE branch from d549714 to 3446e28 Compare August 28, 2024 21:54

Fix merge error.

9bc008a

Ruihan-Yin changed the title ~~[NO MERGE] Update the CPUID and XSAVE logics for APX~~ Update the CPUID and XSAVE logics for APX Aug 28, 2024

bug fixes

44163d7

build-analysis bot mentioned this pull request Aug 29, 2024

restarted. Azure DevOps can't recover from restarts. dotnet/dnceng#3879

Open

3 tasks

Ruihan-Yin added 2 commits August 29, 2024 10:32

Bug fix

600abd0

bug fix

81b47c6

Ruihan-Yin marked this pull request as ready for review August 29, 2024 20:33

Ruihan-Yin requested a review from MichalStrehovsky as a code owner August 29, 2024 20:33

janvorli reviewed Aug 29, 2024

View reviewed changes

resolve commnets.

d516feb

BruceForstall mentioned this pull request Oct 15, 2024

Intel architecture improvements for .NET 10 #108869

Open

24 tasks

jeffschwMSFT added area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI and removed area-PAL-coreclr labels Oct 21, 2024

Merge remote-tracking branch 'origin/main' into apx-cpuid-sept

42f1729

This was referenced Oct 23, 2024

WasmTestOnChrome timeouts in CI #105363

Closed

chrome-DebuggerTests timing out #108078

Open

System.Net.Http.Functional.Tests.TelemetryTest failing with differing string #109024

Open

[debugger] Inspector exceptions in debugger tests #109025

Open

Ruihan-Yin added 2 commits November 4, 2024 15:42

Merge remote-tracking branch 'origin/main' into apx-cpuid-sept

d90bf49

re-generate the ISA changes to propagate the changes in ThunkGenerator.

79b23c8

BruceForstall reviewed Nov 5, 2024

View reviewed changes

jkotas reviewed Nov 5, 2024

View reviewed changes

tannergooding reviewed Nov 7, 2024

View reviewed changes

Ruihan-Yin added 2 commits November 8, 2024 10:21

resolve comments

8e1ba4a

use byte code for EGPR XSAVE logics.

ea8949f

janvorli reviewed Nov 8, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update the CPUID and XSAVE logics for APX #104637

Update the CPUID and XSAVE logics for APX #104637

Ruihan-Yin commented Jul 9, 2024 •

edited

Loading

Ruihan-Yin commented Jul 16, 2024

Ruihan-Yin commented Jul 17, 2024

Ruihan-Yin commented Aug 16, 2024

Ruihan-Yin commented Aug 28, 2024

Ruihan-Yin commented Aug 29, 2024

janvorli left a comment

janvorli Aug 29, 2024

janvorli Aug 29, 2024

mangod9 commented Oct 14, 2024

dotnet-policy-service bot commented Oct 21, 2024

Ruihan-Yin commented Oct 23, 2024

Ruihan-Yin commented Oct 23, 2024

Ruihan-Yin commented Oct 30, 2024

tannergooding commented Oct 30, 2024

BruceForstall commented Nov 4, 2024

BruceForstall commented Nov 4, 2024

BruceForstall Nov 5, 2024

BruceForstall Nov 5, 2024

jkotas Nov 5, 2024

Ruihan-Yin Nov 5, 2024

Ruihan-Yin commented Nov 7, 2024

tannergooding Nov 7, 2024 •

edited

Loading

Ruihan-Yin Nov 7, 2024 •

edited

Loading

tannergooding Nov 7, 2024

Ruihan-Yin Nov 7, 2024

janvorli Nov 8, 2024

janvorli Nov 8, 2024

janvorli Nov 8, 2024

Ruihan-Yin Nov 8, 2024 •

edited

Loading

janvorli Nov 8, 2024

	// These values should be picked up from winrt.h, defining them in case they are missing there.
	// These values should be picked up from winnt.h, defining them in case they are missing there.

		@@ -22,6 +22,8 @@

		; DO NOT CHANGE R2R NUMERIC VALUES OF THE EXISTING SETS. Changing R2R numeric values definitions would be R2R format breaking change.

		; The ISA definiitons should also be mapped to `hwintrinsicIsaRangeArray` in hwintrinsic.cpp.

	; The ISA definiitons should also be mapped to `hwintrinsicIsaRangeArray` in hwintrinsic.cpp.
	; The ISA definitions should also be mapped to `hwintrinsicIsaRangeArray` in hwintrinsic.cpp.

Update the CPUID and XSAVE logics for APX #104637

Are you sure you want to change the base?

Update the CPUID and XSAVE logics for APX #104637

Conversation

Ruihan-Yin commented Jul 9, 2024 • edited Loading

Ruihan-Yin commented Jul 16, 2024

Ruihan-Yin commented Jul 17, 2024

Ruihan-Yin commented Aug 16, 2024

Ruihan-Yin commented Aug 28, 2024

Ruihan-Yin commented Aug 29, 2024

janvorli left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mangod9 commented Oct 14, 2024

dotnet-policy-service bot commented Oct 21, 2024

Ruihan-Yin commented Oct 23, 2024

Ruihan-Yin commented Oct 23, 2024

Ruihan-Yin commented Oct 30, 2024

tannergooding commented Oct 30, 2024

BruceForstall commented Nov 4, 2024

BruceForstall commented Nov 4, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Ruihan-Yin commented Nov 7, 2024

tannergooding Nov 7, 2024 • edited Loading

Choose a reason for hiding this comment

Ruihan-Yin Nov 7, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Ruihan-Yin Nov 8, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Ruihan-Yin commented Jul 9, 2024 •

edited

Loading

tannergooding Nov 7, 2024 •

edited

Loading

Ruihan-Yin Nov 7, 2024 •

edited

Loading

Ruihan-Yin Nov 8, 2024 •

edited

Loading