Move tools to use LKG ILC #107772

agocke · 2024-09-12T23:33:35Z

Starts with crossgen2 and moves shared helpers into a central targets file.

jkoritzinsky · 2024-09-12T23:40:53Z

Are we going to build the non-NativeAOT crossgen2 (the singlefilehost version) against the LKG bits?

If so, could we instead just update/use the crossgen2.csproj project and have 2 projects instead of continuing to have the 3 separate projects?

agocke · 2024-09-12T23:59:54Z

Are we going to build the non-NativeAOT crossgen2 (the singlefilehost version) against the LKG bits?

By that you mean, are we going to pull the single-file host from the LKG as well? I hadn't thought of that, but it makes sense.

jkoritzinsky · 2024-09-13T02:00:18Z

Yep that's what I was thinking of. I think however we ship crossgen2, we should ship it on equivalent runtime bits.

The alternative would likely be more expensive in infra maintenence for minimal gain, unless it's required by source-build or something.

jkotas · 2024-09-13T16:55:14Z

src/coreclr/tools/aot/crossgen2/crossgen2.csproj

@@ -1,14 +1,7 @@
 <Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
+    <CrossHostArch Condition="'$(CrossBuild)' == 'true' or '$(TargetArchitecture)' != '$(BuildArchitecture)' or '$(HostOS)' != '$(TargetOS)' or '$(EnableNativeSanitizers)' != ''">$(BuildArchitecture)</CrossHostArch>


For ilc, this is set unconditionally in the crossarch project

runtime/src/coreclr/tools/aot/ILCompiler/ILCompiler_crossarch.csproj

Line 4 in 2267228

<OutputPath>$(RuntimeBinDir)/$(CrossHostArch)/ilc/</OutputPath>

Also, we should make the crossgen and ilc project files use the same naming scheme. I like the naming scheme for ilc project files (ILCompiler.csproj, ILCompiler_crossarch.csproj).

agocke · 2024-09-13T18:42:23Z

I've been experimenting locally, and I don't think we should get rid of the publish project either.

The problem is that the SDK is architected such that _IsPublishing needs to be set when doing a publish operation.

I can't think of a way to do that without a different project if we also want things to use the non-published copy. If I set _IsPublishing unconditionally than the CoreCLR copy has the wrong settings. If I don't set it, we don't have the right publish settings. There's no condition I can use to set it otherwise.

jkotas · 2024-09-13T18:51:03Z

ilc has two projects (native and cross-arch). It does not have any special handling of publishing. Can crossgen be on the same plan?

agocke · 2024-09-13T19:23:53Z

I think ILC is the incorrect one here -- while not setting _IsPublishing doesn't actively break anything right now, it definitely could in the future. As we've been conditioning more things in the SDK to have different behavior during publish (to specialize for Native AOT and other form factors in combination with multiple architectures), _IsPublishing is becoming more common.

Thus far ILC has not depended on any of those behaviors, but I think that's accidental.

am11 · 2024-09-14T06:17:50Z

Maybe I am missing the bigger picture. Coverage-wise, both approaches are eventually testing what is being shipped:

With live build, we will be testing every version before it is shipped and as the changes are being made.
With LKG, we will be testing every version after it has made it to some public/semi-public nuget feed.

Live build is better IMO, and it gives us control over enabling nativeaot for new platforms without waiting for the entire update cycle, runtime->sdk->runtime (which takes a month or two). End to end tests in dotnet/sdk repo are covering the nuget feed scenario.

agocke · 2024-09-16T17:05:59Z

With live build, we will be testing every version before it is shipped and as the changes are being made.

But it's also testing every version, as you're developing it on a local machine and in CI. So in addition to the complexity in the build system needed to produce the right phase ordering, you also make local development and CI uniquely unstable by changing Native AOT. Add to all of this, debug builds become unusably slow because the debug JIT is incredibly slow.

Optimizing for live builds is optimizing for release validation, while using LKG builds is optimizing for infra simplicity and developer experience.

am11 · 2024-09-16T17:47:03Z

Yup, @jkotas explained it to me about the release vs. debug performance penalty and I agree that live build (for non-testing) should be on release plan.

BTW, are you looking to incorporrate the scenario #105004 is covering? I can close that one. Basically it's this workflow https://github.com/am11/CrossRepoCITesting/actions/runs/10847119420/workflow (also covered in runtime CI, see eng/pipelines changes in the PR).

agocke · 2024-09-18T20:13:17Z

BTW, are you looking to incorporrate the scenario #105004 is covering? I can close that one

In theory, yes. In practice, there isn't an LKG apphost for FreeBSD right? I haven't thought of what we should do in that case. Perhaps the answer is "require an LKG apphost just like we require an LKG runtime"

jkotas · 2024-09-18T20:21:31Z

I haven't thought of what we should do in that case. Perhaps the answer is "require an LKG apphost just like we require an LKG runtime"

I think we should introduce a new build option that builds everything that comes from LKG (including LKG apphost if necessary) and require invoking build with that option before invoking regular build if there is no LKG available for the target platform.

#105004 (comment) has details.

Introducing this build option should be a separate PR from this one.

am11 · 2024-09-19T05:51:39Z

In theory, yes. In practice, there isn't an LKG apphost for FreeBSD right?

Yes, it's about the else case when LKG is not available. These PRs are overlapping it's best to do it one at a time.

Starts with crossgen2 and moves shared helpers into a central targets file.

jkotas · 2024-09-24T20:48:45Z

src/tests/Common/Directory.Build.targets

@@ -9,6 +9,7 @@
          Condition="'$(DisableVersionCheckImported)' != 'true'" />

  <Import Project="$(RepoRoot)eng/liveBuilds.targets" />
+  <Import Project="$(RepositoryEngineeringDir)toolAot.targets" />


Do we still need to build crossgen during test build? Would it make more sense to go back to using the one created during the build?

Looks like there are still reasons to keep this. In particular, I moved all the package overrides here, so if we need to use new ILC packages the overrides are here.

I am talking about https://github.com/dotnet/runtime/blob/main/src/tests/Common/Directory.Build.targets#L66-L78 and UsePublishedCrossgen2 property. Building tests is reaching out into the product build to publish crossgen2. It should not be needed anymore since the ordering problems that this was working around should not exist after this PR. We should be able to use what we have produced during the product build instead. (My assumption was that the line that I have commented on is necessary to make this work.)

I agree that ability to override ILC LKG as necessary is good. But building tests should not need ILC LKG to publish. If tests need to publish, they should use the live ILC.

Do you have thoughts about this? Should triggering build of crossgen from the test partition be deleted? (The pain associated with that actually started the conversation that led to this PR.)

Thefrank · 2024-09-25T01:05:01Z

Will community supported platforms, in my case FreeBSD, still be able to generate a working runtime/ILC/Crossgen when building under Linux?

agocke · 2024-09-25T21:26:05Z

@Thefrank I don't think there's anything preventing it, but the packages will not be built by default

This reverts commit 9bc204d.

agocke · 2024-09-27T21:33:48Z

ugh, turns out we still need the inbuild ILC for the tests

agocke · 2024-10-03T20:55:01Z

I think this is ready to merge

agocke · 2024-10-07T20:53:54Z

@jkoritzinsky Mind also reviewing, as Jan is out?

kg · 2024-10-10T18:59:58Z

This appears to have broken crossgen outerloop (worse than it was already broken) by making the path for crossgen2.dll incorrect. It looks like #108693 will fix it but in the future it might be good to run crossgen outerloop for changes like this.

https://dev.azure.com/dnceng-public/public/_build/results?buildId=834252&view=logs&j=973abf82-0d2a-5516-d7bf-84fe12918d69&t=de5cac76-dfa7-5820-df2c-ae7256ad2c18

dotnet-issue-labeler bot added the area-crossgen2-coreclr label Sep 12, 2024

dotnet-policy-service bot assigned agocke Sep 12, 2024

build-analysis bot mentioned this pull request Sep 13, 2024

restarted. Azure DevOps can't recover from restarts. dotnet/dnceng#3879

Open

3 tasks

jkotas reviewed Sep 13, 2024

View reviewed changes

This was referenced Sep 13, 2024

Intermittent build failure in AfterSourceBuild: "Could not write state file" #76488

Open

SIGKILL (OOM?) while running LibraryImportGenerator.Tests w/o actionable log messages or artifacts dotnet/dnceng#2496

Open

jkotas mentioned this pull request Sep 13, 2024

Use live apphost when publishing ilc as singlefile #105004

Open

This was referenced Sep 19, 2024

Checkout failure: "Git fetch failed with exit code 128" dotnet/arcade#9009

Open

Unable to checkout dotnet/dnceng#4115

Open

agocke added 4 commits September 20, 2024 14:46

Move tools to use LKG ILC

e2500de

Starts with crossgen2 and moves shared helpers into a central targets file.

Remove more properties set centrally.

cf9e021

Remove crossgen2_inbuild and use main project

23814ac

Keep cross-arch configuration

55b59d5

jkotas reviewed Sep 24, 2024

View reviewed changes

agocke added 4 commits September 25, 2024 14:26

Move project back in subsets

b0dc636

Remove old crossarch project

9bc204d

Revert "Remove old crossarch project"

ce83ff1

This reverts commit 9bc204d.

Rename to inbuild

eda26e4

Merge branch 'main' into use-aot-lkg

eb85933

jkoritzinsky approved these changes Oct 7, 2024

View reviewed changes

agocke merged commit 3f28b1a into dotnet:main Oct 8, 2024
157 of 159 checks passed

agocke deleted the use-aot-lkg branch October 8, 2024 21:47

jkotas mentioned this pull request Oct 9, 2024

Delete UsePublishedCrossgen2 property #108693

Merged

tomeksowi mentioned this pull request Oct 9, 2024

[RISC-V] Transfer arguments between calling conventions in shuffling thunks #107282

Merged

steveisok mentioned this pull request Oct 9, 2024

Disable linux-x64 dev-innerloop #108581

Merged

This was referenced Oct 11, 2024

Consolidate LocateNativeCompiler target #103375

Merged

Test failure: readytorun/coreroot_determinism/coreroot_determinism/coreroot_determinism.cmd #106948

Closed

Crossgen2 outer loop fixes #108819

Merged

clamp03 mentioned this pull request Oct 17, 2024

[RISC-V] Test build doesn't make crossgen2 executable in RISC-V. #108973

Open

janvorli mentioned this pull request Oct 17, 2024

Test failure: JIT/jit64/mcc/interop/mcc_i03/mcc_i03.cmd #108640

Open

jakobbotsch mentioned this pull request Oct 29, 2024

SPMI: Use NAOT crossgen2 for collections #109333

Merged

jkotas mentioned this pull request Oct 29, 2024

Update the sdk target to net10.0 dotnet/sdk#43015

Merged

github-actions bot locked and limited conversation to collaborators Nov 10, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Move tools to use LKG ILC #107772

Move tools to use LKG ILC #107772

agocke commented Sep 12, 2024

jkoritzinsky commented Sep 12, 2024

agocke commented Sep 12, 2024

jkoritzinsky commented Sep 13, 2024

jkotas Sep 13, 2024

agocke commented Sep 13, 2024

jkotas commented Sep 13, 2024

agocke commented Sep 13, 2024

am11 commented Sep 14, 2024

agocke commented Sep 16, 2024

am11 commented Sep 16, 2024

agocke commented Sep 18, 2024

jkotas commented Sep 18, 2024

am11 commented Sep 19, 2024

jkotas Sep 24, 2024

agocke Sep 25, 2024

jkotas Sep 25, 2024

jkotas Oct 6, 2024

jkotas Oct 9, 2024

Thefrank commented Sep 25, 2024

agocke commented Sep 25, 2024

agocke commented Sep 27, 2024

agocke commented Oct 3, 2024

agocke commented Oct 7, 2024

kg commented Oct 10, 2024

Move tools to use LKG ILC #107772

Move tools to use LKG ILC #107772

Conversation

agocke commented Sep 12, 2024

jkoritzinsky commented Sep 12, 2024

agocke commented Sep 12, 2024

jkoritzinsky commented Sep 13, 2024

jkotas Sep 13, 2024

Choose a reason for hiding this comment

agocke commented Sep 13, 2024

jkotas commented Sep 13, 2024

agocke commented Sep 13, 2024

am11 commented Sep 14, 2024

agocke commented Sep 16, 2024

am11 commented Sep 16, 2024

agocke commented Sep 18, 2024

jkotas commented Sep 18, 2024

am11 commented Sep 19, 2024

jkotas Sep 24, 2024

Choose a reason for hiding this comment

agocke Sep 25, 2024

Choose a reason for hiding this comment

jkotas Sep 25, 2024

Choose a reason for hiding this comment

jkotas Oct 6, 2024

Choose a reason for hiding this comment

jkotas Oct 9, 2024

Choose a reason for hiding this comment

Thefrank commented Sep 25, 2024

agocke commented Sep 25, 2024

agocke commented Sep 27, 2024

agocke commented Oct 3, 2024

agocke commented Oct 7, 2024

kg commented Oct 10, 2024