Support building sdks that can build self-contained / ReadyToRun / NativeAOT applications without using pre-builts from nuget.org

[tmds]

Thanks to the reference packs, a source-build sdk can now build .NET Core applications that can be considered themselves source-buildable.

This doesn't apply to self-contained applications, and ReadyToRun applications. These require downloading a Microsoft.NETCore.App.Runtime.<rid> package from nuget.org. (This includes framework dependent ReadyToRun, probably because it uses the package to aquire crossgen.)

This issue is for building and including the Runtime package with the sdk. This makes it possible for the maintainer to apply patches during source-build, and the applications built this way using the sdk can be considered source-buildable.

cc @dagood @nguerrera @omajid

[dagood]

This issue is for building and including the Runtime package with the sdk.

Runtime packs are similar to targeting packs, so IMO they should be included in e.g. dotnet-runtime-pack-x-y packages, which the SDK package would have a dependency on.

This makes it possible for the maintainer to apply patches during source-build, and the applications built this way using the sdk can be considered source-buildable.

A note about this from #1202, to put it in context: the runtime pack has limitations on what it supports, due to glibc compatibility and portable vs. non-portable builds. An SCD publish for linux-x64 would need to bring down the extra-compatible prebuilt, but e.g. fedora.30-x64 would be source-built.

---

I believe the SDK and perhaps Core-SDK repos will need to add support for this. The SDK currently locates targeting packs with TargetingPackRoot ($(NetCoreTargetingPackRoot)) which points to e.g. /usr/bin/dotnet/packs/. The runtime pack resolver task doesn't seem to have anything similar.

The definition of NetCoreTargetingPackRoot is over in Core-SDK.

The apphost pack resolver uses TargetingPackRoot too, despite the name, so I guess it might be fine to use it for runtime packs too.

[tmds]

It makes sense to make it work similar to the apphost package.

Can we set a target milestone for this?

I can probably implement this based on these references.

[nguerrera]

Adding @dsplaisted

[tmds]

@dsplaisted @marcpopMSFT @MichaelSimons @dseefeld this issue is popping up as part of the s390x support because without it, users cannot create self-contained deployments.

Single-file publish is not possible either, but that is not a feature with the mono runtime (yet).

Can we look at including the runtime package in all source-built SDKs for .NET 7?

cc @omajid @uweigand

[uweigand]

The current .NET 6 SDK already checks for runtime packs in the packs/ subdirectory, it just doesn't find any because none were installed. I've been able to do using the following straightforward patch to the installer:

Index: installer/src/redist/targets/GenerateLayout.targets
===================================================================
--- installer.orig/src/redist/targets/GenerateLayout.targets
+++ installer/src/redist/targets/GenerateLayout.targets
@@ -134,6 +134,13 @@
         <RelativeLayoutPath>packs/%(PackageName)/%(PackageVersion)</RelativeLayoutPath>
       </BundledLayoutPackage>

+      <BundledLayoutPackage Include="MicrosoftNetCoreAppRuntimePackNupkg">
+        <PackageName>Microsoft.NETCore.App.Runtime.$(SharedFrameworkRid)</PackageName>
+        <PackageVersion>$(MicrosoftNETCoreAppRefPackageVersion)</PackageVersion>
+        <TargetFramework>$(TargetFramework)</TargetFramework>
+        <RelativeLayoutPath>packs/%(PackageName)/%(PackageVersion)</RelativeLayoutPath>
+      </BundledLayoutPackage>
+
       <BundledLayoutPackage Include="MicrosoftNetStandardTargetingPackNupkg">
         <PackageName>NETStandard.Library.Ref</PackageName>
         <PackageVersion>$(NETStandardLibraryRefPackageVersion)</PackageVersion>
@@ -146,6 +153,13 @@
         <PackageVersion>$(MicrosoftAspNetCoreAppRefPackageVersion)</PackageVersion>
         <TargetFramework>$(TargetFramework)</TargetFramework>
         <RelativeLayoutPath>packs/%(PackageName)/%(PackageVersion)</RelativeLayoutPath>
+      </BundledLayoutPackage>
+
+      <BundledLayoutPackage Include="MicrosoftAspNetCoreAppRuntimePackNupkg">
+        <PackageName>Microsoft.AspNetCore.App.Runtime.$(SharedFrameworkRid)</PackageName>
+        <PackageVersion>$(MicrosoftAspNetCoreAppRefPackageVersion)</PackageVersion>
+        <TargetFramework>$(TargetFramework)</TargetFramework>
+        <RelativeLayoutPath>packs/%(PackageName)/%(PackageVersion)</RelativeLayoutPath>
       </BundledLayoutPackage>

       <BundledLayoutPackage Include="MicrosoftNetCoreAppHostPackNupkg">

This likely still needs a bit of tweaking to use an appropriate condition when to do it. Also, it currently installs only the $(SharedFrameworkRid) ... maybe it also should install alternate RIDs as is done for the Apphost?

As a drawback, this approach significantly increases the size of the SDK installation on disk. This is annoying in particular as most of the files installed as part of the target runtime pack are in fact already present in the SDK on-disk installation, under the shared/ directory ... but it looks like that cannot be used as a drop-in replacement because the directory structure is different.

[tmds]

This likely still needs a bit of tweaking to use an appropriate condition when to do it. Also, it currently installs only the $(SharedFrameworkRid) ... maybe it also should install alternate RIDs as is done for the Apphost?

We can only build (rid-specific) packs for the target rid.

[uweigand]

This likely still needs a bit of tweaking to use an appropriate condition when to do it. Also, it currently installs only the $(SharedFrameworkRid) ... maybe it also should install alternate RIDs as is done for the Apphost?

We can only build (rid-specific) packs for the target rid.

I'm not sure I understand what you mean here. This is not about building those packs (they have to have been built previously), this is just about the installer build packaging them up for distribution as tarball.

For example, in the current source-build, we build the runtime twice, and that generates two separate App.Host packages, and both of them get packaged up by the installer. Similarly, those two runtime builds already also generate two App.Runtime packages, so the installer could package both of those as well, if we think that would be useful.

[tmds]

I'm not sure I understand what you mean here.

If you're on Fedora 34, you can claim you're building the host or runtime for fedora.34-x64 but you can't claim to be building the portable linux-x64.

The latter is meant to work across a range of distros. Microsoft builds it and publishes it for others to consume on nuget.org.

For example, in the current source-build, we build the runtime twice, and that generates two separate App.Host packages, and both of them get packaged up by the installer. Similarly, those two runtime builds already also generate two App.Runtime packages, so the installer could package both of those as well, if we think that would be useful.

What is the difference between the packages?

Is one using the target rid (like fedora.34-x64) and the other using the portable rid (like linux-x64)?

If that is the difference, the latter isn't really portable, because it will in some ways depend on Fedora 34.

It's a package that pretends to be the portable rid for the sake of source-build. It is not meant for the user.

[dsplaisted]

Can we look at including the runtime package in all source-built SDKs for .NET 7?

Due to the size of the runtime packs, it might be a good idea to have them separate and optional.

Or maybe it would be a good idea to be able to create source built NuGet packages and have them be acquired via NuGet in the same way the non source-built ones are.

[tmds]

.NET 7 will add support for NativeAOT.

We'd like a source-build SDK to be able to build self-contained NativeAOT applications.

This issue was already tracking adding support for self-contained, and I've added NativeAOT in the title too. We can split it up when that makes sense.

[crummel]

I've summarized what I believe to be the current state of play and the agreed-upon solution for NativeAOT in 7.0 in dotnet/runtime#76206. Please check this out and let me know if you have any concerns.

[jkotas]

I listed what I think are the main challenges with the nuget packages in source-build in an earlier comment: #1215 (comment).

Well, the alternative is to create a parallel path through the whole system where the native AOT toolchain does not look like a package. It sounds very complicated to me. I think addressing the issues with the package-centric approach is easier.

[tmds]

The alternative is to create a parallel path through the whole system where the native AOT toolchain does not look like a package. It sounds very complicated to me. I think addressing the issues with the package-centric approach is easier.

I think in the alternative it would still look like a package, but one that was already extracted.

Searching in the GitHub issues I found this feature request which matches with what I was asking earlier: NuGet/Home#7826.

The issue mentioned using a fallback folder and reading https://github.com/NuGet/Home/wiki/%5BSpec%5D-Fallback-package-folders#folder-format, it seems that placing the nupkg in this folder is optional.

If we can work with extracted packages, it would be easier to tackle the challenges mentioned in #1215 (comment). The stripping would be done automatically by the packaging tooling, so would debug symbol handling, and we can look at reducing the size by using symlinks (like we did for the runtime packs).

Can we use the fallback folder mechanism to include these packages instead of having them as packed nupkg files?

[dsplaisted]

Hi folks,

Some updates, and (I think) some good news.

While thinking this over I eventually came up with an idea that I think would let .props and .targets files be imported from packs in the packs folder. The idea is that ProcessFrameworkReferences would output the list of files to be imported, and the SDK would have a target that ran during restore that would write out generated .props and .targets files that would import the specified files and would be automatically imported during the build. This is how NuGet supports importing .props and .targets files from packages, so the SDK would do something similar also during restore.

However, while I was writing up a design for this idea, I looked more at how the three packages (Microsoft.DotNet.ILCompiler, Microsoft.NET.ILLink.Tasks, and runtime.<rid>.Microsoft.DotNet.ILCompiler) needed for NativeAOT are used. It turns out that the runtime one, which is the large one, doesn't have any .props and .targets in it and is already downloaded as a PackageDownload. So I think that package could be placed in the packs folder, and slight modifications to ProcessFrameworkReferences (probably mostly in the AddToolPack method) would allow it to be resolved from there.

The other two packages are way smaller, so I think including them as .nupkg files in the library-packs folder should work. You still might want to change the package ID to something source-build specific if there are concerns that the Microsoft-built ones from NuGet.org could be used instead of the ones that ship with the source-built SDK in the library-packs folder. We could also use the idea I had to allow these two packages to also be shipped in the packs folder if that was necessary.

As for fallback folders, I had forgotten about them, but was remembering the lessons we learned from them. I believe fallback folders aren't supported (at least not well) anymore. I believe we stopped using fallback folders around .NET Core 3, partly because we switched to framework references and package downloads, but also because fallback folders didn't work with features NuGet was adding such as package signing and hash validation.

[tmds]

I've started to implement the design proposed by @dsplaisted.

After patching the ProcessFrameworkReferences.cs so it looks for the source-built runtime.<rid>.Microsoft.DotNet.ILCompiler in packs, there is still another reference which triggers a nuget.org download for this package.

That reference comes from the runtime.json file that is in the source-built Microsoft.DotNet.ILCompiler package. It looks like:

{
  "runtimes": {
    "fedora.39-x64": {
      "#import": [],
      "Microsoft.DotNet.ILCompiler": {
        "runtime.fedora.39-x64.Microsoft.DotNet.ILCompiler": "[9.0.0-preview.5.24272.2, )"
      }
    },

We need the Microsoft.DotNet.ILCompiler property to be empty, like:

{
  "runtimes": {
    "fedora.39-x64": {
      "#import": [],
      "Microsoft.DotNet.ILCompiler": { }
    },

@dsplaisted @baronfel @ViktorHofer do you have a suggestion how/where to do this?

[dsplaisted]

Interesting. I imagine that reference in the runtime.json file is there to support explicitly referencing the Microsoft.DotNet.ILCompiler package. However, it seems that the PackageDownload added by the ProcessFrameworkReference task would be redundant. @jkotas, do you know how this works?

I don't know where the Microsoft.DotNet.ILCompiler package is created, but for source build I think you could omit the runtime.json file entirely.

[jkotas]

Yes, explicit reference of Microsoft.DotNet.ILCompiler package is expected to add the RID-specific Microsoft.DotNet.ILCompiler package. runtime.json in Microsoft.DotNet.ILCompiler package on nuget.org has entries for all portable RIDs supported by native AOT to make that work.

[tmds]

explicit reference of Microsoft.DotNet.ILCompiler package is expected to add the RID-specific Microsoft.DotNet.ILCompiler package

I assume this means we can not just remove the runtime.json file.
We need to change it then as described in #1215 (comment).

This package is built from the runtime repo: https://github.com/dotnet/runtime/tree/main/src/installer/pkg/projects/Microsoft.DotNet.ILCompiler. The source-built rid gets added here: https://github.com/dotnet/runtime/blob/03bc51f54dac5b8c7ba94ef798dfc93bfbbe6ef9/src/installer/pkg/projects/Microsoft.DotNet.ILCompiler/ILCompilerRIDs.props#L18. I don't know how this goes further into generating runtime.json. @dsplaisted @jkotas @ViktorHofer do you have a suggestion on where to tackle this?

[ViktorHofer]

The runtime.json feature is undocumented and got added in the .NET Core 1.x days to support including RID specific packages. The ILCompiler package is one of the remaining ones that use runtime.json and our plan is to eventually get rid off them all and replace them with alternative solutions (i.e. the Known* items in the SDK for anything in-built). dotnet/runtime#49137 is also related.

Mentioning that as I think changing the ILCompiler package and make it use the common SDK infrastructure (framework references / known* items) would make things way easier here and wouldn't require changing the SDK task.

[dsplaisted]

explicit reference of Microsoft.DotNet.ILCompiler package is expected to add the RID-specific Microsoft.DotNet.ILCompiler package

I assume this means we can not just remove the runtime.json file.

I think the effect of removing the file entirely would be the same as removing the element you are suggesting. The only thing that runtime.json file does is add the RID-specific package reference. If you remove that element, then that should be the same as removing the entire file (except possibly for projects that are targeting other RuntimeIdentifiers).

I don't know if things will work if you remove the reference. I'm not sure how the ILCompiler tasks locate the assets in the RID-specific ILCompiler package. As a first step though I'd suggest trying out removing the runtime.json entirely and seeing what happens.

[tmds]

As a first step though I'd suggest trying out removing the runtime.json entirely and seeing what happens.

Yes, this works.

I don't know if things will work if you remove the reference.

Removing the reference (as in #1215 (comment)) also works.

I think the effect of removing the file entirely would be the same as removing the element you are suggesting.

I interpreted #1215 (comment) as: when there is an explicit reference to the Microsoft.DotNet.ILCompiler we still need to restore these packages from runtime.json.

I think you are suggesting we drop runtime.json and rely on ProcessFrameworkReferences to add the packages for download.

@jkotas are there cases where that won't work?

[jkotas]

@jkotas are there cases where that won't work?

Is explicit reference of Microsoft.DotNet.ILCompiler package going to pick RID-specific Microsoft.DotNet.ILCompiler of the same version?

The explicit references of Microsoft.DotNet.ILCompiler are used e.g. here: https://github.com/dotnet/runtime/blob/main/src/coreclr/nativeaot/docs/compiling.md#using-daily-builds

[tmds]

Is explicit reference of Microsoft.DotNet.ILCompiler package going to pick RID-specific Microsoft.DotNet.ILCompiler of the same version?

I don't think so. We'd break this scenario by removing runtime.json.

[tmds]

How can we update/patch the runtime.json file so we eliminate the package reference only for the source-built rid?

[tmds]

@jkotas would it be ok if we'd remove runtime.json from Microsoft.DotNet.ILCompiler only when it gets built with source-build configuration?

[jkotas]

It is probably ok. It is a paper-cut behavior difference (not the first one) between the Microsoft build and the Linux distro build. We need to be watching for these paper-cut behavior differences and do something about them in case they show up on the radar.

Here is my understanding of how the different configurations are going to work:

• dotnet new console --aot && dotnet publish is going to use Microsoft.DotNet.ILCompiler provided by the Linux distro and runtime.linux-x64.Microsoft.DotNet.ILCompiler of the matching version downloaded from nuget.org
• dotnet new console --aot && dotnet publish -r rhel-x64 is going to use both Microsoft.DotNet.ILCompiler and runtime.rhel-x64.Microsoft.DotNet.ILCompiler provided by the Linux distro
• dotnet new console --aot && dotnet publish with explicit Microsoft.DotNet.ILCompiler package reference is going to use both Microsoft.DotNet.ILCompiler and runtime.linux-x64.Microsoft.DotNet.ILCompiler of the given version downloaded from nuget.org (unless the version matches the version provided by the Linux distro - Microsoft.DotNet.ILCompiler provided by the Linux distro is going to be used that case).

Is my understanding correct?

Also, have you considered installing the native AOT compilers as separate distro package (e.g. dnf install dotnet-sdk-aot-8.0) to reduce the baseline size of .NET SDK? This package can transitively install the native toolchain and other dependencies so that users do not have to install them manually.

[tmds]

Is my understanding correct?

This is a good overview, and it matches with how I expect it to behave.

Also, have you considered installing the native AOT compilers as separate distro package (e.g. dnf install dotnet-sdk-aot-8.0) to reduce the baseline size of .NET SDK? This package can transitively install the native toolchain and other dependencies so that users do not have to install them manually.

Yes, we'll put this in a separate package to reduce the default SDK size.

I like the idea of making this package install the native toolchain through dependencies. It will be up to the package maintainer to actually implement this.

We'll extend https://learn.microsoft.com/en-us/dotnet/core/distribution-packaging with these guidelines.

[agocke]

This all sounds reasonable to me, but I want to make sure we have a long-term design in place that's specified and maintained. Do we have a doc that explains exactly what library-packs and packs directories are, how they interact with PackageDownload, how they are used as hooks for source build, and what semantics we have to maintain in the future?

This area has been difficult to understand in the past.

[tmds]

These are the PRs that implement #1215 (comment):

• source-build: bundle NativeAOT libraries with the SDK. sdk#41198
• Packaging: support explicit runtime.json opt-out by setting IncludeRuntimeJson=false. arcade#14805
• ILCompiler.pkgproj: remove 'runtime.json' when DotNetBuildSourceOnly. runtime#102908

I verified that with these changes the resulting source-built SDK can be used to publish native-aot apps, and that it works as an SDK to build the vmr.

There is still some work to be done, esp on the sdk PR. I hope we can get these into the preview that the main branches are currently targeting.

Once these changes are in the vmr, we may look to do some optimizations that reduce the size of the ILCompiler pack.

[tmds]

@baronfel @dsplaisted @jkotas @MichaelSimons @ViktorHofer friendly reminder these PRs are still open, and this is something we'd really like to be in preview6.

cc @omajid

[MichaelSimons]

@tmds, @omajid, @ashnaga - Can we close this issue now? I am thinking any remaining bugs found can be tracked with instance specific issues.

[tmds]

We still need to add tests, and update https://learn.microsoft.com/en-us/dotnet/core/distribution-packaging.

[MichaelSimons]

@tmds - Is testing the last piece here? Is that still something you are planning to contribute?

[tmds]

Yes. I forgot about this. I'll look into it.