[NativeAOT] Simplify build integration with external bundlers for Apple platforms

[ivanpovazan]

Description

In order to support targeting Apple platforms NativeAOT build targets need to be integrated with external build systems and bundlers like:

• Xamarin.Sdk
• AppleAppBuilder

To achieve this, several project properties need to be set in the correct way and we are currently collecting all that is necessary to achieve the integration.

• For example: in case of targeting an iOS application with NativeAOT (PublishAot=true) and integrating it with an external build pipeline, these are the properties that need to be set:

  <PropertyGroup>
      <!-- This works around an issue in NativeAOT: https://github.com/dotnet/runtime/issues/86186 -->
      <IlcKeepManagedDebuggerSupport>true</IlcKeepManagedDebuggerSupport>
      <!-- We must find the BCL libraries using the runtime pack instead of using the built-in NativeAOT BCL: https://github.com/dotnet/runtime/issues/87060 -->
      <PublishAotUsingRuntimePack>true</PublishAotUsingRuntimePack>
      <!-- Build as a static lib -->
      <NativeLib>static</NativeLib> 
      <!-- Split init for the custom native host -->
      <CustomNativeMain>true</CustomNativeMain>
      <!-- Escape NativeAOT bundling targets -->
      <NativeCompilationDuringPublish>false</NativeCompilationDuringPublish>
      <IlcCompileDependsOn>Compile;ComputeIlcCompileInputs;SetupOSSpecificProps;PrepareForILLink</IlcCompileDependsOn>
  </PropertyGroup>
  <ItemGroup>
      <DirectPInvoke Include="__Internal" />
  </ItemGroup>

NOTE: Please note that the list above is not complete, it merely lists the current workarounds to enable building an iOS application.

To improve the integration process and maintainability it would be beneficial to simplify the integration process when targeting Apple platforms with a single "opt-in" MSBuild property.

[ghost]

Tagging subscribers to 'os-ios': @steveisok, @akoeplinger
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

In order to support targeting Apple platforms NativeAOT build targets need to be integrated with external build systems and bundlers like:

• Xamarin.Sdk
• AppleAppBuilder

To achieve this, several project properties need to be set in the correct way.

• For example: in case of targeting an iOS application with NativeAOT (PublishAot=true) and integrating it with an external build pipeline, these are the properties that need to be set:

  <!-- This works around an issue in NativeAOT: https://github.com/dotnet/runtime/issues/86186 -->
  <IlcKeepManagedDebuggerSupport>true</IlcKeepManagedDebuggerSupport>
  <!-- We must find the BCL libraries using the runtime pack instead of using the built-in NativeAOT BCL: https://github.com/dotnet/runtime/issues/87060 -->
  <PublishAotUsingRuntimePack>true</PublishAotUsingRuntimePack>
  <!-- Build as a static lib -->
  <NativeLib>static</NativeLib> 
  <!-- Split init for the custom native host -->
  <CustomNativeMain>true</CustomNativeMain>
  <!-- Escape NativeAOT bundling targets -->
  <NativeCompilationDuringPublish>false</NativeCompilationDuringPublish>
  <IlcCompileDependsOn>Compile;ComputeIlcCompileInputs;SetupOSSpecificProps;PrepareForILLink</IlcCompileDependsOn>

To improve the integration process and maintainability it would be beneficial to simplify the integration process when targeting Apple platforms with a single "opt-in" MSBuild property.

Author:	ivanpovazan
Assignees:	-
Labels:	os-ios, area-NativeAOT-coreclr
Milestone:	8.0.0

[MichalStrehovsky]

I'm not sure I understand the IlcCompileDependsOn and NativeCompilationDuringPublish parts. Is the intention that we would not run the Publish target when building for iOS?

[ivanpovazan]

I'm not sure I understand the IlcCompileDependsOn and NativeCompilationDuringPublish parts. Is the intention that we would not run the Publish target when building for iOS?

These are just currently used workarounds to avoid build issues like:

• In NativeAOT publish targets: ComputeLinkedFilesToPublish depends on LinkNative, while in iOS setup we would want to do our own native linking, so currently we would just turn it off
• The following causes build errors:

  runtime/src/coreclr/nativeaot/BuildIntegration/Microsoft.NETCore.Native.Publish.targets

  Line 52 in 20ca966

  <Error Condition="'$(OutputType)' != 'Library' and '$(NativeLib)' != ''" Text="NativeLib requires OutputType=Library." />

  • in the iOS build set up, the OutputType is Exe and we are compiling with <NativeLib>static</NativeLib> (this should probably include an additional condition like: and '$(CustomNativeMain)' != 'true' )
• CopyNativeBinary fails as the binary ends up in the .app bundle
• etc..

To summarize, I think I was not precise enough in the description of the issue (I will update it), as we are still collecting all that we would need for easier integration and the goal is to hopefully have this all accessible under a single switch.

[jkotas]

it would be beneficial to simplify the integration process when targeting Apple platforms with a single "opt-in" MSBuild property.

I do not think that the settings are the only one right way to target Apple platforms. They are specific to Xamarin app-model. Other app models (e.g. Avalonia) may want to do the integration differently.

We try to avoid hardcoding app-model specific settings in the core runtime .target files. For example, we have the same relationship with ASP.NET app models: ASP.NET SDKs have number of custom overrides of the defaults. I think we should follow the same pattern for all appmodels/SDKs, including Xamarin.

It is fine to expose individual features in cleaner way to make the integration easier.

[rolfbjarne]

It is fine to expose individual features in cleaner way to make the integration easier.

In particular having to set IlcCompileDependsOn seems like could be implemented in a better way.

[filipnavara]

• IlcKeepManagedDebuggerSupport is fixed by Allow duplicate feature switch settings #87207.
• PublishAotUsingRuntimePack will hopefully go away if we move everything to NativeAOT runtime packs ( [NativeAOT] Extend the runtime pack approach to all supported platforms #87060)

I feel the only fragile part is IlcCompileDependsOn, and the main reason why we need to fiddle with it is to allow running both ILLink and ILCompiler. While there are some discussions about avoiding that in the long-term I feel it would be useful to actually fix the scenario to be allowed. That would primarily mean allowing RunILLink=true opt-in here, and implementing the logic to pass ILLink output to ILCompiler on dotnet/runtime or dotnet/sdk side.

[kotlarmilos]

In particular having to set IlcCompileDependsOn seems like could be implemented in a better way.

While working on the runtime tests, I'm exploring several ideas to address this issue:

• One option is to parameterize the dependencies in Microsoft.NETCore.Native.Publish.targets and exclude the LinkNative target. However, this approach is similar to what has already been done.
• Another option is to create an entry target that invokes both common and platform-specific targets.
• We could also consider creating an abstract project with core targets, where each platform would then have its own project/targets that inherits the common targets and overall workflow.

For example, we have the same relationship with ASP.NET app models: ASP.NET SDKs have number of custom overrides of the defaults. I think we should follow the same pattern for all appmodels/SDKs, including Xamarin.

It would be good to check out an example of the ASP.NET app models, so we can consider the existing approach.

[MichalStrehovsky]

In particular having to set IlcCompileDependsOn seems like could be implemented in a better way.

What targets in Microsoft.NETCore.Native.Publish.targets are problematic? We could condition them on some property that the Xamarin props would set so that they don't interfere with how Xamarin wants to set up the flow of inputs/outputs.

The same with LinkNative - I'd suggest looking into why it's even executed when we're not doing a publish. Eventually, we'd want to split LinkNative into two targets - one that constructs the command line arguments we want to pass to the linker (so that the Xamarin targets can pick it up) and another one that actually runs linker (Xamarin would not participate in that).

[rolfbjarne]

In particular having to set IlcCompileDependsOn seems like could be implemented in a better way.

What targets in Microsoft.NETCore.Native.Publish.targets are problematic? We could condition them on some property that the Xamarin props would set so that they don't interfere with how Xamarin wants to set up the flow of inputs/outputs.

The same with LinkNative - I'd suggest looking into why it's even executed when we're not doing a publish. Eventually, we'd want to split LinkNative into two targets - one that constructs the command line arguments we want to pass to the linker (so that the Xamarin targets can pick it up) and another one that actually runs linker (Xamarin would not participate in that).

I'd suggest looking into why it's even executed when we're not doing a publish.

Note that we're doing "dotnet publish".

The immediate result I get (with the changes in xamarin/xamarin-macios#17374) if I set NativeCompilationDuringPublish=true is this:

error MSB4006: There is a circular dependency in the target dependency graph involving target "_ComputeLinkerArguments". Since "IlcCompile" has "DependsOn" dependence on "_ComputeLinkerArguments", the circular is "_ComputeLinkerArguments<-IlcCompile<-LinkNative<-ComputeLinkedFilesToPublish<-ComputeResolvedFilesToPublishList<-_ComputeFrameworkAssemblies<-_ComputeLinkerArguments".

Looks like that's because of:

runtime/src/coreclr/nativeaot/BuildIntegration/Microsoft.NETCore.Native.Publish.targets

Lines 20 to 22 in 5b900dd

	<Target Name="ComputeLinkedFilesToPublish"
	BeforeTargets="ComputeResolvedFilesToPublishList"
	DependsOnTargets="_ComputeAssembliesToCompileToNative;LinkNative">

where Microsoft.NETCore.Native.Publish.targets injects the ComputeLinkedFilesToPublish into the build chain, and indirectly LinkNative.

So here's problem 1: we don't want to execute NativeAOT's LinkNative target, because we're using the static library ILC creates, and not the final linked output.

Maybe it would make sense to create a property the SDK could set to indicate that we only need the static library, and not the final linked result?

[jkotas]

Maybe it would make sense to create a property the SDK could set to indicate that we only need the static library, and not the final linked result?

It is what <NativeLib>Static</NativeLib> does today. It may need some tweaks so that it works well for Xamarin config.

[kotlarmilos]

There is more we can do to simplify the integration, so keeping this issue open.