[ios][tests] Run functional tests with Mono and Native AOT on Helix #87773

kotlarmilos · 2023-06-19T16:37:42Z

Description

Initial work can be found at https://github.com/steveisok/runtime/tree/ios-sample-lib-targets.

This PR adds support for Native AOT compilation on Helix. It improves test coverage for Mono and Native AOT by running functional tests on Helix. The proxy project is updated with Native AOT props and apple build targets are updated to support Native AOT compilation. Additionally, it simplifies the sample app by utilizing the shared Apple targets.

Fixes #85294

ghost · 2023-06-19T16:37:57Z

Tagging subscribers to this area: @directhex
See info in area-owners.md if you want to be subscribed.

Issue Details

Work in progress.

This PR aims to add the Mono and Native AOT iOS samples as functional tests on a device to the Helix. Currently, it includes some temporary workarounds that will be resolved once the CI reports green.

initial work can be found at https://github.com/steveisok/runtime/tree/ios-sample-lib-targets.

Description

TBD

Author:	kotlarmilos
Assignees:	kotlarmilos
Labels:	`NO-MERGE`, `NO-REVIEW`, `area-Infrastructure-mono`, `os-ios`
Milestone:	8.0.0

steveisok · 2023-06-19T21:10:38Z

These are the artifacts we need to replace when UseNativeAOTRuntime is true. That should give us enough to aot on helix with.

Replace this with the native aot in tree runtime pack. The build still produces one, but w/ only the system.native bits

runtime/src/libraries/sendtohelix-mobile.targets

Line 67 in e834d42

    
           <HelixCorrelationPayload Include="$(MicrosoftNetCoreAppRuntimePackDir)"    Destination="build/microsoft.netcore.app.runtime.$(TargetOS)-$(TargetArchitecture.ToLower())" />

Replace MonoAotCrossDir with the ILC equivalent. This item probably should be renamed to something more generic.

runtime/src/libraries/sendtohelix-mobile.targets

Line 70 in e834d42

    
           <HelixCorrelationPayload Include="$(MonoAotCrossDir)"                      Destination="build/cross" />

Replace MonoTargetsTaskDir with something that points to the NativeAOT BuildIntegration targets

runtime/src/libraries/sendtohelix-mobile.targets

Line 71 in e834d42

    
           <HelixCorrelationPayload Include="$(MonoTargetsTasksDir)"                  Destination="build/MonoTargetsTasks" />

kotlarmilos · 2023-06-20T11:23:25Z

Replace this with the native aot in tree runtime pack. The build still produces one, but w/ only the system.native bits

~~It should be the same path, right?~~

2. Replace MonoAotCrossDir with the ILC equivalent. This item probably should be renamed to something more generic.

~~How does the AppleBuild.targets find the compiler at build/cross on Helix? I want to make the name more generic, but not sure if it is predefined.~~

Update: When adding new artifacts to the Helix, update the HelixCorrelationPayload property in sendtohelix-mobile.targets with the mappings from the AppleBuild.LocalBuild.props.

src/mono/msbuild/common/LibraryBuilder.targets

kotlarmilos · 2023-08-04T13:36:32Z

/azp run runtime-ioslike

azure-pipelines · 2023-08-04T13:36:41Z

Azure Pipelines successfully started running 1 pipeline(s).

ivanpovazan

Apart from the two nits, this looks good.
However, since it is a quite large PR I would suggest someone else also to take a look before merging.

src/tests/FunctionalTests/iOS/Simulator/AOT-LLVM/iOS.Simulator.Aot-Llvm.Test.csproj

src/mono/sample/iOS-NativeAOT/Program.csproj

kotlarmilos · 2023-08-11T08:21:48Z

/azp run runtime-ioslike

azure-pipelines · 2023-08-11T08:21:57Z

Azure Pipelines successfully started running 1 pipeline(s).

kotlarmilos · 2023-08-11T08:28:04Z

/azp run runtime-ioslike

azure-pipelines · 2023-08-11T08:28:14Z

Azure Pipelines successfully started running 1 pipeline(s).

steveisok

Looks good! Thanks for sticking with it.

src/libraries/sendtohelix-mobile.targets

steveisok · 2023-08-10T16:10:30Z

eng/testing/tests.ioslike.targets

-    <_AOTBuildCommand>$(_AOTBuildCommand) /p:XHARNESS_EXECUTION_DIR=&quot;$XHARNESS_EXECUTION_DIR&quot; /p:RunAOTCompilation=$(RunAOTCompilation) /p:TargetOS=$(TargetOS) /p:TargetArchitecture=$(TargetArchitecture) /p:MonoForceInterpreter=$(MonoForceInterpreter) /p:MonoEnableLLVM=true /p:DevTeamProvisioning=$(DevTeamProvisioning) /p:UsePortableRuntimePack=true /p:Configuration=$(Configuration)</_AOTBuildCommand>
+    <_AOTBuildCommand>$(_AOTBuildCommand) /p:XHARNESS_EXECUTION_DIR=&quot;$XHARNESS_EXECUTION_DIR&quot; /p:RunAOTCompilation=$(RunAOTCompilation) /p:UseNativeAOTRuntime=$(UseNativeAOTRuntime) /p:TargetOS=$(TargetOS) /p:TargetArchitecture=$(TargetArchitecture) /p:MonoForceInterpreter=$(MonoForceInterpreter) /p:DevTeamProvisioning=$(DevTeamProvisioning) /p:UsePortableRuntimePack=true /p:Configuration=$(Configuration)</_AOTBuildCommand>


Looking at the other functional tests project files, most of them have MonoForceInterpreter set, but some have TreatAsLocalProperty="MonoForceInterpreter" some don't, so I believe the TreatAsLocalProperty was added as a workaround to prevent overwriting what a project specifies through the command line - like building on Helix.

To conclude, I feel like this needs to be refactored (maybe not in this PR), with either removing any notion of MonoForceInterpreter from the functional test's project file (and test the change), or leave it how it was to prevent any unwanted regression.

@steveisok what do you think?

I would suggest keeping it and slowly unwinding in a follow up.

steveisok · 2023-08-11T16:56:08Z

eng/pipelines/runtime.yml

+      # iOS/tvOS devices
+      # Build the whole product using Native AOT and run libraries tests
+      #
+      - template: /eng/pipelines/common/platform-matrix.yml


~~We don't have the capacity to run these on every PR. Even when we kick off runtime-ioslike manually, there may be a capacity issue running both mono and nativeaot together.~~

Nevermind, my concern was all libraries tests (when we get to it) and I see it's running smoke tests only.

agocke · 2023-09-07T18:31:20Z

Hey, I just noticed this is running libraries tests using Native AOT in the runtime pipeline. We've previously found that this was too expensive to run on every PR. Can we move these legs out of the runtime pipeline and into runtime-extra-platforms or similar?

filipnavara · 2023-09-07T18:36:21Z

This concern was already previously voiced in a comment. The conclusion was that it runs only small subset of the tests (smoke tests). Are you seeing a bigger subset now? If it becomes an issue, is it possible to trigger it on specific path changes automatically?

agocke · 2023-09-07T18:37:05Z

Ah, nevermind, I see now this is only smoke tests. OK, lemme look in more detail at average run times. If it's not too expensive, we can continue running smoke tests.

kotlarmilos added 2 commits June 19, 2023 16:40

Initial commit

74481af

Add Native AOT library pipeline for iOS and disable other tests

ca86967

kotlarmilos added NO-MERGE The PR is not ready for merge yet (see discussion for detailed reasons) NO-REVIEW Experimental/testing PR, do NOT review it area-Infrastructure-mono os-ios Apple iOS labels Jun 19, 2023

kotlarmilos added this to the 8.0.0 milestone Jun 19, 2023

kotlarmilos self-assigned this Jun 19, 2023

kotlarmilos added 4 commits June 19, 2023 18:41

Disable other tests

98522fd

Disable other tests

6ade906

Test build configuration

0e7a920

Test build configuration

1aca721

steveisok self-requested a review June 19, 2023 17:45

Test build configuration

774ed6b

kotlarmilos mentioned this pull request Jun 20, 2023

[nativeaot][tests] Add smoke unit runtime tests for Native AOT on iOS platforms #87260

Merged

kotlarmilos added 2 commits June 20, 2023 17:03

Add properties to the proxy AOT project

034fff1

Merge branch 'main' into feature/ios-functional-tests

a12779b

build-analysis bot mentioned this pull request Jun 20, 2023

Failed USB connection via port 54050, error 61, in tvOS arm64 Release AllSubsets_Mono #82637

Open

kotlarmilos added 4 commits June 20, 2023 19:28

Add AppName to the proxy props

abfe7c1

Remove AppName from the proxy props

09c6097

Merge branch 'main' into feature/ios-functional-tests

063995d

Add ILC payload to Helix

3693155

steveisok reviewed Jun 21, 2023

View reviewed changes

src/mono/msbuild/common/LibraryBuilder.targets Outdated Show resolved Hide resolved

kotlarmilos mentioned this pull request Jun 22, 2023

Enable dotnet/runtime runtime and functional tests for iOS platforms to run with NativeAOT #85294

Closed

3 tasks

Update the Helix targets to support Native AOT tests

5344255

kotlarmilos mentioned this pull request Jun 23, 2023

Resolve path before initializing objwriter #87962

Closed

Fix ILLink and ILC paths on Helix

7c426a5

build-analysis bot mentioned this pull request Aug 4, 2023

[mono][ios] System.Formats.Tar.Tests are failing with System.ArgumentException #88049

Closed

Add globalization static libraries for Native AOT builds

26955e0

build-analysis bot mentioned this pull request Aug 4, 2023

Test failure readytorun/GenericCycleDetection/Depth3Test/Depth3Test.sh #88586

Closed

kotlarmilos added 2 commits August 8, 2023 13:56

Add MonoForceInterpreter property to the functional tests

7637762

Merge branch 'main' into feature/ios-functional-tests

e152e73

kotlarmilos requested a review from ivanpovazan August 8, 2023 12:12

Don't use runtime components for Native AOT builds

4af46f8

ivanpovazan approved these changes Aug 9, 2023

View reviewed changes

src/tests/FunctionalTests/iOS/Simulator/AOT-LLVM/iOS.Simulator.Aot-Llvm.Test.csproj Outdated Show resolved Hide resolved

src/mono/sample/iOS-NativeAOT/Program.csproj Outdated Show resolved Hide resolved

build-analysis bot mentioned this pull request Aug 9, 2023

[coreclr] Native AOT runtime and functional tests on apple mobile report undefined symbols for architecture arm64 #90218

Closed

kotlarmilos added 2 commits August 9, 2023 13:29

Resolve PR comments

8a59bca

Merge branch 'dotnet:main' into feature/ios-functional-tests

7dd5b27

Use item group for native dependencies

8663185

build-analysis bot mentioned this pull request Aug 11, 2023

Tracking issue for "WORKLOAD TIMED OUT" #90309

Closed

Add conditions to the CI jobs

2238ba2

build-analysis bot mentioned this pull request Aug 11, 2023

Crash in Microsoft.Extensions.Logging.Generators.Roslyn4.0.Tests.WorkItemExecution #90019

Open

steveisok approved these changes Aug 11, 2023

View reviewed changes

Update formatting

cc5313a

kotlarmilos merged commit 55828b9 into dotnet:main Aug 11, 2023

kotlarmilos mentioned this pull request Aug 15, 2023

[release/8.0-rc1] [mono][ios] Revert LLVM mode for apple mobile tests #90624

Merged

ghost locked as resolved and limited conversation to collaborators Oct 7, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ios][tests] Run functional tests with Mono and Native AOT on Helix #87773

[ios][tests] Run functional tests with Mono and Native AOT on Helix #87773

kotlarmilos commented Jun 19, 2023 •

edited

Loading

ghost commented Jun 19, 2023

Description

steveisok commented Jun 19, 2023

kotlarmilos commented Jun 20, 2023 •

edited

Loading

kotlarmilos commented Aug 4, 2023

azure-pipelines bot commented Aug 4, 2023

ivanpovazan left a comment

kotlarmilos commented Aug 11, 2023

azure-pipelines bot commented Aug 11, 2023

kotlarmilos commented Aug 11, 2023

azure-pipelines bot commented Aug 11, 2023

steveisok left a comment

steveisok Aug 10, 2023

steveisok Aug 11, 2023

steveisok Aug 11, 2023

agocke commented Sep 7, 2023

filipnavara commented Sep 7, 2023

agocke commented Sep 7, 2023

		<_AOTBuildCommand>$(_AOTBuildCommand) /p:XHARNESS_EXECUTION_DIR="$XHARNESS_EXECUTION_DIR" /p:RunAOTCompilation=$(RunAOTCompilation) /p:TargetOS=$(TargetOS) /p:TargetArchitecture=$(TargetArchitecture) /p:MonoForceInterpreter=$(MonoForceInterpreter) /p:MonoEnableLLVM=true /p:DevTeamProvisioning=$(DevTeamProvisioning) /p:UsePortableRuntimePack=true /p:Configuration=$(Configuration)</_AOTBuildCommand>
		<_AOTBuildCommand>$(_AOTBuildCommand) /p:XHARNESS_EXECUTION_DIR="$XHARNESS_EXECUTION_DIR" /p:RunAOTCompilation=$(RunAOTCompilation) /p:UseNativeAOTRuntime=$(UseNativeAOTRuntime) /p:TargetOS=$(TargetOS) /p:TargetArchitecture=$(TargetArchitecture) /p:MonoForceInterpreter=$(MonoForceInterpreter) /p:DevTeamProvisioning=$(DevTeamProvisioning) /p:UsePortableRuntimePack=true /p:Configuration=$(Configuration)</_AOTBuildCommand>

[ios][tests] Run functional tests with Mono and Native AOT on Helix #87773

[ios][tests] Run functional tests with Mono and Native AOT on Helix #87773

Conversation

kotlarmilos commented Jun 19, 2023 • edited Loading

Description

ghost commented Jun 19, 2023

Description

steveisok commented Jun 19, 2023

kotlarmilos commented Jun 20, 2023 • edited Loading

kotlarmilos commented Aug 4, 2023

azure-pipelines bot commented Aug 4, 2023

ivanpovazan left a comment

Choose a reason for hiding this comment

kotlarmilos commented Aug 11, 2023

azure-pipelines bot commented Aug 11, 2023

kotlarmilos commented Aug 11, 2023

azure-pipelines bot commented Aug 11, 2023

steveisok left a comment

Choose a reason for hiding this comment

steveisok Aug 10, 2023

Choose a reason for hiding this comment

steveisok Aug 11, 2023

Choose a reason for hiding this comment

steveisok Aug 11, 2023

Choose a reason for hiding this comment

agocke commented Sep 7, 2023

filipnavara commented Sep 7, 2023

agocke commented Sep 7, 2023

kotlarmilos commented Jun 19, 2023 •

edited

Loading

kotlarmilos commented Jun 20, 2023 •

edited

Loading