Alter amount of tests that run per PR for mono mobile configurations #60727

steveisok · 2021-10-21T18:41:45Z

In an effort to better utilize CI resources, this change will only run the System.Runtime library test per PR for Android and Wasm configurations. The full libraries and runtime tests will move to only run on the post-PR validation pipeline.

The summary of changes are:

System.Runtime tests will only trigger per PR for Android x86/x64 and Wasm AOT / EAT
All mono runtime builds will only run on the post PR validation build (rolling build)
WasmBuildTests will only run on the post PR validation build (rolling build)
Wasm on Windows will only build

Note: iOS/tvOS/MacCatalyst are excluded until #59503 lands

In an effort to better utilize CI resources, this change will only run the System.Runtime library test per PR for Android and Wasm configurations. The full libraries and runtime tests will move to only run on the post-PR validation pipeline. Note: iOS/tvOS/MacCatalyst are excluded until dotnet#59503 lands

ghost · 2021-10-21T18:41:53Z

Tagging subscribers to this area: @directhex
See info in area-owners.md if you want to be subscribed.

Issue Details

In an effort to better utilize CI resources, this change will only run the System.Runtime library test per PR for Android and Wasm configurations. The full libraries and runtime tests will move to only run on the post-PR validation pipeline.

Note: iOS/tvOS/MacCatalyst are excluded until #59503 lands

Author:	steveisok
Assignees:	-
Labels:	`area-Infrastructure-mono`
Milestone:	-

lewing · 2021-10-21T18:52:22Z

This will cause a lot of regressions to be discovered post PR.

radical · 2021-10-21T18:54:06Z

eng/pipelines/runtime.yml

@@ -323,7 +323,9 @@ jobs:
    buildConfig: Release
    runtimeFlavor: mono
    platforms:
-    - Browser_wasm
+    # BuildWasmApps should only happen on the rolling build. No need to duplicate the build on PR's
+    ${{ if eq(variables['isFullMatrix'], true) }}:


Maybe instead of this, we can have tighter constraints on which changes trigger running Wasm.Build.Tests? Eg. src/mono, src/tests/BuildWasmApps, src/tasks, etc? though it will get kinda detailed, because we would want to include the various nugets consumed for workloads.

Just fyi, this job does not run the library tests as we pass TestAssemblies=false, so it builds, and then runs only the Wasm.Build.Tests .
cc @lewing

I'd rather drop Scenario='normal' from Browser_wasm and keep EAT with Scenario='normal', I think this is actually a little beyond guidance and it would catch a lot of test configuation changes that would otherwise cause us problems.

radical · 2021-10-21T18:55:51Z

eng/pipelines/runtime.yml

@@ -415,7 +417,7 @@ jobs:
    jobParameters:
      testGroup: innerloop
      nameSuffix: AllSubsets_Mono_AOT
-      buildArgs: -s mono+libs+host+packs+libs.tests -c $(_BuildConfig) /p:ArchiveTests=true /p:EnableAggressiveTrimming=true /p:BuildAOTTestsOnHelix=true /p:RunAOTCompilation=true
+      buildArgs: -s mono+libs+host+packs+libs.tests -c $(_BuildConfig) $(_runSmokeTestsOnlyArg) /p:ArchiveTests=true /p:EnableAggressiveTrimming=true /p:BuildAOTTestsOnHelix=true /p:RunAOTCompilation=true


If we don't want to run EAT/AOT tests for every PR, then it would be useful to have some way to opt-in, or trigger these builds on a PR.

Yeah, we need a way to trigger aot manually for sure, it would be nice to auto trigger it with any src/mono/mono change as well.

fanyang-mono · 2021-10-21T20:54:21Z

I feel the same as Larry. Are we experiencing any machine shortage or PR running for too long?

fanyang-mono · 2021-10-21T20:57:36Z

I feel the same as Larry. Are we experiencing any machine shortage or PR running for too long?

I just thought about it again. Maybe it is okay, since you are only disabling runs on Android. (I am not sure how much it would impact WASM, since WASM is actively experiencing big changes.) The catch is that we will need to keep close eye on the rolling builds.

steveisok · 2021-10-22T15:55:55Z

There has been some offline discussion where I think it makes sense to pull this back for the time being. I plan on resubmitting soon.

danmoseley · 2021-10-27T04:34:19Z

If we have the ability to select particular libraries to test pre commit, we could fault in more as and when we see regressions leak.

eng/pipelines/libraries/base-job.yml

src/libraries/tests.proj

lewing

lewing · 2021-10-29T18:17:48Z

@steveisok should we have an issue to track the future adjustments/refinements?

lewing · 2021-10-30T00:09:26Z

🤷

dotnet-issue-labeler bot added the area-Infrastructure-mono label Oct 21, 2021

steveisok requested review from akoeplinger, a team, lewing, radical, SamMonoRT, fanyang-mono and imhameed October 21, 2021 18:44

radical reviewed Oct 21, 2021

View reviewed changes

Missing tab on condition

5e47dda

fanyang-mono approved these changes Oct 22, 2021

View reviewed changes

steveisok closed this Oct 22, 2021

steveisok reopened this Oct 27, 2021

Steve Pfister added 5 commits October 26, 2021 23:43

Merge remote-tracking branch 'upstream/main' into full-tests-on-rolling

45516d9

See if the yml is better

e37e411

Variable in wrong place

ea4afbf

yml :-(

7b2652d

silly checks

2649f5d

safern reviewed Oct 27, 2021

View reviewed changes

eng/pipelines/libraries/base-job.yml Outdated Show resolved Hide resolved

Steve Pfister added 6 commits October 27, 2021 19:07

yml syntax

67e444a

Rollback runtime.yml and see why it's not triggering

3415612

Add smoke test args back in

7c3c210

Add condition back in

218a770

Merge remote-tracking branch 'upstream/main' into full-tests-on-rolling

5b203e1

Even more conditions

71a7521

runfoapp bot mentioned this pull request Oct 28, 2021

System.Net.Sockets.Tests.SendPacketsAsync.SendPacketsElement_FileZeroCount_Success sometimes fails #60017

Closed

Remove echo middleware for wasm when smoke tests are run

01d5578

steveisok force-pushed the full-tests-on-rolling branch 2 times, most recently from dd96e90 to f601e38 Compare October 29, 2021 02:48

Merge remote-tracking branch 'upstream/main' into full-tests-on-rolling

d01b7b5

steveisok force-pushed the full-tests-on-rolling branch from f601e38 to d01b7b5 Compare October 29, 2021 04:57

akoeplinger approved these changes Oct 29, 2021

View reviewed changes

src/libraries/tests.proj Outdated Show resolved Hide resolved

Exclude more runtime test PR runs. PR feedback

3956da7

This was referenced Oct 29, 2021

1ES Pools often run out of space #60610

Closed

Linker tests failing with no space left on device on Linux x64 #60927

Closed

akoeplinger approved these changes Oct 29, 2021

View reviewed changes

lewing approved these changes Oct 29, 2021

View reviewed changes

radical approved these changes Oct 29, 2021

View reviewed changes

steveisok mentioned this pull request Oct 29, 2021

Mono CI Utilization Improvements #61017

Closed

6 tasks

steveisok merged commit 0cff4a2 into dotnet:main Oct 29, 2021

ghost locked as resolved and limited conversation to collaborators Nov 29, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Alter amount of tests that run per PR for mono mobile configurations #60727

Alter amount of tests that run per PR for mono mobile configurations #60727

steveisok commented Oct 21, 2021 •

edited

Loading

ghost commented Oct 21, 2021

lewing commented Oct 21, 2021

radical Oct 21, 2021

lewing Oct 21, 2021 •

edited

Loading

radical Oct 21, 2021

lewing Oct 21, 2021

fanyang-mono commented Oct 21, 2021

fanyang-mono commented Oct 21, 2021 •

edited

Loading

steveisok commented Oct 22, 2021

danmoseley commented Oct 27, 2021

lewing left a comment

lewing commented Oct 29, 2021

lewing commented Oct 30, 2021

Alter amount of tests that run per PR for mono mobile configurations #60727

Alter amount of tests that run per PR for mono mobile configurations #60727

Conversation

steveisok commented Oct 21, 2021 • edited Loading

ghost commented Oct 21, 2021

lewing commented Oct 21, 2021

radical Oct 21, 2021

Choose a reason for hiding this comment

lewing Oct 21, 2021 • edited Loading

Choose a reason for hiding this comment

radical Oct 21, 2021

Choose a reason for hiding this comment

lewing Oct 21, 2021

Choose a reason for hiding this comment

fanyang-mono commented Oct 21, 2021

fanyang-mono commented Oct 21, 2021 • edited Loading

steveisok commented Oct 22, 2021

danmoseley commented Oct 27, 2021

lewing left a comment

Choose a reason for hiding this comment

lewing commented Oct 29, 2021

lewing commented Oct 30, 2021

steveisok commented Oct 21, 2021 •

edited

Loading

lewing Oct 21, 2021 •

edited

Loading

fanyang-mono commented Oct 21, 2021 •

edited

Loading