Rename coreclr ci.yml to outerloop and change triggers to be scheduled

[safern]

It is confusing that it's name is ci.yml when it is the definition for outerloop builds.

Currently CI Rolling builds for innerloop tests will happen as part of runtime pipeline which contains the full matrix added yesterday.

Given this discussion: #98 (comment) moving this outerloop build trigger to be on a daily schedule rather than on commit triggers.

Removed crossgen-comparison job as it happens as part of the runtime pipeline.

cc: @BruceForstall @jkotas @dotnet/runtime-infrastructure

[BruceForstall]

One thing I'm worried about is outerloop runs Pri-1 coreclr tests, whereas innerloop runs Pri-0. If we make this a daily schedule, we will end up running Pri-1 much less often and thus will have batches of commits in which a regression can occur.

If we're worried about resources, can we make this rolling/batched instead of daily scheduled? (i.e., run continuously if there has been a change, but batches all commits since the last run)

[safern]

If we're worried about resources, can we make this rolling/batched instead of daily scheduled? (i.e., run continuously if there has been a change, but batches all commits since the last run)

That's another good alternative... we can try that and see how resource consuming is. I'll wait for others to add their opinion as well.

[safern]

Should we instead, add the Pri1 test runs then into runtime pipeline when the build is a Rolling CI build instead of having 2 separate builds? That will avoid duplication of building libraries twice and libraries tests, etc.

[safern]

Should we instead, add the Pri1 test runs then into runtime pipeline when the build is a Rolling CI build instead of having 2 separate builds? That will avoid duplication of building libraries twice and libraries tests, etc.

Ping on this question?

[jaredpar]

Should we instead, add the Pri1 test runs then into runtime pipeline when the build is a Rolling CI build instead of having 2 separate builds? That will avoid duplication of building libraries twice and libraries tests, etc.

What is the relative stability of these tests? My biggest concern with including them in the rolling build for runtime is the impact it will have on quality tracking. Essentially if the Pri 1 tests are flaky then that will cause CI Council to believe our PR builds are less stable than they actually are.

[jkoritzinsky]

They're generally very stable. The main reason a test becomes Pri 1 in the CoreCLR testing infra is because it never fails and usually when a Pri1 test fails something else is also broken. The Pri0/Pri1 system is meant to cut down PR test run time, not to push flaky tests to run outside of PRs only. Although, it is possible (but uncommon) for someone to break a Pri1 test without breaking any Pri0 tests and merge in a test break.

[BruceForstall]

Should we instead, add the Pri1 test runs then into runtime pipeline when the build is a Rolling CI build instead of having 2 separate builds?

I believe this is how coreclr worked back in the Jenkins (and AzDO coreclr?) days. Namely, a single "pipeline" that did Pri-0 for PR testing, and Pri-1 when run in "post-checkin, CI mode". Is that what you're proposing?

What is the relative stability of these tests?

+1 to what @jkoritzinsky said. The distinction was made intending to create an interesting variety of code that would run quickly (to meet some maximum PR time/resource usage goal), whereas Pri-1 was everything else. Something like 2000 versus 11000 tests.

[safern]

Is that what you're proposing?

Yes, that's exactly what I'm proposing. The reason for that is because we would avoid building libraries and coreclr twice every checkin and use less agents.

[safern]

Closing since we don't want to have a different matrix in between PRs and CI.