SuperPMI: add JIT memory diff measurements in per-PR pipeline runs

[BruceForstall]

Now we have (SPMI-based) asmdiff and tpdiff, each running in CI testing on all platforms for every JIT pull request. These are also easy to run locally.

Proposal: Add memdiff to these pre-PR CI test runs, and also make it possible to run locally. Memdiff would measure JIT memory usage during compilation and compare it against the memory used by a baseline JIT compiler. As with other "diff" types, it would use SuperPMI replay.

We don't need to be too picky about the JIT using more memory -- certainly not as picky as how we pay attention to TP diff regressions -- but memdiff could catch egregious changes, for example a 25% increase in memory usage by the JIT with the implementation of a new data structure/algorithm.

Implementation

Memory diffs need to be run using Release compilers (to avoid counting various memory we allocate during DEBUG builds, such as during various checking phases) that have memory tracking enabled. Release build memory tracking is not currently enabled by default. Either a very low-cost method to track memory needs to be added to default Release builds, or a special Release build with low-cost collection should be built. This might just be the existing MEASURE_MEM_ALLOC code, or be based on that.

To implement, we need the jitrollingbuild to produce a baseline Release build that can be used for memory allocation tracking, as well as have the CI produce such a Release JIT for the PR itself.

Ideally, the Release build we create can also be used for tpdiff, otherwise we would end up with yet another Release flavor, which would be costly in time and AzDO storage, and probably lead to confusion about which is which.

It's possible we could just use Checked builds for memory tracking, and just ignore the CMK_DebugOnly memory pool. It's not clear that would give us an accurate picture of Release compiler memory usage (which is what actually matters). Possibly an experiment could tell us if that's useful enough. If so, we wouldn't need any new builds.

category:eng-sys
theme:super-pmi
skill-level:beginner
cost:medium
impact:small

[ghost]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Now we have (SPMI-based) asmdiff and tpdiff, each running in CI testing on all platforms for every JIT pull request. These are also easy to run locally.

Proposal: Add memdiff to these pre-PR CI test runs, and also make it possible to run locally. Memdiff would measure JIT memory usage during compilation and compare it against the memory used by a baseline JIT compiler. As with other "diff" types, it would use SuperPMI replay.

We don't need to be too picky about the JIT using more memory -- certainly not as picky as how we pay attention to TP diff regressions -- but memdiff could catch egregious changes, for example a 25% increase in memory usage by the JIT with the implementation of a new data structure/algorithm.

Implementation

Memory diffs need to be run using Release compilers (to avoid counting various memory we allocate during DEBUG builds, such as during various checking phases) that have memory tracking enabled. Release build memory tracking is not currently enabled by default. Either a very low-cost method to track memory needs to be added to default Release builds, or a special Release build with low-cost collection should be built. This might just be the existing MEASURE_MEM_ALLOC code, or be based on that.

To implement, we need the jitrollingbuild to produce a baseline Release build that can be used for memory allocation tracking, as well as have the CI produce such a Release JIT for the PR itself.

Ideally, the Release build we create can also be used for tpdiff, otherwise we would end up with yet another Release flavor, which would be costly in time and AzDO storage, and probably lead to confusion about which is which.

It's possible we could just use Checked builds for memory tracking, and just ignore the CMK_DebugOnly memory pool. It's not clear that would give us an accurate picture of Release compiler memory usage (which is what actually matters). Possibly an experiment could tell us if that's useful enough. If so, we wouldn't need any new builds.

category:eng-sys
theme:super-pmi
skill-level:beginner
cost:medium
impact:small

Author:	BruceForstall
Assignees:	-
Labels:	area-CodeGen-coreclr
Milestone:	Future