Merge the remaining groups of runtime tests #71732
Comments
ghost
added
the
|
Tagging subscribers to this area: @hoyosjs Issue DetailsRelated to: #54512 As of 7/6/2022, about 1/3 of the total runtime test set has been converted to using merged wrappers and running most tests in-proc to improve Helix perf. This issue is about fanning out the effort to the various runtime feature teams to follow up by converting the remaining runtime tests. According to the analytic tool ILTransform I created for the purpose of the conversion, the biggest remaining buckets of non-converted tests are in the following subtrees: () JIT/Regression (1312 tests) By merging these subtrees we should be able to reach about 70% conversion rate after which we can focus on the remaining smaller test groups. Thanks Tomas
|
|
|
Can we document how to do this work here? If I know nothing about this and I want to volunteer to pick up one of the categories, how do I go about it? |
Count me in as well. |
|
Basic conversion instructions Typically, the conversion is easiest for csproj projects that are unconditional - they don't depend on the targeting OS, architecture and / or execution modes like GC stress. For ilproj test projects and for conditional csproj test projects the situation is slightly more complicated. I have created an internal app to assist with a semi-automated conversion. You can find it in the private branch https://github.com/trylek/runtime/tree/ILTransform After you clone this branch (it's easier to clone it into a different tree than the one we're trying to convert), the new project ILTransform will appear under src/ILTransform. I'm typically opening and running the project from the Visual Studio. It has several execution modes but the one most interesting for the actual conversion is as follows: <path to the built ILTransform app> -f <subfolder of src/tests to process> This syntax causes the app to carry out the standard conversion steps for all csproj and ilproj projects in the specified subtree. In particular, it renames Main methods in csproj projects based on the project name and attaches the [Fact] attributes to them. For ilproj projects, it suffices to attach the [Fact] attribute, the entrypoint method remains in place so that the test can be run individually (this is a fundamental requirement for some of the low-level tests). In the root folder of the subtree being converted, it's also necessary to modify the Directory.Build.props by including the following property group: <PropertyGroup>
<BuildAsStandalone Condition="'$(BuildAsStandalone)' == ''">false</BuildAsStandalone>
<AssemblyName Condition="'$(BuildAsStandalone)' != 'true'">$(MSBuildProjectName.Replace('_il_d', '').Replace('_il_r', ''))</AssemblyName>
</PropertyGroup>
as seen in the file The BuildAsStandalone property overrides the still default policy of building the tests individually, the AssemblyName caters for subtle shenanigans involving naming of ilproj tests and assembly names in them. Furthermore you need to add a new project (possibly multiple projects) to represent the merged test wrappers in your tree. Here is an example from the JIT/Methodical subtree where it's slightly more involved as due to the large number of tests we use seven different wrappers (Methodical_d1, Methodical_d2, Methodical_do, Methodical_r1, Methodical_r2, Methodical_ro, Methodical_others) to partition the large number of tests (~2000): https://github.com/dotnet/runtime/blob/main/src/tests/JIT/Methodical/Methodical_d1.csproj According to our experience from merging the first 1/3 of the test tree the sweet spot for test merging seems to be several hundred tests per merged wrapper. This may vary with complexity of the individual tests and might require a bit of local experimentation. If everything works as expected, these instructions should be sufficient to convert a particular subtree of the test tree. I'll add more instructions to cover more complicated cases when the described workflow fails. |
Remaining usability issues are listed in #71732 Conversion steps: - static-none uses args - add comment to RequiresProcessIsolation - remove unused arg in Runtime_80731.il - Rewrite 4 IL tests to use newarr instead of args - [cs-main] remove unused args in C# tests - Update 9 tests with Main(args) - Workaround: add RequiresProcessIsolation to LdfldaHack - Move [Fact] to correct location in b12263 - In b06020, wrap $ class so that the C# wrapper can call it - [ILTransform -public] - [ILTransform -sr] - Manually rename b598031/test and test2. These were missed by the previous ILTransform -n run. They no longer clash because run renamed all of their conflicts, but this renames them for consistency. - [ILTransform -prociso] Rerun for new tests - Wrappers - [ILTransform -ilfact] - Manually do missed -collapse-main-sig and fix placement of [Fact] - Fix b89946 that referred to Main - Fix Runtime_59444 reflection on Test* * Fix TestSummary.cs to handle invalid XML characters * [ILTransform -a] * Manual fixes * Update test groupings and add extern aliases * Finish merge - incorporate changes to csproj/Dir.B.props - fix xunit.analyzers errors * [ILTransform -public] * [ILTransform -prociso] * [ILTransform -ilfact] (undo GitHub_26491) * Fix xUnit1003 in Runtime_83003 * Update new test Runtime_83941 for merged groups * 59444 - Changing methods to internal broke reflection calls. Use this opportunity to remove the reflection and simply mark the callees as [Fact]s. - 64883/76273 - Suppress the xunit warning so that the code can continue using the normal (public-only) reflection search. - Fix TestSummary.cs to handle invalid XML characters - Add RequiresProcessIsolation for existing issues.target entry (AOT compiler fails) - undo JIT/opt/And/Regressions/Regression1.csproj (possibly from manual merge from main)
Related to: #54512
As of 7/6/2022, about 1/3 of the total runtime test set has been converted to using merged wrappers and running most tests in-proc to improve Helix perf. This issue is about fanning out the effort to the various runtime feature teams to follow up by converting the remaining runtime tests.
Original conversions:
According to the analytic tool ILTransform I created for the purpose of the conversion, the biggest remaining buckets of non-converted tests are in the following subtrees:
By merging these subtrees we should be able to reach about 70% conversion rate after which we can focus on the remaining smaller test groups.
Leading JIT test directories (updated 03/08/2023):
Merged test groups impact day-to-day work, and directories such as JIT\Regression are actively modified. We also need to address usability issues.
Highest priority - can cause tests to be skipped or impact the team's ability to disable failing tests:
Mainmethods because they will fail ifBuildAsStandaloneis enabled when it tries to build another entry point.High priority - impact day-to-day work or debugging issues with test merging:
Lower priority - these can (somewhat) be worked around but most aren't sustainable (e.g., poor documentation means all testing issues will come to Mark/Tomas/Jeremy)
test --version)? [ANSWER: sort of - You can keep aMainmethod, but you need to then setReferenceXUnitWrapperGeneratortofalseor BuildAsStandalone will break.Mainwill need a[Fact]or[Theory], and arguments (if any) will need to match across the [xxxData] and csproj property for BuildAsStandalone=true/false to be consistent.]This isn't a specific work item but a change we all need to make: We need to aggressively disable tests that take down the runtime (and hide other tests). We should consider having JIT assertions throw managed exceptions, but these can be caught. Ideally we want monitoring (either within the wrapper - with the cost of extra code - or external to it).
Partitioning of subtrees under JIT/Regression
This was originally intended for tracking conversion progress and is useful for understand the set of tests there. However, the entire batch was converted at once, so it won't be used for tracking progress.
Thanks
Tomas
The text was updated successfully, but these errors were encountered: