[Bug]: CustomAnalyzerTest tests fail on macOS 13 and 14

[jrdodds]

Issue Description

Recently PR #10076 added a new CustomAnalyzerTest [Theory] with two sets of data. The PR build checks passed running on a macos-12 image.

However, when I build locally and run tests on macOS 14 on ARM and macOS 13 on Intel, the tests fail.

Steps to Reproduce

1. Run ./build.sh.
2. Run ./build.sh --test

Expected Behavior

All tests should pass.

Actual Behavior

CustomAnalyzerTest tests fail.

Failed tests
Microsoft.Build.BuildCheck.UnitTests.EndToEndTests.CustomAnalyzerTest(customAnalyzerNames: [\"CustomAnalyzer\", \"CustomAnalyzer2\"], analysisCandidate: \"AnalysisCandidateWithMultipleAnalyzersInjected\", expectedRegisteredRules: [\"CustomRule1\", \"CustomRule2\", \"CustomRule3\"])
Shouldly.ShouldAssertException : success\r\n    should be\r\nTrue\r\n    but was\r\nFalse
   at Microsoft.Build.BuildCheck.UnitTests.EndToEndTests.BuildAnalyzerRules(TestEnvironment env, String[] customAnalyzerNames) in /Users/jrdodds/Projects/dotnet/msbuild/src/BuildCheck.UnitTests/EndToEndTests.cs:line 174
   at Microsoft.Build.BuildCheck.UnitTests.EndToEndTests.CustomAnalyzerTest(String[] customAnalyzerNames, String analysisCandidate, String[] expectedRegisteredRules) in /Users/jrdodds/Projects/dotnet/msbuild/src/BuildCheck.UnitTests/EndToEndTests.cs:line 145
   at InvokeStub_EndToEndTests.CustomAnalyzerTest(Object, Span`1)
   at System.Reflection.MethodBaseInvoker.InvokeWithFewArgs(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
Microsoft.Build.BuildCheck.UnitTests.EndToEndTests.CustomAnalyzerTest(customAnalyzerNames: [\"CustomAnalyzer\"], analysisCandidate: \"AnalysisCandidate\", expectedRegisteredRules: [\"CustomRule1\", \"CustomRule2\"])
Shouldly.ShouldAssertException : success\r\n    should be\r\nTrue\r\n    but was\r\nFalse
   at Microsoft.Build.BuildCheck.UnitTests.EndToEndTests.BuildAnalyzerRules(TestEnvironment env, String[] customAnalyzerNames) in /Users/jrdodds/Projects/dotnet/msbuild/src/BuildCheck.UnitTests/EndToEndTests.cs:line 174
   at Microsoft.Build.BuildCheck.UnitTests.EndToEndTests.CustomAnalyzerTest(String[] customAnalyzerNames, String analysisCandidate, String[] expectedRegisteredRules) in /Users/jrdodds/Projects/dotnet/msbuild/src/BuildCheck.UnitTests/EndToEndTests.cs:line 145
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodBaseInvoker.InvokeDirectByRefWithFewArgs(Object obj, Span`1 copyOfArgs, BindingFlags invokeAttr)

The Microsoft.Build.BuildCheck.UnitTests_net8.0_x64.html and .Microsoft.Build.BuildCheck.UnitTests_net8.0_x64xml log files are attached in a .zip.

Microsoft.Build.BuildCheck.UnitTests_net8.0_x64.zip

Analysis

The tests fail in BuildAnalyzerRules() after calling RunnerUtilities.ExecBootstrapedMSBuild():

msbuild/src/BuildCheck.UnitTests/EndToEndTests.cs

Lines 164 to 174 in 393c2fe

	private IList<string> BuildAnalyzerRules(TestEnvironment env, string[] customAnalyzerNames)
	{
	var candidatesNugetFullPaths = new List<string>();
	foreach (var customAnalyzerName in customAnalyzerNames)
	{
	var candidateAnalysisProjectPath = Path.Combine(TestAssetsRootPath, customAnalyzerName, $"{customAnalyzerName}.csproj");
	var nugetPackResults = RunnerUtilities.ExecBootstrapedMSBuild(
	$"{candidateAnalysisProjectPath} /m:1 -nr:False -restore /p:OutputPath={env.CreateFolder().Path} -getTargetResult:Build", out bool success, attachProcessId: false);
	success.ShouldBeTrue();

Versions & Configurations

MSBuild version 17.11.0-dev-24263-01+a8e224f80 for .NET
17.11.0.26301

macOS Sonoma 14.4.1
Apple M1 Pro (ARM)

macOS Ventura 13.6.6
Intel Core i7

[YuliiaKovalova]

Hi @jrdodds,

Thank you for reporting the bug!
I have recently merged the changed version for these tests: #10117
Could you please check it out?

[jrdodds]

Hi @YuliiaKovalova,
The tests still fail. The log files are attached.

Microsoft.Build.BuildCheck.UnitTests_net8.0_x64.zip

MSBuild version 17.11.0-dev-24304-01+db79545e5 for .NET
17.11.0.3040

macOS Sonoma 14.5
Apple M1 Pro (ARM)

[YuliiaKovalova]

@jrdodds, It has to be resolved now, after merging #10300

Please validate it when you have a chance .

[jrdodds]

@YuliiaKovalova Tested/validated and confirmed that the issue is resolved. Thanks

[YuliiaKovalova]

@jrdodds , thank you for checking it!
Your issue got it inspired to revisit the approach to bootstrapping MSBuild and testing the instance.
Feel free to dive deeper if you are interested: https://github.com/dotnet/msbuild/blob/main/documentation/wiki/Bootstrap.md