Regression - Upgrading MSTest.TestAdapter from 2.2.3 to 2.2.6 skips all tests

[majastrz]

Description

We upgraded MSTest.TestAdapter from 2.2.3 to 2.2.6 and later on realized that our .net tests were completely skipped as a result. The issue appears to be caused by some bug in test discovery. This also caused the exit code from dotnet test in this situation to be 0, which was considered a pass by our CI jobs.

This is a serious regression that actually caused invalid PRs to be merged into our main branch!

Separately from the bug fix for this issue, is there a way to force tests to fail even if 0 tests are discovered?

Steps to reproduce

1. Clone https://github.com/Azure/bicep
2. Upgrade MSTest.TestAdapter to 2.2.6.
3. Run dotnet test from repo root.

Expected behavior

Tests are actually discovered and run.

Actual behavior

Tests are skipped and the following output appears from every test project:

Test run for D:\a\bicep\bicep\src\Bicep.Core.UnitTests\bin\Release\net5.0\Bicep.Core.UnitTests.dll (.NETCoreApp,Version=v5.0)
Microsoft (R) Test Execution Command Line Tool Version 16.9.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
[MSTest][Discovery][D:\a\bicep\bicep\src\Bicep.Core.UnitTests\bin\Release\net5.0\Bicep.Core.UnitTests.dll] Failed to discover tests from assembly D:\a\bicep\bicep\src\Bicep.Core.UnitTests\bin\Release\net5.0\Bicep.Core.UnitTests.dll. Reason:Could not load type 'Microsoft.VisualStudio.TestTools.UnitTesting.TestDataSourceDiscoveryOption' from assembly 'Microsoft.VisualStudio.TestPlatform.TestFramework, Version=14.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'.
No test is available in D:\a\bicep\bicep\src\Bicep.Core.UnitTests\bin\Release\net5.0\Bicep.Core.UnitTests.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.

Environment

Sample GitHub actions run demonstrating the regression: https://github.com/Azure/bicep/pull/4251/checks?check_run_id=3479570333
Issue appears to be cross-platform (repro'd on windows, mac, and linux)

[nohwnd]

I would have guessed this is because of outdated Microsoft.NET.Test.SDK package, but you are up to date on 16.10 it seems.

Separately from the bug fix for this issue, is there a way to force tests to fail even if 0 tests are discovered?

There is TreatNoTestAsError test platform setting. microsoft/vstest#2707

[Haplois]

@majastrz thank you for reporting this issue. It seems like you're only updating MSTest.TestAdapter package. You need to keep MSTest.TestAdapter and MSTest.TestFramework packages in sync. If you update both it will fix the issue.

See the patch file for the fixes.

[danielchalmers]

related to #930?

[Haplois]

@danielchalmers, no they are not related.

[majastrz]

@majastrz thank you for reporting this issue. It seems like you're only updating MSTest.TestAdapter package. You need to keep MSTest.TestAdapter and MSTest.TestFramework packages in sync. If you update both it will fix the issue.

See the patch file for the fixes.

Sure, I'll give it a try. If there's such a strong coupling between the packages, why isn't it expressed in NuGet dependencies?

[Haplois]

@majastrz It was designed that way, but we'll probably start releasing a meta package to use in tests, instead of two.

[majastrz]

A meta package would be great. In our scenario dependabot is the one updating dependencies and it isn't smart enough to understand the relationship between the two packages, so that would avoid the problem entirely.