Microsoft.TestPlatform (17.4.0-preview-20220726-02) not working

[alexanderjung-sdxag]

Description

Since updating to the latest version of Microsoft.TestPlatform, unittests don't run anymore in Azure DevOps pipelines.
It seems as if there are some assembly reference mismatches, and test attributes are no longer recognized.

https://www.nuget.org/packages/Microsoft.TestPlatform
17.4.0-preview-20220726-02 <== error
17.2.0 (latestStable) <== works
17.4.0-preview-20220707-01 <== works

Steps to reproduce

Build pipeline step:

- task: VisualStudioTestPlatformInstaller@1
  displayName: 'Install Test Platform'

Expected behavior

Tests running fine.

Actual behavior

example 1:

  Failed TestFileArchiveOptions
  Error Message:
   Unable to set TestContext property for the class xxx.FileArchiveTest. Error: System.ArgumentException: Object of type 'Microsoft.VisualStudio.TestPlatform.MSTestFramework.TestContextImplementation' cannot be converted to type 'Microsoft.VisualStudio.TestTools.UnitTesting.TestContext'..
  Stack Trace:
      at System.RuntimeType.TryChangeType(Object value, Binder binder, CultureInfo culture, Boolean needsSpecialCast)
   at System.Reflection.MethodBase.CheckArguments(Object[] parameters, Binder binder, BindingFlags invokeAttr, CultureInfo culture, Signature sig)
   at System.Reflection.RuntimeMethodInfo.InvokeArgumentsCheck(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.Reflection.RuntimePropertyInfo.SetValue(Object obj, Object value, Object[] index)
   at Microsoft.VisualStudio.TestPlatform.MSTestFramework.TestMethodRunner.SetTestContext()

The test class in question is fine:

    [TestClass]
    public class FileArchiveTest 
    {
        public TestContext TestContext { get; set; }

example 2:

  Failed RunActivity_SingleActivityWithBaggage_BaggageIsSet
  Error Message:
   Method xxx.ClassInitialize has wrong signature. The method must be static, public, does not return a value and should take a single parameter of type TestContext.

the method looks fine:

        [ClassInitialize]
        public static void ClassInitialize(TestContext context)

example 3:

  Failed TarjanTest5 [1 ms]
  Error Message:
   Test method xxx.TopoSortTest.TarjanTest5 threw exception: 
System.InvalidOperationException: Graph is not complete. Missing node 'v7' in adjacency list. Check if node is defined as key in the dictionary.

the ExpectedException attibute is not honored:

        [TestMethod]
        [ExpectedException(typeof(InvalidOperationException))]
        public void TarjanTest5()
        {

Diagnostic logs

failing output:

##[section]Starting: Install Test Platform
==============================================================================
Task         : Visual Studio test platform installer
Description  : Acquire the test platform from nuget.org or the tool cache. Satisfies the ‘vstest’ demand and can be used for running tests and collecting diagnostic data using the Visual Studio Test task.
Version      : 1.151.3
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/tool/vstest-platform-tool-installer
==============================================================================
Starting VsTest platform tools installer task.
==============================================================================
Looking for the latest pre-release version of the Microsoft.Testplatform.
[command]C:\CI\7\_work\_tasks\VisualStudioTestPlatformInstaller_2c65196a-54fd-4a02-9be8-d9d1837b7111\1.151.3\nuget.exe list packageid:Microsoft.TestPlatform -PreRelease -NonInteractive -Source https://api.nuget.org/v3/index.json
Microsoft.TestPlatform 17.4.0-preview-20220726-02
Found tool in cache: VsTest 17.4.0-preview-20220726-02 x64
VsTest will use the Test Platform package found in C:\CI\7\_work\_tool\VsTest\17.4.0-preview-20220726-02\x64
##[section]Finishing: Install Test Platform

working output:

##[section]Starting: Install Test Platform
==============================================================================
Task         : Visual Studio test platform installer
Description  : Acquire the test platform from nuget.org or the tool cache. Satisfies the ‘vstest’ demand and can be used for running tests and collecting diagnostic data using the Visual Studio Test task.
Version      : 1.151.3
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/tool/vstest-platform-tool-installer
==============================================================================
Starting VsTest platform tools installer task.
==============================================================================
Looking for the latest pre-release version of the Microsoft.Testplatform.
[command]C:\CI\1\_work\_tasks\VisualStudioTestPlatformInstaller_2c65196a-54fd-4a02-9be8-d9d1837b7111\1.151.3\nuget.exe list packageid:Microsoft.TestPlatform -PreRelease -NonInteractive -Source https://api.nuget.org/v3/index.json
Microsoft.TestPlatform 17.4.0-preview-20220707-01
Found tool in cache: VsTest 17.4.0-preview-20220707-01 x64
VsTest will use the Test Platform package found in C:\CI\1\_work\_tool\VsTest\17.4.0-preview-20220707-01\x64
##[section]Finishing: Install Test Platform

Environment

Azure Devops pipelines

Workarouds:

use earlier versions:

- task: VisualStudioTestPlatformInstaller@1
  displayName: 'Install Test Platform (latestStable!)'
  inputs:  
    packageFeedSelector: 'nugetOrg'
    versionSelector: 'latestStable'

or

- task: VisualStudioTestPlatformInstaller@1
  displayName: 'Install Test Platform (pinned version!)'
  inputs:
    packageFeedSelector: 'nugetOrg'
    versionSelector: 'specificVersion'
    testPlatformVersion: '17.4.0-preview-20220707-01'

[nohwnd]

Thanks for the report does this happen on any test project of just one specific test project?

[alexanderjung-sdxag]

Several different projects in our solution. Not all, but certainly not one specific one.

[greengumby]

This just occurred on our Azure Pipelines. Only thing to change is the version

Microsoft (R) Test Execution Command Line Tool Version 17.4.0 (x64)

[MarcoRossignoli]

Does it fail again on 17.4.0?
Which version of MSTest are you using?

[brandscill]

We have the same problem. Be good to know if there is any solution.

[nohwnd]

@brandscill which exact version are you having issues with please?

[brandscill]

@nohwnd 17.4.1

[GoAvalon]

Started facing something similar.
Tests were running fine till v17.3.2
But in v17.5.0 (which our build now uses), the tests are failing with

Error Message:
Method Microsoft.XYZ.Tests.TestEnvironment.AssemblyInitialize has wrong signature. The method must be static, public, does not return a value and should take a single parameter of type TestContext.

[nohwnd]

@GoAvalon Can you double check your Microsoft.XYZ.Tests.TestEnvironment.AssemblyInitialize method, does it meet what is described in the error? What version of testing framework are you using?

[GoAvalon]

@nohwnd
Yes, our AssemblyInitialize method already satisfies everything mentioned in the error.
We are using Microsoft.NET.Test.Sdk version 16.5.0
Locally all tests run fine on the latest Visual Studio 2022
If I manually install vstest 17.3 then our tests run in the build pipeline too.

Only with vstest 17.5 in our build pipelines we see this error that I mentioned above.

[nohwnd]

@GoAvalon do you use mstest or xunit or nunit? Which version? :)

[GoAvalon]

@nohwnd We are using MSTest version 2.1.1

[nohwnd]

I am doing this, and it works (2.1.1. was not working with 17.3.0, but 17.5.0 works).

<!-- file mstest68.csproj -->
<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>netcoreapp3.1</TargetFramework>

    <IsPackable>false</IsPackable>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.5.0" />
    <PackageReference Include="MSTest.TestAdapter" Version="2.1.1" />
    <PackageReference Include="MSTest.TestFramework" Version="2.1.1" />
    <PackageReference Include="coverlet.collector" Version="3.1.2" />
  </ItemGroup>

</Project>

// file UnitTest1.cs
using Microsoft.VisualStudio.TestTools.UnitTesting;
using System;

namespace mstest68
{

    [TestClass]
    public class UnitTest1
    {
        [AssemblyInitialize]
        public static void AssemblyInitialize(TestContext context)
        {
            Console.WriteLine("AssemblyInitialize");
        }

        [TestMethod]
        public void TestMethod1()
        {
        }
    }
}

Any suggestions how to repro?

[GoAvalon]

@nohwnd
I was trying to repro this but ended up solving the issue in my code.
What was happening is that one of our test projects referenced another test project. When VSTest ran, it discovered the referenced project multiple times, and the first time the test ran it was successful, but when it ran again (due to it being detected twice), it failed with the above AssemblyInitialize error.

To fix this I removed the test project reference and now vstest step succeeds.
Something probably changed in VSTest from 17.3 to 17.5 because we had this project reference since a long time, but at the very least my issue is resolved by removing all extra references.

[Evangelink]

Hey there,

What's the status here? Is there still some issue going-on?

[riQQ]

I have a repro for this using MSTest 3.4.3 and VSTest 17.10.0
VstestIssue3934.zip

Basically what @GoAvalon said.
Have 2 test projects A and B. A defines a base class with the TestContext property that is inherited by the tests in both projects. This requires a reference A -> B which cause A to be copied to B's output folder.

Steps to reproduce the issue

• Compile the solution in Debug configuration
• Run vstest.console **\bin\Debug\**\AnotherTestAssembly.dll **\bin\Debug\**\TestBase.dll from the directory containing the .sln file in the Developer Command Prompt for VS 2022
• error

  Microsoft (R) Test Execution Command Line Tool Version 17.10.0 (x64)
  Copyright (c) Microsoft Corporation.  All rights reserved.
  
  Starting test execution, please wait...
  A total of 3 test files matched the specified pattern.
    Passed TestMethodAnotherTestAssembly [9 ms]
    Passed TestMethod1 [5 ms]
    Passed TestMethod1 [5 ms]
    Failed TestMethod1
    Error Message:
     Unable to set TestContext property for the class TestBase.UnitTest1. Error: System.ArgumentException: Object of type 'Microsoft.VisualStudio.TestPlatform.MSTestFramework.TestContextImplementation' cannot be converted to type 'Microsoft.VisualStudio.TestTools.UnitTesting.TestContext'..
    Stack Trace:
        at System.RuntimeType.TryChangeType(Object value, Binder binder, CultureInfo culture, Boolean needsSpecialCast)
     at System.Reflection.MethodBase.CheckArguments(Object[] parameters, Binder binder, BindingFlags invokeAttr, CultureInfo culture, Signature sig)
     at System.Reflection.RuntimeMethodInfo.InvokeArgumentsCheck(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
     at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
     at System.Reflection.RuntimePropertyInfo.SetValue(Object obj, Object value, Object[] index)
     at Microsoft.VisualStudio.TestPlatform.MSTestFramework.TestMethodRunner.SetTestContext()
  
  
  Test Run Failed.
  Total tests: 4
       Passed: 3
       Failed: 1
   Total time: 1.1689 Seconds
  

[nohwnd]

I must be doing something wrong, it works for me on stable:

[nohwnd]

Moved the issue here, it is problem with MSTest, and possibly some conflict with vstest.

@Evangelink please try if you can repro.

[Evangelink]

Yes I do repro the issue @nohwnd. As discussed offline yesterday, output is good if I use dotnet test or Test Explorer but calling vstest.console.exe is producing the error. Given that the dll contains a single test, I don't expect to see that many results.

[nohwnd]

Ah that was this issue :D

[nohwnd]

I've investigated this with @Evangelink, he noticed that your wildcard pattern is including the TestBase library in both your TestBase/bin and AnotherTestAssembly/bin. Running the dll twice is probably not intended and avoiding this will stop the failure from happening.

This is also the reason why I can't repro, I am using full paths and not wildcards (I though you are "anonymizing" your path 😅). And this is why you don't see the same issue in VS, because there you will run with the 2 exact paths that you get from build, not with a wildcard.

We've investigated this to find the root cause. The failure itself is because when inspecting the dll MSTest/v1 will load due to the rules in the custom assembly resolver in VSTest, this will try to run the tests as MSTest/v1 tests and will fail to assign the property because the types are not compatible. This is why we could not reproduce this in standalone vstest.console, and only in the on that is under VS installation.

You don't have to use this workaound, but new vstest (17.11) comes with a runsetting that is -- RunConfiguration.SkipDefaultAdapters=true which will skip all the built in adapters that ship with VS.

[riQQ]

This is just the fastest repro I came up with.

Our actual situation is that we have a Tests folder, where we build every test project in a subfolder and then just glob all test DLLs via **\*.Tests.dll in the Tests folder. Noone noticed that some tests were run multiple times.

The solution for us was the same as GoAvalon's. We extracted the shared code to a library without any tests, thus avoiding mutliple copies of an assembly with tests ending up in our build output.

You don't have to use this workaound, but new vstest (17.11) comes with a runsetting that is -- RunConfiguration.SkipDefaultAdapters=true which will skip all the built in adapters that ship with VS.

Just for reference, I found the change you mentioned here: microsoft/vstest#5018

[nohwnd]

We extracted the shared code to a library without any tests, thus avoiding mutliple copies of an assembly with tests ending up in our build output.

That is a good solution, I think it is good practice to not mix test project, and test infrastructure projects. You can also still reference MSTest in your test infrastructure projects, but it is a good idea to make all test base classes abstract.