Upgrading MSTest.TestAdapter from 2.2.3 to 2.2.4 breaks data-driven tests

[majastrz]

Description

IMPORTANT: if the defect is reproduced only in a workflow from within the Visual Studio IDE then do not report the issue here - instead, please report it using Visual Studio's "Send Feedback" option that can be accessed from the Help menu OR using this link https://developercommunity.visualstudio.com.

For a defect specific to the MSTest V2 test framework, describe the issue you've observed.

Upgrading MSTest.TestAdapter from 2.2.3 to 2.2.4 breaks data-driven tests. See PR opened by dependabot against our repo: Azure/bicep#2849. None of the release notes mention any breaking changes that would explain the errors below, so I'm assuming this is a bug.

Defect occurs with dotnet test, but also breaks the test discovery in Visual Studio.

Steps to reproduce

I don't have a minimal repro. Based on the error, it seems creating a data-driven test with a type that isn't annotated with the DataContract attribute should repro it.

Expected behavior

I expect tests to pass normally as before.

Actual behavior

CI build link: https://github.com/Azure/bicep/pull/2849/checks?check_run_id=2688878286

Test failures via dotnet test:

An exception occurred while invoking executor 'executor://mstestadapter/v2': Type 'Bicep.Core.Samples.DataSet' cannot be serialized. Consider marking it with the DataContractAttribute attribute, and marking all of its members you want serialized with the DataMemberAttribute attribute. Alternatively, you can ensure that the type is public and has a parameterless constructor - all public members of the type will then be serialized, and no attributes will be required.
Stack trace:
   at System.Runtime.Serialization.DataContract.DataContractCriticalHelper.ThrowInvalidDataContractException(String message, Type type)
   at System.Runtime.Serialization.DataContract.DataContractCriticalHelper.CreateDataContract(Type type)
   at System.Runtime.Serialization.DataContract.DataContractCriticalHelper.CreateDataContract(Int32 id, RuntimeTypeHandle typeHandle, Type type)
   at System.Runtime.Serialization.DataContract.DataContractCriticalHelper.GetDataContractSkipValidation(Int32 id, RuntimeTypeHandle typeHandle, Type type)
   at System.Runtime.Serialization.DataContract.GetDataContract(RuntimeTypeHandle typeHandle, Type type, SerializationMode mode)
   at System.Runtime.Serialization.DataContractSerializer.GetDataContract(DataContract declaredTypeContract, Type declaredType, Type objectType)
   at System.Runtime.Serialization.Json.DataContractJsonSerializerImpl.InternalWriteObjectContent(XmlWriterDelegator writer, Object graph)
   at System.Runtime.Serialization.Json.DataContractJsonSerializerImpl.InternalWriteObject(XmlWriterDelegator writer, Object graph)
   at System.Runtime.Serialization.XmlObjectSerializer.InternalWriteObject(XmlWriterDelegator writer, Object graph, DataContractResolver dataContractResolver)
   at System.Runtime.Serialization.XmlObjectSerializer.WriteObjectHandleExceptions(XmlWriterDelegator writer, Object graph, DataContractResolver dataContractResolver)
   at System.Runtime.Serialization.Json.DataContractJsonSerializerImpl.WriteObject(XmlDictionaryWriter writer, Object graph)
   at System.Runtime.Serialization.Json.DataContractJsonSerializerImpl.WriteObject(Stream stream, Object graph)
   at Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Helpers.DataSerializationHelper.Serialize(Object[] data)
   at Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.ObjectModel.UnitTestElement.ToTestCase()
   at Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.UnitTestDiscoverer.SendTestCases(String source, IEnumerable`1 testElements, ITestCaseDiscoverySink discoverySink, IDiscoveryContext discoveryContext, IMessageLogger logger)
   at Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.UnitTestDiscoverer.DiscoverTestsInSource(String source, IMessageLogger logger, ITestCaseDiscoverySink discoverySink, IDiscoveryContext discoveryContext)
   at Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestExecutionManager.RunTests(IEnumerable`1 sources, IRunContext runContext, IFrameworkHandle frameworkHandle, TestRunCancellationToken cancellationToken)
   at Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.MSTestExecutor.RunTests(IEnumerable`1 sources, IRunContext runContext, IFrameworkHandle frameworkHandle)
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.RunTestsWithSources.InvokeExecutor(LazyExtension`2 executor, Tuple`2 executorUriExtensionTuple, RunContext runContext, IFrameworkHandle frameworkHandle)
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Execution.BaseRunTests.RunTestInternalWithExecutors(IEnumerable`1 executorUriExtensionMap, Int64 totalTests)

Results File: D:\a\bicep\bicep\TestResults\runneradmin_fv-az177-165_2021-05-27_21_01_32.trx

In Visual Studio, the test discovery is also broken and prevents test execution. Output in VS:

---------- Starting test discovery ----------
An exception occurred while test discoverer 'MSTestDiscoverer' was loading tests. Exception: Type 'Bicep.Core.IntegrationTests.DecompilationTests+ExampleData' cannot be serialized. Consider marking it with the DataContractAttribute attribute, and marking all of its members you want serialized with the DataMemberAttribute attribute. Alternatively, you can ensure that the type is public and has a parameterless constructor - all public members of the type will then be serialized, and no attributes will be required.
An exception occurred while test discoverer 'MSTestDiscoverer' was loading tests. Exception: Type 'Bicep.Core.Samples.DataSet' cannot be serialized. Consider marking it with the DataContractAttribute attribute, and marking all of its members you want serialized with the DataMemberAttribute attribute. Alternatively, you can ensure that the type is public and has a parameterless constructor - all public members of the type will then be serialized, and no attributes will be required.
No test is available in C:\GitHub\Azure\bicep\src\Bicep.LangServer.IntegrationTests\bin\Debug\net5.0\Bicep.LangServer.IntegrationTests.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
An exception occurred while test discoverer 'MSTestDiscoverer' was loading tests. Exception: Type 'Bicep.Core.Samples.DataSet' cannot be serialized. Consider marking it with the DataContractAttribute attribute, and marking all of its members you want serialized with the DataMemberAttribute attribute. Alternatively, you can ensure that the type is public and has a parameterless constructor - all public members of the type will then be serialized, and no attributes will be required.
An exception occurred while test discoverer 'MSTestDiscoverer' was loading tests. Exception: Type 'System.String[]' with data contract name 'ArrayOfstring:http://schemas.microsoft.com/2003/10/Serialization/Arrays' is not expected. Add any types not known statically to the list of known types - for example, by using the KnownTypeAttribute attribute or by adding them to the list of known types passed to DataContractSerializer.
An exception occurred while test discoverer 'MSTestDiscoverer' was loading tests. Exception: Type 'Bicep.Core.Samples.DataSet' cannot be serialized. Consider marking it with the DataContractAttribute attribute, and marking all of its members you want serialized with the DataMemberAttribute attribute. Alternatively, you can ensure that the type is public and has a parameterless constructor - all public members of the type will then be serialized, and no attributes will be required.
An exception occurred while test discoverer 'MSTestDiscoverer' was loading tests. Exception: Type 'Bicep.Core.Samples.DataSet' cannot be serialized. Consider marking it with the DataContractAttribute attribute, and marking all of its members you want serialized with the DataMemberAttribute attribute. Alternatively, you can ensure that the type is public and has a parameterless constructor - all public members of the type will then be serialized, and no attributes will be required.
No test is available in C:\GitHub\Azure\bicep\src\Bicep.Core.Samples\bin\Debug\net5.0\Bicep.Core.Samples.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
An exception occurred while test discoverer 'MSTestDiscoverer' was loading tests. Exception: Type 'Bicep.Core.TypeSystem.PrimitiveType' cannot be serialized. Consider marking it with the DataContractAttribute attribute, and marking all of its members you want serialized with the DataMemberAttribute attribute. Alternatively, you can ensure that the type is public and has a parameterless constructor - all public members of the type will then be serialized, and no attributes will be required.
========== Test discovery finished: 150 Tests found in 11.1 sec ==========

Environment

Windows 10
MSTest.TestAdapter 2.2.4
MSTest.TestFramework 2.2.4 (test framework update doesn't break tests - only adapter upgrade)
Microsoft.NET.Test.Sdk 16.10.0

[Haplois]

Thank you for reporting this, it's a known issue - we're working on it. It's tracked here, there is a preview version addressing this issue is out, you could switch to it meanwhile we're working on releasing the fix.