TestAdapter: Wrongly grouped DataTestMethod

[HightowerCZ]

Describe the bug

DataTestMethods are wrongly grouped with MSTest.TestAdapter in version 3.0.x
(Grouping in VS is by Traits)

Wrong grouping:

Expected grouping (achieved with MSTest.TestAdapter 2.2.x):

Steps To Reproduce

Explore tests in this class:

using Microsoft.VisualStudio.TestTools.UnitTesting;

namespace TestProject
{
    [TestClass]
    [TestCategory("L3")]
    public class BasicUnitTest : UnitTestBase
    {
        public override int TypeCount => 3;
    }
    
    [TestClass]
    [TestCategory("L3")]
    public class ExtendedUnitTest : UnitTestBase
    {
        public override int TypeCount => 8;
    }

    [TestClass]
    [TestCategory("L3")]
    public class UnitTestBase
    {
        public virtual int TypeCount => 5;

        [DataRow(1)]
        [DataRow(2)]
        [DataTestMethod]
        public void DocumentCostCurrentDataFunctions(int count)
        {
            Assert.IsTrue(count < TypeCount);
        }
    }
}

Expected behavior

Expected grouping (achieved with MSTest.TestAdapter 2.2.x):

Actual behavior

Wrong grouping:

Additional context

We also run tests in Azure DevOps wth vstest.console.exe "C:\abc.dll" /TestCaseFilter:"TestCategory=L3" "/logger:trx;LogFileName=C:\abc.trx" /TestAdapterPath:"C:\UnitTests\Tests" command. The test report file has changed after the libraries update.

AB#1813781

[Evangelink]

Hi @HightowerCZ,

Thanks for reporting this bug. I can also reproduce the issue, we will need to investigate to understand what changed.

As for the report file being changed, are you talking about the change of ID or are you referring to something else?

[HightowerCZ]

Hi @Evangelink,

We have two issues. One is showing in Visual Studio; the second is with trx reporting file. I am including the correct report and the incorrect report. It is difficult to find which test failed since the name of the test is the same for all runs.
TestResults.zip

We noticed the issue because we are upgrading our solution from .Net Framework 4.8 to .NET 7.0.

Newest findings:

• vstest.console.exe produce grouped results only for MSTest.Adapter 2.2.3, but this version shows tests in VisualStudio incorrectly
• vstest.console.exe in different versions of MSTest.Adapter than 2.2.3 shows mostly correctly in VisualStudio, but produce the ungrouped report

Is this what you asked for?

[Evangelink]

Hi @HightowerCZ,

I am waiting for a little more info about the change of behaviour in VS because it's not yet fully clear if this is coming from a change caused by MSTest or VS Test Explorer. I will let you know more when I have some news.

Regarding the change in the trx, this is not linked to traits grouping but linked to a change of behavior (breaking change) that was introduced in 2.2.4. This is unfortunate but sadly we cannot easily change this on our side because we need to synchronize with Azure DevOps team to ensure the UI will reflect the change. To get more information about this change, I encourage you to refer to #1024.

[Evangelink]

In MSTest v3+ we have started using some new features from newer versions of Test Platform and one of them is causing this change of display in VS. The Test Explorer team (cc @peterwald) told me that if you update your grouping to be "Traits" + "Class" you will have an output similar to the previous one:

I will update the release notes to specify that change of behaviour and this "workaround".

I hope this new display is convenient enough for you?!

[HightowerCZ]

Hi @Evangelink,

This is completely OK for us as developers.

The only issue is with the AzureDev ops results. We still have to have grouping there. Any idea how to achieve this?

Thank you!

[Evangelink]

A little upvote on the linked issue.

Sadly, we were forced to update the generated trx to comply with some requests from AzDo (to match handling of xUnit and NUnit). We know this is causing some pain for our users but we are still waiting for them to allow mixed mode so we can use the grouped mode.

Please feel free to discuss the AzDo results on the linked issue.

[HightowerCZ]

OK, I put the comment to the linked issue. Thank you for being so helpful!