Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Data Driven tests - Individual results of data rows are not in tree structure, missing parentID in trx file #1024

Open
Legoldos opened this issue Dec 14, 2021 · 5 comments

Comments

@Legoldos
Copy link

Legoldos commented Dec 14, 2021

Description

I recently updated MSTest adapter to version 2.2.5 from 2.2.3 and noticed following change in behavior of results of data driven tests.
There are results for individual data rows, but they are missing parent test. Therefor results arent structuralized and I cant tell which data row result belong to which test.
I would like to know if this is a bug or expected behavior.

Steps to reproduce

  1. Create simple test project targeting .net framework 4.8 or .net core 3.1
  2. Get MSTest framework and adapter 2.2.5+
  3. Create these simple data driven tests:
        [TestMethod]
        [DataRow("1", DisplayName = "A")]
        [DataRow("2", DisplayName = "AA")]
        public void Test1(string test)
        {
        }

        [TestMethod]
        [DataRow("1", DisplayName = "B")]
        [DataRow("2", DisplayName = "BB")]
        public void Test2(string test)
        {
        }
  1. Run tests via cmd with /Logger:trx
  2. Repeat steps but this time with MSTest Framework and adapter version 2.2.3
  3. Compare trx files or console outputs

Expected behavior

This is result structure with adapter version 2.2.3 (sorry its not in English but the idea should be clear)
266544445_282482210466585_1769694899886715917_n
This is how trx file looks:
trxGood

Actual behavior

This is result structure with adapter version 2.2.5 (sorry its not in English but the idea should be clear)
264242296_287803316627445_271656889904929547_n
This is how trx file looks:
trxbad

Environment

Windows 10
Visual studio 2019
vs.console 16.11
MSTest adapter/framework 2.2.5+

@ChristoWolf
Copy link

I second this, see also #1026.

@Evangelink
Copy link
Member

Hey folks. As explained in the linked issue, this change was done to align with how xUnit and NUnit are working. There was a big issue on our side as the change was done within minor version and no clear communication so I understand the frustration.

Note that with v3, we introduce an assembly level attribute that allows you to revert to legacy test ID generation which keeps the hierarchical structure BUT keeps all the bugs related to proper discovery (unfolding) of parameterized tests in VS Test Explorer UI.

I will discuss with teams internally to see how we could update tools to support this mode.

@HightowerCZ
Copy link

Hi,

We get stuck with the old MSTest libraries until this is solved. We are slowly migrating the code base to .Net 7, so maybe one day we will be forced to, but it would be nice to have this fixed before.

Thank you,
Jakub

@Evangelink
Copy link
Member

Hi there,

I have open a ticket on Dev Community https://developercommunity.visualstudio.com/t/Incorrect-test-result-view-in-Azure-DevO/10359804?q=Incorrect+test+result+view for you all to track. Please do upvote the ticket.

@ldsenow
Copy link

ldsenow commented Jul 5, 2024

I have been using the older version over a year now. I am wondering where the assembly level config to revert back?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants