-
Notifications
You must be signed in to change notification settings - Fork 253
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
TestAdapter: Wrongly grouped DataTestMethod #1634
Comments
Hi @HightowerCZ, Thanks for reporting this bug. I can also reproduce the issue, we will need to investigate to understand what changed. As for the report file being changed, are you talking about the change of ID or are you referring to something else? |
Hi @Evangelink, We have two issues. One is showing in Visual Studio; the second is with trx reporting file. I am including the correct report and the incorrect report. It is difficult to find which test failed since the name of the test is the same for all runs. We noticed the issue because we are upgrading our solution from .Net Framework 4.8 to .NET 7.0. Newest findings:
Is this what you asked for? |
Hi @HightowerCZ, I am waiting for a little more info about the change of behaviour in VS because it's not yet fully clear if this is coming from a change caused by MSTest or VS Test Explorer. I will let you know more when I have some news. Regarding the change in the trx, this is not linked to traits grouping but linked to a change of behavior (breaking change) that was introduced in 2.2.4. This is unfortunate but sadly we cannot easily change this on our side because we need to synchronize with Azure DevOps team to ensure the UI will reflect the change. To get more information about this change, I encourage you to refer to #1024. |
In MSTest v3+ we have started using some new features from newer versions of Test Platform and one of them is causing this change of display in VS. The Test Explorer team (cc @peterwald) told me that if you update your grouping to be "Traits" + "Class" you will have an output similar to the previous one: I will update the release notes to specify that change of behaviour and this "workaround". I hope this new display is convenient enough for you?! |
Hi @Evangelink, This is completely OK for us as developers. The only issue is with the AzureDev ops results. We still have to have grouping there. Any idea how to achieve this? Thank you! |
A little upvote on the linked issue. Sadly, we were forced to update the generated trx to comply with some requests from AzDo (to match handling of xUnit and NUnit). We know this is causing some pain for our users but we are still waiting for them to allow mixed mode so we can use the grouped mode. Please feel free to discuss the AzDo results on the linked issue. |
OK, I put the comment to the linked issue. Thank you for being so helpful! |
Describe the bug
DataTestMethods are wrongly grouped with MSTest.TestAdapter in version 3.0.x
(Grouping in VS is by Traits)
Wrong grouping:
![image](https://user-images.githubusercontent.com/22144797/232417126-dfad8209-df9e-4972-87c8-94cce83c8435.png)
Expected grouping (achieved with MSTest.TestAdapter 2.2.x):
![image](https://user-images.githubusercontent.com/22144797/232417595-86831a5f-cae1-41b6-8cb6-98f442c15a65.png)
Steps To Reproduce
Explore tests in this class:
Expected behavior
Expected grouping (achieved with MSTest.TestAdapter 2.2.x):
![image](https://user-images.githubusercontent.com/22144797/232417595-86831a5f-cae1-41b6-8cb6-98f442c15a65.png)
Actual behavior
Wrong grouping:
![image](https://user-images.githubusercontent.com/22144797/232417126-dfad8209-df9e-4972-87c8-94cce83c8435.png)
Additional context
We also run tests in Azure DevOps wth
vstest.console.exe "C:\abc.dll" /TestCaseFilter:"TestCategory=L3" "/logger:trx;LogFileName=C:\abc.trx" /TestAdapterPath:"C:\UnitTests\Tests"
command. The test report file has changed after the libraries update.AB#1813781
The text was updated successfully, but these errors were encountered: