MergeWith in dotnet test is not working con vstest integration(collectors)

[StefanOssendorf]

When I try to use the runsettings file to merge the results of our ~40 Test projects the merge file is not created.

Here is a mwe with two test projects and a runsettings-file. The mwe is derived from the HelloWorld-Documentation project.

I have tried several path-combinations (only merged-file, relative path, absolute path) in the runsettings but neither has worked.

[MarcoRossignoli]

Thank's @StefanOssendorf for reporting this, there is a bug MergeWith doesn't work as expected on collectors.

[MarcoRossignoli]

For the moment you could merge report directly to report viewer if supported.

[MarcoRossignoli]

After some investigation I found the problem, I'll try to explain:

at the moment merge functionality is based on fact that full path of output report file is know and can be specified as we can see from msbuild sample https://github.com/tonerdo/coverlet/blob/master/Documentation/Examples/MSBuild/MergeWith/HowTo.md

with /p:CoverletOutput we know what report path will be and so we can merge files.

When we use collectors the story is different, we're integrated inside vstest plat and we don't have full control on the path of generated report file as we can see on repo

https://github.com/microsoft/vstest/blob/7b6248203164f8ea821f6795632bd22e0b69afb0/src/Microsoft.TestPlatform.Common/DataCollection/DataCollectionAttachmentManager.cs#L117

We can only specify --results-directory where report will be written but first part of path is always a runtime generated guid.
Due to this by design issue we cannot today support merging on this driver.

More info #500 (comment)

cc: @vagisha-nidhi

[MarcoRossignoli]

sorry @StefanOssendorf for the inconvenience vstest integration is pretty new so we don't have all use-cases at the moment.

[StefanOssendorf]

@MarcoRossignoli No problem. Thank you for your very detailed explanation and investigation to this issue.
But on the other hand I can achieve the merge with the MSBuild integration, right?

Just for some context: We are using coverlet in our Azure Pipeline to get the code coverage of our 40+ test projects. The inconvinient thing is, the upload of those 40+ result files take ~8minutes. But after a bit thinking I think the problem is more that we selfhost the build agents than using the hosted ones from ms.

[StefanOssendorf]

Since this issue is a "by vstest design not fixable" I'll close it now.

[MarcoRossignoli]

But on the other hand I can achieve the merge with the MSBuild integration, right?

Yes take a look at our samples!

We'll keep this issue open for future reference.

[MarcoRossignoli]

@StefanOssendorf out contributor @pape77 found a clever way to use merge with vstest, run tests sequentially and override merge one take a look maybe could help #225 (comment)

[github-actions]

This issue is stale because it has been open for 3 months with no activity.

[Bertk]

Tip

Some alternative solutions to merge coverage files

• use dotnet-coverage tool and merge multiple coverage files

  dotnet-coverage merge artifacts/coverage/**/coverage.cobertura.xml -f cobertura -o artifacts/coverage/coverage.xml*

• use dotnet-reportgenerator-globaltool to create a HTML report and a merged coverage file

  reportgenerator -reports:"**/*.cobertura.xml" -targetdir:"artifacts\reports.cobertura" -reporttypes:"HtmlInline_AzurePipelines_Dark;Cobertura"