VSTest publishRunAttachments has no effect on what is published

[bancey]

Type: Bug

Enter Task Name: VSTestV2

Environment

Azure Pipelines

Hosted Agent - "Hosted VS2017" pool

Issue Description

The "publishRunAttachments" parameter has no effect on whether attachments actually get published. They are published regardless.

Snippet from my pipeline yaml:

This results in the .coverage file generated being published, which prevents me from publishing processed coverage results later in my pipeline.

Should be fairly easy to reproduce, happy to provide further information if required.

[nehsharmMS]

@bancey We acknowledge that the parameter does not work as expected. You can try out our new richer code coverage experience which is in preview

If you are interested, we will enable this on your account in couple of days.
You can try this and give us further feedback before this feature becomes GA.

[ShreyasRmsft]

This option has been this way for quite some time. Since we have not had the need so far to disable attachment uploads we have not prioritized fixing this. Let us know if the above first class rich coverage experience does not satisfy your needs and we can revisit this. Thanks.

[bancey]

Thanks for looking into this - would be good to get access to the richer code coverage experience, how would I go about getting this enabled for my organisations?

[nehsharmMS]

@bancey, Please mail your account details on devops_tools@microsoft.com.

[jesse-msp]

Will the new code coverage features be able to handle multiple coverage results? And additional formats?

I have a project that contains both JavaScript and C# tests - currently, I produce cobertura results and merge them together to generate the HTML coverage report on the build report. I ran into this same issue - uploading the .coverage results would prevent the HTML report appearing.

My work around was to call vstest via PowerShell to avoid the attachment upload. Ideally, I could run the C# tests and publish the results, and then run the JavaScript tests and publish the results.

Thanks!

[soutchilin]

Hi,
I have a problem in the other way around. The publishRunAttachments is set to true but no attachments are made to the test.

My task configuration - that is the last task in the build, no tasks and no stages after that. The task is in a yaml file that is used as template for a job from the root yaml. The root yaml file doesn't has any other jobs currently. The build is successful and all tests are passed:

• task: VSTest@2
  displayName: "Run .NET Tests"
  inputs:
  testAssemblyVer2: |
  ${{ parameters.build_configuration }}**test.dll
  !\obj**
  !\node_modules
  !\Microsoft.VisualStudio.TestPlatform*.dll
  vsTestVersion: "toolsInstaller"
  testRunTitle: ".NET Tests"
  codeCoverageEnabled: true
  runSettingsFile: "$(base_path)/Backend/default.runsettings"
  platform: "Any CPU"
  configuration: ${{ parameters.build_configuration }}
  publishRunAttachments: true

The log output
Results File: d:\a_temp\TestResults\VssAdministrator_fv-az659_2020-06-17_19_06_31.trx
Attachments:
d:\a_temp\TestResults\d370fbcf-9b38-4dee-bd8e-1bbce53c2665\VssAdministrator_fv-az659 2020-06-17 19_06_21.coverage
Test Run Successful.
Total tests: 10
Passed: 10
Total time: 13.7713 Seconds
Vstest.console.exe exited with code 0.
**************** Completed test execution *********************
Test results files: d:\a_temp\TestResults\VssAdministrator_fv-az659_2020-06-17_19_06_31.trx
Created test run: 1002926
Publishing test results: 10
Publishing test results to test run '1002926'.
TestResults To Publish 10, Test run id:1002926
Test results publishing 10, remaining: 0. Test run id: 1002926
Published test results: 10
Publishing Attachments: 2
Completed TestExecution Model...
Finishing: Run .NET Tests

Thanks!

[MS-Manikandan]

@bancey We acknowledge that the parameter does not work as expected. You can try out our new richer code coverage experience which is in preview

If you are interested, we will enable this on your account in couple of days. You can try this and give us further feedback before this feature becomes GA.

Hi @nehsharmMS, could u let us know is the above code coverage details is rendered from the visual studio .coverage file by a seperate step task, or is this actually generated from Vstest Task or the .coverage file is parsed to xml then published using publish code coverage results task, Keen to know how this coverage details are published in the above shared images.

[simonachmueller]

Hi, I have a problem in the other way around. The publishRunAttachments is set to true but no attachments are made to the test.

My task configuration - that is the last task in the build, no tasks and no stages after that. The task is in a yaml file that is used as template for a job from the root yaml. The root yaml file doesn't has any other jobs currently. The build is successful and all tests are passed:

* task: VSTest@2
  displayName: "Run .NET Tests"
  inputs:
  testAssemblyVer2: |
  **${{ parameters.build_configuration }}*_*test_.dll
  !**\obj**
  !**\node_modules
  !**\Microsoft.VisualStudio.TestPlatform*.dll
  vsTestVersion: "toolsInstaller"
  testRunTitle: ".NET Tests"
  codeCoverageEnabled: true
  runSettingsFile: "$(base_path)/Backend/default.runsettings"
  platform: "Any CPU"
  configuration: ${{ parameters.build_configuration }}
  publishRunAttachments: true

The log output Results File: d:\a_temp\TestResults\VssAdministrator_fv-az659_2020-06-17_19_06_31.trx Attachments: d:\a_temp\TestResults\d370fbcf-9b38-4dee-bd8e-1bbce53c2665\VssAdministrator_fv-az659 2020-06-17 19_06_21.coverage Test Run Successful. Total tests: 10 Passed: 10 Total time: 13.7713 Seconds Vstest.console.exe exited with code 0. **************** Completed test execution ********************* Test results files: d:\a_temp\TestResults\VssAdministrator_fv-az659_2020-06-17_19_06_31.trx Created test run: 1002926 Publishing test results: 10 Publishing test results to test run '1002926'. TestResults To Publish 10, Test run id:1002926 Test results publishing 10, remaining: 0. Test run id: 1002926 Published test results: 10 Publishing Attachments: 2 Completed TestExecution Model... Finishing: Run .NET Tests

Thanks!

Could you set system.debug variable to true and run the pipeline, it will provide you more detailed log about what happening behind the hood of the ADO.

Like this in the yml:

[vjoshi-msft]

@simonachmueller Just to confirm is this bug resolved?

[simonachmueller]

@vjoshi-msft it's not my bug, I just pointed that user can enable system.debug
But this is definitely not resolved, the "first class rich coverage experience" mentioned above is not documented and I face the same problem @bancey reported. Dunno why this ticket was closed.

[ngcdev]

I'm facing the same problem as well. While I appreciate the "We acknowledge that the parameter does not work as expected", the solution given of "send us an email" should not be considered a valid solution for closing this ticket. I understand that it's not a priority, but unless there is a ticket somewhere, it will never get fixed. If there is a ticket already tracking this, please link it, if not, reopen this one.
Thanks

[Josephpul]

Any update on this? I am having the same issue where the results in the .coverage file generated being published, which prevents me from publishing processed coverage results later in my pipeline.

This 'richer code coverage' screen still doesn't appear to be released as i am not seeing output as shown in that screenshot.

@nehsharmMS