Code Coverage do not match with Azure DevOps summary

[sriksmachi]

Required Information

The build summary page of Azure DevOps pipelines show a different code coverage (less code coverage) when compared to TRX file.

Question, Bug, or Feature?
Type: Bug

Enter Task Name: Azure DevOps build summary

list here (V# not needed):
https://github.com/microsoft/azure-pipelines-tasks/tree/master/Tasks/PublishTestResultsV2

Environment

• Server - Azure Pipelines
• Agent - Private:
  

Issue Description

Azure DevOps build summary shows different code coverage than what is show on Visual studio when we download the trx file.

From build summary

From VS when I downloaded the TRX file.

The task does not contain any Test filters, RunSettings - below is the configuration

[TianyuHou]

Any updates on this issue? our project faced the same issue.

[paulroreilly]

Same problem for our project - DevOps Pipeline shows 70% code coverage, but same tests in VS2019 show 95%.

Not ideal to have to lower the threshold so much to make sure the builds are failed when the coverage is good.

[StingyJack]

Same problem here, but for me its the line counts that are deficient from the pipeline task thats causing the numbers to be off

This is VS displaying the coverage file that was produced from VSTest task (Single file depicted). That total line count is WAY off.

This is from running the code coverage in VS for the same commit. VS has made a bunch of lambdas out of things but the "testable and not covered" line count of 529 would be closer to correct for a 900 line file, sans namespaces, imports, braces, etc.

[nk111]

In my case it was the difference between "% of blocks covered" which is displayed in VS by default and "% of lines covered" which seems to be displayed in the build summary!

[StingyJack]

@nk111 You should be able to change the columns in VS to see if the counts match up

[nk111]

@nk111 You should be able to change the columns in VS to see if the counts match up

Yepp, your screenshot made me aware of this option actually ;)

[eduralph]

So, this error happens if you have one SUT but several Test Assemblies which run after each other. This error occurred for me when I had 2 different test runs - a Unit Test & then later an Integration Test. In this case, the percentage shown on the summary page was from the Unit Test only while on the detail page, it showed the accumulated coverage by Unit & Integration Test.

[Patafoin]

Same problem here : multiple tests projects running after each other.
Report is showing 97% of covered code
Azure pipeline summary is showing about 25% percent.
Seems that azure pipelines doesn't take care of the same results than ReportGenerator do.
And it's not only a problem of using last project results because all my included projects are about 95-97% tested. I think Azure pipelines doesn't take care of tests settings neither (for me, they exclude some tools projects that don't have to be covered but seems to be included in these 25% of code coverage shown by azure).

[lsiddiquee]

Seems like this is a very common issue. Any update on this issue? I am seeing a coverage of 23% in summary in reality when the tests are executed in VS I have over 90% coverage.

[FrankRua]

Same here, VS showing ~80% test coverage, Azure showing 50.51%.

[nk111]

Everyone please make sure you are not comparing "% of lines covered" vs "% of blocks covered"! You can select more columns to be displayed in the VS Test results window! The Devops result means "% of lines covered" which might be lower than % of blocks covered displayed in VS by default!

[FrankRua]

@nk111 Azure is wrong in both ways, see the following output on the UI vs what is in the actual report:

VS What Visual Studio says (notice Covered % of Lines)

[thomaslevesque]

I'm having the same problem, but with a more extreme difference... 90.7% in the report, 0,51% in the summary

Where does the value in the summary come from?

[FrankRua]

@phanikmmsft - Any update on this issue? This is critical to CI/CD pipelines ensuring good code coverage.

[nadesu]

@v-pkoneti - pls look into this.

[gioce90]

I have noticed some similar problems.

My pipeline run these tasks:

# ... DotNetCoreCLI@2 restore task ...

  - task: DotNetCoreCLI@2
    displayName: 'DotNetCoreCLI Test with report (cobertura format)'
    inputs:
      command: test
      projects: '**/**.Tests.csproj'
      arguments: '--no-restore --collect "Code Coverage" --logger trx --results-directory "TestResults/Coverage/" -- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Format=cobertura'
      publishTestResults: false

  - task: reportgenerator@5
    inputs:
      reports: '$(Build.SourcesDirectory)/TestResults/Coverage/**/**.cobertura.xml'
      targetdir: '$(Build.SourcesDirectory)/TestResults/Coverage/Reports'
      reporttypes: 'HtmlInline_AzurePipelines_Dark;Cobertura'
      assemblyfilters: '+MyOrganization.**;-MyOrganization.**.Tests'

  - task: PublishTestResults@2
    inputs:
      testResultsFormat: 'VSTest'
      testResultsFiles: '**/*.trx'
      mergeTestResults: true
      failTaskOnFailedTests: true

  - task: PublishCodeCoverageResults@1
    inputs:
      codeCoverageTool: 'cobertura'
      summaryFileLocation: $(Build.SourcesDirectory)/TestResults/Coverage/Reports/Cobertura.xml
      reportDirectory: $(Build.SourcesDirectory)/TestResults/Coverage/Reports
      failIfCoverageEmpty: false

Please note that I set a variable disable.coverage.autogenerate = true and use a preview "Microsoft.NET.Test.Sdk" nuget package (17.1.0-preview-20211118-03, as indicated here)

This works well and pubblish tests and code coverage. In the "code coverage" tab I see this:

And in the Summary I see the same %. So the reportgenerator@5 task with that assemblyfilters field fix the problem (when is used the native microsoft code coverage).

Just for your knowledge... if I switch the test task to coverlet in this way:

# same restore...

  - task: DotNetCoreCLI@2
    displayName: 'DotNetCoreCLI Test with report (XPlat Code Coverage - cobertura format)'
    inputs:
      command: test
      projects: '**/**.Tests.csproj' # if multiple test projects are to be executed, create a dirs.proj and use it here. That will ensure the tests are executed in parallel
      arguments: '--no-restore --collect "XPlat Code Coverage" --logger trx --results-directory "TestResults/Coverage/" '
      publishTestResults: false

# same tasks...

This is what I see in the "code coverage" tab this time:

Why? And this is totally incompatible with the Summary. So it is a methodology to be discarded.
My advice: stay stick with the new native Microsoft code coverage.

Another try is with this:

# same restore...
  - task: DotNetCoreCLI@2
    displayName: 'DotNetCoreCLI Test with report (.coverage format)'
    inputs:
      command: test
      projects: '**/**.Tests.csproj' # if multiple test projects are to be executed, create a dirs.proj and use it here. That will ensure the tests are executed in parallel
      arguments: '--collect "Code Coverage" '
      publishTestResults: true
# no other tasks needed...

I don't like that because doesn't populate the "Code coverage" tab on the build. I use this on another pipeline for obtain the Pull Request Diff coverage (I talk how to obtain both here microsoft/vstest#981 (comment) )
And is awkward, I know, but for now is the only way to have full and diff coverage). However, is possible to download the .coverage file and I noticed something interesting:

Pay attention: I visualizing Covered/Not covered LINES and not blocks. The comparison must be made on lines.
In the screenshot it is not seen, but the "Code Coverage Results" in Visual Studio include also my "*.Tests.dll".
I want to find a way to do something similar to what I have done with the assemblyfilters field in reportgenerator task, to exclude/include assembly of my interest, so I can have a total match ever. It would be preferable without a .runsettings file.

What I understood (please correct me if I am wrong): probably all these different methods to obtain code coverage use different filters. We need to find a (convenient) way to best set filters according to our needs.

[github-actions]

This issue is stale because it has been open for 180 days with no activity. Remove the stale label or comment on the issue otherwise this will be closed in 5 days

[StingyJack]

@phanikmmsft or @nadesu Why is the actions bot trying to close an issue to which there has been no response from the engineering team?

[lzaromskis]

This is still an issue. DevOps still reports incorrect code coverage. @nadesu @v-pkoneti are there any plans to fix this?

[gioce90]

Hi @lzaromskis can I ask you what results you have launching Code Coverage on Visual Studio? (Test explorer > Analyze Code Coverage for All Tests). Can you put a screen here?

[lzaromskis]

@gioce90 I use Visual Studio Professional and it does not have code coverage.

[StingyJack]

@lzaromskis there are FOSS packages and extensions you can get for visual studio pro and community like Coverlet, Run Coverlet Report and Fine Code Coverage.

Also I saw it mentioned somewhere that async code causes a lot of these problems where the coverage doesn't report correctly. I'll see if I can find that again

[lzaromskis]

After some investigation, found that the issue is related with Report Generator.
One of the fixes can be found here:
https://github.com/danielpalme/ReportGenerator/tree/main/src/AzureDevopsTask#attention

Need to set an environment variable in the pipeline:

disable.coverage.autogenerate: 'true'

[gioce90]

After some investigation, found that the issue is related with Report Generator. One of the fixes can be found here: https://github.com/danielpalme/ReportGenerator/tree/main/src/AzureDevopsTask#attention

Need to set an environment variable in the pipeline:

disable.coverage.autogenerate: 'true'

Hi @lzaromskis , this was also my solution. You used Coverlet or MS's native Code Coverage?

However, if you look carefully I had already written it in this thread :) (with native coverage... with coverlet I haven't tried much)

I asked you if you could use VS code coverage because I wanted to check the reason for the mismatch.
I believe it is due to the fact that the MS's native coverage, when performed on Azure DevOps, also counts the coverage of third party dlls. Or at least that's what I noticed when, instead of having the report generated with ReportGenerator, I download the .coverage file from DevOps and upload it to Visual Studio.
Instead, generating coverage locally from Visual Studio does not cause this problem.

[lzaromskis]

@gioce90 I used Coverlet. The mismatch from ReportGenerator was also because it counted generated C# files, so adding a file filter helped to get correct code coverage.

[StingyJack]

Instead, generating coverage locally from Visual Studio does not cause this problem.

That's weird because visual studio always used to include test dlls and sometimes third party dlls in the coverage. Had to make many test settings files to get them to stop including those unwanted targets. I guess they fixed it.

[github-actions]

This issue is stale because it has been open for 180 days with no activity. Remove the stale label or comment on the issue otherwise this will be closed in 5 days

[Aculeo]

This issue should not be closed as this is still happening. The summary tab shows completely different numbers than the Code Coverage tab. And there is no way to validate the number shown on Summary, since it's not transparent at all how this is calculated.

[gioce90]

HI @Aculeo are you sure this is still happening? I'm collecting code coverage in two different ways (one for full coverage and the other for diff coverage) and to me the summary and the report seems to be fine.

Results of my pipeline used for Diff coverage
Summary:

Code coverage tab:

How you can see, they match.
I noticed that with this pipeline my coverage report contains also info on test libraries and third-party libraries. Is not a real problem for my diff coverage, so is okay.
However, I think that soon with the use of IncludeTestAssembly=false option the test libraries will disappears.
I hope that in the future we will also see something to easily exclude third party libraries....

Results of my pipeline used for Full coverage
Summary:

Code coverage tab:

How you can see, also in this case they match.
Here I produced the report with arguments: '--no-restore --configuration $(buildConfiguration) --collect "Code Coverage" --logger trx --results-directory "TestResults/Coverage/" -- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Format=cobertura'
so I generated a binary .coverage file with cobertura format and after that I used reportgenerator to show the report (reportgenerator is usefull also to hide test and third-party libraries from the report).

This two builds that I have shown here are runs for the same Pull Request, so for the same code. If you see a little mismatch between the two "Line Coverage %", I think it is due the first pipeline include also test and third-party libraries, so is not a problem.

[Aculeo]

I need to check again with a newer version of Azure DevOps Server, but currently we're still seeing this behavior

[gioce90]

Someone should reopen this if there is no validation of the fix :/

[Aculeo]

After some more troubleshooting, for me it is related to this bug: #13448

We are using the task PublishCodeCoverageResultsV1, and it has a known limitation that it cannot merge multiple coverage reports together. We have more than one test project, so this leads to weird behavior: Sometimes it picks the first coverage report, sometimes the second. We can see that in the log for the publish step:

Multiple file or directory matches were found. Using the first match: E:\BuildAgent.v2.185.005\_work\_temp\94a48701-02a1-4237-812d-5dad8cd8bc16\coverage.cobertura.xml

Now PublishCodeCoverageResultsV2 supposedly has this fixed, as it will merge multiple reports together, but we do not have it yet in our Azure DevOps Server, so I cannot verify that. Will have to wait for this to trickle down with future updates.

[epicbugs]

Can verify that the problem is fixed in PublishCodeCoverageResults@2. Happy coding!