Wrong coverage report with DynamicCodeCoverage xml file

[mschnelte]

Used reportgenerator version: 5.0.0

Take the following c# application and compile it.

using System;
using System.Threading;
using System.Threading.Tasks;

namespace AsyncCoverage
{
    class Program
    {
        static void Main(string[] args)
        {
            var c = int.Parse(args[0]);
            Console.WriteLine("Hello World!");
            var monitor = new ManualResetEvent(false);
            Task.Run( () => { 
                if (c>10)
                {
                    Console.WriteLine("Hit >10");
                } else
                {
                    Console.WriteLine("Hit <=10");
                }
                monitor.Set();
            });
            monitor.WaitOne();
        }
    }
}

Collect coverage using the global dotnet-coverage tool.

dotnet-coverage collect -f xml "dotnet .\bin\Debug\netcoreapp3.1\AsyncCoverage.dll 12"

Generate the report

reportgenerator -targetdir:./testrepMS -reports:output.xml

Result is a report with 100% coverage although the resulting xml clearly shows partial coverage only.

<module block_coverage="86,67" line_coverage="88,46" blocks_covered="13" blocks_not_covered="2" lines_covered="23" lines_partially_covered="0" lines_not_covered="3" name="AsyncCoverage.dll" path="AsyncCoverage.dll" id="9275470DA29C4C4B8D0E45E421E3247201000000">
      <functions>
        <function block_coverage="100,00" line_coverage="100,00" blocks_covered="8" blocks_not_covered="0" lines_covered="16" lines_partially_covered="0" lines_not_covered="0" id="8272" token="0x6000001" name="Main(string[])" namespace="AsyncCoverage" type_name="Program">
          <ranges>
            <range source_id="0" covered="yes" start_line="10" start_column="9" end_line="10" end_column="10" />
            <range source_id="0" covered="yes" start_line="11" start_column="13" end_line="11" end_column="40" />
            <range source_id="0" covered="yes" start_line="12" start_column="13" end_line="12" end_column="47" />
            <range source_id="0" covered="yes" start_line="13" start_column="13" end_line="13" end_column="55" />
            <range source_id="0" covered="yes" start_line="14" start_column="13" end_line="23" end_column="16" />
            <range source_id="0" covered="yes" start_line="24" start_column="13" end_line="24" end_column="31" />
            <range source_id="0" covered="yes" start_line="25" start_column="9" end_line="25" end_column="10" />
          </ranges>
        </function>
        <function block_coverage="71,43" line_coverage="70,00" blocks_covered="5" blocks_not_covered="2" lines_covered="7" lines_partially_covered="0" lines_not_covered="3" id="8380" token="0x6000004" name="&lt;Main&gt;b__0()" namespace="AsyncCoverage" type_name="Program.&lt;&gt;c__DisplayClass0_0">
          <ranges>
            <range source_id="0" covered="yes" start_line="14" start_column="29" end_line="14" end_column="30" />
            <range source_id="0" covered="yes" start_line="15" start_column="17" end_line="15" end_column="26" />
            <range source_id="0" covered="yes" start_line="16" start_column="17" end_line="16" end_column="18" />
            <range source_id="0" covered="yes" start_line="17" start_column="21" end_line="17" end_column="50" />
            <range source_id="0" covered="yes" start_line="18" start_column="17" end_line="18" end_column="18" />
            <range source_id="0" covered="no" start_line="19" start_column="17" end_line="19" end_column="18" />
            <range source_id="0" covered="no" start_line="20" start_column="21" end_line="20" end_column="51" />
            <range source_id="0" covered="no" start_line="21" start_column="17" end_line="21" end_column="18" />
            <range source_id="0" covered="yes" start_line="22" start_column="17" end_line="22" end_column="31" />
            <range source_id="0" covered="yes" start_line="23" start_column="13" end_line="23" end_column="14" />
          </ranges>
        </function>
      </functions>

Additional info:

Generating a cobuerta file with coverlet is reporting the expected results.

coverlet --target dotnet --targetargs ".\bin\Debug\netcoreapp3.1\AsyncCoverage.dll 12" .\bin\Debug\netcoreapp3.1\AsyncCoverage.dl --format cobertura

(The .dl is not a typo, somehow this is a way to make coverlet report correctly without the need of unit tests)

I have attached the c# project with the coverage reports (xml, cobertura and json)
AsyncCoverage.zip

[danielpalme]

Thanks for the detailed description and the sample project.

I found the root cause for this issue:
The second <function> element contains those three elements:

<range source_id="0" covered="no" start_line="19" start_column="17" end_line="19" end_column="18" />
<range source_id="0" covered="no" start_line="20" start_column="21" end_line="20" end_column="51" />
<range source_id="0" covered="no" start_line="21" start_column="17" end_line="21" end_column="18" />

Those elements indicate that the lines 19-21 are not covered (covered="no").

The first <function> element contains the following element:

<range source_id="0" covered="yes" start_line="14" start_column="13" end_line="23" end_column="16" />

This indicates, that the lines 14-23 are covered.

ReportGenerator merges all the <range> elements and therefore also the lines 19-21 are covered.

For me the XML file generated by dotnet-coverage seems to be incorrect.
Maybe you want to create an issue here?

I'm not able to fix the issue within ReportGenerator.

[mschnelte]

HI @danielpalme ,

thank you for the quick reply.
I am not sure if this is the right interpretation of the semantics of a range here.

My understanding of the range

<range source_id="0" covered="yes" start_line="14" start_column="13" end_line="23" end_column="16" />

would be that this is the call of the Task.Run method. This method is called and therefore covered.
I am not really sure what the vstest team could do differently here. If the semantic of a range is in fact one coverable statement then the call to Task.Run is spanning over all this lines and is covered. How should they split it otherwise?

And the fact that these lines are also including an anonymous method that then includes some not covered lines is stated in the xml file as well.
But the DynamicCodeCoverage parser is not acknowledging that. So I would say that this is a bug in the ReportGenerator and not in vstest.

I think this boils down to the question of how the semantics of a range is defined.
I did not find any resources on that? What was your input on this when writing the parser? Did you find any official description of the format?

On a related matter:

A similar issue occurs when changing the output format to cobertura with the collect-coverage:

dotnet-coverage collect -f cobertura "dotnet .\bin\Debug\netcoreapp3.1\AsyncCoverage.dll 12"

Produces the following xml:

<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<coverage line-rate="0.8235294117647058" branch-rate="0.5" complexity="3" version="1.9" timestamp="1638530614" lines-covered="14" lines-valid="17" branches-covered="1" branches-valid="2">
  <sources />
  <packages>
    <package line-rate="0.8235294117647058" branch-rate="0" complexity="3" name="AsyncCoverage">
      <classes>
        <class line-rate="1" branch-rate="1" complexity="1" name="AsyncCoverage.Program" filename="C:\source\coverageSamples\AsyncCoverage\Program.cs">
          <methods>
            <method line-rate="1" branch-rate="1" complexity="1" name="Main" signature="(string[])">
              <lines>
                <line number="10" hits="1" branch="False" />
                <line number="11" hits="1" branch="False" />
                <line number="12" hits="1" branch="False" />
                <line number="13" hits="1" branch="False" />
                <line number="14" hits="1" branch="False" />
                <line number="24" hits="1" branch="False" />
                <line number="25" hits="1" branch="False" />
              </lines>
            </method>
          </methods>
          <lines>
            <line number="10" hits="1" branch="False" />
            <line number="11" hits="1" branch="False" />
            <line number="12" hits="1" branch="False" />
            <line number="13" hits="1" branch="False" />
            <line number="14" hits="1" branch="False" />
            <line number="24" hits="1" branch="False" />
            <line number="25" hits="1" branch="False" />
          </lines>
        </class>
        <class line-rate="0.7" branch-rate="0.5" complexity="2" name="AsyncCoverage.Program.&lt;&gt;c__DisplayClass0_0" filename="C:\source\coverageSamples\AsyncCoverage\Program.cs">
          <methods>
            <method line-rate="0.7" branch-rate="0.5" complexity="2" name="&lt;Main&gt;b__0" signature="()">
              <lines>
                <line number="14" hits="1" branch="False" />
                <line number="15" hits="1" branch="True" condition-coverage="50% (1/2)">
                  <conditions>
                    <condition number="0" type="jump" coverage="50%" />
                  </conditions>
                </line>
                <line number="16" hits="1" branch="False" />
                <line number="17" hits="1" branch="False" />
                <line number="18" hits="1" branch="False" />
                <line number="19" hits="0" branch="False" />
                <line number="20" hits="0" branch="False" />
                <line number="21" hits="0" branch="False" />
                <line number="22" hits="1" branch="False" />
                <line number="23" hits="1" branch="False" />
              </lines>
            </method>
          </methods>
          <lines>
            <line number="14" hits="1" branch="False" />
            <line number="15" hits="1" branch="True" condition-coverage="50% (1/2)">
              <conditions>
                <condition number="0" type="jump" coverage="50%" />
              </conditions>
            </line>
            <line number="16" hits="1" branch="False" />
            <line number="17" hits="1" branch="False" />
            <line number="18" hits="1" branch="False" />
            <line number="19" hits="0" branch="False" />
            <line number="20" hits="0" branch="False" />
            <line number="21" hits="0" branch="False" />
            <line number="22" hits="1" branch="False" />
            <line number="23" hits="1" branch="False" />
          </lines>
        </class>
      </classes>
    </package>
  </packages>
</coverage>

but the parser seems to be ignoring the anonymous method completely.
Creating a report does not show any coverage information whatsoever for lines 15 - 23.

[danielpalme]

First: I don't have an official description of the format. Especially the Coberatura format is quite difficult, since many tools produce output in Cobertura format. Any every tool does it in a slightly different way.

Same is true for anonymous method in your latest Cobertura file.
Here the name of the anonymous class is AsyncCoverage.Program.<>c__DisplayClass0_0. Tools like coverlet or altcover use the following format: AsyncCoverage.Program/<>c__DisplayClass0_0 (note the / instead of .).
This is the reason why ReportGenerator ignores that class. I probably will change that behavior.

Regarding the DynamicCode format:
I'm still not sure that I can do something about this problem. What criteria can be used to ignore the <range source_id="0" covered="yes" start_line="14" start_column="13" end_line="23" end_column="16" /> element?
The DynamicCodeCoverage parser only parses the XML file, it does not know about the class itself.

[danielpalme]

Same is true for anonymous method in your latest Cobertura file.
Here the name of the anonymous class is AsyncCoverage.Program.<>c__DisplayClass0_0. Tools like coverlet or altcover use the following format: AsyncCoverage.Program/<>c__DisplayClass0_0 (note the / instead of .).
This is the reason why ReportGenerator ignores that class. I probably will change that behavior.
I committed a change to support the better support the Cobertura format by dotnet-coverage (1107b2e)

If you have an idea how to handle the problem with the DynamicCode format, let me know.