Zero coverage in .net core 3 when using MassTransit

[ittennull]

I installed .net sdk 3.0.100, now I have zero coverage if I include MassTransit. My system:

coverlet --version
Cross platform .NET Core code coverage tool
1.6.0.0


dotnet --info
.NET Core SDK (reflecting any global.json):
 Version:   3.0.100
 Commit:    04339c3a26

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.18362
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\3.0.100\

Host (useful for support):
  Version: 3.0.0
  Commit:  7d57652f33

.NET Core SDKs installed:
  2.2.402 [C:\Program Files\dotnet\sdk]
  3.0.100 [C:\Program Files\dotnet\sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.1.13 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.2.7 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.13 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.2.7 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.0.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.1.13 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.2.7 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.0.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 3.0.0 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

Here is a sample with reproducible bug: https://gitlab.com/ittennull/coverletdemo
There is one test:

        [Fact]
        public void Test1()
        {
            var service = new Service();
            
            //comment out next line to get some coverage
            service.StartMassTransit(); 

            service.Foo(true);
            service.Foo(false);
        }

If I build the solution and run it like this:

coverlet .\CoverletDemo.Test\bin\Debug\netcoreapp3.0\CoverletDemo.Test.dll --target "dotnet" --targetargs "test CoverletDemo.Test\CoverletDemo.Test.csproj --no-build"
Test run for C:\tmp\CoverletDemo\CoverletDemo.Test\bin\Debug\netcoreapp3.0\CoverletDemo.Test.dll(.NETCoreApp,Version=v3.0)
Microsoft (R) Test Execution Command Line Tool Version 16.3.0
Copyright (c) Microsoft Corporation.  All rights reserved.
Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
10
Test Run Successful.
Total tests: 1
     Passed: 1
 Total time: 2,1295 Seconds

Calculating coverage result...
  Generating report 'C:\tmp\CoverletDemo\coverage.json'
+--------------+------+--------+--------+
| Module       | Line | Branch | Method |
+--------------+------+--------+--------+
| CoverletDemo | 0%   | 0%     | 0%     |
+--------------+------+--------+--------+

+---------+------+--------+--------+
|         | Line | Branch | Method |
+---------+------+--------+--------+
| Total   | 0%   | 0%     | 0%     |
+---------+------+--------+--------+
| Average | 0%   | 0%     | 0%     |
+---------+------+--------+--------+

If I comment out service.StartMassTransit(); then

coverlet .\CoverletDemo.Test\bin\Debug\netcoreapp3.0\CoverletDemo.Test.dll --target "dotnet" --targetargs "test CoverletDemo.Test\CoverletDemo.Test.csproj --no-build"
Test run for C:\tmp\CoverletDemo\CoverletDemo.Test\bin\Debug\netcoreapp3.0\CoverletDemo.Test.dll(.NETCoreApp,Version=v3.0)
Microsoft (R) Test Execution Command Line Tool Version 16.3.0
Copyright (c) Microsoft Corporation.  All rights reserved.
Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
10
Test Run Successful.
Total tests: 1
     Passed: 1
 Total time: 1,4602 Seconds

Calculating coverage result...
  Generating report 'C:\tmp\CoverletDemo\coverage.json'
+--------------+--------+--------+--------+
| Module       | Line   | Branch | Method |
+--------------+--------+--------+--------+
| CoverletDemo | 27,77% | 20%    | 33,33% |
+--------------+--------+--------+--------+

+---------+--------+--------+--------+
|         | Line   | Branch | Method |
+---------+--------+--------+--------+
| Total   | 27,77% | 20%    | 33,33% |
+---------+--------+--------+--------+
| Average | 27,77% | 20%    | 33,33% |
+---------+--------+--------+--------+

[MarcoRossignoli]

Could be a dup of #506 but thak's for repro...now I can test on my local.

[MarcoRossignoli]

@ittennull I found the issue.
It's a know issue of interaction between vstest and exit process.
To solve you have to use "collectors" and not dotnet tool.
Again during debug I found also a possible bug on vstest platform(dotnet test) so you need to pass a custom "runsetting" file.
Try with this:

1. reference collectors

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>netcoreapp2.2</TargetFramework>

    <IsPackable>false</IsPackable>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="coverlet.collector" Version="1.1.0" />
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.3.0" />
    <PackageReference Include="xunit" Version="2.4.1" />
    <PackageReference Include="xunit.runner.visualstudio" Version="2.4.1">
      <PrivateAssets>all</PrivateAssets>
      <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
    </PackageReference>
  </ItemGroup>

  <ItemGroup>
    <ProjectReference Include="..\CoverletDemo\CoverletDemo.csproj" />
  </ItemGroup>

</Project>

2. Add custom runsetting file in the same place where you run test command

<?xml version="1.0" encoding="utf-8" ?>
<RunSettings>
  <DataCollectionRunSettings>
    <DataCollectors>
      <DataCollector friendlyName="XPlat code coverage">
        <Configuration>
          <Format>cobertura</Format>
        </Configuration>
      </DataCollector>
    </DataCollectors>
  </DataCollectionRunSettings>
  <InProcDataCollectionRunSettings>
    <InProcDataCollectors>
      <InProcDataCollector assemblyQualifiedName="Coverlet.Collector.DataCollection.CoverletInProcDataCollector, coverlet.collector, Version=1.1.0.0, Culture=neutral, PublicKeyToken=null"
                     friendlyName="XPlat Code Coverage"
                     enabled="True"
                     codebase="coverlet.collector.dll" />
    </InProcDataCollectors>
  </InProcDataCollectionRunSettings>
</RunSettings>

And use integrated collector https://github.com/tonerdo/coverlet#vstest-integration

C:\git\coverletissue\coverletdemo-master\CoverletDemo.Test                                                                                      
λ dotnet test --collect:"XPlat Code Coverage" --settings runsettings                                                                            
Test run for C:\git\coverletissue\coverletdemo-master\CoverletDemo.Test\bin\Debug\netcoreapp2.2\CoverletDemo.Test.dll(.NETCoreApp,Version=v2.2) 
Microsoft (R) Test Execution Command Line Tool Version 16.2.0-preview-20190606-02                                                               
Copyright (c) Microsoft Corporation.  All rights reserved.                                                                                      
                                                                                                                                                
Starting test execution, please wait...                                                                                                         
                                                                                                                                                
                                                                                                                                                
Attachments:                                                                                                                                    
  C:\git\coverletissue\coverletdemo-master\CoverletDemo.Test\TestResults\2c5e3f36-7ced-46eb-926a-b87909ab0eec\coverage.cobertura.xml            
                                                                                                                                                
Test Run Successful.                                                                                                                            
Total tests: 1                                                                                                                                  
     Passed: 1                                                                                                                                  

On my local will be generated a valid cobertura file with hit on lines

<?xml version="1.0" encoding="utf-8"?>
<coverage line-rate="0.2" branch-rate="0.2" version="1.9" timestamp="1569853633" lines-covered="5" lines-valid="25" branches-covered="2" branches-valid="10">
  <sources>
    <source></source>
  </sources>
  <packages>
    <package name="CoverletDemo" line-rate="0.2" branch-rate="0.2" complexity="11">
      <classes>
        <class name="CoverletDemo.Service" filename="C:\git\coverletissue\coverletdemo-master\CoverletDemo\Service.cs" line-rate="0.2083" branch-rate="0.5" complexity="4">
          <methods>
            <method name="Foo" signature="(System.Boolean)" line-rate="1" branch-rate="1">
              <lines>
                <line number="9" hits="2" branch="False" />
                <line number="10" hits="2" branch="True" condition-coverage="100% (2/2)">
                  <conditions>
                    <condition number="4" type="jump" coverage="100%" />
                  </conditions>
                </line>
                <line number="11" hits="1" branch="False" />
                <line number="13" hits="1" branch="False" />
                <line number="14" hits="2" branch="False" />
              </lines>
            </method>
            <method name="StartMassTransit" signature="()" line-rate="0" branch-rate="0">
              <lines>
                <line number="17" hits="0" branch="False" />
                <line number="18" hits="0" branch="True" condition-coverage="0% (0/2)">
                  <conditions>
                    <condition number="12" type="jump" coverage="0%" />
                  </conditions>
                </line>
                <line number="19" hits="0" branch="False" />
                <line number="20" hits="0" branch="False" />
                <line number="21" hits="0" branch="False" />
                <line number="22" hits="0" branch="False" />
                <line number="23" hits="0" branch="False" />
                <line number="24" hits="0" branch="False" />
                <line number="25" hits="0" branch="False" />
                <line number="26" hits="0" branch="False" />
                <line number="27" hits="0" branch="False" />
                <line number="28" hits="0" branch="False" />
                <line number="29" hits="0" branch="False" />
                <line number="30" hits="0" branch="False" />
                <line number="31" hits="0" branch="False" />
...

Unfortunately using collector we cannot show percentage so you need to use some tool like report generator.

TL;DR;

You have to use collector because when test is end rabbitMQ is slow to "close" this lead vstest plat to "kill" process.
For msbuild and dotnet tool we use a pair of AppDomain event to trigger hit file flush(where we collect hits). But vstest by design after some time kill host process if pending and so we cannot flush file to disk.
No file result in 0% of coverage.
With collector we're integrated to vstest so there is a communication between testhost and vstest, vstest doesn't close host process until we flush file.

@fkucuk I think that this is the reason also for your issue...you should try this way.

[dmitry-semenov-rtl]

Yes, it works as you suggested.
Is there a way to do it without adding to csproj?

[MarcoRossignoli]

Is there a way to do it without adding to csproj?

What does it mean?
VsTest platform must find and load collectors so the ref to <PackageReference Include="coverlet.collector" Version="1.1.0" /> is mandatory and it's the default for new dotnet xunit template

dotnet new xunit

All other reference to coverlet are unuseful.
And the BEST WAY TO USE COVERLET TODAY if possible is through collectors because they're fully integrated inside vstest pipeline.
Collectors are young and there are some issue(way to pass parameter with running settings file and not by command line and a new issue on load upgraded collectors), but to me will be the future.

[dmitry-semenov-rtl]

Ok, sounds clear. I asked because previously we didn't reference any collectors from test csproj files and coverlet dotnet tool somehow just worked. I got it that now it's mandatory. I will try to update our pipelines using new rules

[MarcoRossignoli]

coverlet dotnet tool somehow just worked

TL;DR;

Yep, coverlet tool is a simple wrapper on core instrumentation steps https://github.com/tonerdo/coverlet#how-it-works
Actually tool instrument dll and run some "user choice runner" i.e. dotnet test/vstest | vstest.console.exe etc... and when that process exits we collect coverage.
So the issue with test platform remains, collectors is the unique way to tell to test plat "don't kill me just wait untill I've finished my job".
So we're sure that collectors works always in a deterministic way, tool and msbuild could suffer from this issue in a random way(i.e. in this case rabbit shutdown is slow and it's unrelated to coverlet)

[ittennull]

Thanks, I managed to get it work as needed