0% coverage of library when using source generator

[dimarusyy]

Hello,

I have a library targeted .Net Standard 2.0 and a test project using NUnit and targeted .Net 5.0.
I used coverlet to calculate code-coverage as described in https://docs.microsoft.com/en-us/dotnet/core/testing/unit-testing-code-coverage?tabs=windows.

Recently I started to use ISourceGenerator to generate some files during build.
After that I noticed that code coverage emits coverage.cobertura.xml file with the following content :

<?xml version="1.0" encoding="utf-8"?>
<coverage line-rate="0" branch-rate="0" version="1.9" timestamp="1638209313" lines-covered="0" lines-valid="0" branches-covered="0" branches-valid="0">
  <sources>
    <source></source>
  </sources>
  <packages />

I played much with coverlet.collector v3.1.0 and coverlet.msbuild v3.1.0 but always got an empty report (0 lines being covered).
Here is an output of command :

# dotnet test --configuration Debug /p:CollectCoverage=true --diag:log.txt --settings runsettings.xml
  Determining projects to restore...
  All projects are up-to-date for restore.
  Assure.EnumsGenerator -> ../Assure.EnumsGenerator/bin/Debug/netstandard2.0/Assure.EnumsGenerator.dll
  Assure -> ../dotnet/Assure/bin/Debug/netstandard2.0/Assure.dll
  Assure.Tests -> ../Assure.Tests/bin/Debug/net5.0/Assure.Tests.dll
Test run for ../Assure.Tests/bin/Debug/net5.0/Assure.Tests.dll (.NETCoreApp,Version=v5.0)
Microsoft (R) Test Execution Command Line Tool Version 16.11.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
Logging Vstest Diagnostics in file: ../log.txt
A total of 1 test files matched the specified pattern.

Attachments:
  ../Assure.Tests/TestResults/777aab11-c4d4-45b7-9acf-639bc68fa0bd/coverage.info
  ../Assure.Tests/TestResults/777aab11-c4d4-45b7-9acf-639bc68fa0bd/coverage.cobertura.xml
Passed!  - Failed:     0, Passed:  2029, Skipped:     0, Total:  2029, Duration: 764 ms - ../Assure.Tests/bin/Debug/net5.0/Assure.Tests.dll (net5.0)

Calculating coverage result...
  Generating report ../Assure.Tests/coverage.json'

+--------+------+--------+--------+
| Module | Line | Branch | Method |
+--------+------+--------+--------+

+---------+------+--------+--------+
|         | Line | Branch | Method |
+---------+------+--------+--------+
| Total   | 0%   | 0%     | 0%     |
+---------+------+--------+--------+
| Average | 0%   | 0%     | 0%     |
+---------+------+--------+--------+

#

File log.datacollector.21-11-29_18-08-30_12299_4.txt keeps a message :

TpTrace Verbose: 0 : 21376, 1, 2021/11/29, 18:08:30.641, 90805477623404, datacollector.dll, [coverlet]Unable to instrument module: ../Assure.Tests/bin/Debug/net5.0/Assure.dll, pdb without local source files, [.../Assure/Assure.EnumsGenerator/Assure.EnumsGenerator.EnumsGenerator/AccessRight.cs]
TpTrace Verbose: 0 : 21376, 1, 2021/11/29, 18:08:30.695, 90805531432922, datacollector.dll, [coverlet]Unable to instrument module: .../Assure.Tests/bin/Debug/net5.0/Moq.dll, embedded pdb without local source files, [C:\projects\moq4\src\Moq\obj\TypeNameFormatter.cs]
TpTrace Information: 0 : 21376, 1, 2021/11/29, 18:08:30.734, 90805570278990, datacollector.dll, DataCollectionRequestHandler.ProcessRequests : DataCollection started.
TpTrace Information: 0 : 21376, 1, 2021/11/29, 18:08:30.765, 90805601531262, datacollector.dll, DataCollectionRequestHandler.ProcessRequests : Datacollector received message: (DataCollection.TestHostLaunched) -> {
  "ProcessId": 21394
}
TpTrace Information: 0 : 21376, 1, 2021/11/29, 18:08:33.808, 90808644847167, datacollector.dll, DataCollectionRequestHandler.ProcessRequests : Datacollector received message: (DataCollection.AfterTestRunEnd) -> false
TpTrace Verbose: 0 : 21376, 1, 2021/11/29, 18:08:33.809, 90808645919498, datacollector.dll, [coverlet]CoverletCoverageDataCollector: SessionEnd received

In log.host.21-11-29_18-08-30_74105_4.txt I noticed a bunch of messages like :

TpTrace Warning: 0 : 21394, 13, 2021/11/29, 18:08:31.922, 90806758548843, testhost.dll, TestRunCache: No test found corresponding to testResult 'CreateSubclasses(Assure.Common.ServerAlreadyActivatedError: [AlreadyExists] : ,AlreadyExists)' in inProgress list.

Could somebody please assist to understand what is wrong ?
Also, coverlet works fine when I remove compile-time code-generation.

PS :
runsettings.xml :

<?xml version="1.0" encoding="utf-8"?>

<RunSettings>
	<DataCollectionRunSettings>
		<DataCollectors>
			<DataCollector friendlyName="XPlat code coverage">
				<Configuration>
					<Format>lcov,cobertura</Format>
					<UseSourceLink>true</UseSourceLink>
					<Include>Assure.Client*</Include>
				</Configuration>
			</DataCollector>
		</DataCollectors>
	</DataCollectionRunSettings>
</RunSettings> 

[Andreas-Huber]

I have the same problem using XUnit and .NET 5.0 projects.
There is no indication that something went wrong in the logs, including the MSBuild bin logs - or at least I could not find it.
I created a sample project, to demonstrate the problem: https://github.com/Andreas-Huber/reproduce-coverlet-coverage-issue-1262

Run the code coverage, and the resulting coverage.cobertura.xml file will be empty.
dotnet test -c Release --collect:"XPlat Code Coverage"

To show it is working without the generators, remove the following line from Playground.csproj. IDE might need a restart for it to work. Then the code coverage report is filled.

<ProjectReference Include="..\Playground.CodeGeneration\Playground.CodeGeneration.csproj" OutputItemType="Analyzer" ReferenceOutputAssembly="false" />

The code that is generated has no influence when it is added without the generator. I also tried various ways to call the generators (--no-build), or to ignore generated files a settings file. Unfortunately, I did manage to find a workaround.

[petli]

TpTrace Verbose: 0 : 21376, 1, 2021/11/29, 18:08:30.641, 90805477623404, datacollector.dll, [coverlet]Unable to instrument module: ../Assure.Tests/bin/Debug/net5.0/Assure.dll, pdb without local source files, [.../Assure/Assure.EnumsGenerator/Assure.EnumsGenerator.EnumsGenerator/AccessRight.cs]

This may indicate the issue, but very obliquely. Coverlet uses a heuristic to determine which libraries are relevant to instrument, to avoid instrumenting third-party libraries. This is to look for the source code files, and if not all of them are found consider the DLL external.

Source generators produce source files that coverlet can't see, so that will lead to this issue. However, it knows about the conventional suffix .g.cs, so if you make sure the generated source files have this suffix you should be able to get coverage. In the example above, it should probably be AccessRight.g.cs.

I've logged an issue on making this heuristic configurably less harsh, but it is yet open since I haven't had time to work on it (nor anyone else): #1164

[dimarusyy]

The code that is generated has no influence when it is added without the generator. I also tried various ways to call the generators (--no-build), or to ignore generated files a settings file. Unfortunately, I did manage to find a workaround.

@Andreas-Huber , thanks for reply.
I tried your test-project and somehow I got non-empty coverlet report :

<coverage line-rate="0.5" branch-rate="1" version="1.9" timestamp="1638270724" lines-covered="1" lines-valid="2" branches-covered="0" branches-valid="0">

I'm just wondering if the issue is related to suffix .g.cs in generated files. In my project I used only .cs.
My environment : image: mcr.microsoft.com/dotnet/sdk:5.0.

[Andreas-Huber]

Ah sorry, I was already ahead and pushed a new commit after Peters comment with the .g.cs suffix - it was not there before.
So that seems to fix it in the playground project.
I was just about to test that in the production project because I thought adding the .g.cs suffix was one of the things I tried already.

So in this case adding .g.cs definitely fixed it.

[Andreas-Huber]

@petli Thanks again for the explanation. The .g.cs suffix fixed the issue for me everywhere.

[dimarusyy]

Thanks for help! Changing filename to .g.cs fixed the issue for me too.