Code coverage of System.Private.CoreLib is broken #51058

stephentoub · 2021-04-10T23:57:30Z

According to the directions, running a test suite like System.Runtime.Tests should be able to get coverage of the code in corelib via /p:Coverage=true /p:TestRuntime=true, but while doing so does add CoreLib to the command-line, the resulting code coverage report is empty.

cc: @ViktorHofer

dotnet-issue-labeler · 2021-04-10T23:57:34Z

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

ghost · 2021-04-11T16:50:42Z

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

Issue Details

According to the directions, running a test suite like System.Runtime.Tests should be able to get coverage of the code in corelib via /p:Coverage=true /p:TestRuntime=true, but while doing so does add CoreLib to the command-line, the resulting code coverage report is empty.

cc: @ViktorHofer

Author:	stephentoub
Assignees:	-
Labels:	`area-Infrastructure`, `untriaged`
Milestone:	-

danmoseley · 2021-04-11T23:10:14Z

This has broken enough times, ideally we'd protect it with an outerloop test (just as we should protect launch and build in VS). As far as I know we have no existing example of a test that does a build, but presumably we can assume dotnet is on the path (at least we do in the build)

trylek · 2021-04-12T00:35:06Z

For Helix executions of CoreCLR tests Simon added provisions for passing dotnet path around for the purpose of running Crossgen2, please see here:

runtime/src/tests/run.py

Line 909 in 94641d8

os.environ["__TestDotNetCmd"] = args.dotnetcli_script_path

runtime/src/tests/Common/CLRTest.CrossGen.targets

Line 114 in ef6c694

if [ ! -z ${__TestDotNetCmd+x} ] %3B then

I guess it shouldn't be that hard to introduce a similar logic to library tests.

ViktorHofer · 2021-04-21T16:30:14Z

Marking as 6.0 and as a bug. Will look into this as soon as I have some cycles. Assigning to myself.

stephentoub · 2022-01-19T02:04:28Z

@danmoseley, is there someone who can look at this in Viktor's absence?

danmoseley · 2022-01-19T03:47:45Z

@ericstj owns our infra backlog.

ViktorHofer · 2022-01-31T09:59:40Z

CoreLib code coverage measurement is broken because the following negative path is hit during coverlet's CanInstrument() check: https://github.com/coverlet-coverage/coverlet/blob/df968b862687aeb9ac86ae47938da04d95c10452/src/coverlet.core/Helpers/InstrumentationHelper.cs#L191.

coverlet.core.dll!Coverlet.Core.Helpers.InstrumentationHelper.MatchDocumentsWithSources(System.Reflection.Metadata.MetadataReader metadataReader) Line 191 C#
coverlet.core.dll!Coverlet.Core.Helpers.InstrumentationHelper.PortablePdbHasLocalSource(string module, out string firstNotFoundDocument) Line 162 C#
coverlet.core.dll!Coverlet.Core.Instrumentation.Instrumenter.CanInstrument() Line 108 C#
coverlet.core.dll!Coverlet.Core.Coverage.PrepareModules() Line 128 C#
coverlet.console.dll!Coverlet.Console.Program.Main.AnonymousMethod__1() Line 112 C#
[External Code]
coverlet.console.dll!Coverlet.Console.Program.Main(string[] args) Line 299 C#

Coverlet fails to find the generated ArayPoolEventSource file on disk: C:\git\runtime3\src\coreclr\System.Private.CoreLib\System.Private.CoreLib.Generators\Generators.EventSourceGenerator\ArrayPoolEventSource.Generated.cs.

The options that I see are:

Change coverlet's MatchDocumentsWithSources code path to also exclude ".Generated.cs" files or come up with a better heuristic. Potentially breaking customers.
Change the suffix of the generated code files in dotnet/runtime from .Generated.cs to .g.cs so that they are excluded by coverlet.

FWIW this was easy to debug by building coverlet myself, adding a Debugger.Attach() call and then invoking the coverlet.console.dll with the dotnet.exe from the testhost:
C:\git\runtime3\.dotnet\dotnet.exe C:\git\coverlet\src\coverlet.console\bin\Debug\net5.0\coverlet.console.dll "System.Runtime.Tests.dll" --target "C:\git\runtime3\artifacts\bin\testhost\net7.0-windows-Debug-x64\dotnet.exe" --targetargs " exec --runtimeconfig System.Runtime.Tests.runtimeconfig.json --depsfile System.Runtime.Tests.deps.json xunit.console.dll System.Runtime.Tests.dll -xml testResults.xml -nologo -notrait category=OuterLoop -notrait category=failing" --format "opencover" --output "coverage.opencover.xml" --verbosity "normal" --use-source-link --exclude-by-file "C:\git\runtime3\src\libraries\Common\src\System\SR.*" --exclude-by-file "C:\git\runtime3\src\libraries\Common\src\System\NotImplemented.cs" --include-directory "C:\git\runtime3\artifacts\bin\testhost\net7.0-windows-Debug-x64\shared\Microsoft.NETCore.App\7.0.0" --include "[System.Runtime]*" --include "[System.Private.CoreLib]*"

I hope that helps :)

stephentoub · 2022-01-31T13:29:24Z

Thanks.

CoreLib code coverage measurement is broken because the following negative path is hit during coverlet's CanInstrument()

Fixing that (#64534) helps but does not solve the problem. Everything in corelib still shows as 0%.

ViktorHofer · 2022-01-31T15:05:35Z

I believe the remaining issue is that the instrumented IL code is at the wrong place:

	internal static void OnProcessExit()
	{
		Coverlet.Core.Instrumentation.Tracker.System.Private.CoreLib_a5472523-6bb8-47da-bbb9-359dc829635b.RecordHitInCoreLibrary(20343);
		AssemblyLoadContext.OnProcessExit();
		Coverlet.Core.Instrumentation.Tracker.System.Private.CoreLib_a5472523-6bb8-47da-bbb9-359dc829635b.RecordHitInCoreLibrary(20344);
		if (EventSource.IsSupported)
		{
			Coverlet.Core.Instrumentation.Tracker.System.Private.CoreLib_a5472523-6bb8-47da-bbb9-359dc829635b.RecordHitInCoreLibrary(20345);
			Coverlet.Core.Instrumentation.Tracker.System.Private.CoreLib_a5472523-6bb8-47da-bbb9-359dc829635b.RecordHitInCoreLibrary(20346);
			EventListener.DisposeOnShutdown();
		}
		Coverlet.Core.Instrumentation.Tracker.System.Private.CoreLib_a5472523-6bb8-47da-bbb9-359dc829635b.RecordHitInCoreLibrary(20347);
		Coverlet.Core.Instrumentation.Tracker.System.Private.CoreLib_a5472523-6bb8-47da-bbb9-359dc829635b.RecordHitInCoreLibrary(20348);
		EventHandler processExit = AppContext.ProcessExit;
		if (processExit == null)
		{
			Coverlet.Core.Instrumentation.Tracker.System.Private.CoreLib_a5472523-6bb8-47da-bbb9-359dc829635b.RecordHitInCoreLibrary(20349);
		}
		else
		{
			Coverlet.Core.Instrumentation.Tracker.System.Private.CoreLib_a5472523-6bb8-47da-bbb9-359dc829635b.RecordHitInCoreLibrary(20350);
			processExit(AppDomain.CurrentDomain, EventArgs.Empty);
			Coverlet.Core.Instrumentation.Tracker.System.Private.CoreLib_a5472523-6bb8-47da-bbb9-359dc829635b.RecordHitInCoreLibrary(20351);
			Coverlet.Core.Instrumentation.Tracker.System.Private.CoreLib_a5472523-6bb8-47da-bbb9-359dc829635b.UnloadModule(null, null);
		}
	}

This is from the ildasm my instrumented System.Private.CoreLib assembly. The code indicates that the processExit branch is hit and the Coverlet.Core.Instrumentation.Tracker.System.Private.CoreLib_a5472523-6bb8-47da-bbb9-359dc829635b.UnloadModule method is never invoked and hence the result file is never written.

EDIT: The code responsible for injecting the UnloadModule method can be found here: https://github.com/coverlet-coverage/coverlet/blob/df968b862687aeb9ac86ae47938da04d95c10452/src/coverlet.core/Instrumentation/Instrumenter.cs#L303.

stephentoub · 2022-01-31T15:19:05Z

@sharwell, seems that change came from coverlet-coverage/coverlet@96b1594. Can you take a look?

ViktorHofer · 2022-01-31T15:19:22Z

Yes that was it. If I change

        internal static void OnProcessExit()
        {
            AssemblyLoadContext.OnProcessExit();
            if (EventSource.IsSupported)
            {
                EventListener.DisposeOnShutdown();
            }


-            ProcessExit?.Invoke(AppDomain.CurrentDomain, EventArgs.Empty);
+           // Don't optimize this code path to inline the != null check via the ?. operator as otherwise
+           // coverlet can't inject the UnloadModule instrumentation code path and CoreLib's coverage results will be empty.
+           if (ProcessExit != null)
+           {
+               ProcessExit.Invoke(AppDomain.CurrentDomain, EventArgs.Empty);
+           }
        }

internal static void OnProcessExit()
{
	Coverlet.Core.Instrumentation.Tracker.System.Private.CoreLib_0612f6fa-768c-4cd7-9fc6-d31e1f1280a5.RecordHitInCoreLibrary(20343);
	AssemblyLoadContext.OnProcessExit();
	Coverlet.Core.Instrumentation.Tracker.System.Private.CoreLib_0612f6fa-768c-4cd7-9fc6-d31e1f1280a5.RecordHitInCoreLibrary(20344);
	if (EventSource.IsSupported)
	{
		Coverlet.Core.Instrumentation.Tracker.System.Private.CoreLib_0612f6fa-768c-4cd7-9fc6-d31e1f1280a5.RecordHitInCoreLibrary(20345);
		Coverlet.Core.Instrumentation.Tracker.System.Private.CoreLib_0612f6fa-768c-4cd7-9fc6-d31e1f1280a5.RecordHitInCoreLibrary(20346);
		EventListener.DisposeOnShutdown();
	}
	Coverlet.Core.Instrumentation.Tracker.System.Private.CoreLib_0612f6fa-768c-4cd7-9fc6-d31e1f1280a5.RecordHitInCoreLibrary(20347);
	Coverlet.Core.Instrumentation.Tracker.System.Private.CoreLib_0612f6fa-768c-4cd7-9fc6-d31e1f1280a5.RecordHitInCoreLibrary(20348);
	if (AppContext.ProcessExit != null)
	{
		Coverlet.Core.Instrumentation.Tracker.System.Private.CoreLib_0612f6fa-768c-4cd7-9fc6-d31e1f1280a5.RecordHitInCoreLibrary(20349);
		Coverlet.Core.Instrumentation.Tracker.System.Private.CoreLib_0612f6fa-768c-4cd7-9fc6-d31e1f1280a5.RecordHitInCoreLibrary(20350);
		AppContext.ProcessExit(AppDomain.CurrentDomain, EventArgs.Empty);
	}
	Coverlet.Core.Instrumentation.Tracker.System.Private.CoreLib_0612f6fa-768c-4cd7-9fc6-d31e1f1280a5.RecordHitInCoreLibrary(20351);
	Coverlet.Core.Instrumentation.Tracker.System.Private.CoreLib_0612f6fa-768c-4cd7-9fc6-d31e1f1280a5.RecordHitInCoreLibrary(20352);
	Coverlet.Core.Instrumentation.Tracker.System.Private.CoreLib_0612f6fa-768c-4cd7-9fc6-d31e1f1280a5.UnloadModule(null, null);
}

ViktorHofer · 2022-01-31T15:45:05Z

There is one more remaining issue which unfortunately needs to be fixed in coverlet itself: coverlet-coverage/coverlet#1285. If I fix that locally and apply the AppContext.ProcessExit workaround from above I finally get code coverage results for CoreLib:

Calculating coverage result...
  Generating report 'C:\git\runtime3\artifacts\bin\System.Runtime.Tests\net7.0-Browser-Debug\coverage.opencover.xml'
+------------------------+--------+--------+--------+
| Module                 | Line   | Branch | Method |
+------------------------+--------+--------+--------+
| System.Private.CoreLib | 41.12% | 38.28% | 33.73% |
+------------------------+--------+--------+--------+
| System.Runtime         | 100%   | 100%   | 100%   |
+------------------------+--------+--------+--------+

+---------+--------+--------+--------+
|         | Line   | Branch | Method |
+---------+--------+--------+--------+
| Total   | 41.12% | 38.28% | 33.73% |
+---------+--------+--------+--------+
| Average | 70.56% | 69.14% | 66.86% |
+---------+--------+--------+--------+

Unfortunately I don't have any more time to help out here but I think that all the information needed is now on the table.

jkotas · 2022-01-31T15:52:22Z

AppContext.ProcessExit workaround

Should this better be fixed in Coverlet too?

danmoseley · 2022-01-31T16:08:43Z

Presumably this broke when we picked up the previous coverlet 1.7.2? If we ensure we always verify corelib coverage before updating our coverlet version, we should at least be protected against coverlet breaks, presumably?

Incidentally apparently this workaround can be removed now.
https://github.com/danmoseley/runtime/blob/9df283248256dd880eb1ee5ecce533ad206a55d3/eng/testing/coverage.targets#L62

stephentoub · 2022-01-31T16:44:01Z

Should this better be fixed in Coverlet too?

Yes 😄. cc: @sharwell, @MarcoRossignoli

stephentoub · 2022-01-31T16:44:26Z

Unfortunately I don't have any more time to help out here but I think that all the information needed is now on the table.

Thanks for getting to it, Viktor.

MarcoRossignoli · 2022-01-31T21:54:21Z

Should this better be fixed in Coverlet too?

I'll take a look asap(weekend I think).

Thanks @ViktorHofer 🙇

MarcoRossignoli · 2022-02-05T18:41:28Z

I did a pair of fixes for the issues above, can someone with more "knowledge" and a ready runtime env(build etc...) give it a try before official release? I'm a bit rusty at the moment.

https://github.com/coverlet-coverage/coverlet/blob/master/Documentation/ConsumeNightlyBuild.md

Will be shipped at midnight.

stephentoub · 2022-02-05T18:48:16Z

Thanks, @MarcoRossignoli.

Will be shipped at midnight.

That's on the nightly feed. What about official builds?

MarcoRossignoli · 2022-02-05T18:51:26Z

I do manual deploy for official build. I did one a few days ago 1/30...so the only difference between 3.1.1 and 3.1.2 will be this fix...for this reason I'd like to know before the release if it's working to avoid to ship an useful bugged version.
If it works I'll release asap(in few days).

I've fired manual dogfood release the version with the fix is 3.1.2-preview.4.gc8eafe5297 on https://pkgs.dev.azure.com/tonerdo/coverlet/_packaging/coverlet-nightly/nuget/v3/index.json

stephentoub · 2022-02-05T19:07:49Z

I've fired manual dogfood release the version with the fix is 3.1.2-preview.4.gc8eafe5297

Tried it:

  Unhandled exception. System.InvalidProgramException: Common Language Runtime detected an invalid program.
     at System.AppContext.OnProcessExit()

MarcoRossignoli · 2022-02-05T19:09:00Z

interesting...pls can you send here command list to build/run coverage, so I can go by my own

stephentoub · 2022-02-05T19:11:25Z

Apply stephentoub@009a5af to your dotnet/runtime repo clone.
(You might need to also dotnet tool restore after doing that.)
Build the repo: .\build clr+libs -rc release
Change into a test directory: cd src\libraries\System.Reflection.Emit\tests
Run tests with coverage: dotnet build /t:test /p:Coverage=true

MarcoRossignoli · 2022-02-05T19:13:12Z

thx I'll be back asap(I have my suspect).

MarcoRossignoli · 2022-02-05T22:14:15Z

looks good now I see the report(cannot evaluate the quality so if you see something strange let me know)

I'll ship the new version asap.

MarcoRossignoli · 2022-02-06T16:28:34Z

Release 3.1.2 is out you can give it a try.

dotnet-issue-labeler bot added the untriaged New issue has not been triaged by the area owner label Apr 10, 2021

jeffschwMSFT added the area-Infrastructure label Apr 11, 2021

stephentoub added this to the 6.0.0 milestone Apr 11, 2021

stephentoub mentioned this issue Apr 13, 2021

Add S.R.CompilerServices.InterpolatedStringBuilder #51086

Merged

ViktorHofer removed the untriaged New issue has not been triaged by the area owner label Apr 21, 2021

ViktorHofer added the bug label Apr 21, 2021

ViktorHofer self-assigned this Apr 21, 2021

Anipik modified the milestones: 6.0.0, 7.0.0 Jul 21, 2021

jozkee mentioned this issue Aug 31, 2021

Fix detection of unsupported links #58391

Merged

safern unassigned ViktorHofer Jan 19, 2022

stephentoub mentioned this issue Jan 31, 2022

Fix EventSourceGenerator generated file extension #64534

Merged

ViktorHofer mentioned this issue Jan 31, 2022

CoreLib's coverage measurement is broken coverlet-coverage/coverlet#1285

Closed

MarcoRossignoli mentioned this issue Jan 31, 2022

Fix UnloadModule injection coverlet-coverage/coverlet#1287

Closed

MarcoRossignoli mentioned this issue Feb 6, 2022

Bump coverlet version #64875

Merged

ghost added in-pr There is an active PR which will close this issue when it is merged and removed in-pr There is an active PR which will close this issue when it is merged labels Feb 6, 2022

stephentoub closed this as completed in #64875 Feb 7, 2022

ghost removed the in-pr There is an active PR which will close this issue when it is merged label Feb 7, 2022

ghost locked as resolved and limited conversation to collaborators Mar 9, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Code coverage of System.Private.CoreLib is broken #51058

Code coverage of System.Private.CoreLib is broken #51058

stephentoub commented Apr 10, 2021

dotnet-issue-labeler bot commented Apr 10, 2021

ghost commented Apr 11, 2021

danmoseley commented Apr 11, 2021

trylek commented Apr 12, 2021

ViktorHofer commented Apr 21, 2021

stephentoub commented Jan 19, 2022

danmoseley commented Jan 19, 2022

ViktorHofer commented Jan 31, 2022 •

edited

Loading

stephentoub commented Jan 31, 2022 •

edited

Loading

ViktorHofer commented Jan 31, 2022 •

edited

Loading

stephentoub commented Jan 31, 2022

ViktorHofer commented Jan 31, 2022 •

edited

Loading

ViktorHofer commented Jan 31, 2022 •

edited

Loading

jkotas commented Jan 31, 2022

danmoseley commented Jan 31, 2022

stephentoub commented Jan 31, 2022 •

edited

Loading

stephentoub commented Jan 31, 2022

MarcoRossignoli commented Jan 31, 2022 •

edited

Loading

MarcoRossignoli commented Feb 5, 2022 •

edited

Loading

stephentoub commented Feb 5, 2022

MarcoRossignoli commented Feb 5, 2022 •

edited

Loading

stephentoub commented Feb 5, 2022

MarcoRossignoli commented Feb 5, 2022 •

edited

Loading

stephentoub commented Feb 5, 2022 •

edited

Loading

MarcoRossignoli commented Feb 5, 2022 •

edited

Loading

MarcoRossignoli commented Feb 5, 2022 •

edited

Loading

MarcoRossignoli commented Feb 6, 2022

Code coverage of System.Private.CoreLib is broken #51058

Code coverage of System.Private.CoreLib is broken #51058

Comments

stephentoub commented Apr 10, 2021

dotnet-issue-labeler bot commented Apr 10, 2021

ghost commented Apr 11, 2021

danmoseley commented Apr 11, 2021

trylek commented Apr 12, 2021

ViktorHofer commented Apr 21, 2021

stephentoub commented Jan 19, 2022

danmoseley commented Jan 19, 2022

ViktorHofer commented Jan 31, 2022 • edited Loading

stephentoub commented Jan 31, 2022 • edited Loading

ViktorHofer commented Jan 31, 2022 • edited Loading

stephentoub commented Jan 31, 2022

ViktorHofer commented Jan 31, 2022 • edited Loading

ViktorHofer commented Jan 31, 2022 • edited Loading

jkotas commented Jan 31, 2022

danmoseley commented Jan 31, 2022

stephentoub commented Jan 31, 2022 • edited Loading

stephentoub commented Jan 31, 2022

MarcoRossignoli commented Jan 31, 2022 • edited Loading

MarcoRossignoli commented Feb 5, 2022 • edited Loading

stephentoub commented Feb 5, 2022

MarcoRossignoli commented Feb 5, 2022 • edited Loading

stephentoub commented Feb 5, 2022

MarcoRossignoli commented Feb 5, 2022 • edited Loading

stephentoub commented Feb 5, 2022 • edited Loading

MarcoRossignoli commented Feb 5, 2022 • edited Loading

MarcoRossignoli commented Feb 5, 2022 • edited Loading

MarcoRossignoli commented Feb 6, 2022

ViktorHofer commented Jan 31, 2022 •

edited

Loading

stephentoub commented Jan 31, 2022 •

edited

Loading

ViktorHofer commented Jan 31, 2022 •

edited

Loading

ViktorHofer commented Jan 31, 2022 •

edited

Loading

ViktorHofer commented Jan 31, 2022 •

edited

Loading

stephentoub commented Jan 31, 2022 •

edited

Loading

MarcoRossignoli commented Jan 31, 2022 •

edited

Loading

MarcoRossignoli commented Feb 5, 2022 •

edited

Loading

MarcoRossignoli commented Feb 5, 2022 •

edited

Loading

MarcoRossignoli commented Feb 5, 2022 •

edited

Loading

stephentoub commented Feb 5, 2022 •

edited

Loading

MarcoRossignoli commented Feb 5, 2022 •

edited

Loading

MarcoRossignoli commented Feb 5, 2022 •

edited

Loading