Coverlet fails to instrument F# projects that have "unknown" source (inline functions, anonymous records etc)

[da9l]

I have a F# solution with a web sdk api project, two shared libraries and some windows services.
Coverlet sucessfully instrument and analyze one of the shared libraries and the windows services but fails to instrument the api project and one of the libraries.

I've seen that there are similar issues here in the repo and while this could be a duplicate of one of those I have two detail that I think sets this apart.

• it works for some projects in my solution and not for others. See below.
• I get the "unknown" file reference in the Instrumentation log entry for the failing projects none of the other issues does that. See below.

I've tested with all three variants of coverlet 3.0.3: msbuild, collector and console and all three give the same result.
Inspecting the logs I see this for the failed instrumentations:

Unsuccessful library and unsuccessful api project,

[coverlet] Unable to instrument module: C:\myproject\src\tests\bin\Debug\net5.0\coreLibA.dll, pdb without local source files, [C:\myproject\src\coreLibA\unknown]
[coverlet] Unable to instrument module: C:\myproject\src\tests\bin\Debug\net5.0\myapi.dll, pdb without local source files, [C:\myproject\src\api\unknown]

Things I've tried and checked:

• Creating a new solution with same kind of projects - unable to reproduce.
• Does changing the name of the assembly with the property have any impact? - No, i have one changed and one unchanged assembly name that works and one changed and one unchanged that doesn't work.

I have two questions:

• What else can I do to find what the significant difference is between the projects that work and those that don't?
• What is the significance of the strange "unknown" file reference at the end that I get [C:\myproject\src\api\unknown]. The word "unknown" is not present in the coverlet repo so I assume it comes from something external to coverlet.

[petli]

Do you use type providers in the F# projects? There's been issues with the similar concept of source generators in C#, and it might be that the unknown directory here have to do with those.

[da9l]

Yeah, there is a type provider in there. I will try to isolate it and exclude it and see if the problem goes away. Is there perhaps a known workaround for this?

[petli]

For C# code generators it seems a de facto standard is emerging that generated code should have the suffix .g.cs, which Coverlet filters on to ignore those files. If there is a similar standard for F# type providers, we can add a filter on that too. I don't think there is a workaround you can do outside of a coverlet change, however.

[da9l]

Thanks for the feedback but I don't think f# generate any files when tp are involved. It's more of an dev time VS runtime types on demand if I've understood it right. http://blog.mavnn.co.uk/type-providers-from-the-ground-up/ Now that I understand this I think a workaround might be to just mark all functions that depend on any type provider type. I'll try it out on Monday. My semi educated guess is that when coverlet try to instrument any type provider generated code there is no reference to any source file hence the "unknown" file reference. If coverlet was to be modified I guess it could simply ignore symbols that doesn't have a source file reference or have an option to do that. All this assume I'm right about how tp works tho. Thanks, Daniel Den fre 16 apr. 2021 17:35Peter Liljenberg ***@***.***> skrev:

…

For C# code generators it seems a de facto standard is emerging that generated code should have the suffix .g.cs, which Coverlet filters on to ignore those files. If there is a similar standard for F# type providers, we can add a filter on that too. I don't think there is a workaround you can do outside of a coverlet change, however. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1145 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABD23KS4JI57BZYJV3OYCLTJBKK7ANCNFSM43A64PTQ> .

[MarcoRossignoli]

@da9l that unknown comes from pdb

coverlet/src/coverlet.core/Helpers/InstrumentationHelper.cs

Line 129 in 3ae1976

if (!_fileSystem.Exists(docName) && !docName.EndsWith(".g.cs"))

Coverlet by design instruments only asm for which it has got a source file available. Seems that there is an unknown on pdb(generated by compilers during build, not by coverlet). Are you able to provide a repro?

[da9l]

Not yet but I think I might be able to.

[da9l]

I've created a sample project that use an XMLType provider and I'm unable to recreate the problem.
I Inspected the assembly using JetBrains free dotPeek and found that the pdb file really think there is a source file named "unknown". Maybe if I add a specific exclusion of "unknown" might do it.

[da9l]

Ah... I have good reasons to think the TypeProvider was not involved at all here.
Instead it is a function in the library FSharpPlus that we've been using. So this issue can most likely be closed unless any one would like to implement a feature that allows coverlet to ignore specific source files.

For reference:

        let moveTip (source,target) =
             FilePath.moveTip source target

        zip source' target' 
        // |> map (uncurry FilePath.moveTip) // Usage of the FSharpPlus inline function uncurry will generate a reference to the file unknown in the pdb.
        |> map moveTip // this and the function above is the workaround
        |> Option.defaultValue Error

So if the unknown file reference occurs whenever one uses a f# inline function by design (inline is a #f language keyword) this should be handled by coverlet. So it's probably not a bug in the FSharpPlus lib. More likely its is either by design in the f# compiler or a bug in the f# compiler.

[petli]

As I understand the purpose of this heuristic is to filter out dependent libraries, so that the user doesn't have to exclude them manually from the code coverage. But is seems a bit too aggressive to fail if any source file is missing, rather than just failing if all source files are missing. What do you think, @MarcoRossignoli ?

Alternatively, a way to identify and ignore these compiler generated files (of all kinds) without having to encode every possible file pattern might be to look if the purported path is in a subdirectory of a project file (though then we need to identify all possible project file extensions).

[da9l]

Ok, I might have stumbled upon a more obvious case. I will post a repo with the problem reproduced if I succeed with that.

When I use an Anonymous Record in F# this problem appears again. The type refers to the "unknown" file again. I will try to create a repo that reproduce this error and also ask the fsharp ppl if this is by design or not.

[da9l]

Problem reproduction repo up: https://github.com/da9l/coverletWithAnonymousRecords
In the repo there are two libs and one test project. One of the libs contains an anonymous record and the other one don't.
The one with AR does not show up in the coverlet analysis.

[baronfel]

The unknown file comes from a sort of logical 'zero' range (F# Ranges are a start position/end position/file structure) used by the F# compiler for constructs that have no direct mapping to source code, so it does need to be handled/escaped in some way irrespective of whether or not it should be emitted for anonymous records.

[petli]

Thanks for the clarification @baronfel! I've updated the issue title to make it more clear to others what the problem is.

[rhewitt316]

Hello, I am experiencing the same problem and I have posted a simple example here in the hope that it will help: https://github.com/rhewitt316/coverlet-fsharp-anonymous-records-example

I am using coverlet.console 3.0.3. Coverlet fails to instrument my example as-is. If the anonymous record is removed from my example, then instrumentation succeeds.

[MarcoRossignoli]

Hi @rhewitt316 thank for the repro, we'll look into asap.

[daveMueller]

There already is an issue at the fsharp repo for this dotnet/fsharp#11195. There also is a workaround described even if it isn't pretty.

We could check the document table for entries that end with unknown and have a F# language guid and then ignore them. There also is a really similar issue where the VB compiler has a bug #1100.

I'm personally don't really know if coverlet should handle such compiler bugs? @MarcoRossignoli @petli

@petli I get your idea with 'just fail if all source files are missing'. The benefit would be to get rid of such issues like this one. The disadvantage would be a higher possibility for false positives if a library has just one same document name like the project.

[MarcoRossignoli]

I think we should try to fix this issue F# is growing would be great provide coverage tool to this community.
I would go for recognize F# construct if possible, if it's too hard we can relax a bit the heuristic, maybe with some calculated threshold (not a simple all/none/one).
What you think?
Can we recognize F# project by file name .fs and skip to validate that entry?

[petli]

I get your idea with 'just fail if all source files are missing'. The benefit would be to get rid of such issues like this one. The disadvantage would be a higher possibility for false positives if a library has just one same document name like the project.

I've moved this part of the discussion to a new issue (#1164) to not distract from the F# situation here.

[MarcoRossignoli]

Thanks @petli

[rhewitt316]

Thanks all for your effort in making this tool available and maintaining it. We have implemented the workaround pointed to by @daveMueller until a new version of coverlet console is released with the fix.

[Arshia001]

FYI, I'm on version 3.1.0 and still seeing this issue. The workaround mentioned in dotnet/fsharp#11195 works, but it isn't exactly great. Is there something I can do to diagnose the issue further?

[daveMueller]

@Arshia001 This should be fixed in 3.1.0 but maybe we missed something. I just tried it again with the repro @da9l provided (https://github.com/da9l/coverletWithAnonymousRecords) and everything works fine with 3.1.0.

Could you provide a repro of your issue? Then we could look into this again. By the way what coverlet driver are you using? E.g. the F# xunit template currently comes with coverlet.collector 3.0.2 and maybe you are accidentally using this one.

[Arshia001]

@daveMueller No, I don't think I am. Here's the relevant section from my paket.dependencies:

group Test
    source https://api.nuget.org/v3/index.json
    framework: net5.0

    nuget FSharp.Core 5.0.0
    nuget coverlet.collector 3.1.0
    nuget FSharpPlus
    nuget Microsoft.NET.Test.Sdk 16.5.0
    nuget Unquote 5.0.0
    nuget NUnit
    nuget NUnit3TestAdapter
    nuget MySqlConnector
    nuget Dapper
    nuget Simple.Migrations
    nuget Ply
    nuget Microsoft.NETCore.Platforms 5.0.2

I'll try to set up a minimal repro and get back to you.

[daveMueller]

Yes thanks everything seems fine. Then most likely we missed something and a repro would really help. 👍

[Arshia001]

I deleted every old version of coverlet.collecter.dll from my computer, removed all bin/obj folders and rebuilt everything from scratch; and the error is gone now. I also went throught my PDB's and the changes in #1165 and it seems like everything should be OK. AFAICT updating the version of coverlet.collector in paket.dependencies and restoring was not enough. @daveMueller thanks anyway.