New parameter ExcludeAssembliesWithoutSources to control automatic assembly exclusion

[petli]

Problem

Coverlet automatically excludes third-party assemblies from the coverage analysis, since they are usually not interesting to the user. The heuristic for determining if an assembly is a third-party dependency is if any of its source documents does not have a corresponding source file.

(Ref:

coverlet/src/coverlet.core/Helpers/InstrumentationHelper.cs

Lines 125 to 129 in 3ae1976

	// We verify all docs and return false if not all are present in local
	// We could have false negative if doc is not a source
	// Btw check for all possible extension could be weak approach
	// We exlude from the check the autogenerated source file(i.e. source generators)
	if (!_fileSystem.Exists(docName) && !docName.EndsWith(".g.cs"))

)

C# source generators and some F# constructs add source documents to firsrt-party assemblies, causing them to be excluded by this heuristic, and compiler bugs may also trigger this. See e.g. #1084 #1145 #1100 #1297

The C# source generators are handled by identifying them using a best practice that the document names should end in ".g.cs", and similar logic can be added for the other issues. But the source generator naming is but a best practice and not all projects will follow it, and new compiler issues will likely turn up in the future, so this heuristic will continue to cause issues.

Suggested solution

Add a new parameter ExcludeAssembliesWithoutSources with the values MissingAny, MissingAll and None. MissingAny is the current behaviour, but MissingAll would be the new default behaviour to avoid these and future issues with generators. None would be a new setting to never filter on source files, for those rare cases where it may not make sense. Explicit include/exclude settings would override any filtering from this parameter.

Changing the default to include an assembly in the coverage if at least one source document exist may cause some third-party documents to be included, if the same file name exists in the project being analysed with coverlet (pointed out by @daveMueller). Users affected by this can then either change the setting to MissingAny to revert to the old behaviour, or explicitly exclude those assemblies.

Coverlet should still have a heuristic to ignore generated source files according to conventions, but now to avoid logging those cases.

(Edit: change parameter name in feature description to ExcludeAssembliesWithoutSources)

[petli]

A more consistent parameter name may be ExcludeAssembliesWithoutSources.

(Edit: the description above have been updated to this name now, replacing the original idea IgnoreAssembliesWithoutSources.)

[TomasMorton]

Ran into this because we heavily use this library: https://github.com/mcintyre321/OneOf with the source generator. It generates GenerateOneOfAttribute.cs in all of our projects. This causes coverlet to fail:
TpTrace Verbose: 0 : 8714, 1, 2022/06/17, 11:18:29.732, 9406716979570, datacollector.dll, [coverlet]Unable to instrument module: /Users/../src/bin/Debug/net6.0/redacted.dll, pdb without local source files, [/Users/../src/../OneOf.SourceGenerator/OneOf.SourceGenerator.OneOfGenerator/GenerateOneOfAttribute.cs]

Due to this only 2 of our 61 projects are able to be covered. We also use internal source generators. Relying on file suffix isn't really an option when you start using 3rd parties.

[daveMueller]

I start working on this. But it could take some time 🙏.

[daveMueller]

OK I just stumbled across this pr #1360, which basically already covers ExcludeAssembliesWithoutSources = None.
Should I remove the new parameter name InstrumentModulesWithoutLocalSources now and align it with the definition in this issue?

[MelGrubb]

I know this is closed, but ".generated.cs" is a pretty common naming style, as it ".designer.cs". Is there some reason that coverlet doesn't support all three?

[daveMueller]

@MelGrubb I think they are all supported in the current version https://github.com/coverlet-coverage/coverlet/pull/1392/files#diff-be914c2ef40b4434ccb0927c453da5f7f2a4f828d601367f585bd7284349ba85L486-L490 but I also could be wrong.

At least I'm sure that in the upcoming version we removed this restriction completely and generated files can be named arbitrary.