using altcover in a application that contains c++ c++/cli

[jmecosta]

hi,

well my fast try to use this, was not very successful. My use case:

1. Windows x64 application (>2GB binaries). Single exe entry
2. multiple c++ core dlls
3. c# dlls all over the place, \plugins, \feature, \dialogs etc
4. assemblies in GAC

my first try was to run "altcover.exe" alone... and of course i get those:
ERROR *** Instrumentation phase failed
Writing mixed-mode assemblies is not supported
Details written to D:\prod\structures\BuildDrop\Work\bin_release_x64__Instrumented\AltCover-2019-10-09--19-46-27.log

second, was to try to use -s assembly, but this is far from optimal since the code base keeps changing and i would rather aovid the need to adjust those everytime a user adds on.

third, after using some -s ive seen pure c# assemblies being reported as mixed mode. yes those are probably doing some interop, dll import and stuff but they are still .net and would bvery much like to measure the coverage one those

is my kind of scenario supported.

thanks

[SteveGilham]

The "Writing mixed-mode assemblies is not supported" failure is a limitation inherited from the use of Mono.Cecil to do the assembly rewriting; that library makes no attempt to handle non-IL code within an assembly, and in fact deliberately fails with that message on assemblies that don't have the IL-only flag set in their metadata.

While fixing that would not be practical, it would be feasible for AltCover to treat this as a soft failure, just automatically skipping mixed-mode assemblies, in the same way as assemblies with no debug symbols.

[jmecosta]

hum, but this means that most of my assemblies will not get coverage? as far as i tested there are a lot of them... i do keep central settings, how can i set this IL-only flag?

[SteveGilham]

It's the compiler that that sets that flag in the first place. The corflags.exe tool lets you manually override whatever value that is, but if there is any non-IL executable code in the assembly, it will be lost in the instrumentation process, as Mono.Cecil only models the IL content of an assembly.

One way to proceed in the mixed-mode environment would be to separate out all the interop calls into pure interface assemblies with no logic, so that the .net side of the operation can be instrumented in isolation.

Alternatively, as this currently has to be a .net Framework on Windows project, you may find that OpenCover would be a better fit for your particular use case; with OpenCppCoverage available for the native part of the equation as required.

[jmecosta]

It's the compiler that that sets that flag in the first place. The corflags.exe tool lets you manually override whatever value that is, but if there is any non-IL executable code in the assembly, it will be lost in the instrumentation process, as Mono.Cecil only models the IL content of an assembly.

Looks like looking for troubles with this one! Not sure if it will not cause other problems during runtime?

One way to proceed in the mixed-mode environment would be to separate out all the interop calls into pure interface assemblies with no logic, so that the .net side of the operation can be instrumented in isolation.

Yes, but not realistic in a project of this dimension! cant convince managers to spend years updating all their assemblies just to collect coverage

Alternatively, as this currently has to be a .net Framework on Windows project, you may find that OpenCover would be a better fit for your particular use case; with OpenCppCoverage available for the native part of the equation as required.

We are using opencover, but trying to find a replacement for it because of #59, we are doing test impact analysis and collecting only the relevant coverage for the tests. OpenCover doesn't support this use case. Your tool one the other hand does, and it would great if you could support our use case! This is a very common scenario in enterprise environments

[SteveGilham]

As well as setting the flags, corflags.exe will by default just dump the flag settings like

CLR Header: 2.5
PE        : PE32
CorFlags  : 0x1
ILONLY    : 1
32BITREQ  : 0 
32BITPREF : 0 
Signed    : 0

I've been doing some experiments, and I've not yet managed to get a mixed-mode C# assembly from using P/Invoke or with COM interop as client or server, so a simple example of this would be of interest, even if just to be able to say "don't do that".

OTOH with C++/CLI, even with the (now deprecated) /clr:pure option to stop it linking in the 'C' runtime, the ILOnly flag is not (or at least not reliably) set, so essentially no C++/CLI code can currently be handled by any of the Cecil-based coverage tools for .net (AltCover, coverlet, minicover, ...)

For the bigger problem, it is possible to hijack the Cecil assembly writing process (I've done it before for strong-naming on .net core), so it might be possible to bit-wise copy unmanaged code from the source assembly into the rewritten one, but I have no idea if that would still function (if function entry-points are passed by offsets, those might well break). I'm afraid I can't promise any speedy turn-around on this one, let alone a positive result.

Alternatively, have you lodged a similar issue to #59 with OpenCover? Putting together a PR that ports the watch-and-toggle mechanism from AltCover to OpenCover would be rather less complex, and carry less uncertainties.

[jmecosta]

As well as setting the flags, corflags.exe will by default just dump the flag settings like

CLR Header: 2.5
PE        : PE32
CorFlags  : 0x1
ILONLY    : 1
32BITREQ  : 0 
32BITPREF : 0 
Signed    : 0

I've been doing some experiments, and I've not yet managed to get a mixed-mode C# assembly from using P/Invoke or with COM interop as client or server, so a simple example of this would be of interest, even if just to be able to say "don't do that".

when i tried, even the nunit.framework.dll is been reported as "Writing mixed-mode assemblies is not supported"

Can this be because the dll in fact is not in that folder? our setup as i mention contains dozens of folders that we configure in probing paths.

OTOH with C++/CLI, even with the (now deprecated) /clr:pure option to stop it linking in the 'C' runtime, the ILOnly flag is not (or at least not reliably) set, so essentially no C++/CLI code can currently be handled by any of the Cecil-based coverage tools for .net (AltCover, coverlet, minicover, ...)

C++/CLI we have issues even with OpenCover, but mostly works. But if we dont get those i dont get bother that much... Those are a handfull of assemblies...

For the bigger problem, it is possible to hijack the Cecil assembly writing process (I've done it before for strong-naming on .net core), so it might be possible to bit-wise copy unmanaged code from the source assembly into the rewritten one, but I have no idea if that would still function (if function entry-points are passed by offsets, those might well break). I'm afraid I can't promise any speedy turn-around on this one, let alone a positive result.

Alternatively, have you lodged a similar issue to #59 with OpenCover? Putting together a PR that ports the watch-and-toggle mechanism from AltCover to OpenCover would be rather less complex, and carry less uncertainties.

Yes i have discuss this with @sawilde i cant find the thread there, but i remenber that it was not possible to do with OpenCover

[SteveGilham]

AltCover as currently released will just stop dead at the first mixed-mode (i.e. with the ILOnly flag not set) assembly it tries to instrument, after potentially having successfully instrumented one or more other assemblies (i.e. all but the last in the console messages). The plural in the Mono.Cecil error message means it will not write any assembly with that bit not set, not that multiple such offending assemblies have been encountered.

Whereas OpenCover does instrumentation on the fly in memory as a response to assembly loading at run-time, AltCover has to be given the files to instrument in advance. The files given are specified grouped by directory -- in the simple case of unit test coverage, all the files in the unit test assembly project's output directory, for example. Where files are scattered, then the v6.0 feature of multiple input directories can be used to capture them in many locations.

While OpenCover does not currently support pause/resume behaviour, implementing it along AltCover's lines would be simple, if slightly tedious

• Add a _recording_paused flag to the ProfilerCommunications class, cleared by default
• in ProfilerCommunication::AddVisitPointToBuffer exit without writing if the flag is set
• Look to an environment variable, say OpenCover_ControlFile, for a file path, and if that variable and that file both exist, set the flag, and if the file is at any point deleted, clear it (using a file system watcher to handle this toggling would be simplest)
• in the C# side of things, add a file path value to the key OpenCover_ControlFile in the launched process environment dictionary; this could either by generated from the report file path or be specified as a separate argument (absent by default).

Then the user can control recording through manipulating that file just like with AltCover, the main difference being that the recording defaults on when the file is absent, rather than off. The file path would have to be read-only visible to the executing process, thought that would typically only be a concern for instrumenting services running with low privilege

[jmecosta]

This is very helpful, thank you. I will definitely give it a go👍

…

On Wed, 16 Oct 2019, 16:54 Steve Gilham, ***@***.***> wrote: AltCover as currently released will just stop dead at the first mixed-mode (i.e. with the ILOnly flag not set) assembly it tries to instrument, after potentially having successfully instrumented one or more other assemblies (i.e. all but the last in the console messages). The plural in the Mono.Cecil error message means it will not write any assembly with that bit not set, not that multiple such offending assemblies have been encountered. Whereas OpenCover does instrumentation on the fly in memory as a response to assembly loading at run-time, AltCover has to be given the files to instrument in advance. The files given are specified grouped by directory -- in the simple case of unit test coverage, all the files in the unit test assembly project's output directory, for example. Where files are scattered, then the v6.0 feature of multiple input directories can be used to capture them in many locations. While OpenCover does not currently support pause/resume behaviour, implementing it along AltCover's lines would be simple, if slightly tedious - Add a _recording_paused flag to the ProfilerCommunications class, cleared by default - in ProfilerCommunication::AddVisitPointToBuffer exit without writing if the flag is set - Look to an environment variable, say OpenCover_ControlFile, for a file path, and if that variable and that file both exist, set the flag, and if the file is at any point deleted, clear it (using a file system watcher to handle this toggling would be simplest) - in the C# side of things, add a file path value to the key OpenCover_ControlFile in the launched process environment dictionary; this could either by generated from the report file path or be specified as a separate argument (absent by default). Then the user can control recording through manipulating that file just like with AltCover, the main difference being that the recording defaults on when the file is absent, rather than off. The file path would have to be read-only visible to the executing process, thought that would typically only be a concern for instrumenting services running with low privilege — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#77?email_source=notifications&email_token=AAH6HSH5P47PIKLTKHRHZYDQO4MHZA5CNFSM4I7CFOR2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBMSERI#issuecomment-542712389>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAH6HSD3E75MNRLXTZCSCVDQO4MHZANCNFSM4I7CFORQ> .

[SteveGilham]

I'll close this issue at the moment, but I've added a long-term research TODO regarding a proper resolution.