Jenkins rejects testreport

[jjvdgeer]

We upgraded NUnit-Console from 3.10 to 3.15.5 and after that Jenkins failed some of our builds because it could no longer read the result file:

ERROR: Step ‘Publish xUnit test result report’ failed: Conversion error: Cannot convert string "" to double

Further investigation revealed that the test report originally had looked like this:

<environment nunit-version="3.6.0.0" clr-version="4.0.30319.42000" os-version="Microsoft Windows NT 10.0.20348.0" ....>

However with 3.15.5 we get:

<environment nunit-version="" clr-version="4.0.30319.42000" os-version="Microsoft Windows NT 6.2.9200.0" ....>

The fact that nunit-version is empty is what causes the problem. However, it's curious that the os-version also seems wrong, as the machine it ran on was Windows 2022 which I think might be 10.0.20348.0 while 6.2.9200.0 may be Windows 8 which was not at all involved here.

We fixed the problem by migrating to nunit3 format.

The full command line being used:
"C:\Program Files (x86)\NUnit.org\nunit-console\nunit3-console.exe" "C:\Jks\ws\Classic-AMIS-AkuttDb_master\Classic-AMIS-AkuttDb-Service\Classic.Amis.AkuttDb.AdLib.UnitTests\bin\Release\Classic.Amis.AkuttDb.AdLib.UnitTests.dll" -result MyTestResults.xml;format=nunit2

Method of installation (e.g. NuGet, zip, msi):
msi

Log files (Run the console with --trace=Debug to produce these):
InternalTrace.4840.log

Version of the NUnit Engine/Console:
3.15.5

All versions of the NUnit Framework in use:
We use nunit2 format most places but most places it worked fine for unknown reasons. This project included nunit.framework.dll 2.6.3.13283

Whether the behaviour is any different when the tests are run using the --inprocess flag:
Not checked

[OsirisTerje]

Sorry for the late response.
Have you checked the source code to see where the version information is added?
Do I read you correct that this only happens when the format is NUnit2, but format NUnit3 it works fine?

[couperyork]

Hi @OsirisTerje ,

I am also receiving this error - ERROR: Conversion error: Cannot convert string "" to double

This occurs when the format is specified as NUnit2.

This was working until we went from Nunit.Console 3.4.0 to 3.16.0 (installed on the CI machine). We then needed to introduce NUnit V2 Result Writer 3.7.0 to output NUnit2 files. as we were getting:

07:03:39  Copyright (c) 2022 Charlie Poole, Rob Prouse
07:03:39  Tuesday, 20 February 2024 07:03:39
07:03:39  
07:03:39  Unknown result format: nunit2

The version of the Nunit nuget package in the project containing the unit tests was 3.11.0

We use the Jenkins Xunit Publishing plugin version 3.1.3 to publish the results.

Any ideas?
Thanks.

[couperyork]

It seems as though the issue was resolved once I fixed a seperate parsing problem.

I had been loading into the nunit3-console a test assembly (part of loading in many test assemblies) that no longer existed... which output the following:

1) Invalid : D:\Jenkins\workspace\API\IntegrationTest\bin\Release\IntegrationTest.dll
File not found: D:\Jenkins\workspace\API\IntegrationTest\bin\Release\IntegrationTest.dll

Although this wasn't an obvious problem previous to the upgrade to NUnit Console 3.16, the later versions of NUnit Console now outputs these sorts of results to the output XML file subtly differently (missing nunit-version as @jjvdgeer mentioned), which the Jenkins Xunit Publishing plugin doesn't respect.

Output With Error:

<environment nunit-version="" clr-version="4.0.30319.42000" os-version="Microsoft Windows NT 6.2.9200.0" platform="Win32NT" cwd="D:\Jenkins\workspace\API" machine-name="CI1" user="CIUser" />
  <culture-info current-culture="en-GB" current-uiculture="en-US" />
  <test-suite type="Assembly" name="D:\Jenkins\workspace\API\IntegrationTest\bin\Release\IntegrationTest.dll" executed="True" result="Failure" success="False" time="0.000" asserts="">
    <properties>
      <property name="_SKIPREASON" value="File not found: D:\Jenkins\workspace\API\IntegrationTest\bin\Release\IntegrationTest.dll" />
    </properties>
    <reason>
      <message><![CDATA[File not found: D:\Jenkins\workspace\API\IntegrationTest\bin\Release\IntegrationTest.dll]]></message>
    </reason>
    <results />
  </test-suite>

Ouput without error assembly loaded:

  <environment nunit-version="3.11.0.0" clr-version="4.0.30319.42000" os-version="Microsoft Windows NT 6.2.9200.0" platform="Win32NT" cwd="D:\Jenkins\workspace\API" machine-name="CI1" user="CIUser" />
  <culture-info current-culture="en-GB" current-uiculture="en-US" />
  <test-suite type="Assembly" name="UnitTest.dll" executed="True" result="Success" success="True" time="7.653" asserts="597">

[OsirisTerje]

@couperyork Is it the same with the latest, 3.17 ?

And, can you upload a small repro for this?

[couperyork]

@OsirisTerje It is the same with the latest, 3.17 yes.

Here is a small repro for this issue.
https://github.com/couperyork/NUnitOutputVersionIssue

Let me know if you have any questions.

[OsirisTerje]

Thanks @couperyork !

The resultwriter (nunit-v2-result-writer) picks the first assembly and grabs the nunit framework version from that one.
If it is an invalid assembly, no environment info at all comes from the framework. The other fields there are put in by the reportwriter itself.

The nunit3 output from the framework with ONLY an invalid assembly, looks like this:

The code used for the 3.4 version must have been very different from what we have now. I actually doubt it would work with later framework versions at all. I have not tested those though. It is anyway an error that is a combination of multiple things:

1. An invalid assembly doesn't return any environments information
2. The resultwriter expects the first assembly to be valid, and picks the framework version from that one.
3. The other environment data does not come from the framework (it is present there), but the resultwriter populates it itself. (This might not be a problem, but explains why the other information is present.)

Further, this is probably only a problem when using the console. Using dotnet test, the assemblies are run one-by-one, so an invalid assembly will not be loaded at all.

Why @jjvdgeer gets it to work with the nunit3 format is a bit strange, because as shown in the image, a non-existing assembly will not give out any environment information at all. But @jjvdgeer might have changed something else to fix the issue. Not sure how the Jenkings XUnit converter handles the NUnit3 format though, but.....

Is this an edge case only?
Do you always have non-existing assemblies in your to-be-tested list?

A fix could be to loop through all assemblies and pick the version number from one that has a version number.

[OsirisTerje]

@jjvdgeer @couperyork Transferred the issue to the resultwriter repo

[OsirisTerje]

@jjvdgeer
I see you wrote:

We fixed the problem by migrating to nunit3 format.

The full command line being used:
"C:\Program Files (x86)\NUnit.org\nunit-console\nunit3-console.exe" "C:\Jks\ws\Classic-AMIS-AkuttDb_master\Classic-AMIS-AkuttDb-Service\Classic.Amis.AkuttDb.AdLib.UnitTests\bin\Release\Classic.Amis.AkuttDb.AdLib.UnitTests.dll" -result MyTestResults.xml;format=nunit2

I don't think this fixed the issue, but more that the sequence of assemblies was changed.

The other issue you mention about the OS version can be something that was fixed in the framework lately, but not in the resultwriter. Would be interesting to see if you get different results if you run using the nunit3 format.

[couperyork]

@OsirisTerje, Thanks for taking a look and transferring the issue to the correct place.

Agreed that this is definitely an atypical combination of inputs which gave the output we received, hopefully there is now enough visibility in case others experience this issue.

It's possible that some other combination of assembly loading/some other silent error causes the version grief for others.

[couperyork]

I've been doing some more testing today @OsirisTerje, It appears it's not only an assembly loading problem. I loaded a single valid assembly and did a run that had a few tests errors, failures and ignores mixed in with successes. The nunit-v2-result-writer has not output the nunit version even though 491 tests passed.

<test-results name="D:\git\Api.UnitTests\bin\Debug\Api.UnitTests.dll" total="536" errors="6" failures="9" not-run="30" inconclusive="0" ignored="30" skipped="0" invalid="0" date="2024-02-21" time="08:31:47">
  <environment nunit-version="" clr-version="4.0.30319.42000" os-version="Microsoft Windows NT 6.2.9200.0" platform="Win32NT" cwd="D:\git\Api.UnitTests" machine-name="DEV" user="couper" />
  <culture-info current-culture="en-GB" current-uiculture="en-US" />
  <test-suite type="Assembly" name="D:\git\Api.UnitTests\bin\Debug\Api.UnitTests.dll" executed="True" result="Failure" success="False" time="958.715" asserts="0">
    <properties>
      <property name="_PID" value="20728" />
      <property name="_APPDOMAIN" value="domain-" />
    </properties>
    <failure>
      <message><![CDATA[One or more child tests had errors]]></message>
      <stack-trace />
    </failure> ....

[jjvdgeer]

@OsirisTerje Sorry for late replies :) I've reverted the project back to when it failed and it still has the same problem with 3.17 of NUnit-Console. It is not an invalid test-dll, at least the dll is an existing file with valid nunit tests that run. Not sure what you meant by 'invalid', but I assume we can agree that is valid.

The NUnit3 format does not seem to have the Windows version number (or I oversaw it because it isn't filled out?) so I don't think the problem is valid for NUnit3.

I enclose the NUnit2 and NUnit3 testreports for that project, in case it helps. Oh, I don't. XML is not a supported file type to attach, apparently...

[CharliePoole]

The use of nunit-version at the top level in the nunit2 format is an historical anomaly. Even in NUnit V2, it was possible to have multiple test assemblies using different versions of NUnit. In version 3, we corrected this by moving the framework version to the assembly nodes.

As @OsirisTerje noted, we simply used the first assembly's value in the original v2 result writer, which worked at a time when invalid assemblies were not retained in the XML output, but no longer works.

I have "fixed" this for the 3.8.0 release so that we use the first non-empty attribute found. It will still be blank if there are NO valid assemblies, but I think that's less of a problem.

[CharliePoole]

🎉 This issue has been resolved in version 3.8.0 🎉

The release is available on:

• GitHub Release
• NuGet Package
• Chocolatey Package