NUnit.Console.exe sometimes returns with -100 (SocketException) #1276
Comments
|
@Lorilatschki Did this process work before you upgraded to 3.16? This may be related to #1275. |
|
@CharliePoole this also fails sporadically with 3.15.0 and 3.15.2. Other versions we didn‘t tested yet because we recently changed from v2.x. |
|
@CharliePoole I did't see the mentioned related issue #1275 realy related to this issue because it then would fail constantly, but it only fails in a sporadic manner. |
|
The logs should all be found in the same directory. The agent could simply be failing before it creates the named log. Does the log appear when there is no failure? |
|
Unforunately not. I have run a test locally which succeeded. The InternalTrace.*.log has been created in the The nunit-agent_39648.log does not exist on my drive. I have searched the whole drive, I doesn't found it. |
|
I have checked the failing case scenario and indeed I found the second trace file by searching for PID of the nunit-agent.exe process. The trace output is: So no trace output other than Info/Debug 😞 and compared to the success case the same output at the end (Set stop signal). |
|
@CharliePoole any idea how to proceed on that? We still getting this error regularly in a sporadic manner. |
|
@Lorilatschki Best best would be for you to create a non-intermittent repro, if possible. There seems to be an associated issue here as well: the main log tells you to look for an FYI I'm actually no longer working on this project but I'm sure someone on the @nunit/engine-team will have to look at this. |
This issue is inconsistent and I am not sure where to find agent logs. |
|
@saiparth do you get this error sporadically or constantly? |
|
@Lorilatschki |
|
@ nunit/engine-team do you have any idea to get more details about the issue? |
|
Hi @OsirisTerje |
|
@saiparth Sorry for the delay here, we're re-organizing, so..... But will see if we can figure this one out. |
|
Hi @OsirisTerje |
|
Sorry, no update yet. These issues are hard to figure out.
|
|
@OsirisTerje Framework was upgraded from 4.6 to 6. hence I wont be able to use any previous version. I cant because my reports have some dependency and I created utility for retry based on Nunit xml results. |
|
@saiparth Sorry, I didn't mean to share your production code. But if you could create a small repro with some slow tests that would repro this. You know the situation, so more straightforward for you to create something. |
|
I could not find anything so far, I will get back to you when I am able to reproduce the issue. there is no specific use-case, its 4000 tests running in app server everyday for over a week. Selenium and Rest sharp are used |
|
Same on our site. We run about 2000 tests/fixtures each run (up to 128 in parallel dependend on our agent hardware set). Almost every second run we do have at least one failure saying It is always a different test which fails. |
|
Yes is also occurs locally!
Due to our company policy we are not able to share this, sorry! |
|
@Lorilatschki We don't want nor need your company code. We need a repro from you, fake code, which is organized so that the error you report can be reproduced. Then we have a better chance of fixing it. The other option is that you simply build the console code yourself, in debug mode, use that in your local build and then grab more information from that, possibly also being able to debug into the console/engine. |
Since we do not know which is the root cause of the failure, it is hard to create a repo for that. I assume it is enough to run a banch of tests in parallel and from time to time this failure ocurrs. I can give it a try if I can easily create a repo to repoduce the issue.
What do you mean grap more information from that? What information do you expect are part of the console code in debug mode rather than release? |
|
@OsirisTerje meanwhile we have enabled windows dump file creation for nunit3-console.exe
nunit-agent.exeon all our agents. Hopefully we get a dump created in the -100 scenario. If so, would that help as well? |
|
Hi there, we were able to reproduce the issue on our end. This issue appears to be reproducible reliably when run with the -x86 flag for the nunit3-console. It's also flaky on our end when run without that. I attached a zipped test project to reproduce the issue. It contains a dummy test project targeted at net7.0 and a PowerShell file with the exact command we used along with the outcome that resulted in on our end, including the console output and an additional windows application error event with another interesting stack trace. Feel free to contact me for more information. |
|
We kind of stick to net48 for the product we are facing the sporadic issues. |
|
@Lorilatschki Do you see the same sporadic issues if you use |
|
@OsirisTerje we try to setup several runs using the dotnet test instead of the nunit.console. We will update on this once we do have succeeded on that. |
|
@OsirisTerje we are using the
I have also tried to setup the |
|
@Lorilatschki The adapter can also use the where filter, see https://docs.nunit.org/articles/vs-test-adapter/Tips-And-Tricks.html#where And don't use the UseNUnitIdforTestCaseId :-) |
|
@OsirisTerje unfortunately it doesn't work. The name of the test fixture is When I run dotnet test "C:\temp\myTeset.dll" --test-adapter-path C:\temp\NUnit3TestAdapter\build\net48 -- NUnit.Where="id == 1-1012"it states that there is no matching test dotnet test "C:\temp\myTeset.dll" --test-adapter-path C:\temp\NUnit3TestAdapter\build\net48 -- NUnit.Where="name == MyTestCase"it works as expected and the test can be found and is executed. Any idea what I am doing wrong? |
|
I would not trust the id to be the same. To determine the id in this case, use the dump flag and check if that actually is the same. https://docs.nunit.org/articles/vs-test-adapter/Tips-And-Tricks.html#dumpxmltestdiscovery-and-dumpxmltestresults But, do you actually need the id's ? |
I will give it a try.
This is hard to explain but let me try it shortly. Because we do have a lot of test dependency tools for our tests, they cannot be run in parallel out of the box, because some of them may not run together with some others. Since we do have a lot of tests and we wanted to take advantage of a parallel test execution we defined some attributes on our tests which are kind of blueprint how the test can run in parallel. We can change the filter under the hood to use the testname instead of the test id, but all that expense to run |
|
In fact the id's between the |
|
If you're using [Thinking] It may be possible to get more info out (without using the dump file) by using the --logger "console;verbosity=normal".... |
I cannot see a dump file in case I execute it with |
|
NOTE: There are seperate dump commands for discovery and execution One below your workfolder, aka bin folder E.g. bin\debug\net6.0\dump |
|
Thanks for the hint, the dump file is created and the ids are part of it. One last question: In order to give it a try, I need to specify the nunit result file name. Within the nunit.console3.exe we do it that way: NOTE: We are using the nunit2 format since we have a test result overview generation tool which was developed against the nunit2 schema. In order to avoid adapting this tool we use the nunit2 format extension. Is there a way to easily achive this via |
|
You should really try to get off that format :-) The best is if you can check if it works to run the tests with dotnet test, and if that fails or not the same way. I have never seen any errors like this with the dotnet test/adapter, so that is why I am asking. To set the result file, use the TestOutputXml command |
|
Regarding the nunit2 format, we can get rid of it by adapting our overview generator to that.
Unfortunately this way you can only define the output folder rather than the output filename! Since we are running tests in parallel and the default name is the name of the test assembly (independend to what the fixture name is) there will be confilicts across the filenames. As I found in a separate issue, setting up the name of the result file is not possible yet within |
|
You can set the NewOutputXmlFileForEachRun to True and it will create new files if there already is a file with that name. You will then get |
|
I don't think this is suiteable since the filenames should be predictable as we have some post processing of the files. So without the possibilty to define a own result file name, we cannot easily change to We do have another issue which will arise when moving to As a consequence we would love to get this fixed within the NUnit.ConsoleRunner itself, because switching to dotnet test will be huge effort and has some downsites due to the above mentioned aspects. We much appreciate your support! |
|
To change things in the adapter is rather easy, so nunit/nunit3-vs-adapter#1114 could be fixed. Also including a mutex for the file creation (good point), alternatively added a guid to the filename to make them unique. Fixing the socket exception in the console though, is quite a different thing. We do need a working repro, which we don't have, and then we have some uncharted territory to go through there. The NUnit.Console covers much more than the adapter based system, and we don't have anyone focused on the console right now. I'll try to collect together the different console issues that seems related and see if there are any good hints on what is actually causing this. Note also that you could build the console/engine code yourself try to get some more debugging information in there. In particular in the NUnit.Engine.ITestAgent.Stop() method (this is an interface, so there are two possible implementation which could be the culprit) where the crash seems to originate. Running a debug version might also give some more information. First, figure out which implementation is actually being used here, by adding some debug output in those two methods. Then perhaps try a catch socketexception there. ALTERNATIVELY: I could do some changes to get more debug info in, and release that out to our Myget feed. This would be more slow though, but the fact you have something where this is reproduced makes it interesting. |
That sounds great, can you give a update once you have deployed such a version to the Myget feed?
If I have some free time I can fix it and creating MRs out of it. Regarding the naming issue, shall I add a new RunSettings property |
|
@OsirisTerje here are the two pull requests. |
|
@Lorilatschki Thanks a lot for this work! It's really appreciated! |
Same to me, your support is amazing! |
|
The adapter with @Lorilatschki 's PRs are here: https://www.myget.org/feed/nunit/package/nuget/NUnit3TestAdapter/4.5.0-dev-02467 |
|
@OsirisTerje since we figured out that using the
|
|
Yeah... I am on travels right now, but back next week. I'll try to get it out during that week. |
|
we are moving our system from net48 to net6.0 (with its tests of course) and started to see this issue: is this is related with the topicstarter's issue? should I create a new issue? if this is the same issue,. am I correct, that the reasons are not fully known? thank you! |
|
@lsoft Yes, It might be the same issue. And yes, we're not sure how to handle this. Also, BInaryFormatter is going away, so this has to be changed anyway. |
|
We ran into this including the CI test suite being evaluated falsely positive. The circumstances are complicated and multi threading is involved. There was a trivial exception within a thread, the UnhandledExceptionHandler catching and KILLing the process after logging. That is totally unrelated to NUnit. But when running within the test agent that kills it. I stripped it down and this test case in turn always produces the SocketException (Agent), NUnitEngineException ("Remote test agent believes agency process has exited") and ExitCode -100 (Console):
|
|
Got this very similar issue with our test suite recently (occures after all tests had run and report is about to be generated): stumbled upon this issue: #3446 Tweaking our logging output to be more compact did the trick. Hope this helps someone |
We are using NUnit.Console.exe v3.16.0 and we are getting a -100 failure sporadically with a System.Net.Sockets.SocketException.
NUnit v3.13.3
NUnit.Console.exe v3.16.0
NUnit.Extension.NUnitV2ResultWriter v3.6.1
NUnit.Extension.OverlayConfig v3.16.0.13 (custom extension built by us)
Hint: I have adjusted all paths/names due to our company policy.We are running our tests like so (). The test-ids are collected by using the --explore console argument beforehand:
In case such an error occurs, only one trace file is being created. In that case InternalTrace.36104.log but no InternalTrace.36104.TestAssembly.dll.log. In case the error do not occur, the second log file is created.
InternalTrace.36104.log output:
Test output:
The text was updated successfully, but these errors were encountered: