Connection is closed after long step (~200s) #92

ludekcakl · 2016-08-19T11:48:43Z

Steps to reproduce

create step with wait implementation:

        [Step("Wait for <secconds> secconds")]
        public static void Wait(int secconds)
        {
            Thread.Sleep(TimeSpan.FromSeconds(secconds));
        }

run steps with increasing wait time. Gauge will fail with:

Gauge.CSharp.Runner.SandboxBuilder Creating a Sandbox in: C:\Projects\NBS\src\Acc.Tests\Acc.Gui.Tests\bin\Debug\
Checking updates...
Sandbox Loaded Instance Manager of Type:Gauge.CSharp.Lib.DefaultClassInstanceManager
Executing in 1 parallel streams.
[runner: 1] Gauge.CSharp.Runner.SandboxBuilder Creating a Sandbox in: C:\Projects\NBS\src\Acc.Tests\Acc.Gui.Tests\bin\Debug\[runner: 1]
[runner: 1] Sandbox Loaded Instance Manager of Type:Gauge.CSharp.Lib.DefaultClassInstanceManager[runner: 1]
[runner: 1] # Wait test
[runner: 1] Sandbox Acc.Gui.Tests.Implementation.Shared[runner: 1]
[runner: 1] Starting ChromeDriver 2.23.409699 (49b0fa931cda1caad0ae15b7d1b68004acd05129) on port 61395
Only local connections are allowed.
[runner: 1]   ## Waitings
[runner: 1]       * Wait for "100" secconds
[runner: 1] Sandbox Acc.Gui.Tests.Implementation.Shared[runner: 1]
[runner: 1]       * Wait for "150" secconds
[runner: 1] Sandbox Acc.Gui.Tests.Implementation.Shared[runner: 1]
[runner: 1]       * Wait for "200" secconds
[runner: 1] Sandbox Acc.Gui.Tests.Implementation.Shared[runner: 1]
[runner: 1]       * Wait for "250" secconds
[runner: 1]
        Failed Step: Wait for "250" secconds
        Specification: Acc.Gui.Tests\Specs\Settlement\Wait.spec:9
        Error Message: Connection closed [127.0.0.1:61394] cause: EOF
        Stacktrace:

[runner: 1]         Error Message: write tcp 127.0.0.1:61392->127.0.0.1:61394: use of closed network connection
        Stacktrace:

[runner: 1]     Error Message: write tcp 127.0.0.1:61392->127.0.0.1:61394: use of closed network connection
    Stacktrace:

[runner: 1]   Error Message: write tcp 127.0.0.1:61392->127.0.0.1:61394: use of closed network connection
  Stacktrace:

[runner: 1]
[runner: 1] Gauge.CSharp.Runner.GaugeListener System.Runtime.Remoting.RemotingException: Object '/06fbe031_26cd_443f_852b_923e7dc21366[runner: 1] vyb5sepifie83_12.rem' has been disconnected or does not exist at the server.[runner: 1]
[runner: 1]
[runner: 1] Server stack trace: [runner: 1]
[runner: 1]    at System.Runtime.Remoting.Channels.ChannelServices.CheckDisconnectedOrCreateWellKnownObject(IMessage msg)[runner: 1]
[runner: 1]    at System.Runtime.Remoting.Channels.ChannelServices.SyncDispatchMessage(IMessage msg)[runner: 1]
[runner: 1]
[runner: 1] Exception rethrown at [0]: [runner: 1]
[runner: 1]    at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)[runner: 1]
[runner: 1]    at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)[runner: 1]
[runner: 1]    at Gauge.CSharp.Runner.Models.GaugeMethod.get_ParameterCount()[runner: 1]
[runner: 1]    at Gauge.CSharp.Runner.Processors.ExecuteStepProcessor.Process(Message request) in c:\Program Files (x86)\Go Agent\pipe[runner: 1] -Package\gauge-csharp\Runner\Processors\ExecuteStepProcessor.cs:line 49[runner: 1]
[runner: 1]    at Gauge.CSharp.Runner.GaugeListener.PollForMessages() in c:\Program Files (x86)\Go Agent\pipelines\Csharp-Package\gauge-csharp\Runner\GaugeListener.cs:line 52[runner: 1]
  Error Message: write tcp 127.0.0.1:61392->127.0.0.1:61394: use of closed network connection

Gauge version

Gauge version: 0.6.1

Plugins
-------
csharp (0.9.1)
html-report (2.1.1)
xml-report (0.1.3)

The text was updated successfully, but these errors were encountered:

Buthrakaur · 2016-08-24T11:03:02Z

Hello @sriv ,
this issue is probably the root cause of errors I already discussed with you on gitter few weeks ago. I reported the same exceptions being thrown on TeamCity agents and you suggested it's caused by some firewall or antivirus, but I was probably also facing the problem of long step execution duration. Is it possible to increase the timeout or rather make the timeout configurable, please?

sriv · 2016-08-26T04:34:23Z

I can reproduce this. Thanks for the steps @ludekcakl .

I was under the assumption that @hakito's commit would address this, but for some reason it doesn't.

Need to investigate this further.

@Buthrakaur - this isn't the runner timing out, but it is the runner's sandbox timing out :(, which is internal to Gauge-CSharp and should not have to be visible to the user. Apologies if I misdirected you on your earlier comment.

sswaroopgupta · 2016-08-30T03:12:32Z

Tested with time wait in concepts and getting the error below.

Concept

# Wait for <sleepTime> to execute cpt
* Wait for <sleepTime> to execute

Spec

With Long Steps
===============

     |sleepTime|
     |---------|
     |1      |
     |10     |
     |100     |
     |200    |

* Wait for <sleepTime> to execute cpt

With long Steps1
---------------
* Wait for <sleepTime> to execute cpt

With long Steps2
---------------
* Wait for <sleepTime> to execute cpt

With long Steps3
---------------
* Wait for <sleepTime> to execute cpt

With long Steps4
---------------
* Wait for <sleepTime> to execute cpt

With long Steps5
---------------
* Wait for <sleepTime> to execute cpt

With long Steps6
---------------
* Wait for "10" to execute cpt

Test Name:  With long Steps6
Test Outcome:   Failed
Result StandardOutput:  
Gauge.CSharp.Runner.SandboxBuilder Creating a Sandbox in: Z:\Documents\Visual Studio 2015\Projects\GaugeProject1\GaugeProject1\bin\Debug\
Sandbox Loaded Instance Manager of Type:Gauge.CSharp.Lib.DefaultClassInstanceManager
# With Long Steps

     |sleepTime|
     |---------|
     |1        |
  ## With long Steps6
Sandbox _GaugeProjectTemplate.StepImplementation
Sandbox _GaugeProjectTemplate.StepImplementation

     |sleepTime|
     |---------|
     |10       |
  ## With long Steps6
Sandbox _GaugeProjectTemplate.StepImplementation
Sandbox _GaugeProjectTemplate.StepImplementation

     |sleepTime|
     |---------|
     |100      |
  ## With long Steps6
Sandbox _GaugeProjectTemplate.StepImplementation
Sandbox _GaugeProjectTemplate.StepImplementation

     |sleepTime|
     |---------|
     |200      |
  ## With long Steps6
Sandbox _GaugeProjectTemplate.StepImplementation

            Failed Step: Wait for <sleepTime> to execute
            Specification: specs\example.spec
            Error Message: Connection closed [127.0.0.1:65525] cause: EOF
            Stacktrace: 

            Error Message: write tcp 127.0.0.1:65522->127.0.0.1:65525: use of closed network connection
            Stacktrace: 

    Error Message: write tcp 127.0.0.1:65522->127.0.0.1:65525: use of closed network connection
    Stacktrace: 

  Error Message: write tcp 127.0.0.1:65522->127.0.0.1:65525: use of closed network connection
  Stacktrace: 


  Error Message: write tcp 127.0.0.1:65522->127.0.0.1:65525: use of closed network connection
  Stacktrace: 

Gauge.CSharp.Runner.GaugeListener System.Runtime.Remoting.RemotingException: Object '/06594202_921d_4dd8_9978_9ab5a3a31187/iskqroirur+mxs2u_zxpd6rr_4.rem' has been disconnected or does not exist at the server.

Server stack trace: 
   at System.Runtime.Remoting.Channels.ChannelServices.CheckDisconnectedOrCreateWellKnownObject(IMessage msg)Unable to connect to plugin Html Report 2.2.0. proto: required field "SuiteExecutionResult.SuiteResult.SpecResults.ProtoSpec.Items.TableDrivenScenario.Scenarios.ScenarioItems.Concept.Steps.Step.StepExecutionResult.PostHookFailure.StackTrace" not set


   at System.Runtime.Remoting.Channels.ChannelServices.SyncDispatchMessage(IMessage msg)

Exception rethrown at [0]: 
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
   at Gauge.CSharp.Runner.Models.GaugeMethod.get_ParameterCount()
   at Gauge.CSharp.Runner.Processors.ExecuteStepProcessor.Process(Message request) in c:\Program Files (x86)\Go Agent\pipelines\Csharp-Package\gauge-csharp\Runner\Processors\ExecuteStepProcessor.cs:line 49
   at Gauge.CSharp.Runner.GaugeListener.PollForMessages() in c:\Program Files (x86)\Go Agent\pipelines\Csharp-Package\gauge-csharp\Runner\GaugeListener.cs:line 52
Specifications: 1 executed  0 passed    1 failed    0 skipped
Scenarios:  1 executed  0 passed    1 failed    0 skipped

Total time taken: 5m43.509s

Buthrakaur · 2016-08-31T11:07:24Z

Hi @sriv , how is it going with this issue? I don't understand the QA "roundtrip" and it also looks like the 0.9.2 version containing your fix for this issue is broken with some errors from travis and appveyor builds...

sriv · 2016-09-01T03:09:15Z

Hi @Buthrakaur - the QA roundtrip is just for us in the team to track issues - @sguptatw was looking at various use cases to see if this issue is unresolved in any scenario. Her case is that when using concepts with table driven execution, the issue is persistent. I am unable to replicate it, hence the back and forth.

@sguptatw: I tried this project, output is in the gist. Could you please share a project that I can replicate the issue on?

Note that the whole run took about 1hr20mins, and the execution passed without failure.

@Buthrakaur: on your point about failed appveyor and travis builds, I believe those are failures independant of this fix. The two tests that are failing are features that are yet to be implemented in gauge core. I need to fix the tagging of tests<->app to have appropriate version of tests running against the app.

Buthrakaur · 2016-09-07T11:21:11Z

Hi @sriv @sguptatw , any progress? We're blocked by this issue at the moment :(

sriv · 2016-09-12T12:48:21Z

@Buthrakaur , @ludekcakl - the fix is available in the latest nightly version of gauge-csharp. Could you please try this out and see if it fixes your cases?

Buthrakaur · 2016-09-13T09:46:27Z

I just tried it on the original timeout test provided by @ludekcakl and it looks ok:

Gauge version: 0.6.1

Plugins
-------
csharp (0.9.2.nightly-2016-09-12)
html-report (2.1.1)
xml-report (0.1.3)
Gauge.CSharp.Runner.SandboxBuilder Creating a Sandbox in: C:\Temp\Acc.Tests\bin\Debug
Checking updates...
Sandbox Loaded Instance Manager of Type:Gauge.CSharp.Lib.DefaultClassInstanceManager
Executing in 1 parallel streams.
[runner: 1] Gauge.CSharp.Runner.SandboxBuilder Creating a Sandbox in: C:\Temp\Acc.Tests\bin\Debug[runner: 1]
[runner: 1] Sandbox Loaded Instance Manager of Type:Gauge.CSharp.Lib.DefaultClassInstanceManager[runner: 1]
[runner: 1] # timeout test
[runner: 1] Sandbox Acc.Tests.Implementation.Shared[runner: 1]
[runner: 1]   ## tst
[runner: 1]       * Wait for "100" seconds
[runner: 1] Sandbox Acc.Tests.Implementation.Shared[runner: 1]
[runner: 1]       * Wait for "250" seconds
[runner: 1] Sandbox Acc.Tests.Implementation.Shared[runner: 1]
[runner: 1]       * Wait for "500" seconds
[runner: 1] Sandbox Acc.Tests.Implementation.Shared[runner: 1]
[runner: 1]
Sucessfully generated xml-report to => C:\Temp\Acc.Tests\reports\xml-report
Plugin [Xml Report] with pid [16308] has exited
Successfully generated html-report to => C:\Temp\Acc.Tests\reports\html-report
Plugin [Html Report] with pid [18616] has exited
Specifications: 1 executed      1 passed        0 failed        0 skipped
Scenarios:      1 executed      1 passed        0 failed        0 skipped

Total time taken: 14m15.045s

I'll try it on our real tests too and report back.

Buthrakaur · 2016-09-13T10:40:02Z

I just tried it on our real test suite which suffered by the timeout issue before and it seems to work just fine with the 0.9.2 nightly build. So could you release 0.9.2 as a regular version? I'll use --install csharp --file gauge-csharp-0.9.2.nightly-2016-09-12.zip in the meantime.

sriv · 2016-09-13T10:53:09Z

@Buthrakaur - neat! I will work on releasing 0.9.2, but it requires some cleanup in our build scripts and tests. Am working on that at the moment.

sriv · 2016-09-14T10:37:20Z

Gauge-C# 0.9.2 is now released and should have this fix. Am closing this issue, please report back if the issue persists.

sriv self-assigned this Aug 26, 2016

sriv added the bug label Aug 26, 2016

sriv added a commit that referenced this issue Aug 29, 2016

set infinite lifetime for proxies, #92

e006828

sriv added the ready for QA label Aug 29, 2016

sswaroopgupta added in QA and removed ready for QA labels Aug 30, 2016

sswaroopgupta closed this as completed Aug 30, 2016

sswaroopgupta removed the in QA label Aug 30, 2016

sswaroopgupta reopened this Aug 30, 2016

sswaroopgupta added in QA in progress and removed in QA labels Aug 30, 2016

sriv closed this as completed Sep 14, 2016

sriv removed the in progress label Sep 14, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Connection is closed after long step (~200s) #92

Connection is closed after long step (~200s) #92

ludekcakl commented Aug 19, 2016 •

edited by sswaroopgupta

Loading

Buthrakaur commented Aug 24, 2016

sriv commented Aug 26, 2016

sswaroopgupta commented Aug 30, 2016 •

edited

Loading

Buthrakaur commented Aug 31, 2016

sriv commented Sep 1, 2016

Buthrakaur commented Sep 7, 2016 •

edited

Loading

sriv commented Sep 12, 2016

Buthrakaur commented Sep 13, 2016

Buthrakaur commented Sep 13, 2016 •

edited

Loading

sriv commented Sep 13, 2016

sriv commented Sep 14, 2016 •

edited

Loading

Connection is closed after long step (~200s) #92

Connection is closed after long step (~200s) #92

Comments

ludekcakl commented Aug 19, 2016 • edited by sswaroopgupta Loading

Steps to reproduce

Gauge version

Buthrakaur commented Aug 24, 2016

sriv commented Aug 26, 2016

sswaroopgupta commented Aug 30, 2016 • edited Loading

Buthrakaur commented Aug 31, 2016

sriv commented Sep 1, 2016

Buthrakaur commented Sep 7, 2016 • edited Loading

sriv commented Sep 12, 2016

Buthrakaur commented Sep 13, 2016

Buthrakaur commented Sep 13, 2016 • edited Loading

sriv commented Sep 13, 2016

sriv commented Sep 14, 2016 • edited Loading

ludekcakl commented Aug 19, 2016 •

edited by sswaroopgupta

Loading

sswaroopgupta commented Aug 30, 2016 •

edited

Loading

Buthrakaur commented Sep 7, 2016 •

edited

Loading

Buthrakaur commented Sep 13, 2016 •

edited

Loading

sriv commented Sep 14, 2016 •

edited

Loading