Cypress 4.6.0 hangs when running headless chrome in Jenkins CI with "Received unexpected number of handles"

[uchalasani]

Current behavior:

Cypress 4.6.0 hangs when running headless chrome in Jenkins CI with "Received unexpected number of handles"

cypress:launcher chrome stderr: [0522/135920.301344:ERROR:broker_posix.cc(46)] Received unexpected number of handles +480ms
cypress:launcher chrome stderr: [0522/135920.301543:FATAL:memory.cc(38)] Out of memory. size=491520 +1ms
cypress:server:util:socket_whitelist whitelisted socket closed, removing { localPort: 43432�[39m } +609ms
cypress:server:util:socket_whitelist whitelisted socket closed, removing { localPort: 38188�[39m } +1ms
cypress:server:util:process_profiler current & mean memory and CPU usage by process group:
cypress:server:util:process_profiler ┌─────────┬───────────────────┬──────────────┬────────────────────────────────┬────────────┬────────────────┬──────────┬──────────────┬─────────────┐
cypress:server:util:process_profiler │ (index) │       group       │ processCount │              pids              │ cpuPercent │ meanCpuPercent │ memRssMb │ meanMemRssMb │ maxMemRssMb │
cypress:server:util:process_profiler ├─────────┼───────────────────┼──────────────┼────────────────────────────────┼────────────┼────────────────┼──────────┼──────────────┼─────────────┤
cypress:server:util:process_profiler │    0    │     'cypress'     │      1       │              '33'              │    0.17    │      3.84      │  246.34  │    221.94    │    247.1    │
cypress:server:util:process_profiler │    1    │     'Chrome'      │      6       │ '445, 450, 451, 453, 466, 468' │    7.85    │      9.03      │  196.11  │    378.07    │   497.99    │
cypress:server:util:process_profiler │    2    │ 'electron-shared' │      3       │         '45, 202, 227'         │     0      │       0        │  85.96   │    73.61     │    85.96    │
cypress:server:util:process_profiler │    3    │     'plugin'      │      1       │             '284'              │     0      │      0.02      │  44.13   │    44.53     │    45.71    │
cypress:server:util:process_profiler │    4    │      'other'      │      2       │           '560, 561'           │     0      │       0        │   2.07   │     2.07     │    2.08     │
cypress:server:util:process_profiler │    5    │      'TOTAL'      │      13      │              '-'               │    8.03    │     11.09      │  574.61  │    635.69    │   876.03    │
cypress:server:util:process_profiler 

Desired behavior:

Cypress 4.6.0 hangs when running headless chrome in Jenkins CI should run successfully.

Test code to reproduce

Versions

Cypress 4.6.0 Docker image (cypress/included:4.6.0)
Jenkins 2.222.3

[flotwig]

it looks like Chrome closes because it ran out of available RAM:

:FATAL:memory.cc(38)] Out of memory. size=491520 

how much RAM do your CI machines have?

[uchalasani]

It has 12 GiB. Which I bumped up from 4GiB. Which is plenty for these tests in my opinion. The issue is Cypress does not fail. It does not end the process. Last night it ran for 10 hours and 46 minutes before it got killed. Thanks for your prompt help, Ugandhar

…

On Fri, May 22, 2020 at 10:28 AM Zach Bloomquist ***@***.***> wrote: it looks like Chrome closes because it ran out of available RAM: :FATAL:memory.cc(38)] Out of memory. size=491520 how much RAM do your CI machines have? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#7461 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AINV2TLSZMSMYYUQVIARE4LRS2DXZANCNFSM4NH2EHFA> .

--

-- Thanks, Ugandhar Chalasani

[brian-mann]

@uchalasani do you think the root cause of what you're experiencing is actually this issue? #7464

[uchalasani]

I am not sure if that is it or not. Symptom description looks to be similar. I can test if there a code fix. Please let me know.

[jennifer-shehane]

@uchalasani Do you have this issue in previous versions of Cypress? Can you try our 4.5.0 docker image? Please let me know the results. Thanks.

[uchalasani]

Yes. Looks like this issue is not new to 4.6.0. The following is the result of 4.5.0 testing.

[32m'cartocdn_b.global.ssl.fastly.net'�[39m, getDelayMsForRetry: �[36m[Function: getDelayForRetry]�[39m }, iteration: �[33m0�[39m } �[32m+18ms�[0m
  �[32;1mcypress:https-proxy �[0mreceived upstreamSocket callback for request { port: �[32m'443'�[39m, hostname: �[32m'cartocdn_b.global.ssl.fastly.net'�[39m, err: �[90mundefined�[39m } �[32m+18ms�[0m
  �[32;1mcypress:server:util:socket_whitelist �[0mwhitelisting socket { localPort: �[33m46760�[39m } �[32m+17ms�[0m
  �[31;1mcypress:launcher �[0mchrome stderr: [0526/131613.953882:ERROR:broker_posix.cc(46)] Received unexpected number of handles �[31m+2s�[0m
  �[31;1mcypress:launcher �[0mchrome stderr: [0526/131613.968399:ERROR:broker_posix.cc(46)] Received unexpected number of handles �[31m+15ms�[0m
  �[31;1mcypress:launcher �[0mchrome stderr: [0526/131613.975966:ERROR:broker_posix.cc(46)] Received unexpected number of handles �[31m+9ms�[0m
  �[31;1mcypress:launcher �[0mchrome stderr: [0526/131613.994346:ERROR:broker_posix.cc(46)] Received unexpected number of handles �[31m+17ms�[0m
  �[31;1mcypress:launcher �[0mchrome stderr: [0526/131614.010272:ERROR:broker_posix.cc(46)] Received unexpected number of handles �[31m+18ms�[0m
  �[31;1mcypress:launcher �[0mchrome stderr: [0526/131614.026131:ERROR:broker_posix.cc(46)] Received unexpected number of handles �[31m+13ms�[0m
  �[31;1mcypress:launcher �[0mchrome stderr: [0526/131614.042388:ERROR:broker_posix.cc(46)] Received unexpected number of handles �[31m+18ms�[0m
  �[31;1mcypress:launcher �[0mchrome stderr: [0526/131614.052616:INFO:CONSOLE(495)] "[Cesium WebGL] Fragment shader compile log: ", source: http://laika-4dhd.test.svc.cluster.local:8080/static/Cesium.js (495)
[0526/131614.052698:INFO:CONSOLE(495)] "WebGL: INVALID_OPERATION: getTranslatedShaderSource: attempt to use a deleted object", source: http://laika-4dhd.test.svc.cluster.local:8080/static/Cesium.js (495)
[0526/131614.052726:INFO:CONSOLE(495)] "[Cesium WebGL] Fragment shader translation failed.", source: http://laika-4dhd.test.svc.cluster.local:8080/static/Cesium.js (495) �[31m+9ms�[0m
  �[31;1mcypress:launcher �[0mchrome stderr: [0526/131614.054692:INFO:CONSOLE(559)] "An error occurred while rendering.  Rendering has stopped.
undefined
RuntimeError: Fragment shader failed to compile.  Compile log: 
Error
    at new t (http://laika-4dhd.test.svc.cluster.local:8080/static/Cesium.js:471:23527)
    at f (http://laika-4dhd.test.svc.cluster.local:8080/static/Cesium.js:495:14987)
    at y (http://laika-4dhd.test.svc.cluster.local:8080/static/Cesium.js:495:17404)
    at d._bind (http://laika-4dhd.test.svc.cluster.local:8080/static/Cesium.js:495:18419)
    at G (http://laika-4dhd.test.svc.cluster.local:8080/static/Cesium.js:525:28429)
    at k.draw (http://laika-4dhd.test.svc.cluster.local:8080/static/Cesium.js:526:671)
    at n.execute (http://laika-4dhd.test.svc.cluster.local:8080/static/Cesium.js:494:17779)
    at Ge (http://laika-4dhd.test.svc.cluster.local:8080/static/Cesium.js:551:15680)
    at Ke (http://laika-4dhd.test.svc.cluster.local:8080/static/Cesium.js:551:17137)
    at nt (http://laika-4dhd.test.svc.cluster.local:8080/static/Cesium.js:551:25620)", source: http://laika-4dhd.test.svc.cluster.local:8080/static/Cesium.js (559) �[31m+1ms�[0m
  �[31;1mcypress:launcher �[0mchrome stderr: [0526/131614.176093:INFO:CONSOLE(0)] "WebGL: CONTEXT_LOST_WEBGL: loseContext: context lost", source: http://laika-4dhd.test.svc.cluster.local:8080/space (0) �[31m+122ms�[0m
  �[34;1mcypress:server:project �[0monMocha hook end �[34m+7s�[0m

[uchalasani]

I tested with Cypress 4.4.1 docker image also. It ran successfully 2 times, and the 3rd run failed with the same error message. i.e. 4.4.1 is little better but still had this issue.

  �[31;1mcypress:proxy:http �[0mRunning middleware { stage: �[32m'IncomingResponse'�[39m, middlewareName: �[32m'SendResponseBodyToClient'�[39m } �[31m+0ms�[0m
  �[31;1mcypress:launcher �[0mchrome stderr: [0526/143956.692031:ERROR:broker_posix.cc(46)] Received unexpected number of handles
[0526/143956.692311:FATAL:memory.cc(22)] Out of memory. size=491520 �[31m+637ms�[0m
  �[32;1mcypress:server:util:socket_whitelist �[0mwhitelisted socket closed, removing { localPort: �[33m34864�[39m } �[32m+749ms�[0m
  �[32;1mcypress:server:util:socket_whitelist �[0mwhitelisted socket closed, removing { localPort: �[33m54606�[39m } �[32m+0ms�[0m
  �[32;1mcypress:server:util:process_profiler �[0mcurrent & mean memory and CPU usage by process group:
  �[32;1mcypress:server:util:process_profiler

[uchalasani]

Any further help on this?

[adasq]

Hi, I am receiving a similar issue:

error 28-May-2020 13:11:24 [661:0528/131124.084172:ERROR:broker_posix.cc(46)] Received unexpected number of handles

after calling "reload page" (cy.visit('/')).

But there are no logs pointing to the lack of memory in my case.

[uchalasani]

Hi, I am receiving a similar issue:

error 28-May-2020 13:11:24 [661:0528/131124.084172:ERROR:broker_posix.cc(46)] Received unexpected number of handles

after calling "reload page" (cy.visit('/')).

But there are no logs pointing to the lack of memory in my case.

In our case I already bumped up memory from 4GiB to 12 GiB and I still see issue. Fact that 4.4.1 is successful 2 out of 3 times tells us that it is not necessarily memory. Unfortunately, 4.5.0 and 4.6.0 is always hangs.

I am looking for either a work around or a fix. Please let me know.

[uchalasani]

@uchalasani Do you have this issue in previous versions of Cypress? Can you try our 4.5.0 docker image? Please let me know the results. Thanks.

I replied with the information.
Can you suggest either a work around or a fix. Please let me know.

[adasq]

Hi, I am receiving a similar issue:

error 28-May-2020 13:11:24 [661:0528/131124.084172:ERROR:broker_posix.cc(46)] Received unexpected number of handles

after calling "reload page" (cy.visit('/')).

But there are no logs pointing to the lack of memory in my case.

I should mention here, that my tests are being executed within docker container. I managed to resolve the issue. Adding --ipc=host to docker run did the job.

[uchalasani]

In my case there are few different issues we resolved. Previous releases have few different issues, I am ignoring them as I am now using most recent 4.7.0 version.

The errors relating memory I thought I addressed, but it turned out that Cypress needs more shared memory. But it is not clear from the error messages. It is easy to get lost in the flood of Debug errors which are mis-leading.
Solution: I mapped my hosts /tmp as /dev/shm and Cypress running in the docker is able to use it and run tests successfully. In our case --ipc=host was not needed.

[uchalasani]

I am closing this issue as my above solution worked.
Ours is a unique case: Jenkins is running in Docker container in Kubernetes cluster in the OpenShift environment.

[tamsky]

regarding use of docker run flag: --ipc=host:
the Chrome flag --disable-dev-shm-usage may be another possible fix.