Connections hang in CLOSED state

[Yozhig]

We have faced an issue when connections (ports and processes owning them) hang forever (we use these connections for infinite streams of events).

...
4071 inet_tcp 249       69938937   <0.15260.394>                   *:*                    CLOSED       STREAM
4072 inet_tcp 249       141447159  <0.17014.480>                   *:*                    CLOSED       STREAM
4078 inet_tcp 249       176131209  <0.6376.600>                    *:*                    CLOSED       STREAM
4084 inet_tcp 249       80625928   <0.22290.881>                   *:*                    CLOSED       STREAM
4087 inet_tcp 249       101426932  <0.2095.695>                    *:*                    CLOSED       STREAM
4095 inet_tcp 249       147349243  <0.23210.849>                   *:*                    CLOSED       STREAM

Our investigation showed it happens due to combination of configuration, error handling from cowboy side and inet_drv behavior. The interesting thing here is when send_timeout_close configuration parameter is used and inet_drv closes a connection due to timeout it does not send tcp_closed or tcp_error message to a port owner. The only thing we can do in this case is always check a return value of Transport:send/2.
UPD: I've made a test for this case in #1590

[stolen]

@essen could you react on this please?
We faced this issue too. Long running connections (MPEG-TS over HTTP in our case) tend to leak.
This leads to a growing number of useless processes spending CPU and memory.
On this picture there is a significant leak (about 5K processes which is about 1.5K cowboy_loop workers per day) on a very low load (about 2 new requests per second and max 100 concurrent connections).

[essen]

What I don't understand is why the idle/inactivity timeouts don't help.

Anyway have you tried the patch in the PR, does it fix your case?

[stolen]

why the idle/inactivity timeouts don't help

I should have mentioned, sorry for that.
These are long-living responses. Client connects and receives data as it appear, and it may last for hours (e.g. internet radio via shoutcast or SSE), without any data sent by a client. Dowloading 2-gigabyte file over slow 3G link is also affected.
Meaningful idle_timeout kills these connections because there is no client activity in the socket, leading to loss of data in perfect network conditions.
So, we have to disable it explicitly with cowboy_req:cast({set_options, #{idle_timeout => infinity}}, Req); for these long requests.

have you tried the patch in the PR

Not yet. Will check it soon.
I've traced ssl:send/2 -> {error,enotconn}, so the patch seems to address exactly our case.

[essen]

Yes I've merged a similar patch for Gun, didn't think there was users disabling these timeouts so didn't follow up here (priorities and all that). But sounds like it'd be useful for the next release.

[stolen]

Looking forward for the release. Thank you!

[stolen]

The issue is fixed with PR.
Service was restarted at 11/15 12:00 (large step down), and there is no growth in process count.

[essen]

This will now work properly on current master. I made a slightly different fix so you may want to test again before I release the new version. Closing, thanks!