unhandled {ssl_closed, _} message when using async
#464
Comments
|
I can reproduce this - I'm also using the async option with streaming. It seems that my chunk gets mixed in with it, as the last item in the tuple.
Update/Edit: When I run the streaming process in an Elixir Task, the message doesn't occur. Which makes me think there's a mailbox issue? Only occurs in our deployment environment, (cloudfoundry, ubuntu trusty). Locally on OSX 10.12.5, there's no issue. Using hackney 1.11 code snippet: # options for request - [async: :once, stream_to: self()]
defp handle_async(%{status: status} = conn, ref) do
:ok = :hackney.stream_next(ref)
receive do
{:hackney_response, ^ref, {:status, code, _reason}} ->
conn
|> Plug.Conn.put_status(code)
|> handle_async(ref)
{:hackney_response, ^ref, {:headers, headers, }} ->
downcased_headers =
headers
|> Stream.map(fn {k,v} -> {String.downcase(k), v} end)
|> Enum.filter(fn {k, _v} -> k !== "cache-control" end)
conn
|> Plug.Conn.merge_resp_headers(downcased_headers)
|> Plug.Conn.delete_resp_header("content-length") #RFC spec dictates no content-length on chunked requests
|> Plug.Conn.put_resp_header("connection", "close")
|> Plug.Conn.send_chunked(status)
|> handle_async(ref)
{:hackney_response, ^ref, :done} ->
conn
{:hackney_response, ^ref, chunk} ->
case Plug.Conn.chunk(conn, chunk) do
{:ok, new_conn} ->
handle_async(new_conn, ref)
{:error, :closed} ->
Logger.info("Client canceled request.")
:hackney.close(ref)
conn
{:error, reason } ->
Logger.error("Unable to complete request: #{inspect(reason)}")
:hackney.close(ref)
conn
end
message ->
Logger.warn("Hackney.handle_async got an unexpected message: #{inspect(message)}")
handle_async(conn, ref)
end
end |
|
hrm it's probably happening on close. @paulswartz is your code similar to the one posted by @matthewoden ? |
|
I'm using hackney via httpoison, but here's the relevant code: https://github.com/edgurgel/httpoison/blob/master/lib/httpoison/base.ex#L369-L387 |
|
If it matters, I've done a quick port on my code over to I'd prefer to use hackney, so just let me know if you need any other usage info. |
|
well the issue is that a message is still on the process heap after cloising the socket. I have a fix for it. Will take care of it later today. Thanks for the code! |
|
can you try #482? |
|
can anyone provides me a link on which I can reproduce the issue? |
|
I can't provide our source, but I'll see if I can get #482 out in a test environment today or tomorrow. (Edit: I've never been able to reproduce it locally, even after dockerizing the thing.) |
|
This Like @matthewoden, I also can't reproduce it locally.... probably something to do with loopbacks? This is in stdout on travis container startup... |
|
@stephenmoloney do you mean the link? I need to reporduce it apart of the another lib ... |
|
@benoitc sorry, I don't full understand... When the tests run, that error is printed...
I still didn't get time to try the tracing. |
|
@benoitc Take your example of async receiving in the documentation, convert to require stream_next, and put a sleep of a second or more between requesting stream_next. This is how I have done it via elixir with only hackney. There is something possibly a race condition of receiving the last of the web request and the receiving a message to stream_next. The stream has fully closed, but the buffer has not been flushed out and the done message sent. |
|
@critch thanks. Quite possible indeed. I think that rather than pausing i should rather receive actively at some point. I will have a closer looks, thanks for the reproducible way. |
|
fixed in latest master. |
|
I still see this happening with hackney One thing I don't understand from I'm going to keep looking into it. |
|
I've got two traces, one where it works and one where it crashes. This is part of the working trace that differs from the broken one. This is the broken one If you look at them side by side, we see the working one has But the broken one has the So I think the |
|
Just going to leave a comment here if anyone else encounters this behavior: the reason for this seemed to be calling |
|
@rozap you mean 2 consecutive call without receiving the message? can you show a simple snippet for it? just fyi right after the next release i will revisit the stream code to make it simpler to handle, but if we can fix that issue firsy/ |
|
Seen with |
|
I get the same error with hackney 1.15.1 It works with hackney 1.15.1 |
|
Leaking messages with: Elixir 1.8.2 |
|
Leaking messages with: Elixir 1.9.2 If it makes sense, I can take a look at it. Also, is there any pattern of usage that sidesteps the issue? |
|
We're getting this when using it with |
This commit fixed it for us: opencensus-beam/oc_google_reporter@bb578a5 It seems that if you don't close hackney either explicitly with |
They happen a lot (see benoitc/hackney#464) but aren't worthy of a warning at the moment.
They happen a lot (see benoitc/hackney#464) but aren't worthy of a warning at the moment.
|
Hi @benoitc |
|
Hi @benoitc, |
|
Looks like this is back in Erlang 23 and the newest version of hackney, but this time it happens for streaming and non-streaming interfaces. For the streaming case, it is easy to reproduce it by running the following: {:ok, ref} = :hackney.get("https://data.seattle.gov/api/views/tpvk-5fr3/rows.csv?accessType=DOWNLOAD", [{"Accept-Encoding", "gzip, deflate"}], <<>>, [
async: :once
])
Enum.each(0..1000, fn _ ->
# receive do
# {:hackney_response, ^ref, _} -> :ok
# end
:hackney.stream_next(ref)
end)Uncommenting the That change doesn't fix the non-streaming interface though. I'm going to continue to look into that. |
|
I've traced the crash a bit for the non-async path. Here is a trace where things behave correctly. And here is one where we don't handle the In the bad trace (2nd one), it looks like the sslsocket gets closed automatically, before we call close explicitly and get the alert. In the other instance, I see the genserver call to close and then the ssl closed alert (generated on When we do call close, we get back So ultimately my understanding is that the server closed the connection before the client, which leads to this message not getting handled and leaking into the caller. But may not be interpreting it correctly. |
|
Fix submitted as #640 |
We're seeing something similar to benoitc/hackney#464 that is causing the genserver to crash, because it's not handling a message that seems to be leaking from hackney / httpoison. This change should prevent that crash from happening.
We're seeing something similar to benoitc/hackney#464 that is causing the genserver to crash, because it's not handling a message that seems to be leaking from hackney / httpoison. This change should prevent that crash from happening.
and others
I see the following message in my Elixir logs periodically:
That looks like a close message from the SSL socket, which I'd expect hackney to manage the same as a close of the TCP connection (which are handled properly). I'm using the
asyncoption to stream the response data as it comes in.The text was updated successfully, but these errors were encountered: