Client invalid state exception when connection times out

[zaphoyd]

Issue posted to the mailing list by Geoff:

This might be related to #361

Client connection behavior during timeouts / high load needs some additional testing.

I think I am also seeing this problem.

m_state: state::closed
m_local_close_code: 1006 (abnormal_close)
m_local_close_reason: "The opening handshake timed out"

In my case the server is under high load explaining the timeout. The client throws the "invalid state" exception and exits. The abnormal_close description suggests this state is not a result of messages passed between client/server. Rather the client is giving up on the server and initiating the closing of the connection. So as far as I can work out, in this scenario it is possible to receive the handshake response while in the closed state.

In my opinion it would be better for the state machine to ignore the message rather than terminate.

A workaround is to try and avoid the situation by making the timeouts longer.

[glam2000]

I can also get this to happen on the server side. Probably also related to load - eg too many clients and not being able to keep up. Nginx is forwarding between the client/server and terminating ssl from client. Maybe the client has stopped (crashed) but the link has not gone down immediately?

If you need any additional information let me know.

#6  0xb740a02e in __cxa_throw () from /usr/lib/libstdc++.so.6
#7  0x0821b649 in websocketpp::connection<dtasio::WebSocketServer::config>::atomic_state_change (this=0xafbee9f8, internal_req=websocketpp::session::internal_state::PROCESS_HTTP_REQUEST, internal_dest=websocketpp::session::internal_state::PROCESS_CONNECTION, external_req=websocketpp::session::state::connecting, external_dest=websocketpp::session::state::open, msg=...) at include/websocketpp/impl/connection_impl.hpp:1653
#8  0x0821b095 in websocketpp::connection<dtasio::WebSocketServer::config>::handle_send_http_response (this=0xafbee9f8, ec=...) at include/websocketpp/impl/connection_impl.hpp:1201
...
(gdb) print m_state
$1 = websocketpp::session::state::closed
(gdb) print m_local_close_code
$2 = 1006
(gdb) 

[zaphoyd]

Thank you, that gdb trace is super helpful. This issue will have a patch later tonight.

[zaphoyd]

I've pushed an overhaul of the internal state machine that removes the use of exceptions in handling state errors during handshake processing. All such errors are demoted to rerror/recoverable error and will terminate the connection via standard (non-exception based) methods rather than the whole application.

In addition, there are some cases where we expect to see a state error due to non-deterministic interleaving of timeout handlers and read/write handlers. In these expected cases state errors are ignored entirely.

I was able to capture the issue demonstrated by the above stack trace in both servers and clients in a unit test and the new code passes all common inter-leavings that I've seen. If those who've seen this issue can try testing with the latest commit on the develop branch and let me know it resolves the above issue that would be very helpful.

[zaphoyd]

I believe that the fix for this issue is now present in the 0.5.0 release on the master branch. If anyone has additional similar issues please reopen.