-
Notifications
You must be signed in to change notification settings - Fork 436
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
grpcwebproxy: "panic: concurrent write to websocket connection" #713
Comments
The command line is
|
Thanks for the error report, your assessment sounds about right. It would be cool to have a test that exercises the websocket writer concurrently to see if we could reproduce it. In any case, I don't know if any of the current maintainers have the bandwidth to pick this up, and the websocket transport is experimental. If you'd like to try to tackle this, I'd be happy to review a PR. |
I'm not a go developer, so hopefully one of the maintainers can look at this. If I get some time this weekend I could probably create a test setup that triggers the issue. |
Looks like the spawned goroutine responsible for pings is doing unguarded I'm not entirely sure if these exported APIs, such as |
I'm seeing the same problem. I think writes to the websocket should be protected by a mutex lock. I can create a PR, but a proper test for this may be tricky.
|
I wonder if we can avoid writing in two goroutines and have the spawned goroutine communicate back instead? Introducing a lock seems messy. |
I'm not familiar with the code other than looking at the spot where the write failed, so I guess a suggestion on a technical fix may be premature. I'll try to dig deeper but any suggestions are definitely welcome @johanbrandhorst! |
Yeah, I don't have much context on this, unfortunately, it's been a while, but it would be great to understand exactly where these two writers exist, and if we can streamline writes into a single goroutine instead of introducing a lock. |
May be some prior art in https://github.com/sacOO7/gowebsocket. Edit: using locks so maybe not the best example for concurrency control with gorilla websocket. https://github.com/gorilla/websocket/tree/master/examples/chat is probably a better example. |
I had a look at fixing this with a mutex; go mutexes are very fast when there are no locks on the mutex (which should be almost all the time, as it's only the keepalive ping that happens concurrently with other writes on the socket). |
I put together a draft PR that refactors the websocket handler using the overall design of the gorilla mux chat example (uses channels as @johanbrandhorst suggested). Still a WIP. |
I get the following error when running the grpxwebproxy service:
Versions of relevant software used
grpcwebproxy-v0.12.0-linux-x86_64
grpcwebproxy-v0.12.0-win64.exe
What happened
Very intermittently, the proxy process crashes with the message
panic: concurrent write to websocket connection
What you expected to happen
No crash :-)
How to reproduce it (as minimally and precisely as possible):
I can't reproduce this reliably, it is intermittent on our system.
Anything else we need to know
According to https://godoc.org/github.com/gorilla/websocket#hdr-Concurrency this is because there must only be one writer at a time.
My guess is that the ping happened at exactly the same time as some other message.
We use both linux and windows pre-built versions of the proxies, I saw this error on linux.
The text was updated successfully, but these errors were encountered: