"IOException: Connection was closed" when retrying non-WebApplicationExceptions #37323
Comments
|
/cc @FroMage (resteasy-reactive), @stuartwdouglas (resteasy-reactive) |
|
Is this still an issue with the latest Quarkus versions? |
|
@geoand I just tried it with Quarkus 3.9.3 and the behavior doesn't seem to have changed. |
|
Thanks for checking |
|
Just want to add to this, I'm having similar issues with a container health check implementation utilizing a number of resteasy reactive rest clients. Despite completely avoiding any kind of retry logic, after a number of requests the health check fails and all subsequent rest client requests start failing with the same IOException: Connection was closed error.
|
|
Setting quarkus.rest-client.keep-alive-enabled=false seems to resolve the issue, but obviously has significant performance impacts and isn't really production viable. |
Just to make sure I understand, you are using the REST Client in order to call the health check of other services in the cluster? |
Yup, exactly. Not sure if its a factor, but the downstream APIs are using the experimental "wellness check" of the quarkus smallrye health extension. Will be doing some testing today exclusively with microprofile health endpoints to see if it makes any difference. |
Made no difference. |
|
@vietj @cescoffier I assume the Vert.x HTTP client should remove closed connections from the pool, right? |
|
It should, but let's wait for @vietj response. |
|
@geoand I confirm that closed client connections are evicted from the pool |
|
Thanks @vietj |
|
So what does this mean for this bug? Likely related to resteasy reactive rest client extension if underlying vertx http is not root cause? |
|
It's unclear and needs further investigation |
|
So there is an issue where the server was closing the connection when it shouldn't have, resulting in the exception. The client should be able to handle a connection close though, but its just that in this case because the close was sent after the response (instead of having FIN on the last packet of the response) there is basically a race condition where the client does not know the connection is closed yet, and the retry will use the same connection that is about to close. There is not really anything that can be done about this on the client side, pooled HTTP connections are inherently racy if the server decides to close them in between connections. In this case adding a small back off delay to the retry would have fixed it from the client side. |
|
Thanks all. Will this be added to LTS since its a bugfix? |
|
That's the plan |
|
Thanks guys, highly appreciated! |
Fixes quarkusio#37323 (cherry picked from commit 23f07ab)
Fixes quarkusio#37323 (cherry picked from commit 23f07ab)
Fixes quarkusio#37323 (cherry picked from commit 23f07ab)
Fixes quarkusio#37323 (cherry picked from commit 23f07ab)
Fixes quarkusio#37323 (cherry picked from commit 23f07ab)
* Coverage for JSON tokens https://issues.redhat.com/browse/QUARKUS-4179 * Cover client handling server failures https://issues.redhat.com/browse/QQE-692 quarkusio/quarkus#37323 * Cover case of failing security tests They were failing in a presence of a router filter and resteasy-reactive dependency quarkusio/quarkus#40307 https://issues.redhat.com/browse/QQE-692 * Add coverage for max parameters of multipart tweak mulripart tests fix for loop initial from 0 * Update to 3.8.5 --------- Co-authored-by: jcarranzan <jcarranz@redhat.com>
Describe the bug
Disclaimer: I asked it here already, because I am not sure if it is a bug. At least the behavior seems strange and not logical to me.
I have two local Quarkus applications, one - I will call it server - offers a simple REST endpoint, the other uses a RESTEasy Reactive RestClient to access this endpoint and has a retry configured in case of errors.
When I throw a WebApplicationException or a sub-type of it in the server endpoint, the client does a retry as expected.
When I throw any other Exception like e.g. a simple RuntimeException, the client receives a bunch of
on the first retry attempt. No further retries are carried out then, it simply stops.
Expected behavior
I would expect that the client does a retry in all HTTP 500 error cases because I cannot guarantee that only WebApplicationExceptions are thrown on the server side.
Actual behavior
The client receives a HTTP 500 in both cases (the body differs though) but it is not able to retry on Exceptions which are not WebApplicationExceptions.
How to Reproduce?
I created a reproducer here. Follow the README.md to start the two applications.
Output of
uname -aorverLinux xxx 6.1.0-13-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.55-1 (2023-09-29) x86_64 GNU/Linux
Output of
java -versionopenjdk version "17.0.7" 2023-04-18 OpenJDK Runtime Environment Temurin-17.0.7+7 (build 17.0.7+7) OpenJDK 64-Bit Server VM Temurin-17.0.7+7 (build 17.0.7+7, mixed mode, sharing)
Quarkus version or git rev
3.5.3
Build tool (ie. output of
mvnw --versionorgradlew --version)mvnw 3.8.8
Additional information
No response
The text was updated successfully, but these errors were encountered: