Reactive connection handling on Tomcat #32249
Comments
spring-projects-issues
added
the
status: waiting-for-triage
jhoeller
added
the
in: web
|
@mschweyen you mention using WebFlux on Tomcat, but tomcat-demo in the example has |
mschweyen
changed the title
mschweyen
changed the title
|
@rstoyanchev thank you for your feedback and sorry for my unclear description. You are right the problem I described was Reactive connection handling on Spring MVC and Tomcat. I have done further tests. It seems the same issue occurs when using Webflux on Tomcat. Here the updated demo project webflux-problem.zip.
We observed the problem in springboot version 3.2.2. |
|
Is there anything I can do to support you with the analysis? |
|
Could this message indicate that the connections are not cleaned up correctly: The latch counter increments to 8192 and the the server stops to respond to new connections. |
|
@mschweyen I've had a look at the sample. I'm not sure what For a single run, with the spring-mvc-demo, I see one streamed event after which Tomcat triggers a timeout notification, which we handle with an ASYNC dispatch, and the request completes. I don't know why a timeout and not a closed connection error notification, but we handle both similarly. For the webflux-tomcat-demo, we use the non-blocking I/O API and the handling is a bit different (no ASYNC dispatch), but the sequence of events is the same. For the long test with 30000 requests, each request streams its one event after which the client closes and performs the next request. On the server side, however, there are no timeout notifications, and as the controller stream is infinite, nothing further happens. This is both with Spring MVC and WebFlux. After the configured 10 second client response timeout, I see a flood of timeout notifications on the server side and those are handled as expected. This is probably expected behavior due to Servlet containers not providing notifications for closed connections, see jakartaee/servlet#44, so the only way to find out if there is a network issue is to keep writing (e.g. heartbeats) which eventually fails as the socket is not usable. For example, changing to: @PostMapping(value = "/api/pdp/decide", produces = MediaType.TEXT_EVENT_STREAM_VALUE, consumes = MediaType.APPLICATION_JSON_VALUE)
public Flux<ServerSentEvent<String>> decide(
@Valid @RequestBody Object input) {
return Flux.merge(
Flux.interval(Duration.ofSeconds(5))
.map(aLong -> ServerSentEvent.<String>builder().comment("heartbeat").build()),
Flux.create((FluxSink<String> fluxSink) -> fluxSink.next(PERMIT))
.map(decision -> ServerSentEvent.<String>builder().data(decision).build()));
}I start to see "Broken pipe" It is a little surprising that the Servlet non-blocking API doesn't do better, but that might be expected. It is perhaps worth finding out why Tomcat is able to notify us immediately for a single request, but not for many requests where it doesn't notice until the server is closed. It might be possible to improve something, /cc @markt-asf. |
rstoyanchev
added
the
status: waiting-for-feedback
|
@rstoyanchev the option SO_LINGER=0 is set to avoid that the socket is held in TIME_WAIT state on the client site. Depending on the host config it can runs out of available sockets during load tests. The same can be reproduced with one server and multiple client hosts. Here some additional information about our usecase: The usecase is about a solution for stream based access control where a client sends an AuthroizationSubscription and the server is answering with a flow of AuthrizationDecisions. Now it's on the client to apply the decision. The Decision changes over time depending on the date, time, rule changes or any other exteral infromation. Actually we can not use netty because we are using Vaadin as Web UI. Geat, thanks a lot for this solution. I understand that the heartbeat may be a way to ensure that tomcat detects a dead client. I will have a try now with our application and I'm confident that it solves the problem. May be a good idea to find a gerneric solution for tomcat as this can easily lead to problems or even be used for DOS attacs. |
spring-projects-issues
added
status: feedback-provided
|
Thanks for the extra context. I can't speak for Tomcat, but the above referenced Servlet issue is probably where something needs to change first, but I could be wrong. |
rstoyanchev
added
status: invalid
|
Hi, even though we have a heartbeat, some connections don't close when we have too many subscriptions at the same time. What should I do? |
We observed a problem with the connection handling when using Webflux on Tomcat.
Connections remain open in CLOSE_WAIT status on the server site when the client closes the connection.
This may lead to a situation where the server is not reachable if there are so many client connections that the server reaches its limit.
There is no problem when using Netty. Here is a small example (webflux-problem.zip) demonstrating that the same tests succeed on Netty and fails on Tomcat.
The problem may have a similar origin as Issue #30393.
The text was updated successfully, but these errors were encountered: