Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

io.netty.channel.unix.Errors$NativeIoException: recvAddress(..) failed: Connection reset by peer #3555

Closed
sheldon-pacvue opened this issue Dec 19, 2024 · 1 comment
Labels
status/invalid We don't feel this issue is valid

Comments

@sheldon-pacvue
Copy link

spring gateway to invoke http api meet error
1.The error only occurs after the interface has not been called for a long time; it appears immediately when the interface is called again.
2.Removing the newly added filter functionality still results in the same issue, indicating that the problem is unrelated to the new code.
3.This issue only occurs in the production environment; it cannot be reproduced in the staging environment or locally.

Dependencies:
spring:6.1.6
spring boot: 3.2.5
spring cloud: 4.1.2
netty:4.1.109.Final
netty-reactor: 1.1.18

Error message:
2024-12-19 07:34:48.719 [reactor-http-epoll-1] trace_id=6763cc98b7580915df2015741d955008 span_id=df2015741d955008 ERROR c.p.h.g.auth.AuthExceptionHandler - recvAddress(..) failed: Connection reset by peer
io.netty.channel.unix.Errors$NativeIoException: recvAddress(..) failed: Connection reset by peer
Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
Error has been observed at the following site(s):
*__checkpoint ⇢ org.springframework.cloud.gateway.filter.WeightCalculatorWebFilter [DefaultWebFilterChain]
*__checkpoint ⇢ AuthorizationWebFilter [DefaultWebFilterChain]
*__checkpoint ⇢ ExceptionTranslationWebFilter [DefaultWebFilterChain]
*__checkpoint ⇢ LogoutWebFilter [DefaultWebFilterChain]
*__checkpoint ⇢ ServerRequestCacheWebFilter [DefaultWebFilterChain]
*__checkpoint ⇢ SecurityContextServerWebExchangeWebFilter [DefaultWebFilterChain]
*__checkpoint ⇢ ReactorContextWebFilter [DefaultWebFilterChain]
*__checkpoint ⇢ HttpHeaderWriterWebFilter [DefaultWebFilterChain]
*__checkpoint ⇢ ServerWebExchangeReactorContextWebFilter [DefaultWebFilterChain]
*__checkpoint ⇢ org.springframework.security.web.server.WebFilterChainProxy [DefaultWebFilterChain]
*__checkpoint ⇢ HTTP GET "/api/v1/customers/me/ads?accountId=1547060389" [ExceptionHandlingWebHandler]
Original Stack Trace:

@sheldon-pacvue sheldon-pacvue added status/need-triage A new issue that still need to be evaluated as a whole type/bug A general bug labels Dec 19, 2024
@sheldon-pacvue
Copy link
Author

sheldon-pacvue commented Dec 19, 2024

I have resolved this issue. The root cause was that AWS's firewall has a default maximum TCP idle timeout of 350 seconds. When a connection is reused after being idle for a long time, it gets blocked by the firewall. By adding the following HTTP client configuration, the problem can be resolved by reclaiming connections after 300 seconds and setting the maximum idle time to 60 seconds:

spring:
  cloud:
    gateway:
      httpclient:
        pool:
          max-life-time: 300s # Reclaim connections after 300 seconds
          max-idle-time: 60s  # Maximum idle time for connections

@violetagg violetagg added status/invalid We don't feel this issue is valid and removed type/bug A general bug status/need-triage A new issue that still need to be evaluated as a whole labels Dec 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
status/invalid We don't feel this issue is valid
Projects
None yet
Development

No branches or pull requests

2 participants