Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

After the dubbo connection from consumer to provider is broken, the consumer does not rebuild the connection immediately, but waits for more than a minute before rebuilding the connection. #5234

Closed
2 tasks done
dangit815 opened this issue Oct 24, 2019 · 8 comments
Closed
2 tasks done

After the dubbo connection from consumer to provider is broken, the consumer does not rebuild the connection immediately, but waits for more than a minute before rebuilding the connection. #5234

dangit815 opened this issue Oct 24, 2019 · 8 comments
Milestone
2.7.6

Comments

@dangit815
Copy link

dangit815 commented Oct 24, 2019 •
edited
Loading

  • I have searched the issues of this repository and believe that this is not a duplicate.
  • I have checked the FAQ of this repository and believe that this is not a duplicate.

I am using the new dubbo version 2.7.4, and I know the bugfix of #1968 and #4694 has been merge into 2.7.4 version, but I still met this problem.

Environment

  • Dubbo version: 2.7.4
  • Operating System version: CentOS
  • Java version: JDK 1.8

Steps to reproduce this issue

  1. start the consumer and provider services.
  2. tcpkill the connection from consumer to provider.

Expected Result

Fast rebuild new connection when the connection is broken.

Actual Result

What actually happens?

The connection is rebuilt after more than one minute.
@dangit815
Copy link
Author

cc @chickenlj

@luyuanwan
Copy link
Contributor

I want to know how to kill -9 the connection. Kill -9 consumer ?

@dangit815
Copy link
Author

@luyuanwan I use tcpkill to kill the connection from consumer to provider.

@dangit815
Copy link
Author

I got the following log information. Obviously, after the heartbeat timeout, a new tcp connection is initiated.

2019-10-25 06:52:48.723 INFO 1 --- [eCheck-thread-1] o.a.d.r.e.s.header.ReconnectTimerTask : [DUBBO] Initial connection to HeaderExchangeClient [channel=org.apache.dubbo.remoting.transport.netty4.NettyClient [172.16.0.15:0 -> dubboprovider/10.247.251.34:20880]], dubbo version: 2.7.4, current host: 172.16.0.15
2019-10-25 06:52:48.731 INFO 1 --- [eCheck-thread-1] o.a.d.r.transport.netty4.NettyClient : [DUBBO] Close old netty channel [id: 0xe4e6a862, L:/172.16.0.15:36250 ! R:dubboprovider/10.247.251.34:20880] on create new netty channel [id: 0xa8c79251, L:/172.16.0.15:55844 - R:dubboprovider/10.247.251.34:20880], dubbo version: 2.7.4, current host: 172.16.0.15
2019-10-25 06:52:48.732 INFO 1 --- [eCheck-thread-1] o.a.d.remoting.transport.AbstractClient : [DUBBO] Succeed connect to server dubboprovider/10.247.251.34:20880 from NettyClient 172.16.0.15 using dubbo version 2.7.4, channel is NettyChannel [channel=[id: 0xa8c79251, L:/172.16.0.15:55844 - R:dubboprovider/10.247.251.34:20880]], dubbo version: 2.7.4, current host: 172.16.0.15

@dangit815
Copy link
Author

cc @luyuanwan @chickenlj

@cs-charles
Copy link

Ask for an answer

@cs-charles
Copy link

00:24:26 [DubboClientReconnectTimer-thread-2] INFO com.alibaba.dubbo.remoting.transport.netty.NettyClient[101]: [DUBBO] Close old netty channel [id: 0xd4fa664e, /172.17.41.81:50104 :> /172.17.41.44:20880] on create new netty channel [id: 0xfa7fba17, /172.17.41.81:50526 => /172.17.41.44:20880], dubbo version: 2.8.5, current host: 172.17.41.81
00:24:26 [DubboClientReconnectTimer-thread-2] INFO com.alibaba.dubbo.remoting.transport.AbstractClient[282]: [DUBBO] Successed connect to server /172.17.41.44:20880 from NettyClient 172.17.41.81 using dubbo version 2.8.5, channel is NettyChannel [channel=[id: 0xfa7fba17, /172.17.41.81:50526 => /172.17.41.44:20880]], dubbo version: 2.8.5, current host: 172.17.41.81
00:26:25 [DubboClientHandler-172.17.41.44:20880-thread-570] INFO com.alibaba.dubbo.rpc.protocol.dubbo.DubboProtocol[127]: [DUBBO] disconnected from /172.17.41.44:20880,url:dubbo://172.17.41.44:20880/com.souche.onsmock.OnsMockService?application=BDDTester-consumer&codec=dubbo&dubbo=2.8.5&heartbeat=60000&interface=com.souche.onsmock.OnsMockService&logger=slf4j&pid=99&qos.enable=false&register.ip=172.17.41.81&revision=1.0-SNAPSHOT&side=consumer&timeout=100000&timestamp=1575635124518, dubbo version: 2.8.5, current host: 172.17.41.81
00:26:26 [DubboClientReconnectTimer-thread-2] INFO com.alibaba.dubbo.remoting.transport.netty.NettyClient[101]: [DUBBO] Close old netty channel [id: 0xfa7fba17, /172.17.41.81:50526 :> /172.17.41.44:20880] on create new netty channel [id: 0xbfa72568, /172.17.41.81:50904 => /172.17.41.44:20880], dubbo version: 2.8.5, current host: 172.17.41.81
00:26:26 [DubboClientReconnectTimer-thread-2] INFO com.alibaba.dubbo.remoting.transport.AbstractClient[282]: [DUBBO] Successed connect to server /172.17.41.44:20880 from NettyClient 172.17.41.81 using dubbo version 2.8.5, channel is NettyChannel [channel=[id: 0xbfa72568, /172.17.41.81:50904 => /172.17.41.44:20880]], dubbo version: 2.8.5, current host: 172.17.41.81
00:28:25 [DubboClientHandler-172.17.41.44:20880-thread-571] INFO com.alibaba.dubbo.rpc.protocol.dubbo.DubboProtocol[127]: [DUBBO] disconnected from /172.17.41.44:20880,url:dubbo://172.17.41.44:20880/com.souche.onsmock.OnsMockService?application=BDDTester-consumer&codec=dubbo&dubbo=2.8.5&heartbeat=60000&interface=com.souche.onsmock.OnsMockService&logger=slf4j&pid=99&qos.enable=false&register.ip=172.17.41.81&revision=1.0-SNAPSHOT&side=consumer&timeout=100000&timestamp=1575635124518, dubbo version: 2.8.5, current host: 172.17.41.81
00:28:26 [DubboClientReconnectTimer-thread-2] INFO com.alibaba.dubbo.remoting.transport.netty.NettyClient[101]: [DUBBO] Close old netty channel [id: 0xbfa72568, /172.17.41.81:50904 :> /172.17.41.44:20880] on create new netty channel [id: 0xd3118857, /172.17.41.81:51274 => /172.17.41.44:20880], dubbo version: 2.8.5, current host: 172.17.41.81
00:28:26 [DubboClientReconnectTimer-thread-2] INFO com.alibaba.dubbo.remoting.transport.AbstractClient[282]: [DUBBO] Successed connect to server /172.17.41.44:20880 from NettyClient 172.17.41.81 using dubbo version 2.8.5, channel is NettyChannel [channel=[id: 0xd3118857, /172.17.41.81:51274 => /172.17.41.44:20880]], dubbo version: 2.8.5, current host: 172.17.41.81
00:30:25 [DubboClientHandler-172.17.41.44:20880-thread-573] INFO com.alibaba.dubbo.rpc.protocol.dubbo.DubboProtocol[127]: [DUBBO] disconnected from /172.17.41.44:20880,url:dubbo://172.17.41.44:20880/com.souche.onsmock.OnsMockService?application=BDDTester-consumer&codec=dubbo&dubbo=2.8.5&heartbeat=60000&interface=com.souche.onsmock.OnsMockService&logger=slf4j&pid=99&qos.enable=false&register.ip=172.17.41.81&revision=1.0-SNAPSHOT&side=consumer&timeout=100000&timestamp=1575635124518, dubbo version: 2.8.5, current host: 172.17.41.81
00:30:26 [DubboClientReconnectTimer-thread-1] INFO com.alibaba.dubbo.remoting.transport.netty.NettyClient[101]: [DUBBO] Close old netty channel [id: 0xd3118857, /172.17.41.81:51274 :> /172.17.41.44:20880] on create new netty channel [id: 0x70f61582, /172.17.41.81:51576 => /172.17.41.44:20880], dubbo version: 2.8.5, current host: 172.17.41.81
00:30:26 [DubboClientReconnectTimer-thread-1] INFO com.alibaba.dubbo.remoting.transport.AbstractClient[282]: [DUBBO] Successed connect to server /172.17.41.44:20880 from NettyClient 172.17.41.81 using dubbo version 2.8.5, channel is NettyChannel [channel=[id: 0x70f61582, /172.17.41.81:51576 => /172.17.41.44:20880]], dubbo version: 2.8.5, current host: 172.17.41.81
00:32:25 [DubboClientHandler-172.17.41.44:20880-thread-575] INFO com.alibaba.dubbo.rpc.protocol.dubbo.DubboProtocol[127]: [DUBBO] disconnected from /172.17.41.44:20880,url:dubbo://172.17.41.44:20880/com.souche.onsmock.OnsMockService?application=BDDTester-consumer&codec=dubbo&dubbo=2.8.5&heartbeat=60000&interface=com.souche.onsmock.OnsMockService&logger=slf4j&pid=99&qos.enable=false&register.ip=172.17.41.81&revision=1.0-SNAPSHOT&side=consumer&timeout=100000&timestamp=1575635124518, dubbo version: 2.8.5, current host: 172.17.41.81
00:32:27 [DubboClientReconnectTimer-thread-1] INFO com.alibaba.dubbo.remoting.transport.netty.NettyClient[101]: [DUBBO] Close old netty channel [id: 0x70f61582, /172.17.41.81:51576 :> /172.17.41.44:20880] on create new netty channel [id: 0x890a5186, /172.17.41.81:51840 => /172.17.41.44:20880], dubbo version: 2.8.5, current host: 172.17.41.81
00:32:27 [DubboClientReconnectTimer-thread-1] INFO com.alibaba.dubbo.remoting.transport.AbstractClient[282]: [DUBBO] Successed connect to server /172.17.41.44:20880 from NettyClient 172.17.41.81 using dubbo version 2.8.5, channel is NettyChannel [channel=[id: 0x890a5186, /172.17.41.81:51840 => /172.17.41.44:20880]], dubbo version: 2.8.5, current host: 172.17.41.81
00:34:25 [DubboClientHandler-172.17.41.44:20880-thread-576] INFO com.alibaba.dubbo.rpc.protocol.dubbo.DubboProtocol[127]: [DUBBO] disconnected from /172.17.41.44:20880,url:dubbo://172.17.41.44:20880/com.souche.onsmock.OnsMockService?application=BDDTester-consumer&codec=dubbo&dubbo=2.8.5&heartbeat=60000&interface=com.souche.onsmock.OnsMockService&logger=slf4j&pid=99&qos.enable=false&register.ip=172.17.41.81&revision=1.0-SNAPSHOT&side=consumer&timeout=100000&timestamp=1575635124518, dubbo version: 2.8.5, current host: 172.17.41.81
00:34:27 [DubboClientReconnectTimer-thread-1] INFO com.alibaba.dubbo.remoting.transport.netty.NettyClient[101]: [DUBBO] Close old netty channel [id: 0x890a5186, /172.17.41.81:51840 :> /172.17.41.44:20880] on create new netty channel [id: 0xa7550ff6, /172.17.41.81:52074 => /172.17.41.44:20880], dubbo version: 2.8.5, current host: 172.17.41.81
00:34:27 [DubboClientReconnectTimer-thread-1] INFO com.alibaba.dubbo.remoting.transport.AbstractClient[282]: [DUBBO] Successed connect to server /172.17.41.44:20880 from NettyClient 172.17.41.81 using dubbo version 2.8.5, channel is NettyChannel [channel=[id: 0xa7550ff6, /172.17.41.81:52074 => /172.17.41.44:20880]], dubbo version: 2.8.5, current host: 172.17.41.81
00:36:25 [DubboClientHandler-172.17.41.44:20880-thread-577] INFO com.alibaba.dubbo.rpc.protocol.dubbo.DubboProtocol[127]: [DUBBO] disconnected from /172.17.41.44:20880,url:dubbo://172.17.41.44:20880/com.souche.onsmock.OnsMockService?application=BDDTester-consumer&codec=dubbo&dubbo=2.8.5&heartbeat=60000&interface=com.souche.onsmock.OnsMockService&logger=slf4j&pid=99&qos.enable=false&register.ip=172.17.41.81&revision=1.0-SNAPSHOT&side=consumer&timeout=100000&timestamp=1575635124518, dubbo version: 2.8.5, current host: 172.17.41.81
00:36:27 [DubboClientReconnectTimer-thread-1] INFO com.alibaba.dubbo.remoting.transport.netty.NettyClient[101]: [DUBBO] Close old netty channel [id: 0xa7550ff6, /172.17.41.81:52074 :> /172.17.41.44:20880] on create new netty channel [id: 0x86c64cc2, /172.17.41.81:52368 => /172.17.41.44:20880], dubbo version: 2.8.5, current host: 172.17.41.81
00:36:27 [DubboClientReconnectTimer-thread-1] INFO com.alibaba.dubbo.remoting.transport.AbstractClient[282]: [DUBBO] Successed connect to server /172.17.41.44:20880 from NettyClient 172.17.41.81 using dubbo version 2.8.5, channel is NettyChannel [channel=[id: 0x86c64cc2, /172.17.41.81:52368 => /172.17.41.44:20880]], dubbo version: 2.8.5, current host: 172.17.41.81
00:38:25 [DubboClientHandler-172.17.41.44:20880-thread-578] INFO com.alibaba.dubbo.rpc.protocol.dubbo.DubboProtocol[127]: [DUBBO] disconnected from /172.17.41.44:20880,url:dubbo://172.17.41.44:20880/com.souche.onsmock.OnsMockService?application=BDDTester-consumer&codec=dubbo&dubbo=2.8.5&heartbeat=60000&interface=com.souche.onsmock.OnsMockService&logger=slf4j&pid=99&qos.enable=false&register.ip=172.17.41.81&revision=1.0-SNAPSHOT&side=consumer&timeout=100000&timestamp=1575635124518, dubbo version: 2.8.5, current host: 172.17.41.81

@chickenlj
Copy link
Contributor

I will check this issue later.

@chickenlj chickenlj added this to the 2.7.6 milestone Dec 30, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
2.7.6
Development

No branches or pull requests
5 participants
@qinliujie @luyuanwan @chickenlj @dangit815 @cs-charles