Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sofa client出core #213

Closed
scottzzq opened this issue Nov 1, 2017 · 4 comments
Closed

sofa client出core #213

scottzzq opened this issue Nov 1, 2017 · 4 comments

Comments

@scottzzq
Copy link

scottzzq commented Nov 1, 2017

#0 0x0000000000cb30a4 in async_result_init (orig_handler=..., this=) at thirdparty/boost/asio/async_result.hpp:62
#1 post<boost::_bi::bind_t<void, void ()(google::protobuf::Closure), boost::_bi::list1<boost::_bi::valuegoogle::protobuf::Closure* > >&> (this=,
handler=...) at thirdparty/boost/asio/impl/io_service.hpp:106
#2 post<boost::_bi::bind_t<void, void ()(google::protobuf::Closure), boost::_bi::list1<boost::_bi::valuegoogle::protobuf::Closure* > > > (handler=...,
this=) at ./sofa/pbrpc/thread_group_impl.h:222
#3 post (handle=0x35375bf40, this=) at ./sofa/pbrpc/thread_group_impl.h:232
#4 sofa::pbrpc::SimpleRpcChannelImpl::DoneCallback (this=0x1b5663900, done=0x35375bf40, cntl=...) at sofa/pbrpc/simple_rpc_channel_impl.cc:180
#5 0x0000000000c9936b in operator() (a0=..., this=) at thirdparty/boost/function/function_template.hpp:767
#6 sofa::pbrpc::RpcControllerImpl::Done (this=0x31d82b520, error_code=, reason=...) at ./sofa/pbrpc/rpc_controller_impl.h:222
#7 0x0000000000c8f27e in sofa::pbrpc::RpcClientImpl::CallMethod (this=, request=request@entry=0x31394c480, response=response@entry=0x22442f4a0, cntl=...)
at sofa/pbrpc/rpc_client_impl.cc:227
#8 0x0000000000cb4012 in sofa::pbrpc::SimpleRpcChannelImpl::CallMethod (this=0x1b5663900, method=, controller=0x313135880, request=0x31394c480,
response=0x22442f4a0, done=) at sofa/pbrpc/simple_rpc_channel_impl.cc:147
#9 0x0000000000cb9986 in sofa::pbrpc::DynamicRpcChannelImpl::CallMethod (this=0x1770a3a00, method=0x1b3de0120, controller=0x313135880, request=0x31394c480,
response=0x22442f4a0, done=0x35375bf40) at sofa/pbrpc/dynamic_rpc_channel_impl.cc:204
#10 0x0000000000b84a45 in libzp::Cluster::DoNodeSyncTask (arg=0x313134900) at jtable-sofa-client/libzp/src/zp_cluster.cc:1148
#11 0x0000000000b84b1f in libzp::Cluster::AddNodeAsyncTask (node=..., context=) at jtable-sofa-client/libzp/src/zp_cluster.cc:1154
#12 0x0000000000b89951 in libzp::Cluster::DoMgetAsyncTask (arg=0x34b4aee00) at jtable-sofa-client/libzp/src/zp_cluster.cc:871
#13 0x0000000000b8a895 in libzp::Cluster::DoAsyncTask_get (arg=arg@entry=0x34b4aee00) at jtable-sofa-client/libzp/src/zp_cluster.cc:934
#14 0x0000000000b8a9cd in libzp::Cluster::AddAsyncTask_get (this=this@entry=0x2f06240, context=context@entry=0x34b4aee00) at jtable-sofa-client/libzp/src/zp_cluster.cc:982
#15 0x0000000000b8abae in libzp::Cluster::Amget (this=0x2f06240, table=..., keys=...,

函数调用栈如图, 求解答,谢谢!

@scottzzq
Copy link
Author

scottzzq commented Nov 1, 2017

@qinzuoyan 不知道是否和编译器有关,我们使用gcc5.2编译的

@qinzuoyan
Copy link
Collaborator

看起来应当不是编译器的问题。我觉得很大可能是RpcClient对象销毁后,继续发起RPC请求导致的。原因如下:

  1. 首先这是在调用Closure回调函数的时候出core;
  2. 第7层栈显示,因为进入了_is_running=false的分支,所以调用了cntl->Done()函数以给用户返回错误:rpc_client_impl.cc#L221(注,你的代码应当不是最新的,所以代码行数有偏差)。我推断此时RpcClientImpl应当是调用过Stop()函数,进一步推断是RpcClient对象析构调用了RpcClient::ShutDown()函数导致。
  3. RpcClientImpl::Stop()中会stop掉_callback_thread_group:rpc_client_impl.cc#L114
  4. 第4层栈显示,在调用cntl->Done()的时候,需要将回调函数分派到callback_thread_group中去:simple_rpc_channel_impl.cc#L178。如果此时callback_thread_group已经Stop掉了,可能会导致core。
  5. 因为没有更多信息,以上只是我的初步猜测。

建议:检查下你的RpcClient对象的生命周期,是否在析构后还可能存在继续发送请求的情况?因为创建RpcChannel时,构造函数传入的是RpcClient的裸指针,所以如果RpcClient对象析构了,但是RpcChannel还在继续使用,那么就会使用RpcClient对象的野指针,造成core。

@scottzzq
Copy link
Author

scottzzq commented Nov 2, 2017

经过排查,的确是这种情况,排查了好几天,之前一直以为是boost库的问题和编译器的问题。非常感谢@qinzuoyan

@scottzzq scottzzq closed this as completed Nov 2, 2017
@qinzuoyan
Copy link
Collaborator

qinzuoyan commented Nov 2, 2017 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants