problem when running benchmark on 3 DCs

[aletomsic]

I've been running multi DC benchmarks in grid 5000 and observed this error when antidote gets overloaded, which crashes the system completely.
It even happens when setting the gen_server:call's timeout that generates the problem to infinity.

https://github.com/SyncFree/antidote/blob/master/src/inter_dc_query.erl#L71

`2017-03-22 17:05:06 =SUPERVISOR REPORT====
Supervisor: {local,riak_core_vnode_sup}
Context: child_terminated
Reason: {timeout,{gen_server,call,[inter_dc_query,{any_request,2,{{'antidote@172.16.97.20',{1490,194493,569828}},867766597165223607683437869425293042920709947392},<<131,104,4,100,0,8,114,101,97,100,95,108,111,103,110,20,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,152,97,1,98,0,2,104,148>>,#Fun<inter_dc_sub_vnode.deliver_log_reader_resp.2>}]}}
Offender: [{pid,<0.29266.2>},{name,undefined},{mfargs,{riak_core_vnode,start_link,undefined}},{restart_type,temporary},{shutdown,300000},{child_type,worker}]

2017-03-22 17:05:07 =ERROR REPORT====
** State machine <0.29272.2> terminating
** Last event in was {riak_vnode_req_v1,525227150915793236229449236757414210188850757632,ignore,{txn,{interdc_txn,{'antidote@172.16.97.20',{1490,194493,569828}},525227150915793236229449236757414210188850757632,{op_number,{'antidote@172.16.97.2',{'antidote@172.16.97.20',{1490,194493,569828}}},500443,157595},{dict,0,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},1490195961178313,undefined,undefined,[]}}}
** When State == active
** Data == {state,525227150915793236229449236757414210188850757632,inter_dc_sub_vnode,{state,525227150915793236229449236757414210188850757632,{dict,0,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}}},undefined,none,undefined,undefined,undefined,undefined,undefined,86616}
** Reason for termination =
** {timeout,{gen_server,call,[inter_dc_query,{any_request,2,{{'antidote@172.16.97.20',{1490,194493,569828}},525227150915793236229449236757414210188850757632},<<131,104,4,100,0,8,114,101,97,100,95,108,111,103,110,20,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,92,97,1,98,0,2,103,155>>,#Fun<inter_dc_sub_vnode.deliver_log_reader_resp.2>}]}}
2017-03-22 17:05:07 =CRASH REPORT====
crasher:
initial call: riak_core_vnode:init/1
pid: <0.29272.2>
registered_name: []
exception exit: {{timeout,{gen_server,call,[inter_dc_query,{any_request,2,{{'antidote@172.16.97.20',{1490,194493,569828}},525227150915793236229449236757414210188850757632},<<131,104,4,100,0,8,114,101,97,100,95,108,111,103,110,20,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,92,97,1,98,0,2,103,155>>,#Fun<inter_dc_sub_vnode.deliver_log_reader_resp.2>}]}},[{gen_fsm,terminate,7,[{file,"gen_fsm.erl"},{line,559}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}
ancestors: [riak_core_vnode_sup,riak_core_sup,<0.820.0>]
messages: [{'$gen_event',{riak_vnode_req_v1,525227150915793236229449236757414210188850757632,ignore,{txn,{interdc_txn,{'antidote@172.16.97.20',{1490,194493,569828}},525227150915793236229449236757414210188850757632,{op_number,{'antidote@172.16.97.2',{'antidote@172.16.97.20',{1490,194493,569828}}},500443,157595},{dict,0,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},1490195961178313,undefined,undefined,[]}}}},{'$gen_event',{riak_vnode_req_v1,525227150915793236229449236757414210188850757632,ignore,{txn,{interdc_txn,{'antidote@172.16.97.20',{1490,194493,569828}},525227150915793236229449236757414210188850757632,{op_number,{'antidote@172.16.97.2',{'antidote@172.16.97.20',{1490,194493,569828}}},500443,157595},{dict,0,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},1490195961178313,undefined,undefined,[]}}}},{'$gen_event',{riak_vnode_req_v1,525227150915793236229449236757414210188850757632,ignore,{txn,{interdc_txn,{'antidote@172.16.97.20',{1490,194493,569828}},525227150915793236229449236757414210188850757632,{op_number,{'antidote@172.16.97.2',{'antidote@172.16.97.20',{1490,194493,569828}}},500443,157595},{dict,0,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},1490195961178313,undefined,undefined,[]}}}},{'$gen_event',{riak_vnode_req_v1,525227150915793236229449236757414210188850757632,ignore,{txn,{interdc_txn,{'antidote@172.16.97.20',{1490,194493,569828}},525227150915793236229449236757414210188850757632,{op_number,{'antidote@172.16.97.2',{'antidote@172.16.97.20',{1490,194493,569828}}},500443,157595},{dict,0,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},1490195961178313,undefined,undefined,[]}}}}]
links: [<0.824.0>]
dictionary: [{random_seed,{27839,21123,25074}}]
trap_exit: true
status: running
heap_size: 4185
stack_size: 27
reductions: 3069
neighbours:`

[peterzeller]

Probably related to inter-dc replication, see #402