Bad match in PUT or transfer path [JIRA: RIAK-1850]

[nerophon]

The line:

https://github.com/basho/riak_kv/blob/develop/src/riak_kv_vnode.erl#L1044

... suffers a bad match causing process failure. Encountered by a customer running dual memory/leveldb backend. Triggers during handoff send. Crash causes transfer counts to balloon (on the sender) and eventual node crash.

Example error output:

2015-06-10 13:35:54.390 [info] <0.17404.21>@riak_core_handoff_sender:start_fold:153 Starting ownership transfer of riak_kv_vnode from 'riak@172.16.3.6' 17126972312471518572699431633393941636592959488 to 'riak@172.16.3.8' 17126972312471518572699431633393941636592959488
2015-06-10 13:35:56.089 [error] <0.176.0>@riak_core_handoff_manager:handle_info:340 An outbound handoff of partition riak_kv_vnode 17126972312471518572699431633393941636592959488 was terminated because the vnode died
2015-06-10 13:35:56.089 [error] <0.17276.21> gen_fsm <0.17276.21> in state active terminated with reason: no match of right hand value {reply,{riak_kv_w1c_put_reply_v1,ok,primary},{state,17126972312471518572699431633393941636592959488,riak_kv_multi_backend,false,{state,[{<<"memory">>,riak_kv_memory_backend,{state,624165094,623902949,undefined,undefined,0,undefined}},{<<"leveldb">>,riak_kv_eleveldb_backend,{state,<<>>,"/data/leveldb/leveldb/17126972312471518572699431633393941636592959488",[{block_cache_threshold,33554432},{block_restart_interval,16},{block_size_steps,16},{compression,true},{create_if_missing,true},{delete_threshold,...},...],...}}],...},...}} in riak_kv_vnode:handle_handoff_command/3 line 1044

[fadushin]

Is this a mixed-mode cluster? I.e., are not all nodes in the cluster on 2.1.1?

[sacreman]

$ ansible -m shell -a "riak version" riak -K -s

riakn2 | success | rc=0 >>
2.1.1

riakn5 | success | rc=0 >>
2.1.1

riakn3 | success | rc=0 >>
2.1.1

riakn4| success | rc=0 >>
2.1.1

riakn1 | success | rc=0 >>
2.1.1

riakn6 | success | rc=0 >>
2.1.1

riakn9 | success | rc=0 >>
2.1.1

riakn7| success | rc=0 >>
2.1.1

riakn8 | success | rc=0 >>
2.1.1

riakn10 | success | rc=0 >>
2.1.1

[Basho-JIRA]

Reviewed PR and tests - added rt:wait_for_bucket_type_visible to test to prevent flappiness. Once Fred's run the test for a while longer, will merge PRs.

_[posted via JIRA by Douglas Rohrer]_

[Basho-JIRA]

Reviewed by Doug and merged to 2.1 branch.
2.1 will need to merge forward to develop.

_[posted via JIRA by Jon Meredith]_