Figure out cause of error

[vb0100]

Good day!

I've had some issue on the rabbitmq server. It did not broke cluster, it recovered, but it would be nice if you could provide some insight or ideas, what could possibly be the reason for this behavior.

Preliminary data:

RabbitMQ 3.8.22
Erlang 24.0.5

Running in docker-swarm, it is a cluster using quorum queues feature with consul for service discovery. There are multiple rabbitmq nodes sitting behind HAProxy.
It seems like 1 of the nodes lost a connections with other one and there was a string of errors, but those are a little bit hard to understand what could be the issue in the first place.

The errors that I've managed to recover:

rabbit_rabbit_worker_b: 2021-11-26 11:47:40.598 [error] <0.27216.781> Mnesia(rabbit@rabbit_worker_b): ** ERROR ** mnesia_event got {inconsistent_database, starting_partitioned_network, rabbit@rabbit_master}
rabbit_rabbit_worker_b: 2021-11-26 11:47:10.867 [debug] <0.27017.781> Lager installed handler lager_forwarder_backend into error_logger_lager_event
rabbit_rabbit_worker_b: 2021-11-26 11:47:10.707 [debug] <0.27037.781> Lager installed handler error_logger_lager_h into error_logger
rabbit_rabbit_worker_a: 2021-11-26 11:47:08.538 [error] <0.32580.644> closing AMQP connection <0.32580.644> (10.0.1.6:54060 -> 10.0.1.3:5672):
rabbit_rabbit_worker_c: 2021-11-26 11:47:07.698 [error] <0.4513.641> Supervisor {<0.4513.641>,rabbit_channel_sup} had child channel started with rabbit_channel:start_link(1, <0.4454.641>, <0.4512.641>, <0.4454.641>, <<"10.0.1.11:40682 -> 10.0.1.21:5672">>, rabbit_framing_amqp_0_9_1, {user,<<"rabbit_username">>,[administrator],[{rabbit_auth_backend_internal,none}]}, <<"/">>, [{<<"connection.blocked">>,bool,true},{<<"consumer_cancel_notify">>,bool,true}], <0.4487.641>, <0.4547.641>) at <0.4531.641> exit with reason reached_max_restart_intensity in context shutdown
rabbit_rabbit_worker_c: 2021-11-26 11:47:07.697 [warning] <0.4454.641> Non-AMQP exit reason '{{error,noproc},[{rabbit_fifo_client,enqueue,3,[{file,"src/rabbit_fifo_client.erl"},{line,168}]},{rabbit_quorum_queue,deliver,3,[{file,"src/rabbit_quorum_queue.erl"},{line,746}]},{rabbit_amqqueue,'-deliver/3-fun-3-',4,[{file,"src/rabbit_amqqueue.erl"},{line,2239}]},{lists,foldl,3,[{file,"lists.erl"},{line,1267}]},{rabbit_amqqueue,deliver,3,[{file,"src/rabbit_amqqueue.erl"},{line,2236}]},{rabbit_channel,deliver_to_queues,2,[{file,"src/rabbit_channel.erl"},{line,2204}]},{rabbit_channel,handle_method,3,[{file,"src/rabbit_channel.erl"},{line,1375}]},{rabbit_channel,handle_cast,2,[{file,"src/rabbit_channel.erl"},{line,643}]}]}'
rabbit_rabbit_worker_c: 2021-11-26 11:47:07.697 [info] <0.4513.641> supervisor: {<0.4513.641>,rabbit_channel_sup}, errorContext: shutdown, reason: reached_max_restart_intensity, offender: [{pid,<0.4531.641>},{id,channel},{mfargs,{rabbit_channel,start_link,[1,<0.4454.641>,<0.4512.641>,<0.4454.641>,<<"10.0.1.11:40682 -> 10.0.1.21:5672">>,rabbit_framing_amqp_0_9_1,{user,<<"rabbit_username">>,[administrator],[{rabbit_auth_backend_internal,none}]},<<"/">>,[{<<"connection.blocked">>,bool,true},{<<"consumer_cancel_notify">>,bool,true}],<0.4487.641>,<0.4547.641>]}},{restart_type,intrinsic},{shutdown,70000},{child_type,worker}]
rabbit_rabbit_worker_c: 2021-11-26 11:47:07.697 [error] <0.4513.641> Supervisor {<0.4513.641>,rabbit_channel_sup} had child channel started with rabbit_channel:start_link(1, <0.4454.641>, <0.4512.641>, <0.4454.641>, <<"10.0.1.11:40682 -> 10.0.1.21:5672">>, rabbit_framing_amqp_0_9_1, {user,<<"rabbit_username">>,[administrator],[{rabbit_auth_backend_internal,none}]}, <<"/">>, [{<<"connection.blocked">>,bool,true},{<<"consumer_cancel_notify">>,bool,true}], <0.4487.641>, <0.4547.641>) at <0.4531.641> exit with reason {error,noproc} in rabbit_fifo_client:enqueue/3 line 168 in context child_terminated
rabbit_rabbit_worker_c: 2021-11-26 11:47:07.697 [info] <0.4513.641> supervisor: {<0.4513.641>,rabbit_channel_sup}, errorContext: child_terminated, reason: {{error,noproc},[{rabbit_fifo_client,enqueue,3,[{file,"src/rabbit_fifo_client.erl"},{line,168}]},{rabbit_quorum_queue,deliver,3,[{file,"src/rabbit_quorum_queue.erl"},{line,746}]},{rabbit_amqqueue,'-deliver/3-fun-3-',4,[{file,"src/rabbit_amqqueue.erl"},{line,2239}]},{lists,foldl,3,[{file,"lists.erl"},{line,1267}]},{rabbit_amqqueue,deliver,3,[{file,"src/rabbit_amqqueue.erl"},{line,2236}]},{rabbit_channel,deliver_to_queues,2,[{file,"src/rabbit_channel.erl"},{line,2204}]},{rabbit_channel,handle_method,3,[{file,"src/rabbit_channel.erl"},{line,1375}]},{rabbit_channel,handle_cast,2,[{file,"src/rabbit_channel.erl"},{line,643}]}]}, offender: [{pid,<0.4531.641>},{id,channel},{mfargs,{rabbit_channel,start_link,[1,<0.4454.641>,<0.4512.641>,<0.4454.641>,<<"10.0.1.11:40682 -> 10.0.1.21:5672">>,rabbit_framing_amqp_0_9_1,{user,<<"rabbit_username">>,[administrator],[{rabbit_auth_backend_internal,none}]},<<"/">>,[{<<"connection.blocked">>,bool,true},{<<"consumer_cancel_notify">>,bool,true}],<0.4487.641>,<0.4547.641>]}},{restart_type,intrinsic},{shutdown,70000},{child_type,worker}]
rabbit_rabbit_worker_c: 2021-11-26 11:47:07.697 [error] <0.4531.641> CRASH REPORT Process <0.4531.641> with 0 neighbours exited with reason: {error,noproc} in rabbit_fifo_client:enqueue/3 line 168 in gen_server2:terminate/3 line 1183
rabbit_rabbit_worker_c: 2021-11-26 11:47:07.696 [info] <0.4531.641> [{initial_call,{rabbit_channel,init,['Argument__1']}},{pid,<0.4531.641>},{registered_name,[]},{error_info,{exit,{{error,noproc},[{rabbit_fifo_client,enqueue,3,[{file,"src/rabbit_fifo_client.erl"},{line,168}]},{rabbit_quorum_queue,deliver,3,[{file,"src/rabbit_quorum_queue.erl"},{line,746}]},{rabbit_amqqueue,'-deliver/3-fun-3-',4,[{file,"src/rabbit_amqqueue.erl"},{line,2239}]},{lists,foldl,3,[{file,"lists.erl"},{line,1267}]},{rabbit_amqqueue,deliver,3,[{file,"src/rabbit_amqqueue.erl"},{line,2236}]},{rabbit_channel,deliver_to_queues,2,[{file,"src/rabbit_channel.erl"},{line,2204}]},{rabbit_channel,handle_method,3,[{file,"src/rabbit_channel.erl"},{line,1375}]},{rabbit_channel,handle_cast,2,[{file,"src/rabbit_channel.erl"},{line,643}]}]},[{gen_server2,terminate,3,[{file,"src/gen_server2.erl"},{line,1183}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}},{ancestors,[<0.4513.641>,<0.4486.641>,<0.4477.641>,<0.4506.641>,<0.1099.0>,<0.1098.0>,<0.1097.0>,<0.1095.0>,<0.1094.0>,rabbit_sup,<0.292.0>]},{message_queue_len,0},{messages,[]},{links,[<0.4513.641>]},{dictionary,[{permission_cache_can_expire,false},{{credit_to,<0.4454.641>},199},{credit_flow_default_credit,{400,200}},{channel_operation_timeout,15000},{guid,{{3706790374,781202571,1673689228,4045868044},0}},{rand_seed,{#{jump => #Fun<rand.3.92093067>,max => 288230376151711743,next => #Fun<rand.5.92093067>,type => exsplus},[184184455610661110|151609141466206989]}},{process_name,{rabbit_channel,{<<"10.0.1.11:40682 -> 10.0.1.21:5672">>,1}}},{msg_size_for_gc,347},{permission_cache,[{{resource,<<"/">>,exchange,<<>>},#{},write},{{resource,<<"/">>,queue,<<"testing_queue-three">>},#{},configure}]}]},{trap_exit,true},{status,running},{heap_size,2586},{stack_size,29},{reductions,10306}], []
rabbit_rabbit_worker_c:  {{error,noproc},
rabbit_rabbit_worker_c: 2021-11-26 11:47:07.696 [error] <0.4454.641> Error on AMQP connection <0.4454.641> (10.0.1.11:40682 -> 10.0.1.21:5672, vhost: '/', user: 'rabbit_username', state: running), channel 1:
rabbit_rabbit_worker_c: ** {{error,noproc},[{rabbit_fifo_client,enqueue,3,[{file,"src/rabbit_fifo_client.erl"},{line,168}]},{rabbit_quorum_queue,deliver,3,[{file,"src/rabbit_quorum_queue.erl"},{line,746}]},{rabbit_amqqueue,'-deliver/3-fun-3-',4,[{file,"src/rabbit_amqqueue.erl"},{line,2239}]},{lists,foldl,3,[{file,"lists.erl"},{line,1267}]},{rabbit_amqqueue,deliver,3,[{file,"src/rabbit_amqqueue.erl"},{line,2236}]},{rabbit_channel,deliver_to_queues,2,[{file,"src/rabbit_channel.erl"},{line,2204}]},{rabbit_channel,handle_method,3,[{file,"src/rabbit_channel.erl"},{line,1375}]},{rabbit_channel,handle_cast,2,[{file,"src/rabbit_channel.erl"},{line,643}]}]}
rabbit_rabbit_worker_c: ** Last message in was {'$gen_cast',{method,{'basic.publish',0,<<>>,<<"testing_queue-three">>,false,false},{content,60,none,<<128,0,10,116,101,120,116,47,112,108,97,105,110>>,rabbit_framing_amqp_0_9_1,[<<"[REDACTED]">>]},flow}}
rabbit_rabbit_worker_c: 2021-11-26 11:47:07.695 [error] <0.4531.641> ** Generic server <0.4531.641> terminating
rabbit_master: 2021-11-26 11:46:57.375 [error] <0.1027.0> Supervisor rabbit_shovel_sup had child rabbit_shovel_status started with rabbit_shovel_status:start_link() at <0.1028.0> exit with reason reached_max_restart_intensity in context shutdown
rabbit_master: 2021-11-26 11:46:57.374 [error] <0.1027.0> Supervisor rabbit_shovel_sup had child rabbit_shovel_status started with rabbit_shovel_status:start_link() at <0.1028.0> exit with reason {{nodedown,rabbit@rabbit_worker_b},{gen_server,call,[<18657.1147.0>,which_children,infinity]}} in context child_terminated
rabbit_master: 2021-11-26 11:46:57.374 [error] <0.1028.0> CRASH REPORT Process rabbit_shovel_status with 0 neighbours exited with reason: {{nodedown,rabbit@rabbit_worker_b},{gen_server,call,[<18657.1147.0>,which_children,infinity]}} in gen_server:call/3 line 247
rabbit_master: 2021-11-26 11:46:57.373 [error] <0.1028.0> ** Generic server rabbit_shovel_status terminating 
rabbit_rabbit_worker_b: {badarg,[{ets,insert,[queue_metrics,{{resource,<<"/">>,queue,<<"testing_queue-one">>},[{consumers,1},{consumer_capacity,1.0},{consumer_utilisation,1.0},{message_bytes_ready,0},{message_bytes_unacknowledged,0},{message_bytes,0},{message_bytes_persistent,0},{messages_persistent,0},{policy,<<"dead-letter">>},{operator_policy,''},{effective_policy_definition,[{<<"dead-letter-exchange">>,<<"dead-letter-dlx">>},{<<"dead-letter-routing-key">>,<<"dead-letter-dlx">>},{<<"delivery-limit">>,5}]},{memory,0},{state,down},{garbage_collection,[]},{leader,''},{online,[rabbit@rabbit_worker_a]},{members,[rabbit@rabbit_master,rabbit@rabbit_worker_b,rabbit@rabbit_worker_a]},{open_files,[{rabbit@rabbit_worker_b,0},{rabbit@rabbit_worker_a,0}]},{single_active_consumer_pid,''},{single_active_consumer_ctag,''},{messages_ram,0},{message_bytes_ram,0}],0}],[{error_info,#{cause => id,module => erl_stdlib_errors}}]},{rabbit_core_metrics,queue_stats,2,[{file,"src/rabbit_core_metrics.erl"},{line,250}]},{rabbit_quorum_queue,'-handle_tick/3-fun-0-',10,[{file,"src/rabbit_quorum_queue.erl"},{line,372}]}]}
rabbit_rabbit_worker_b: 2021-11-26 11:46:57.328 [error] emulator Error in process <0.26306.781> on node rabbit@rabbit_worker_b with exit value:
rabbit_rabbit_worker_b: {badarg,[{ets,insert,[queue_metrics,{{resource,<<"/">>,queue,<<"testing_queue-two">>},[{consumers,1},{consumer_capacity,1.0},{consumer_utilisation,1.0},{message_bytes_ready,0},{message_bytes_unacknowledged,0},{message_bytes,0},{message_bytes_persistent,0},{messages_persistent,0},{policy,<<"dead-letter">>},{operator_policy,''},{effective_policy_definition,[{<<"dead-letter-exchange">>,<<"dead-letter-dlx">>},{<<"dead-letter-routing-key">>,<<"dead-letter-dlx">>},{<<"delivery-limit">>,5}]},{memory,0},{state,down},{garbage_collection,[]},{leader,''},{online,[rabbit@rabbit_worker_a]},{members,[rabbit@rabbit_master,rabbit@rabbit_worker_b,rabbit@rabbit_worker_a]},{open_files,[{rabbit@rabbit_worker_b,0},{rabbit@rabbit_worker_a,0}]},{single_active_consumer_pid,''},{single_active_consumer_ctag,''},{messages_ram,0},{message_bytes_ram,0}],0}],[{error_info,#{cause => id,module => erl_stdlib_errors}}]},{rabbit_core_metrics,queue_stats,2,[{file,"src/rabbit_core_metrics.erl"},{line,250}]},{rabbit_quorum_queue,'-handle_tick/3-fun-0-',10,[{file,"src/rabbit_quorum_queue.erl"},{line,372}]}]}
rabbit_rabbit_worker_b: 2021-11-26 11:46:57.328 [error] emulator Error in process <0.26313.781> on node rabbit@rabbit_worker_b with exit value:
rabbit_rabbit_worker_b: {badarg,[{ets,insert,[queue_metrics,{{resource,<<"/">>,queue,<<"testing_queue-one">>},[{consumers,1},{consumer_capacity,1.0},{consumer_utilisation,1.0},{message_bytes_ready,0},{message_bytes_unacknowledged,0},{message_bytes,0},{message_bytes_persistent,0},{messages_persistent,0},{policy,<<"dead-letter">>},{operator_policy,''},{effective_policy_definition,[{<<"dead-letter-exchange">>,<<"dead-letter-dlx">>},{<<"dead-letter-routing-key">>,<<"dead-letter-dlx">>},{<<"delivery-limit">>,5}]},{memory,0},{state,down},{garbage_collection,[]},{leader,''},{online,[rabbit@rabbit_worker_a]},{members,[rabbit@rabbit_master,rabbit@rabbit_worker_b,rabbit@rabbit_worker_a]},{open_files,[{rabbit@rabbit_worker_b,0},{rabbit@rabbit_worker_a,0}]},{single_active_consumer_pid,''},{single_active_consumer_ctag,''},{messages_ram,0},{message_bytes_ram,0}],0}],[{error_info,#{cause => id,module => erl_stdlib_errors}}]},{rabbit_core_metrics,queue_stats,2,[{file,"src/rabbit_core_metrics.erl"},{line,250}]},{rabbit_quorum_queue,'-handle_tick/3-fun-0-',10,[{file,"src/rabbit_quorum_queue.erl"},{line,372}]}]}
rabbit_rabbit_worker_b: 2021-11-26 11:46:57.328 [error] emulator Error in process <0.26233.781> on node rabbit@rabbit_worker_b with exit value:
rabbit_rabbit_worker_b: {badarg,[{ets,insert,[queue_metrics,{{resource,<<"/">>,queue,<<"testing_queue-one">>},[{consumers,1},{consumer_capacity,1.0},{consumer_utilisation,1.0},{message_bytes_ready,0},{message_bytes_unacknowledged,0},{message_bytes,0},{message_bytes_persistent,0},{messages_persistent,0},{policy,<<"dead-letter">>},{operator_policy,''},{effective_policy_definition,[{<<"dead-letter-exchange">>,<<"dead-letter-dlx">>},{<<"dead-letter-routing-key">>,<<"dead-letter-dlx">>},{<<"delivery-limit">>,5}]},{memory,0},{state,down},{garbage_collection,[]},{leader,''},{online,[rabbit@rabbit_worker_a]},{members,[rabbit@rabbit_master,rabbit@rabbit_worker_b,rabbit@rabbit_worker_a]},{open_files,[{rabbit@rabbit_worker_b,0},{rabbit@rabbit_worker_a,0}]},{single_active_consumer_pid,''},{single_active_consumer_ctag,''},{messages_ram,0},{message_bytes_ram,0}],0}],[{error_info,#{cause => id,module => erl_stdlib_errors}}]},{rabbit_core_metrics,queue_stats,2,[{file,"src/rabbit_core_metrics.erl"},{line,250}]},{rabbit_quorum_queue,'-handle_tick/3-fun-0-',10,[{file,"src/rabbit_quorum_queue.erl"},{line,372}]}]}
rabbit_rabbit_worker_b: 2021-11-26 11:46:57.328 [error] emulator Error in process <0.26142.781> on node rabbit@rabbit_worker_b with exit value:
rabbit_rabbit_worker_b: {badarg,[{ets,insert,[queue_metrics,{{resource,<<"/">>,queue,<<"testing_queue-two">>},[{consumers,1},{consumer_capacity,1.0},{consumer_utilisation,1.0},{message_bytes_ready,0},{message_bytes_unacknowledged,0},{message_bytes,0},{message_bytes_persistent,0},{messages_persistent,0},{policy,<<"dead-letter">>},{operator_policy,''},{effective_policy_definition,[{<<"dead-letter-exchange">>,<<"dead-letter-dlx">>},{<<"dead-letter-routing-key">>,<<"dead-letter-dlx">>},{<<"delivery-limit">>,5}]},{memory,0},{state,down},{garbage_collection,[]},{leader,''},{online,[rabbit@rabbit_worker_a]},{members,[rabbit@rabbit_master,rabbit@rabbit_worker_b,rabbit@rabbit_worker_a]},{open_files,[{rabbit@rabbit_worker_b,0},{rabbit@rabbit_worker_a,0}]},{single_active_consumer_pid,''},{single_active_consumer_ctag,''},{messages_ram,0},{message_bytes_ram,0}],0}],[{error_info,#{cause => id,module => erl_stdlib_errors}}]},{rabbit_core_metrics,queue_stats,2,[{file,"src/rabbit_core_metrics.erl"},{line,250}]},{rabbit_quorum_queue,'-handle_tick/3-fun-0-',10,[{file,"src/rabbit_quorum_queue.erl"},{line,372}]}]}
rabbit_rabbit_worker_b: 2021-11-26 11:46:57.328 [error] emulator Error in process <0.26152.781> on node rabbit@rabbit_worker_b with exit value:

Thank you! 🙏

[lukebakken]

There have been many improvements to RabbitMQ and quorum queues since version 3.8.22. Please upgrade RabbitMQ and Erlang to the latest versions. If the issue re-appears after a network partition, provide the full logs from all nodes.

For what it's worth, if your cluster's operation returned to normal after the above events, there is nothing to worry about. Those messages look typical of events that happen during a network partition.

[vb0100]

Hi, thanks for your time. Interestingly cluster work returned to normal, but all consumers lost their connection, is that something that is expected? I will try to update and keep an eye on it 👀

[lukebakken]

all consumers lost their connection

• There's a good chance that the network issue that affected your cluster also affected your consumer connections
• Were all of the consumers connected to one node? If RabbitMQ restarted on that node the connections would be lost

[vb0100]

Current setup is working through HAProxy with roundrobin balancing.
About "all consumers" is a bit of a stretch from my side, sorry for that. Right now it's quite hard to tell if that was the case, but some of consumers definitely lost connectivity.