Leaderless QQs reporting noproc

[Ayanda-D]

Community Support Policy

• I have read RabbitMQ's Community Support Policy

RabbitMQ version used

3.13.6

How is RabbitMQ deployed?

Debian package

Steps to reproduce the behavior in question

Hi Team 👋

We've been running a number of test scenarios over the last couple of months to evaluate QQ performance and resiliency (before upgrading some of our client critical CMQs) and sometimes (not often) we encounter leaderless QQs, where leaders become unavailable and are reported as noproc despite sanity checks such as check_if_node_is_quorum_critical commands succeeding. On our maintenance and upgrade procedures, we consider it a big risk to restart brokers and machines on which leaders are expected to be running but wouldn't, which would result in message loss. Is this a known issue? This looks very similar QQ leader behaviour described for followers here #10423 (although we are prioritizing addressing the leaders, from which we'd still access the queue).

We have been looking at and experimenting with a custom workaround CLI command which (efficiently) checks sanity of leaders, which we're looking to PR for comments and review, and to have available for use in later rabbit releases. Plan is to optionally use this command along with the already provided upgrade and maintenance queue checks when upgrading critical environments. Command signature we have so far (open to other name suggestions 🙏 ):

rabbitmq-queues leader_health_check [--vhost <vhost>] [--global] <pattern>

which spawns concurrent leader checks (with concurrency limit/threshold which we dont want to exceed), where:

["<pattern>", "regular expression pattern used to match quorum queues"],
["--global", "run leader health check for all queues in all virtual hosts on the node"],

We'll follow up with the implementation for further comments. This is to mitigate the major risk factor of noproc leaders (of potential message loss), which is currently a blocker on our CMQs -> QQs upgrade path.

[kjnilsson]

I think the first thing to do is to provide information on how these queues failed so that we can fix that (if still an issue).

[Ayanda-D]

We see this on very rare occasions, unfortunately haven't been able to consistently reproduce. Didn't manage to simulate/reproduce this with exit(Pid, kill) either (ra_state/leaderboards seem to be updated as expected). Bit difficult to see the root cause, but having seen it, couldnt shrug it off & looked at the workaround cli command to protect things a bit on restarts.

[kjnilsson]

I suspect that this could be improved by using whereis(K) here and updating the status accordingly. that is what the quorum_status command does.

[Ayanda-D]

Yep, I think that will also improve the issue, I've just tried it out. I'll forward a PR for this as well soon. The only other thing that the cli command does is an explicit ra_server_proc:ping/1 per leader (slightly more expensive). Let me prepare these and forward for review 👍

[kjnilsson]

cool. ping/1 will be much too expensive for large numbers of QQs and this should provide much of the same guarantees whilst being pretty cheap.

[Ayanda-D]

Ok cool. i've opened #12709 - please check. i'm ignoring this clause - i'm not seeing where/how ra_state can hold anything else? Unless this was for backward compatibility (or should we let it crash if anything else).

[michaelklishin]

Keep in mind that RabbitMQ 3.13.x is out of community support, so any contributions in this area won't be available in OSS RabbitMQ 3.13.x.

[Ayanda-D]

Ok 👍

[michaelklishin]

This should be addressed by #12727 (contributed by @Ayanda-D) and available for testing in alpha 4.0.4-alpha.0caa0664.