rabbitmq-streams delete_replica fails when the node of the replica to be deleted is down.

[kjnilsson]

Describe the bug

rabbit-3 is down in this case:

Removing a replica of queue s1 on node rabbit-3@nkarlVMD6R...
Error:
node_not_running

Reproduction steps

1. create stream in a cluster
2. stop a rabbit node
3. try to delete the stream replica on that node
   ...

Expected behavior

The replica should be deleted irrespective of node status. I.e. even if the node isn't in rabbit cluster anymore.

Additional context

The stream coordinator would keep trying even if the command was let through, we also would need to report success back if the target node is no longer in the rabbit cluster.

https://groups.google.com/g/rabbitmq-users/c/62eUMwZHvOM/m/dnVou9pLAwAJ

[arnaudmorin]

Extra note:
I was able to add a replica on some affected queues.
But some of them are failing adding a replica:

Adding a replica for queue q-agent-notifier-l2population-update_fanout on node rabbit@rabbit1...
Error:
{:disallowed, :out_of_sync_replica}

I have no idea if this could also be related to the same bug...

[kjnilsson]

Extra note: I was able to add a replica on some affected queues. But some of them are failing adding a replica:

Adding a replica for queue q-agent-notifier-l2population-update_fanout on node rabbit@rabbit1...
Error:
{:disallowed, :out_of_sync_replica}

I have no idea if this could also be related to the same bug...

This is part of a protection check that disallows adding another replica if there is already one configured that is too far behind on replication