Add peer again after being removed from jetstream cluster

[tomkretzschmar]

After removing a peer from a jetstream cluster using nats stream cluster peer-remove nats-0 we'd like to add that server again at a later point. However it seems as the information about evicted peers is stores permanently in the raft protocol.

Question: Is there a way to add a server with the same name back to a cluster, after it being removed once via the above command?

Some observations:

• Even after restarting the server in question it will only show in the JetStream summary but not in the Raft Meta Group Information when executing nats server report jetstream
• If we give the server a new, different name it will be added to the cluster immediately. If we change the name back it's gone again.
• Looking into the old nats-streaming-server code, there seemed to be an API (or subject) dedicated for adding peers again, however in the new Jetstream API this appears to be missing.

Some more context:

• We want to update between two helm chart versions without downtime
• We have two distinct deployments running, pointing to the same jetstream cluster and all nodes knowing each other
• We want to move a stream using placement tags
• In a rollback test scenario we removed the new deployment from the cluster after the rollback to the old deployment was executed
• Now we want to do the forward migration again, but the evicted nodes cannot be added to the cluster again

Thanks for your answers or suggestions already 🙏

[ripienaar]

In my experience it's not allowed again, @derekcollison did mention there is a period of time after which it might be removed from the RAFT layer and could come back, I've not observed that working

[tomkretzschmar]

We also did not observe it being removed from the layer, tried it after ~3 days.
However if it is never removed, I wonder wether the RAFT protocol size is growing forever?

[tomkretzschmar]

After running in parallel for ~24h the new nodes are now added to the cluster again. So there is some time period. Restarting the removed peer alone didn't help.

[ripienaar]

Please check it carefully in the past a returning peer had a bunch of problem. See this issue for some things you can check as a start

nats-io/natscli#654

[derekcollison]

An AddPeer will reverse.

[tomkretzschmar]

Can this be triggered from the nats cli or any other way in my cluster? From what I can see this is not possible as it's not even exposed by the jetstream API, correct?

[derekcollison]

Not via NATS cli. IIRC you need to restart the removed peer.