-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
When whole aditional region goes down imposibble to get nomad server-members #3600
Comments
Hi, thanks for submitting this issue. Would you mind submitting logs from the Nomad client when you experience this issue? This issue appears similar to #1515 and looks to be a duplicate. |
@chelseakomlo yes you right, this is a duplicate and we have the same logs:
But what we can do now for cleanup, and verify that we fully made cleanup? |
Thanks for confirming this is a duplicate, and the logs! See here for documentation on outage recovery: https://www.nomadproject.io/guides/outage.html It might be a good idea to try bringing up one server in the "down" region, and allow it to rejoin. You could then have the server leave cleanly, for example by using the following command: https://www.nomadproject.io/docs/commands/server-force-leave.html Let us know how this goes. |
@chelseakomlo Sorry but three doesn't described situation for outage recovery in multiregion configurations. Bring up server in lost region was first idea what come to us, but for some reasons this is not possible. We try to remove servers by memory, but without any success, seems that we doesn't remembers all their names ;-(. Maybe esist any hidden api that can list all serveres in all regions? |
Sure, we will add to our documentation about outage recovery in a multi-region cluster, that is a good suggestion. Here is a link to our HTTP api for nodes: https://www.nomadproject.io/api/nodes.html as well as system maintenance: https://www.nomadproject.io/api/system.html |
Closing as this issue is a duplicate, let us know if you continue to experience difficulty with this. |
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. |
Nomad version
0.6.3
After some time we decide remove someone. We simple remove all nomad servers from that region. And after that when we try to watch nomad server-members in alive regions we got follow error:
Which is on one side is correct because we have dead region, but how can we got to make work
nomad server-members
command?The text was updated successfully, but these errors were encountered: