-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Services registered with Nomad Service Provider stayed forever after a major server outage #16762
Comments
I think this is the same as #15630 - was only a single node test cluster in my case but could see how all server nodes being down would lead to the same situation |
I just had a very similar scenario. I bought down two of the 3 Nomad servers in our dev env. Did manually recovery via peers.json and found my entire dev env down. I was also under impression that service outage should hurt clients but it looks like it did. Furthermore, the recovery required us to restart the jobs impacted. We also have the issue where zombie service allocations are still registered. This all happened on 1.5.3. |
@schmichael any chance you folks can clarify what are the guarantees around scenarios involving Nomad server downtime with regards to Nomad variables & templates? |
Made this small script to clean up. Might be useful for anyone who is stuck with bunch of zombie services: #!/bin/sh
set -e
services=$(nomad service list -t '{{ range (index . 0).Services }}{{printf "%s\n" .ServiceName }}{{ end }}')
for svc in $services; do
echo "checking $svc:"
data=$(nomad service info -t '{{ range . }}{{ printf "%s" .AllocID }}%{{ printf "%s\n" .ID }}{{ end }}' "$svc" | uniq)
for d in $data; do
alloc=$(echo "$d" | cut -d'%' -f1)
svc_id=$(echo "$d" | cut -d'%' -f2)
echo " checking $alloc ($svc_id)"
if ! nomad alloc status "$alloc" > /dev/null 2>&1; then
echo " !! removing $svc_id"
nomad service delete "$svc" "$svc_id" > /dev/null 2>&1
fi
done
echo
done |
@michael-strigo Thanks a lot for this. I made slight modifications to loop over all namespaces and also ignore namespaces where no service registrations are found. Here's the modified version:
|
This issue seems to still be present in 1.6.1. Is there any information we can provide to help debug and solve the issue? |
I'm going to assign myself this issue, considering it a duplicate of #16616. Please see my comment #16616 (comment) if anyone has anything additional to share. Closing as duplicate. |
Nomad version
Nomad v1.5.2
BuildDate 2023-03-21T22:54:38Z
Revision 9a2fdb5
Operating system and Environment details
Issue
This bug was actually discovered together with #16760. But I believe they're not caused by the same reason, so I opened this issue separately.
So the thing is, for some reason, all three of my Nomad servers went down. And I didn't get them back online in time because I was trying to figure out what was causing them to go down. I thought my running jobs wouldn't be affected during this time. But, unexpectedly, all my jobs involving
nomadVar
ornomadService
template functions also went down (other jobs keep running).Anyway, after I brought the servers back online, my jobs that unexpectedly went down are running again. But another thing happened, the services registered by the jobs that down unexpectedly before are not removed.
Reproduction steps
If everything is fine, do
docker container ls
and you should get 1 running container:foobar-{allocId}
. Donomad service list
and you should get a service named "foobar". Donomad service info foobar
and you should only get one entry.Now go to terminal tab 1 and close the server (
Ctrl+C
).Now go to terminal tab 3 and wait a few minutes until the log "exceeded maximum retries" to appear.
At this point, do
docker container ls
and you should not get any running containers.Now go to terminal tab 1 and start the server again:
docker container ls
and you should get 1 running container:foobar-{allocId}
(if not, wait a few seconds). Donomad service list
and you should get a service named "foobar". Donomad service info foobar
and you should get two entries, the one you saw in step 6 should still there.The text was updated successfully, but these errors were encountered: