Registering stale services with Consul

[ccakes]

Nomad version

Nomad v0.9.2 (0283266)
(seen on 0.9.0+)

Operating system and Environment details

Amazon Linux 2

Issue

Nomad is registering services with Consul for old allocations

I have a 3-node Nomad server cluster with 2 Nomad agents, all on separate machines. One of the Nomad agents regularly registers services for previous allocations, resulting in a list of registered services failing health checks.

I've tried using the Consul HTTP API to manually deregister the services but they keep coming back.

Nomad:
Screenshot 2019-06-12 at 17 01 52

Consul:
Screenshot 2019-06-12 at 17 02 06

Reproduction steps

Unsure, doesn't seem to happen consistently but has only started since I upgraded from 0.8.6 to 0.9.0 (and then on to 0.9.2)

---

Let me know what logs/jobspec etc would be helpful and I can share.

[endocrimes]

@ccakes Hey! - This might be related to #5770, which I have a fix in flight for here: #5821 - If you could try running clients on 0.9.3 (coming imminently), that would be really useful.

It would also be helpful if you could share jobspecs for any jobs you've experienced this for. Thanks!

[mayuresh82]

We are hitting this in 0.9.1 as well.
@endocrimes so is the fix in 0.9.3 as you suggest ?

[robloxrob]

I believe that should be fixed on #5821 as suggested above.

[ccakes]

Thanks @endocrimes! Will update now and report back 👍

[stale]

This issue will be auto-closed because there hasn't been any activity for a few months. Feel free to open a new one if you still experience this problem 👍

[ccakes]

Apologies for forgetting to reply, for anyone else that finds this can confirm upgrading resolved the issue

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.