-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
consul service registration failed - invalid port / port label not found #9709
Comments
Can you try if the errors are gone if you move the service stanza also up to the group level? |
Hi @MorphBonehunter , I think this job is actually not quite valid - the |
@shoenig this is the case. |
Today i encounter the problem again.
This time it was also followed from some other messages which seems strange:
Which seems to be clear as the registration doesn't work.
Restarting the task doesn't work, so i have to resubmit the job with an minor change (CPU MHz) to get it working again.
Even after Node drain this was spammed so i had to restart the drained nodes nomad.
|
@MorphBonehunter I believe this issue is fixed and covered by #9736 You should be able to reliably reproduce this by causing an inplace update to occur (updating a service stanza value). A destructive update like updating an environment variable will create a new allocation and avoid the issue. #9736 prevents inplace updates from nullifying the tasks ports, which eventually cause the error logs you have shared. |
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. |
Nomad version
Nomad v1.0.1 (c9c68aa)
Operating system and Environment details
ArchLinux 5.4.84-1-lts
Issue
Service registration fails every now and then on resubmitting job (for example update docker image version).
Reproduction steps
I can not reliable reproduce this in my environment as this occurs not on every job update.
It seems to happen after i moved the deprecated network stanza in the task level to the group level and it is not bound to one job, it happens on different jobs when it appears.
It happen to the job below after i change the following (which doesn't touch the service definiton etc.):
I can fix the problem with an stop/start cycle of the job.
Job file (if appropriate)
Nomad Client logs (if appropriate)
I can see this error in the GUI and in the logs:
The text was updated successfully, but these errors were encountered: