-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
envoyproxy/envoy:v${NOMAD_envoy_version} error="API error (400): invalid tag format" #9887
Comments
Hi @bert2002, from the error message I think what's happening is that it's not getting interpolated at all, as that "API error" should be bubbling up from the driver. Can you share the jobspec? It might help us figure out what's going on there. |
My jobspec has quite a lot of groups, so I will share a limited one. I had to drain the node and the container got started on another node and it was without any problem. Now on the same node (where the problem was) containers are working again without any problem. The only thing I can image is a runtime error or external service was not reachable (docker hub, etc.) Any other idea? |
@bert2002 this is the jobspec you shared and I don't see any setting for the Envoy proxy image. You saw that error without trying to set the image via interpolation? job "app1" {
datacenters = ["staging"]
type = "service"
reschedule {
delay = "10s"
delay_function = "exponential"
max_delay = "120s"
unlimited = true
}
#
# collectd
#
group "collectd" {
count = 1
network {
mode = "bridge"
}
restart {
interval = "2m"
attempts = 8
delay = "15s"
mode = "delay"
}
service {
name = "collectd"
connect {
sidecar_service {
proxy {
upstreams {
destination_name = "redis"
local_bind_port = 6379
}
}
}
}
}
task "collectd" {
driver = "docker"
leader = true
config {
image = var.docker_image_collectd
mounts = [
{
type = "bind"
target = "/etc/collectd/collectd.conf"
source = "/opt/app/collectd/collectd.conf"
},
{
type = "bind"
target = "/etc/collectd/collectd.conf.d"
source = "/opt/app/collectd/collectd.conf.d/"
},
{
type = "bind"
target = "/usr/local/lib/collectd"
source = "/opt/app/collectd/plugins/"
}
]
}
}
task "filebeat" {
driver = "docker"
resources {
memory = 100
cpu = 50
}
config {
image = var.docker_image_filebeat
mounts = [
{
type = "bind"
target = "/usr/share/filebeat/filebeat.yml"
source = "/opt/app/filebeat/filebeat.yml"
}
]
}
}
}
}
|
I ran into that same error today without setting the image (my jobspec is very similar), though I can't reproduce this either unfortunately. |
yes that is correct and it just happened again (on the same node). I drained it again and it works fine on the other nodes. Is there any more information/logs I can provide? |
it just happened again after I wanted to restart a alloc (on a different node) |
I updated to
|
@tgross is it possible to set |
Hey! I have the same problem. Could you please provide any information how this can be solved? |
@empikls as workaround I am using a fixed version. In
|
Thanks a lot @bert2002 ! |
Had the same problem too.
|
same issue for me. it's a test job so I just re-run it and it starts working again.
|
+1 facing the same issue. |
@mr-karan (and others facing the issue), would you mind clicking the 👍 in the issue so we can better track common problems? This issue does seem to be a bit unpredictable, so we don't have an update yet. The workaround from @bert2002 is the best option right now. You can check which Envoy version to use based on your Consul version in the docs. |
I can easily reproduce this by simply restarting the sidecar task via the UI. |
@legege Can confirm, restart via UI causes the error. |
+1 I am experiencing this as well |
+1 got the same error. came out of the blue |
This issue is still present in v 1.3.1. After a restart via UI job fails due to error while pulling envoy image. |
What is not clear to me, where is the ${NOMAD_envoy_version} variable set. The documentation says it comes from a consul query? |
This is happening to me very, very rarely. It's not a big deal because eventually the alloc gets replaced but it's weird to see it every few days |
Still happening in v1.4.3 |
yup I found the same issue in 1.4.3. |
It is still happening, more often than last week or two. Thnx |
This happened in This issue causes service interruption because Nomad scaling actions make existing allocations restart without reason. Those restarts also fail, and introduce delays before retrying and reallocating the allocation. |
The more jobs we add, the more frequently we see this. Initially it was maybe once a month, but now we have service interruptions and downtime on a weekly basis being caused by this. |
+1 We recently experienced this problem as one of our applications running in Prod crashed and we received the error when Nomad was trying to restart the task. We had to issue a restart and the application worked fine after this. Strangely, we cannot reproduce the error just by simply restarting the sidecar task via the UI. The versions we are running are Nomad v1.3.5, Consul v1.12.4. |
Same issue. For us it happened when manually restarting an alloc with a connect proxy sidecar |
One more occurrence of this, and same as in the previous message it happened when I manually restarted an allocation from the Nomad UI. Using Nomad 1.4.3 |
Hi everyone, I am currently taking a time-boxed look into this. If anyone has logs from the client where an allocation and task gets into this problem, could you please share them? This would really help dive into this bug, which I have been unable to reproduce so far. |
Hi everyone, I have been able to reproduce this locally and should have a proposed fix ready at some point soon. I will post my reproduction steps below for other readers and testers. Linux is required.
JobSpec:
|
Nomad version
Operating system and Environment details
Issue
After initiating a restart of a allocation, nomad cannot find envoy sidecar image anymore. I did a couple of restarts earlier today (same config) and did not run into this problem and consul does not report any problems.
Nomad Server logs (if appropriate)
Any way to find out what
NOMAD_envoy_version
is set to?Cheers,
bert2002
The text was updated successfully, but these errors were encountered: