Nomad 0.7.x gives error parsing some ports #4008
Comments
SummaryThis is a duplicate of #3681, but that's an extremely long issue so let me summarize here! The service mentioned in the error is trying to register with an invalid port: DetailsNomad 0.6 allowed empty or invalid Nomad 0.7.1 began requiring valid The error logging has been improved on master for the upcoming 0.8 release: Since service {
name = "catalogue-db"
tags = ["db", "catalogue", "catalogue-db"]
port = "http"
}The port ResolutionThere are a few fixes:
|
|
For reference I used this simplified version of the job file to ease reproduction: job "sockshop" {
datacenters = ["dc1"]
update {
stagger = "10s"
max_parallel = 1
}
# - catalogue - #
group "catalogue" {
count = 1
restart {
attempts = 10
interval = "5m"
delay = "25s"
mode = "delay"
}
# - db - #
task "cataloguedb" {
driver = "docker"
config {
image = "weaveworksdemos/catalogue-db:0.3.5"
hostname = "catalogue-db.service.consul"
network_mode = "weave"
dns_servers = ["172.17.0.1"]
dns_search_domains = ["service.consul"]
}
template {
data = <<EOH
MYSQL_ROOT_PASSWORD="foo"
EOH
destination = "secrets/mysql_root_pwd.env"
env = true
}
env {
MYSQL_DATABASE = "socksdb"
MYSQL_ALLOW_EMPTY_PASSWORD = "false"
}
service {
name = "catalogue-db"
tags = ["db", "catalogue", "catalogue-db"]
}
resources {
cpu = 100 # 100 Mhz
memory = 256 # 256MB
network {
mbits = 10
port "http" {}
}
}
} # - end db - #
} # - end catalogue - #
} |
|
Thanks @schmichael. Just saw these updates. Thanks for looking at this and suggesting a workaround. I will test tomorrow. Can you explain why I got errors for catalogue-db and user-db, but not for other services that did the same thing with regard to ports? Can you also clarify what the doc at https://www.nomadproject.io/docs/job-specification/service.html#port is trying to say about mapping a port in the service to a port in the network stanza? "port: Specifies the label of the port on which this service is running. Note this is the label of the port and not the port number unless address_mode = driver. The port label must match one defined in the network stanza unless you're also using address_mode="driver". Numeric ports may be used when in driver addressing mode." It seemed to me that by specifying the service port as "http" and matching the same under network, I was complying with Nomad job specification rules. Thanks. |
|
Getting rid of the port definition in each service did fix the problem for me. I can now run the demo on Nomad 0.7.1 and see the Nomad UI. Thanks. |
That would have been true if you had used host or bridge networking. Since you're using a custom network and driver (sockshop+weave), Nomad assumes you want to advertise the driver's port. This autodetection is unfortunately subtle and creates a confusing number of configuration options I'm afraid. The easiest way to think about it is:
Hope that helps and please let me know if there's still confusion or if you have any documentation suggestions! This is a classic example of defaults hopefully making things Just Work for most people but failing in very confusing ways for others. |
|
That does help quite a bit, @schmichael. Also, any ideas on why only 2 of my 13 services were affected by the problem? |
Hopefully helps prevent more issues like #3681 and #4008. The port/address_mode logic is really subtle, and it took me a long time to diagnose #4008 despite being the one to have addressed the duplicate issue before! Not to mention I wrote the code! Definitely need to do something to make it more understandable...
Absolutely. I made an attempt in #4055, but I would love advice.
All of your other services had a |
|
Thanks so much for clarifying. I do see the port map settings on the other 11. I have no idea why they were not on the other 2. Will review the other question tomorrow. |
|
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. |
Nomad version
Output from
nomad versionNomad v0.7.1 (0b295d3)
Operating system and Environment details
Ubuntu 16.04.2 LTS (Xenial Xerus)
Issue
Nomad configuration that works on Nomad 0.6.3 does not work on 0.7.0 or 0.7.1
After running
nomad run sockshop.nomad, two of the database services fail to start, giving error:"unable to get address for service "catalogue-db": invalid port "http": strconv.Atoi: parsing "http": invalid syntax"
Note that other services in the Nomad configuration also have http ports which are parsed fine.
Reproduction steps
See README.md for https://github.com/rberlind/nomad-microservices-demo-with-weave/tree/nomad-0.7.1
Nomad Server logs (if appropriate)
Nomad Client logs (if appropriate)
Job file (if appropriate)
See https://github.com/rberlind/nomad-microservices-demo-with-weave/blob/nomad-0.7.1/shared/jobs/sockshop.nomad
The text was updated successfully, but these errors were encountered: