-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Nomad can not use consul ingress-gateways because tasks use protocol tcp #8647
Comments
How about adding below into
sidecar_service {
proxy {
config {
protocol = "http"
}
}
} |
Great idea. I tried setting protocol with no change in behavior
I believe this is the documentation page that lists the available config options |
By default services deployed within Consul service mesh are configured as Any services you wish to associate with an ingress gateway listener must previously be configured to use the same protocol as that listener prior to association, otherwise a configuration error will be returned. |
Hi @spuder there might be some overlap with #8294 (comment) -- apparently Michael got it working there |
Good suggestions. I've modified the job to use
countdash.job
I still am seeing this error when I attempt to register the service
I've ensured that the job is completely stoped and the service is eliminated from consul before submitting the job again. |
I was running into this exact same issue and finally was able to get this working by creating a "service-defaults" config entry in Consul (not through Nomad), with the same name as the service - and specifying the protocol there For example, for your above service (assume it is named "web") resource "consul_config_entry" "web" {
kind = "service-defaults"
name = "web"
config_json = jsonencode({
Protocol : "http"
})
} Stop and start your job in Nomad, then try registering the gateway. It is unfortunate that when registering the service, Consul seems to ignore the specification inside the Nomad Connect proxy config stanza making it impossible to accomplish this in a single Nomad configuration . I dug through the source code a little but was unable to find anything that stood out as to why that is. Seems like a bug but this should provide a workaround in the meantime. |
This PR adds initial support for running Consul Connect Ingress Gateways (CIGs) in Nomad. These gateways are declared as part of a task group level service definition within the connect stanza. ```hcl service { connect { gateway { proxy { // envoy proxy configuration } ingress { // ingress-gateway configuration entry } } } } ``` A gateway can be run in `bridge` or `host` networking mode, with the caveat that host networking necessitates manually specifying the Envoy admin listener (which cannot be disabled) via the service port value. Currently Envoy is the only supported gateway implementation in Consul, and Nomad only supports running Envoy as a gateway using the docker driver. Aims to address #8294 and tangentially #8647
This PR adds initial support for running Consul Connect Ingress Gateways (CIGs) in Nomad. These gateways are declared as part of a task group level service definition within the connect stanza. ```hcl service { connect { gateway { proxy { // envoy proxy configuration } ingress { // ingress-gateway configuration entry } } } } ``` A gateway can be run in `bridge` or `host` networking mode, with the caveat that host networking necessitates manually specifying the Envoy admin listener (which cannot be disabled) via the service port value. Currently Envoy is the only supported gateway implementation in Consul, and Nomad only supports running Envoy as a gateway using the docker driver. Aims to address #8294 and tangentially #8647
Mhm, so while the ingress gateway in 0.12.4 seems to work nicely for tcp services (just tried :D) it seems to fail rather horribly for http. Do we miss something @shoenig or is there simply no support for it yet? |
What's the problem you're seeing @apollo13 ? Using the |
@shoenig, exactly. I was just wondering if there is already something builtin in Nomad to set the type (not that we miss something :)) |
I think we'll publish a learn guide in the near future detailing the ins-and-outs of running Gateways in Nomad. For now though, here's a little example I've been using: set service defaults$ cat ig-service-defaults.json {
"Kind": "service-defaults",
"Name": "uuid-api",
"Protocol": "http"
} consul config write ig-service-defaults.json example job file# $ cat ig-http.nomad
job "ig-http" {
datacenters = ["dc1"]
group "ingress-group" {
network {
mode = "bridge"
port "inbound" {
static = 8080
to = 8080
}
}
service {
name = "my-ingress-service"
port = "8080"
connect {
gateway {
proxy {
connect_timeout = "500ms"
}
ingress {
listener {
port = 8080
protocol = "http"
service {
name = "uuid-api"
hosts = ["example.com", "example.com:8080"]
}
}
}
}
}
}
}
group "generator" {
network {
mode = "host"
port "api" {}
}
service {
name = "uuid-api"
port = "${NOMAD_PORT_api}"
connect {
native = true
}
}
task "generate" {
driver = "docker"
config {
image = "hashicorpnomad/uuid-api:v3"
network_mode = "host"
}
env {
BIND = "0.0.0.0"
PORT = "${NOMAD_PORT_api}"
}
}
}
} $ nomad job run ig-http.nomad inspect
|
Thanks, do you think setting the service type/proto directly in nomad would be in scope in the future, or is that something out of scope for nomad totally? |
It might be possible in the future. We shied away from managing anything but the We've discussed internally some possible mechanics Consul can provide to improve the multi-writer story - if that stuff gets implemented then I don't see why Nomad couldn't make use of it. If you don't mind opening a ticket describing your use case, that would definitely help us gauge the interest for that feature. |
Oh, thank you for the extensive explanation. I didn't realize that service defaults is the only way to set the protocol. I thought it would be possible to do that during service registration (I never looked that closely at consul aside from it's nomad integration). Not sure if a new ticket makes sense; the use-case is simply providing an ingress gateway to the outside world (well mostly internal infra) so everything is encrypted. As it stands currently most people use traefik or so but then (usually) the traffic between traefik and the services is not encrypted. That said traefik just laid the groundwork to support connect services traefik/traefik@76f42a3 EDIT:// To be expand on the usecase a bit: When I said "simply" I ment something along the lines of "people can just submit a job to nomad and the rest will be taken care of". Ie they shouldn't have to know about specific consul quirks for configuration. |
@shoenig Is it possible to gauge interest by simple a thumbs up? I just want to submit a Nomad job that configures an ingress controller to route to internal Connect services via HTTP header. The use case is very simple and not new conceptually. I've had this methodology working in Docker Swarm for about 3 years and would like to finalize the transition over to Nomad + Consul Connect. |
I have the same issue with proxy config stanza in Nomad. I setted protocol = "http" but it seems Consul ignored this config. |
Counter-example if I understand it right: There are many P2P applications (notably Ethereum) that suppose publicly reachable TCP and UDP on the same port. Today Nomad doesn't have that distinction - specifying a port with a service for the port means both TCP and UDP. In that scenario one would need two ingresses to the same service, unless I'm misisng something. |
For future reference, here is our current work around
Here is an example of how you may configure consul using terraform
Note that if you try and change this on a running service, you will get an error because the service will already have the default type of |
Any update on this issue? Like others in this thread I would also like to avoid making updates in Consul directly, outside of the Nomad job. |
Now that Consul versions pre-dating The global nature of Certainly open to feedback! |
@shoenig that certainly seems more than fine for us at SeatGeek. At the moment, we have to configure this and a ServiceResolver separately from registering a Nomad job during a deploy, meaning much more coordination for something that is more or less a unit of work (registering a service against Consul for service discovery). Our initial use case is exactly setting the service protocol correctly so that Consul Connect does the right thing at the proxy level, though there are other things we want to configure as we expand our Consul Connect adoption past the initial phase. It would even be fine if the logic here was gated behind some sort of |
See also #14802 for an example of the challenges around updating the configuration. |
Any progress in this issue? That's realy a mess to maintain the services in that way if you want introduce distrubuted tracing over envoy |
For posterity, this is how to set the proxy defaults as a Terraform resource. resource "consul_config_entry" "proxy_defaults_global" {
kind = "proxy-defaults"
name = "global"
config_json = jsonencode({
Config = {
protocol = "http"
}
})
} |
Nomad 0.11.1
Consul 1.8.2
Consul Ingress-Gateways support
tcp
andhttp
listeners. Http listeners are preferred because they allow for multiple services to listen on a single port and use Host header identification.Problem
Nomad jobs default to service type of
tcp
. There does not appear to be a documented way to change a nomad job to usehttp
as the service type. As a result the user will get the following error when they attempt to create a listener for it.https://www.nomadproject.io/docs/job-specification/service
Steps to reproduce
count-dash.job
Expected result
The service should be added to the ingress controller
Actual result
Consul throws this warning
The text was updated successfully, but these errors were encountered: