-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Consul Connect sidecar proxies require additional configuration for gRPC-TLS listener #15360
Comments
I believe it might be worth it to add this information also to the Consul Update Docs page, as it currently still reads as follows:
I've just tried an update of my Consul cluster to 1.14.3, and because I was running Nomad 1.4.3 I disregarded the warning. The Connect sidecar starts, but the actual workload tasks get "connection refused" errors when trying to connect to their consul connect upstreams. Sorry, please disregard above, I just saw that @jrasell's PR above already corrects the docs. Should be merged ASAP. |
Working on this in #15701 versions
consul.hcldatacenter = "dc1"
advertise_addr = "127.0.0.1"
bind_addr = "127.0.0.1"
client_addr = "127.0.0.1"
data_dir = "./data"
node_name = "s0"
server = true
bootstrap_expect = 1
ports {
http = -1
https = 8501
grpc_tls = 8503
}
ui_config {
enabled = false
}
tls {
defaults {
ca_file = "./tls/consul-agent-ca.pem"
cert_file = "./tls/dc1-server-consul-0.pem"
key_file = "./tls/dc1-server-consul-0-key.pem"
verify_incoming = true
verify_outgoing = true
}
internal_rpc {
verify_server_hostname = true
}
}
auto_encrypt {
allow_tls = true
} nomad.hclserver {
enabled = true
}
client {
enabled = true
}
consul {
grpc_ca_file = "/home/shoenig/Work/tickets/grpc-connect/tls/consul-agent-ca.pem"
ca_file = "/home/shoenig/Work/tickets/grpc-connect/tls/consul-agent-ca.pem"
cert_file = "/home/shoenig/Work/tickets/grpc-connect/tls/dc1-client-consul-0.pem"
key_file = "/home/shoenig/Work/tickets/grpc-connect/tls/dc1-client-consul-0-key.pem"
ssl = true
address = "127.0.0.1:8501"
grpc_address = "127.0.0.1:8503"
} tls files
Start consul
Start nomad
nomad node status
api.nomadjob "api" {
datacenters = ["dc1"]
group "api" {
network {
mode = "bridge"
}
service {
name = "count-api"
port = "9001"
connect {
sidecar_service {}
}
}
task "web" {
driver = "docker"
config {
image = "hashicorpdev/counter-api:v3"
auth_soft_fail = true
}
}
}
} Envoy bootstrap log line
Envoy logs in the alloc
In the alloc,
config_dump
|
Should be able to get this working with fixes on the Consul side: hashicorp/consul#15913 |
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. |
Nomad version
Nomad v1.4.3 (f464aca)
Issue
Consul 1.14 introduced changes to the gRPC listener used for Envoy xDS configuration, splitting support for plaintext and TLS-enabled listeners for the protocol across two ports.
An initial fix for discovering the TLS-enabled listener via Consul's
/agent/self
endpoint landed in #15309, however when utilizing gRPC over TLS theconsul connect envoy [...]
command internally invoked by Nomad to bootstrap Envoy requires the addition of the -grpc-ca-file or -grpc-ca-path flags (or environment variable equivalents) to inject a trusted CA certificate into the generated Envoy config.When the gRPC CA has not been explicitly passed, Envoy sidecar proxy tasks will initially deploy but will never retrieve their running configuration from the local Consul agent, sitting in a fail/retry loop unable to negotiate a connection with the gRPC-TLS listener..
To support this, Nomad will likely need to make a CA certificate available
within the allocation filesystem wherever aon disk on each scheduler, or the Consul CLI will need to be extended to allow other means of providing the CA certificate (i.e. as an inline PEM rather than file path).sidecar_service{}
stanza has been defined,As a temporary workaround to restore proxy connectivity, the plaintext Consul HTTP and gRPC endpoints may be enabled and bound to a loopback interface within the local Consul agent, as Nomad will default to discovering the plaintext gRPC listener as long as HTTP is the URL schema in the
consul.address
configuration.The text was updated successfully, but these errors were encountered: