Consul Connect sidecar proxies require additional configuration for gRPC-TLS listener #15360
Assignees
Labels
stage/accepted
Confirmed, and intend to work on. No timeline committment though.
theme/consul/connect
Consul Connect integration
theme/networking
type/bug
Milestone
Comments
lgfa29
added
the
stage/accepted
|
I believe it might be worth it to add this information also to the Consul Update Docs page, as it currently still reads as follows:
I've just tried an update of my Consul cluster to 1.14.3, and because I was running Nomad 1.4.3 I disregarded the warning. The Connect sidecar starts, but the actual workload tasks get "connection refused" errors when trying to connect to their consul connect upstreams. Sorry, please disregard above, I just saw that @jrasell's PR above already corrects the docs. Should be merged ASAP. |
|
Working on this in #15701 versionsconsul.hcldatacenter = "dc1"
advertise_addr = "127.0.0.1"
bind_addr = "127.0.0.1"
client_addr = "127.0.0.1"
data_dir = "./data"
node_name = "s0"
server = true
bootstrap_expect = 1
ports {
http = -1
https = 8501
grpc_tls = 8503
}
ui_config {
enabled = false
}
tls {
defaults {
ca_file = "./tls/consul-agent-ca.pem"
cert_file = "./tls/dc1-server-consul-0.pem"
key_file = "./tls/dc1-server-consul-0-key.pem"
verify_incoming = true
verify_outgoing = true
}
internal_rpc {
verify_server_hostname = true
}
}
auto_encrypt {
allow_tls = true
}nomad.hclserver {
enabled = true
}
client {
enabled = true
}
consul {
grpc_ca_file = "/home/shoenig/Work/tickets/grpc-connect/tls/consul-agent-ca.pem"
ca_file = "/home/shoenig/Work/tickets/grpc-connect/tls/consul-agent-ca.pem"
cert_file = "/home/shoenig/Work/tickets/grpc-connect/tls/dc1-client-consul-0.pem"
key_file = "/home/shoenig/Work/tickets/grpc-connect/tls/dc1-client-consul-0-key.pem"
ssl = true
address = "127.0.0.1:8501"
grpc_address = "127.0.0.1:8503"
}tls files
Start consul Start nomad nomad node statusapi.nomadjob "api" {
datacenters = ["dc1"]
group "api" {
network {
mode = "bridge"
}
service {
name = "count-api"
port = "9001"
connect {
sidecar_service {}
}
}
task "web" {
driver = "docker"
config {
image = "hashicorpdev/counter-api:v3"
auth_soft_fail = true
}
}
}
}Envoy bootstrap log line Envoy logs in the alloc In the alloc, config_dump |
|
Should be able to get this working with fixes on the Consul side: hashicorp/consul#15913 |
This was referenced Jan 11, 2023
|
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Nomad version
Nomad v1.4.3 (f464aca)
Issue
Consul 1.14 introduced changes to the gRPC listener used for Envoy xDS configuration, splitting support for plaintext and TLS-enabled listeners for the protocol across two ports.
An initial fix for discovering the TLS-enabled listener via Consul's
/agent/selfendpoint landed in #15309, however when utilizing gRPC over TLS theconsul connect envoy [...]command internally invoked by Nomad to bootstrap Envoy requires the addition of the -grpc-ca-file or -grpc-ca-path flags (or environment variable equivalents) to inject a trusted CA certificate into the generated Envoy config.When the gRPC CA has not been explicitly passed, Envoy sidecar proxy tasks will initially deploy but will never retrieve their running configuration from the local Consul agent, sitting in a fail/retry loop unable to negotiate a connection with the gRPC-TLS listener..
To support this, Nomad will likely need to make a CA certificate available
within the allocation filesystem wherever aon disk on each scheduler, or the Consul CLI will need to be extended to allow other means of providing the CA certificate (i.e. as an inline PEM rather than file path).sidecar_service{}stanza has been defined,As a temporary workaround to restore proxy connectivity, the plaintext Consul HTTP and gRPC endpoints may be enabled and bound to a loopback interface within the local Consul agent, as Nomad will default to discovering the plaintext gRPC listener as long as HTTP is the URL schema in the
consul.addressconfiguration.The text was updated successfully, but these errors were encountered: