-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Consul grpc_tls usage cannot deploy nomad jobs anymore #15266
Comments
Hi @suikast42 and thanks for raising this issue. We are currently looking into how this change impacts Nomad's integration, and what, if any fixes would be needed. |
See hashicorp/nomad#15266 for details. I plan to submit a followup PR to update these docs once Nomad releases a fix. At that point Nomad users will still need to be informed that they must upgrade Nomad to a compatible version *before* upgrading Consul to 1.14.
This has been investigated internally and we are currently working on building and testing a fix. A note has been added to the Consul upgrade guide which I will also post here:
OverviewConsul 1.14.0 changed the way in which gRPC listeners are configured, particularly when using TLS. Prior to the change, a single listener was responsible for handling plain-text and encrypted gRPC requests. In 1.14.0 and beyond, separate listeners will be used for each, defaulting to 8502 and 8503 for plain-text and TLS respectively. The change means that Nomad’s Consul Connect integration will not currently work when integrated with Consul clusters using TLS and running 1.14.0 or greater. Consul clusters that do not utilize TLS to protect communication are not affected and continue to work with Nomad’s Connect integration. Required ChangesConsul 1.14.0 clusters running TLS fail to be fingerprinted correctly by Nomad due to the changes described above in combination with modifications to the returns object from Consul’s agent self endpoint. The fingerprint results in the Nomad nodes FingerprinterThe Nomad Consul fingerprinter identifies the gRPC port Consul has exposed using the Consul gRPC Socket HookThe |
Regarding to your comment #15295 (comment) The working consul 1.13.3 sever and agent config Consul 1.14.0 does not booting up if the grpc_tls is missing. So I add only I hope the was the right answer. {
"node_name": "{{host_name}}",
"datacenter": "{{data_center}}",
"data_dir": "{{consul_data_dir}}",
"server": true,
"log_level" : "INFO",
"bind_addr": "0.0.0.0",
"advertise_addr": "{{host_ip}}",
"client_addr": "0.0.0.0",
"encrypt": "{{consul_encrypt_key}}",
"ui_config": {
"enabled" : true
},
"addresses": {
"grpc" : "127.0.0.1"
},
"ports": {
"http": -1,
"grpc" : 8502,
"https": 8501
},
"connect": {
"enabled": true
},
"retry_join":{{masters | to_json }},
"bootstrap_expect": {{masters|length}},
"auto_encrypt": {
"allow_tls": true
},
"performance" :{
"raft_multiplier" : 1
},
"node_meta": {
"node_type": "server"
},
"tls":{
"defaults":{
"ca_file": "{{cluster_intermediate_ca_bundle}}",
"cert_file": "{{consul_cert}}",
"key_file": "{{consul_cert_key}}",
"verify_incoming": true,
"verify_outgoing": true
},
"internal_rpc":{
"verify_server_hostname": true
}
},
"telemetry": {
"disable_hostname" : true
}
}
{
"node_name": "{{host_name}}",
"datacenter": "{{data_center}}",
"data_dir": "{{consul_data_dir}}",
"ports": {"https":8501},
"bind_addr": "0.0.0.0",
"advertise_addr": "{{host_ip}}",
"auto_encrypt": {
"tls": true
},
"retry_join":{{masters | to_json }},
"encrypt": "{{consul_encrypt_key}}",
"node_meta": {
"node_type": "worker"
},
"ports": {
"http": -1,
"grpc" : 8502,
"https": 8501
},
"connect": {
"enabled": true
},
"tls":{
"defaults":{
"ca_file": "{{cluster_intermediate_ca_bundle}}",
"cert_file": "{{consul_cert}}",
"key_file": "{{consul_cert_key}}",
"verify_incoming": false,
"verify_outgoing": true
},
"internal_rpc":{
"verify_server_hostname": true
}
},
"telemetry": {
"disable_hostname" : true
}
}
´´´ |
Hi folks, we've got a patch landed for this and it will ship in Nomad 1.4.3, which is coming out shortly. |
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. |
Nomad version
1.4.2
Consul version
1.14.0
I updated my consul agent today. Consul 1.40 have a breaking change
BREAKING CHANGES:
config: Add new ports.grpc_tls configuration option.
I remove the grpc falg form the config and replace it with grpc_tls . After that nomad is not able to deploy jobs with the error:
Placement Failure
Task Group "keycloak":
The text was updated successfully, but these errors were encountered: