-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[question] Is there any way to disable TCP Sidecar Listening Check? #9773
Comments
The https://github.com/hashicorp/consul/blob/master/agent/sidecar_service.go#L172
Are you saying you're setting The check serves a purpose, combined with |
Hey @shoenig thanks a lot for the fast answer! Really appreciate it. Nope, the check works fine. However, this particular application keeps logging the sidecar ping connection somehow, reporting it as an error (failed attempt to connect). Maybe due to the sudden connection closing, I don't really know. On the Consul side, everything is fine tho. So, since I have no control whatsoever regarding the application, I want to remove (or replace) the sidecar check with something else. So, from Consul's side, as said in the line that you provided, that is possible by passing any check to the sidecar service because the sidecar ping is created only if no checks are given. However, since there's no way to pass checks to the sidecar service from Nomad (neither sidecar_task nor sidecar_service accepts checks), I don't see a straightforward way to do it. I was even considering changing the Envoy image to somehow block those health checks, but after some consideration, I think that these checks aren't registered as Envoy health checks, but they're somehow managed by Consul (might be wrong tho). |
The check is executed by Consul, and should only be establishing a TCP connection with envoy (or whatever your sidecar_task is). If your application is detecting the pings, that's quite perplexing! And suggests there's probably some misconfiguration or very weird iptables rules in play. |
@shoenig don't think so... Like I said before, I don't have the details of the application, but as far as I know the application keeps listening for a TCP connection with some specific data. The TCP check from Consul is proxied through Envoy and since it's not a valid request, just an empty TCP Ping, the application think it's an error and logs it. So that's why I want to disable and pass a custom check to the sidecar service or at least change the sidecar check port - Consul could ping a different port instead of pinging the service port. I had some hope that it could be disabled from Nomad, since it's possible to disable the TCP ping check for the native build-in proxy with the Before trying to use the Ingress Gateway we had a setup with a F5 load balancing that also had a health check, and I verified why I think that F5's healthcheck doesn't trigger the error: instead of fully opening a connection, it uses a TCP half-open connection. So since it's not a fully formed TCP connection, nothing gets logged. |
@shoenig
|
Hi @leonardobsjr! It's probably not feasible to provide that particular knob, but what you're describing also shouldn't be happening either so that suggests there might be a bug at play here somewhere (maybe Nomad, maybe Consul, maybe Envoy!). Can you provide a jobspec that shows what your Connect configuration looks like? That would help us figure out what's going on there. @AndrewChubatiuk can you please open a new issue for that? That doesn't seem to be directly related to @leonardobsjr's issue and I want to make sure we don't lose track of things. |
@tgross my PR with a fix for this issue was just merged |
Ha! Shows what I know! 😊 That was #9975 and it'll land in Nomad 1.0.4. Going to keep this issue open for @leonardobsjr unless @shoenig thinks that should cover this as well? |
@tgross Does @AndrewChubatiuk PR enables overriding Consul checks from Nomad? |
Yeah the original issue is legitimate; there is currently no way to not have this check enabled when using Connect from Nomad. Consul injects this TCP check automagically if there are no checks defined on the sidecar service. Nomad doesn't let you specify checks one way or the other for |
Sure @tgross , it's actually very very easy to replicate. The attached deployment file deploys go-echo, which is a very simple service that listens to a port expecting a connection and echoes whatever you send after connecting. After you deploy it, check the logs on docker and you will see that Consul is opening and closing a connection every 10s. In my case, the application understands that every connection is a business request and expects some stuff to be sent to proceed. Since it is just a check instead of a valid request, nothing is sent, and the app logs it as an error. Pretty annoying. |
Can confirm this becomes an issue for many services that I wouldn't even call legacy - each check yields up to 4 log lines for some things, and that's not even on debug level. @schoenig's suggestion to add a |
Fix merged in #10531 and will ship in the upcoming Nomad 1.1.0-rc1 |
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. |
Nomad version
Nomad 1.0.1
[Consul 1.8.6]
Operating system and Environment details
Linux Ubuntu 24.04
Issue
Both Nomad and Consul are running on, respectively, -dev-connect and -dev. Is there any way for Nomad to disable the creation of the Sidecar Listening Check? It's possible through Consul (just by not passing any checks), but neither sidecar_task nor sidecar_service accepts checks. I have an old service that keeps logging it as a failed connection.
I know that this is not exactly Nomad-related, but if there's no way to do it on Nomad, can you explain how does the check occurs? It's something controlled by Consul or it's relegated to Envoy as an Envoy Health check? I was thinking about changing the Envoy image to somehow block the tcp pinging, but I don't really know how...
The text was updated successfully, but these errors were encountered: