-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Nomad job with envoy escape hatch has no dynamic port interpolation available #14403
Comments
Hi @johnalotoski 👋 #14445 adds interpolation to configuration values, so I think what you described would then be possible. I'm building a binary from that PR here: https://github.com/hashicorp/nomad/actions/runs/2974808759 Once that's done, would you be able to try and validate if this provides what you need? Thanks! |
Hi @lgfa29, thanks very much for the fix! I took the diff from here and compiled it into our Nomad 1.3.2 version currently being used, and tested it works! I did find that the notation I needed to use to get the variable substitution to work correctly was (modifying the example snippet from above slightly):
Where double $ was required and also string quotes which were then properly removed and substituted with a JSON integer once interpolation was done. Presumably the double $$ is for HCL escaping of the "$" char? Thanks! |
Nice! Thanks for test it. I will update the docs and get that PR merged so it's available in the next release 🙂
Ah yeah. The interpolation is done within a string right? |
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. |
Nomad version
Nomad v1.2.9, v1.3.2
Issue
Envoy uses a default stream_idle_timeout of 5 minutes which is in conflict with a few long running APIs we are using. There are a few direct override knobs for envoy timeouts in Consul, but there is no direct knob for this
stream_idle_timeout
. Advanced Consul Escape hatches are available, and in this case, configuring an escape hatch override for envoy_public_listener_json appears that it would solve the problem, as we could add a route public listeneridle_timeout
which will override envoy's defaultstream_idle_timeout
, or directly modify thestream_idle_timeout
itself.However, when Nomad sets up the consul job, the dynamic port the envoy listener will use ahead of time is not known, so some Nomad interpolation appears to be necessary to be able to declare this snippet properly. Taking the example of the Consul
envoy_public_listener_json
from the Consul link above, with slight modifications, I believe we'd need to be able to set theconnect.sidecar_service.proxy.config.envoy_public_listener_json
stanza in the Nomad job declaration to something like the following where the Nomad assigned port is interpolated and passed to the Consul escape hatch json override:However, interpolation of the Nomad assigned dynamic port to the Consul connect service doesn't appear available to the passed json escape hatch override in the testing I've done, which seems to eliminate the possibility of using envoy escape hatches to override parameters there aren't already direct Consul overrides for.
I've tested the above approach by deploying the job with a random hardcoded port substituted in the parameterization above for the escape hatch which will be incorrect. Then, once the job is deployed, adjusting the above job definition for the escape hatch snippet and assigning the correct port that Nomad has utilized for the connect proxy and re-deploying/updating the job, at which point the escape hatch override does work as intended. So it appears the only remaining issue is the ability to interpolate a Nomad dynamic port and pass it to the escape hatch snippet.
Perhaps Nomad interpolation can done in this snippet and I'm not aware of it?
The text was updated successfully, but these errors were encountered: