-
Notifications
You must be signed in to change notification settings - Fork 28
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Signal Handling from Orchestrator #69
Comments
Hello @MillsyBot, Thanks a lot, it is really great to receive a nice comment about our work and it is a real pleasure to share our work with people being grateful. We are using daily this tool with signals to regenerate on the fly our configurations for dozens of different very large configurations in many teams with many instances of Prometheus (Prometheus receiving the signal) and this is working for a very long time, so I suspect a big bug would have been notified, but it is still possible, especially since we are not using very high I also added the signal reload for your exact usage some time ago, so it is exactly intended to do so. But there are a few warnings (that are supposed to be explained in the README.md, but I take the time to re-explain (and feel free to modify the README in a PR if it is not clear). The most common problemFirst, in your example, you are using Second, the order of --sig-reload and --exec are very important: if you consider this example:
/my/monitoring and python servers would be launched once all files are properly generated and will receive signals each time one of those files is changing. However:
It means that for instance appending Another possible issueFrom the doc:
It means that the signal is supposed to be generated when all the files are considered not dirty, it means all the data for all the files have been retrieved and properly computed. Consider this example: <%
counts = {}
services.each do |service_name, tags|
counts[service_name] = service(service_name).count
end
%><%= JSON.pretty_print(counts) %> If you have a cluster where 1 service is renamed every 30s and you used At startup, consul-templaterb ignores --wait and try to fetch data as fast as possible. But once the template has been rendered, it would watch. every 60s, the template is evaluated, then wait... so, at t+60: consul-templaterb finds new services This might be fixed in at least 3 ways:
with #!/bin/bash
# Test if .last_reload is older than 1 min
if test "`find .last_reload -mmin +1`"
then
kill -USR2 $(pidof haproxy)
touch .last_reload
fi => thus, each time haproxy.conf would be modified, the script would be launch, but only reload haproxy every 60s max, meaning that you could use If it something elseYou might consider running it with debug flags (which displays if there are some dirty templates), and if you don't find, try posting the debug messages here. About --wait
This might be fixed by having an option to delay --sig-reload to launch only every x seconds (might not be that hard) and decorrelate it from --wait. I would be glad to accept such PR if you are interested Best |
Hello @MillsyBot, I added support from what I described in my previous comment in 70808c6 This will let you use |
NEW FEATURES: * Added `-W` or `--wait-between-reload-signal` to avoid sending too many signals to a process executed. This avoids for instance reloading too much a HAProxy configuration without having to play with `-w` as described in [#69](#69) BUGFIX: * Removed warnings at runtime with Ruby 2.7+ * Minor JS fix in Consul-UI (Added missing unused parameter to function `serviceTitleGenerator`)
Thanks for all the love here! This really makes my day. |
@robloxrob glad it helps |
@pierresouchay WOW, thank you so much for the detailed response, and for releasing a new version addressing my issues! So cool, I will buy you a beer at the next HAProxy Conf (we meet there last year). Out of curiosity: what type of performance impact are you seeing with HAProxy reloading every 1sec? Are the impacts significant to connection reuse when the FD are constantly passing from one process to another? |
@MillsyBot It really depends on the versions. In my tests a few years ago (Version 1.7.x), it used to break many things (including opened connections, so while it is ok to loose a connection or 2 every minute, not acceptable every sec), in recent, HAProxy has been doing lots of work to avoid this (esp. with 2.x releases), so I suspect it is not such a big issue with recent kernels and recent HAProxy versions. On our side, we know use systems interconnected with Consul to pilot HAProxy (most notably: https://github.com/haproxytech/haproxy-consul-connect that uses dataplane recent additions to change on the fly HAProxy config) But anyway, we also had this exact same issue with our prometheus instances (we used |
@MillsyBot Don't forget to tell me if it is fixing your issue, in that case, please close the issue! |
@pierresouchay Super helpful. |
@chuckyz says thanks as well. |
First of all: Awesome work, this an amazingly fast and awesome tool. Thanks for sharing this!
#! /bin/bash /usr/local/bin/consul-templaterb --wait=60 --no-vault-renew --vault-addr=${VAULT_ADDR} --consul-addr=${CONSUL_HTTP_ADDR} \ --sig-reload=USR2 --exec "/haproxy/bin/haproxy -f /haproxy/hap.conf" "$@"
and the variables being passed is
"--template", "/haproxy/hap.conf.erb:/haproxy/hap.conf", "--template", "/haproxy/routing.map.erb:/haproxy/routing.map", "--template", "/haproxy/secrets_generator.erb:/haproxy/secrets_generator"
This is a watered down version of our entrypoint running in a Ubuntu18.04 container. Our goal is to run this consul-templaterb rendering HAProxy configs and reloading the process when consul values are changed. This is generally working as expected, however we are running some tests and we are not seeing signal handling as expected.
We tried running consul-templaterb in the background and trapping the signals, however I am not sure that this was the intent of this when you wrote consul-templaterb.
When using an orchestrator what is your suggested deployment/signal handling?
The text was updated successfully, but these errors were encountered: