Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SysV init scripts of ::daemon are broken #524

Open
benibr opened this issue Feb 11, 2021 · 2 comments
Open

SysV init scripts of ::daemon are broken #524

benibr opened this issue Feb 11, 2021 · 2 comments
Labels
bug Something isn't working

Comments

@benibr
Copy link

benibr commented Feb 11, 2021

I tried to rollout module version 11 which contains #502 and fixes the broken prometheus SysV init script.
Sadly the PR breaks the daemon parameter in the SysV init script of all additional daemons created by prometheus::daemon

Affected Puppet, Ruby, OS and module versions/distributions

  • Distribution: CentOS 6
  • Module version: 11

How to reproduce (e.g Puppet code you use)

Setting init_style to sysv for node_exporter

What are you seeing

/etc/init.d/node_exporter status
node_exporter dead but pid file exists

Output log

# tail /var/log/node_exporter
....
level=info ts=2021-02-11T15:18:29.085Z caller=node_exporter.go:191 msg="Listening on" address="localhost:9100 --no-collector.hwmon --collector.meminfo_numa "
....
level=error ts=2021-02-11T15:18:29.123Z caller=node_exporter.go:194 err="listen tcp: address tcp/9100 --no-collector.hwmon --collector.meminfo_numa : unknown port"

Any additional information you'd like to impart

I worked around this by setting $init_style = 'none' for the node_exporter which just does not override the init script and keeps the existing one.

@benibr
Copy link
Author

benibr commented Feb 11, 2021

For anyone who's interested in the actual quoting hell of these init scripts:
Here is an excerpt of bash -x /etc/init.d/node_exporter start with the working versrion:

...
Starting prometheus ipmi_exporter: + mkrundir
+ '[' '!' -d /var/run/ipmi_exporter ']'
+ chown ipmi-exporter /var/run/ipmi_exporter
+ '[' -f /var/run/ipmi_exporter/ipmi_exporter.pid ']'
+ retcode=0
+ sleep 1
+ daemon --user=ipmi-exporter --pidfile=/var/run/ipmi_exporter/ipmi_exporter.pid /usr/local/bin/ipmi_exporter '   --freeipmi.path=/usr/local/bin'
+ mkpidfile
+ mkrundir
+ '[' '!' -d /var/run/ipmi_exporter ']'
+ chown ipmi-exporter /var/run/ipmi_exporter
...

And here of the non working version:

...
Starting prometheus ipmi_exporter: + mkrundir
+ '[' '!' -d /var/run/ipmi_exporter ']'
+ chown ipmi-exporter /var/run/ipmi_exporter
+ '[' -f /var/run/ipmi_exporter/ipmi_exporter.pid ']'
+ rm /var/run/ipmi_exporter/ipmi_exporter.pid
+ daemon --user=ipmi-exporter --pidfile=/var/run/ipmi_exporter/ipmi_exporter.pid ''\''/usr/local/bin/ipmi_exporter'\'' \ \ \ --freeipmi.path\=/usr/local/bin >> '\''/var/log/ipm
i_exporter'\'' 2>&1 &'
+ local gotbase= force= nicelevel corelimit
+ local pid base= user= nice= bg= pid_file=
+ local cgroup=
+ nicelevel=0
+ '[' --user=ipmi-exporter '!=' -user=ipmi-exporter ']'
+ case $1 in
+ user=ipmi-exporter
+ shift
+ '[' --pidfile=/var/run/ipmi_exporter/ipmi_exporter.pid '!=' -pidfile=/var/run/ipmi_exporter/ipmi_exporter.pid ']'
+ case $1 in
+ pid_file=/var/run/ipmi_exporter/ipmi_exporter.pid
+ shift
+ '[' ''\''/usr/local/bin/ipmi_exporter'\'' \ \ \ --freeipmi.path\=/usr/local/bin >> '\''/var/log/ipmi_exporter'\'' 2>&1 &' '!=' ''\''/usr/local/bin/ipmi_exporter'\'' \ \ \ --f
reeipmi.path\=/usr/local/bin >> '\''/var/log/ipmi_exporter'\'' 2>&1 &' ']'
+ '[' -z '' ']'
+ base='ipmi_exporter'\'' 2>&1 &'
+ __pids_var_run 'ipmi_exporter'\'' 2>&1 &' /var/run/ipmi_exporter/ipmi_exporter.pid
+ local 'base=ipmi_exporter'\'' 2>&1 &'
+ local pid_file=/var/run/ipmi_exporter/ipmi_exporter.pid
++ /usr/bin/dirname /var/run/ipmi_exporter/ipmi_exporter.pid
+ local pid_dir=/var/run/ipmi_exporter
+ local binary=
+ '[' -d /var/run/ipmi_exporter -a '!' -r /var/run/ipmi_exporter ']'
+ pid=
+ '[' -f /var/run/ipmi_exporter/ipmi_exporter.pid ']'
+ return 3
+ '[' -n '' -a -z '' ']'
+ corelimit='ulimit -S -c 0'
+ '[' -n '' ']'
+ '[' -n '' ']'
+ '[' color = verbose -a -z '' ']'
+ '[' -z ipmi-exporter ']'
+ runuser -s /bin/bash ipmi-exporter -c 'ulimit -S -c 0 >/dev/null 2>&1 ; '\''/usr/local/bin/ipmi_exporter'\'' \ \ \ --freeipmi.path\=/usr/local/bin >> '\''/var/log/ipmi_export
er'\'' 2>&1 &'
+ '[' 0 -eq 0 ']'
+ success 'ipmi_exporter'\'' 2>&1 & startup'
+ '[' color '!=' verbose -a -z '' ']'
+ echo_success
+ '[' color = color ']'
+ echo -en '\033[60G'
                                                           + echo -n '['
[+ '[' color = color ']'
+ echo -en '\033[0;32m'
+ echo -n '  OK  '
  OK  + '[' color = color ']'
+ echo -en '\033[0;39m'
+ echo -n ']'
]+ echo -ne '\r'
+ return 0
+ return 0
+ retcode=0
+ sleep 1
+ mkpidfile
+ mkrundir
+ '[' '!' -d /var/run/ipmi_exporter ']'
+ chown ipmi-exporter /var/run/ipmi_exporter
...

@Derek-Milbert
Copy link

I had a Similar issue on a FedoraCore 6 Server using sysv as init_style. I believe the problem lies in the init.d template.
When generating the init.d file for node exporter a ruby module is called which is supposed to "escape" ( \ ) certain characters, what this ends up doing is breaking the entire init.d file and making the "daemon" section of the init.d file unusable.

Here is the example of the broken "daemon" section (more specifically the extra options) of the init.d file versus the correct/working version.
Broken:
daemon --user=node-exporter --pidfile="$PID_FILE" "'$DAEMON' --web.config\=/etc/web.yml\ --collector.textfile.directory\ /var/lib/node_exporter/textfile_collector\ \ >> '$LOG_FILE' 2>&1 &"

Working:
daemon --user="node-exporter" --pidfile="$PID_FILE" "$DAEMON --web.config=\"/etc/web.yml\" --collector.textfile.directory \"/var/lib/node_exporter/textfile_collector\" >> '$LOG_FILE' 2>&1 &"

in the broken code you can see the the equal symbol is escaped for example and should not be.

Problem code:
<%= Shellwords.escape(@options) %> part in "'$DAEMON' <%= Shellwords.escape(@options) %> >> '$LOG_FILE' 2>&1 &"

"'$DAEMON' <%= Shellwords.escape(@options) %> >> '$LOG_FILE' 2>&1 &"

@TheMeier TheMeier added the bug Something isn't working label May 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants