Router displays error on startup

[smarterclayton]

$ docker logs 9e0e9b14e735
E0320 03:23:35.303123       1 router.go:126] Error reloading router: wait: no child processes
 Reload output: + config_file=/var/lib/haproxy/conf/haproxy.config
+ pid_file=/var/lib/haproxy/run/haproxy.pid
+ old_pid=
+ path_map_file=/var/lib/haproxy/conf/os_http_be.map
+ sort -r /var/lib/haproxy/conf/os_http_be.map -o /var/lib/haproxy/conf/os_http_be.map
+ '[' -f /var/lib/haproxy/run/haproxy.pid ']'
+ old_pid=18
+ '[' -n 18 ']'
+ /usr/sbin/haproxy -f /var/lib/haproxy/conf/haproxy.config -p /var/lib/haproxy/run/haproxy.pid -sf 18
[WARNING] 078/032335 (23) : Setting tune.ssl.default-dh-param to 1024 by default, if your workload permits it you should set it to at least 2048. Please set a value >= 1024 to make this warning disappear.

[smarterclayton]

The first reloading error shouldn't be displayed.

[pweil-]

Looks like it has to do with the reaper not having children. Probably just needs suppressed in all cases. I'll take a look.

[pweil-]

Actually, I tried reproducing this. My container seems fine. I tried deleting and recreating it a couple of times and it always came up ok. I can put a suppression in there just in case though if you think that's necessary

[vagrant@openshiftdev origin]$ docker logs -f 160a
I0402 18:02:49.224641       1 reaper.go:17] Launching reaper
I0402 18:02:49.240737       1 controller.go:26] Running router controller
I0402 18:02:49.280852       1 plugin.go:68] Processing 1 Endpoints for Name: kubernetes (MODIFIED)
I0402 18:02:49.280893       1 plugin.go:71]   Endpoint 0 : {10.0.2.15 %!s(int=8443) %!s(*api.ObjectReference=<nil>)}
I0402 18:02:49.280938       1 plugin.go:83] Modifying endpoints for default/kubernetes
I0402 18:02:49.280946       1 router.go:59] Commiting router changes
I0402 18:02:49.364906       1 plugin.go:68] Processing 1 Endpoints for Name: kubernetes-ro (MODIFIED)
I0402 18:02:49.364943       1 plugin.go:71]   Endpoint 0 : {10.0.2.15 %!s(int=8443) %!s(*api.ObjectReference=<nil>)}
I0402 18:02:49.364950       1 plugin.go:83] Modifying endpoints for default/kubernetes-ro
I0402 18:02:49.364977       1 router.go:59] Commiting router changes
I0402 18:02:49.365469       1 reaper.go:24] Signal received: child exited
I0402 18:02:49.365523       1 reaper.go:27] Waiting to reap
I0402 18:02:49.365552       1 reaper.go:30] No more process to reap.
I0402 18:02:49.400792       1 reaper.go:24] Signal received: child exited
I0402 18:02:49.400932       1 reaper.go:27] Waiting to reap
I0402 18:02:49.400957       1 reaper.go:34] Reaped process with pid 21
I0402 18:02:49.400962       1 reaper.go:27] Waiting to reap
I0402 18:02:49.400967       1 reaper.go:30] No more process to reap.
I0402 18:03:15.627853       1 controller.go:42] Processing Route: hello-nginx
I0402 18:03:15.627870       1 controller.go:43]            Alias: www.example.com
I0402 18:03:15.627875       1 controller.go:44]            Event: ADDED

[danmcp]

Still an issue?

[danmcp]

@smarterclayton ^^

[pweil-]

I was unable to reproduce this @smarterclayton. If you aren't still having this issue this can be closed.

[gravis]

We're having this today.
OS 1.1.0.1
The router is working for some minutes, than completely stop, with this in the logs

[...]
E1223 15:29:05.442754       1 controller.go:100] error reloading router: wait: no child processes
99  ---
100 + config_file=/var/lib/haproxy/conf/haproxy.config
101 + pid_file=/var/lib/haproxy/run/haproxy.pid
102 + old_pid=
103 + haproxy_conf_dir=/var/lib/haproxy/conf
104 + for mapfile in '"$haproxy_conf_dir"/*.map'
105 + sort -r /var/lib/haproxy/conf/os_edge_http_be.map -o /var/lib/haproxy/conf/os_edge_http_be.map
106 + for mapfile in '"$haproxy_conf_dir"/*.map'
107 + sort -r /var/lib/haproxy/conf/os_edge_http_expose.map -o /var/lib/haproxy/conf/os_edge_http_expose.map
108 + for mapfile in '"$haproxy_conf_dir"/*.map'
109 + sort -r /var/lib/haproxy/conf/os_edge_http_redirect.map -o /var/lib/haproxy/conf/os_edge_http_redirect.map
110 + for mapfile in '"$haproxy_conf_dir"/*.map'
111 + sort -r /var/lib/haproxy/conf/os_http_be.map -o /var/lib/haproxy/conf/os_http_be.map
112 + for mapfile in '"$haproxy_conf_dir"/*.map'
113 + sort -r /var/lib/haproxy/conf/os_reencrypt.map -o /var/lib/haproxy/conf/os_reencrypt.map
114 + for mapfile in '"$haproxy_conf_dir"/*.map'
115 + sort -r /var/lib/haproxy/conf/os_sni_passthrough.map -o /var/lib/haproxy/conf/os_sni_passthrough.map
116 + for mapfile in '"$haproxy_conf_dir"/*.map'
117 + sort -r /var/lib/haproxy/conf/os_tcp_be.map -o /var/lib/haproxy/conf/os_tcp_be.map
118 + '[' -f /var/lib/haproxy/run/haproxy.pid ']'
119 + old_pid=254
120 + '[' -n 254 ']'
121 + /usr/sbin/haproxy -f /var/lib/haproxy/conf/haproxy.config -p /var/lib/haproxy/run/haproxy.pid -sf 254

[pweil-]

@gravis are you able to reproduce this consistently? If so could you elaborate on the steps taken. Do you have any details about route creation or any actions taken on the pod prior to this showing?

@mrunalp - this has the same symptoms that were in #1434 but resolved with using syscall.WNOHANG in the reaper.

[gravis]

It was on a production cluster, so I can't run many tests. We just took the errors from the log one by one. Most of them were related to mounted volumes (glusterfs). We destroyed the pod many times, and finally rebooted the nodes, and the issue was gone...

[gravis]

I wonder if it could be related to routes using the default certificate.
I have a lot of:

W1223 21:12:22.086795       1 router.go:524] a edge terminated route with host [redacted] does not have the required certificates.  The route will still be created but no certificates will be written

I have removed these routes for the night, and so far, no occurence of the error reloading router: wait: no child processes message.