[2.10.2] Director deploy crashes the Icinga service [FreeBSD]

[Stefar77]

With Icinga 2.10.2-1 when somebody presses deploy in Icinga director the Icinga2 services fails to start the new process.

Failure: Address already in use on the API socket.

Current Behavior

Use deploy in director and after a couple of seconds the newly created Icinga process will crash with a address in use error.

Possible Solution

Wait a bit longer when starting the new listener of retry when getting the address in use error on reload.
I didn't take the time to see how/when the old process closes the socket so I just added a sleep(1); before creating the primary JSON-RPC listener and that fixed the problem for now. (remote/apilistener.cpp:253)

Steps to Reproduce (for bugs)

1. Install Icingaweb2 and Director (on FreeBSD 11.2 in a jail)
2. Press deploy

Context

Upgraded from 2.8 to 2.10.2-1 and next morning we noticed the bug so I tried to set bind_host and bind_port but that didn't seem to help. I ended up patching the source code a bit.

Your Environment

• Version used (icinga2 --version): 2.10.2
• Operating System and version: FreeBSD 11.2
• Enabled features (icinga2 feature list): api checker command graphite ido-mysql livestatus mainlog notification syslog
• Icinga Web 2 version and modules (System - About): 2.6.2
• Config validation (icinga2 daemon -C): no problems

[Crunsher]

Can you also reproduce this without Director, by sending SIGHUP to the Icinga process?

[dnsmichi]

Sounds a bit like #6815, can you test that patch please? :)

[Stefar77]

@dnsmichi That didn't help, still quits when I deploy
@Crunsher Yes it just makes a Icinga process go to 2000% for a bit and quits all Icinga processes.

I'll add the sleep(1); again for now.

[Stefar77]

Also noticed 2.10.2 on Windows (agents) then you start the daemon from console and press CTRL-C it will hang and spam Socket errors until you close the command prompt.

[dnsmichi]

That's a different problem on Windows, you cannot run it in foreground in cmd. That's tracked with #3029.

[bmccorkle]

I'm experiencing this same issue since updating Icinga2 recently.

FreeBSD 11.2
Icinga2 2.10.1

Disabled features: compatlog elasticsearch gelf graphite livestatus opentsdb perfdata statusdata syslog
Enabled features: api checker command debuglog ido-mysql influxdb mainlog notification

[mat813]

I do not use the director, and I have been experiencing the same issue ever since 2.10.0.

In the logs, I get:

[2018-12-26 16:46:01 +0100] critical/TcpSocket: Invalid socket: Address already in use
Context:
        (0) Activating object 'api' of type 'ApiListener'

[2018-12-26 16:46:01 +0100] critical/ApiListener: Cannot bind TCP socket for host '' on port '5665'.
Context:
        (0) Activating object 'api' of type 'ApiListener'

[2018-12-26 16:46:01 +0100] critical/ApiListener: Cannot add listener on host '' for port '5665'.
Context:
        (0) Activating object 'api' of type 'ApiListener'

To mitigate the issue, I changed our internal documentation saying to run service icinga2 restart instead of reload on the master, and then to go on the salt master and run salt '*' service.start icinga2. It feels like it tries to restart the API before its previous instance being stopped, or something.

[dnsmichi]

It is a problem with re-using the sockets and allowing incoming API requests which block on shutdown. Their handling was changed with 2.10. Can only be seen in environments with many parallel requests and reloads.

[mat813]

Mmmm, I do have ~50 satellites, but I do not reload often. Though, it happens on every reload.

[dnsmichi]

#6898 and #6901 fixes this, seems to be a problem specifically with *BSD.