unicorn startup never returns, causes race condition at setup and during all restarts

[inopinatus]

Expected Behavior

• Unicorn should not be started until assets are fully precompiled.
• Monit should always have correct state of the application server process
• Deploys should always restart/reload the app server

Actual Behavior

The monit config is installed and reloaded early, with the following consequence:

• Monit starts Unicorn before assets are precompiled
• Application now throws continuous server errors due to starting up without its asset manifest.

The monit command for unicorn never returns, with the following consequences:

• Monit thinks Unicorn hasn't started, eventually times out and gives up.
• The restart at the end of setup->deploy, occurs whilst Monit thinks it is still trying to start Unicorn, so it never gets restarted.
• Setup "succeeds" but the app server is returning bad responses.

Then during deploys, similarly, unicorn should be restarted at end of deploy. Assuming monit has noticed the old one:

• Old service is killed and new service is, eventually, started
• Monit never finds out, thinks Unicorn hasn't started, eventually times out and gives up.
• Restart of background job worker is unnecessarily deferred by many minutes, leaving it out of sync with app server.

However, if monit thinks it's still trying to start Unicorn when the new restart comes in:

• Deployments within the timeout period fail to restart at all, leaving old version running.

Logs

Here is monit struggling to deal during setup:

Feb 26 07:18:37 demo-app0 monit[903]: 'demo-app0' Monit reloaded
Feb 26 07:18:37 demo-app0 monit[903]: 'unicorn_myapp' process is not running
Feb 26 07:18:37 demo-app0 monit[903]: 'unicorn_myapp' trying to restart
Feb 26 07:18:37 demo-app0 monit[903]: 'unicorn_myapp' start: '/bin/sh -c cd /srv/www/myapp/current && SECRET_KEY_BASE="this_should_never_be_logged_wtf"...
Feb 26 07:20:07 demo-app0 monit[903]: 'unicorn_myapp' failed to start (exit status 2) -- '/bin/sh -c cd /srv/www/myapp/current && SECRET_KEY_BASE="this_should_never_be_logged_wtf"
Feb 26 07:22:38 demo-app0 monit[903]: 'unicorn_myapp' process is not running
Feb 26 07:22:38 demo-app0 monit[903]: 'unicorn_myapp' trying to restart
Feb 26 07:22:38 demo-app0 monit[903]: 'unicorn_myapp' start: '/bin/sh -c cd /srv/www/myapp/current && SECRET_KEY_BASE="this_should_never_be_logged_wtf"...
Feb 26 07:24:08 demo-app0 monit[903]: 'unicorn_myapp' failed to start (exit status -1) -- Program '/bin/sh -c cd /srv/www/myapp/current && SECRET_KEY_BASE="this_should_never_be_logged_wtf" 
Feb 26 07:24:17 demo-app0 monit[903]: 'unicorn_myapp' restart on user request
Feb 26 07:24:17 demo-app0 monit[903]: Monit daemon with PID 903 awakened
Feb 26 07:24:17 demo-app0 monit[903]: Monit daemon with PID 903 awakened
Feb 26 07:25:05 demo-app0 monit[26179]: Reinitializing monit daemon
Feb 26 07:25:36 demo-app0 monit[903]: Awakened by User defined signal 1
Feb 26 07:25:36 demo-app0 monit[903]: Reinitializing Monit -- control file '/etc/monit/monitrc'
Feb 26 07:25:36 demo-app0 monit[903]: 'demo-app0' Monit reloaded
Feb 26 07:29:52 demo-app0 monit[903]: 'unicorn_myapp' restart on user request
Feb 26 07:29:52 demo-app0 monit[903]: Monit daemon with PID 903 awakened
Feb 26 07:29:52 demo-app0 monit[903]: Awakened by User defined signal 1
Feb 26 07:29:52 demo-app0 monit[903]: 'unicorn_myapp' trying to restart
Feb 26 07:29:52 demo-app0 monit[903]: 'unicorn_myapp' stop: '/bin/sh -c cat /run/lock/myapp/unicorn.pid | xargs --no-run-if-empty kill -QUIT; sleep 5'
Feb 26 07:29:57 demo-app0 monit[903]: 'unicorn_myapp' start: '/bin/sh -c cd /srv/www/myapp/current && SECRET_KEY_BASE="this_should_never_be_logged_wtf"...
Feb 26 07:31:27 demo-app0 monit[903]: 'unicorn_myapp' failed to start (exit status -1) -- Program '/bin/sh -c cd /srv/www/myapp/current && SECRET_KEY_BASE="this_should_never_be_logged_wtf" 
Feb 26 07:31:27 demo-app0 monit[903]: 'unicorn_myapp' restart action failed
Feb 26 07:31:27 demo-app0 monit[903]: 'unicorn_myapp' process is running after previous exec error (slow starting or manually recovered?)
Feb 26 07:31:27 demo-app0 monit[903]: 'unicorn_myapp' failed to get service data

Ultimately, I do not understand why the daemonization of Unicorn was dropped. This was a seriously breaking change, and only apparently necessary for Puma. I don't think I can run with this in production, given the woefully inconsistent states that are occurring as a result.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.