problems with restarting puma

[davidhq]

I'm very often left with non-working web app after deploy...

puma somehow dies...

-----> DB schema unchanged; skipping DB migration        
-----> Skipping asset precompilation        
-----> Cleaning up old releases (keeping 5)        
-----> Build finished        
-----> Moving build to releases/42        
-----> Updating the current symlink        
-----> Launching        
       Command phased-restart sent success 
-----> Done. Deployed v42        
       Elapsed time: 3.00 seconds

(here the website is down)

david@eclipse:~/Projects/meetcasts (master)$ mina puma:restart
-----> Loading rbenv
Puma is not running!
[23126] Puma starting in cluster mode...
[23126] * Version 2.9.0 (ruby 2.1.2-p95), codename: Team High Five
[23126] * Min threads: 1, max threads: 16
[23126] * Environment: production
[23126] * Process workers: 2
[23126] * Preloading application
[23126] * Listening on unix:///var/www/meetcasts/shared/tmp/sockets/puma.sock
[23126] * Listening on tcp://0.0.0.0:3003
[23126] * Daemonizing...
Connection to example.com closed.
       Elapsed time: 4.00 seconds

(note the line Puma is not running)

Sometimes I have to repeat mina puma:restart multiple times to get it working...

Is this a puma problem? I have some sinatra apps and when I send SIGUSR2, it always restarts correctly... Maybe on Rails it has problems and it's not a mina issue? How can I get more debug data about this?

[gabskoro]

So you say when your deploying using mina it doesn't run, but as I can see, when you ssh to your server and run mina puma:restart everything seems to be ok? Can you please copy your deploy.rb file (only the deploy task), and can you run deploy with the -v parameter (mina deploy -v) and also write me the response down.

[davidhq]

I actually run mina puma:restart locally (so I don't ssh there, but mina does), you probably ment that.

I have to run this repeteatly until it suddenly works (page loads). So again after a normal deploy, the site doesn't load and after running mina puma:restart it says that puma was not running and it is starting it now. Then sometimes page loads but many times I have to repeat up to 5 to 7 times before it works.

Here is the deploy file for one project:
https://gist.github.com/davidhq/fe0decad29bb005e9a6d

I thought it actually happens only on this project, but just now it happened on another.

I have examined puma restarting code

cd #{deploy_to}/#{current_path} && #{pumactl_cmd} -S #{puma_state} phased-restart

and it seems ok.. but it's here that things break in the first place..

I don't know how to proceed in finding the fault, but it looks like it's puma's problem, right?

[davidhq]

now I have the webapp in not-running state after a deploy:

david@theta:/var/www/tb/shared/tmp/sockets$ cat puma.state 
---
pid: 30362
config: !ruby/object:Puma::Configuration
  options:
    :min_threads: 1
    :max_threads: 16
    :quiet: true
    :debug: false
    :binds:
    - unix:///var/www/tb/shared/tmp/sockets/puma.sock
    - tcp://0.0.0.0:3007
    :workers: 1
    :daemon: true
    :worker_directory: "/var/www/tb/current"
    :environment: production
    :state: "/var/www/tb/shared/tmp/sockets/puma.state"
    :control_url: unix:///var/www/tb/shared/tmp/sockets/pumactl.sock
    :config_file: config/puma.rb
    :mode: :http
    :worker_timeout: 60
    :preload_app: true
    :rackup: config.ru
    :control_auth_token: 7d18137e9e5e7ea13dc2d3758397e3a
david@theta:/var/www/tb/shared/tmp/sockets$ ps aux | grep 30362
[no process with such PID]

[davidhq]

Maybe it has something to do with rsync I'm using? Because my friends who deploy with mina don't have any issues, but they don't use rsync method... I was thinking that because I haven't been excluding /tmp (as I probably should) when rsyncing, that this got on the server (and puma socket and state are there).. but I don't know how it could be in the way because symlinking to /shared/tmp is working and it happens before puma restart.

[davidhq]

It wasn't that because today it happened again and those directories were excluded... I also upgraded puma from 2.9.0 to 2.9.2... it's a bit frustrating :( Should I ask at Puma's side as well?

[davidhq]

I now tried replacing invoke :'puma:phased_restart' with:

queue "kill -SIGUSR2 `ps aux | grep tcp://0.0.0.0:3003 | grep puma | awk '{ print $2 }'`"

And it happened again on 3rd deploy... so it seems that's not a mina problem in any way...

[davidhq]

I have some new info.... puma/puma#598 ... still not sure where is the problem but apparently puma rejects phased restart and after normal restart it cannot find the Gemfile for some reason...

[davidhq]

Turns out you cannot use phased restart with preloading the app... I hope it will be ok now and that this helps others ... more about this: https://github.com/puma/puma/blob/master/DEPLOYMENT.md#restarting

[ashiksp]

The last link seems returning 404, another links:
https://github.com/puma/puma/blob/master/docs/restart.md
https://github.com/puma/puma/blob/master/docs/deployment.md#restarting

[andrew-aladev]

It's puma issue. The problem i that puma is not actually starts daemon immediately after start. System kills puma process when ssh session is closed. You can fix it by adding sleep 1 after puma start.

For example:

namespace :puma do
  task :custom_start => :environment do
    command %[
      if [ -e '#{fetch(:pumactl_socket)}' ]; then
        echo 'Puma is already running!';
      else
        if [ -e '#{fetch(:puma_config)}' ]; then
          cd '#{fetch(:current_path)}' && #{fetch(:puma_cmd)} -d -e #{fetch(:puma_env)} -C #{fetch(:puma_config)}
          sleep 1
        else
          echo 'Puma config is required'
          exit 1
        fi
      fi
    ]
  end
end