Feature: after stop callback

[rajyan]

Description

Currently we can configure callbacks after the "shutdown" by

shoryuken/lib/shoryuken/launcher.rb

Lines 81 to 87 in 3099c57

	if (callback = Shoryuken.stop_callback)
	logger.debug { 'Calling stop_callback' }
	callback.call
	end
	fire_event(:shutdown, true)
	end

stop_callback or registering a callback to :shutdown event, but we cannot configure a callback after the executor has stopped (after the executor.kill in stop! or after wait_for_termination in stop call).

shoryuken/lib/shoryuken/launcher.rb

Lines 16 to 24 in 3099c57

	def stop!
	initiate_stop
	executor.shutdown
	return if executor.wait_for_termination(Shoryuken.options[:timeout])
	executor.kill
	end

I believe registering a callback is useful in some cases when we want to gracefully shutdown the Shoryuken worker.

Usage example

For example, we have a job that takes about 10 sec ~ 5 min with a long enough visibility timeout.
When the job starts, we take a DB lock and change the status of the job (model) to "job_running" to make sure the job runs one at a time (to avoid at-least once duplication and run it exactlly once).

We want to stop this job gracefully as possible on deploy, so we have configured "timeout" option to wait for the job to finish after the shutdown is triggered by SIGTERM.

shoryuken/lib/shoryuken/launcher.rb

Line 21 in 3099c57

return if executor.wait_for_termination(Shoryuken.options[:timeout])

Although, we are using AWS Fargate (and Fargate spot) for the job worker, which can only wait for 2 minutes at max after the SIGTERM (see https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definition_parameters.html stop timeout), so we are currently configuring ":timeout" options to 60 seconds.

In this case, I want to configure some kind of callback to run after stop to check for "killed while running" jobs (status "job_running"), and rollback the job status to "job_ready" if the job is retryable (which is retried by SQS with visibility timeout) or change the job status to "job_failed" and notify them.