Celeryd replacement or leveraging

[dev-zero]

With the advent of apricotpy and RabbitMQ in the new workflows it is likely a good time to revisit the usage of celery.

The current tasks of celery are (as per aiida/daemon/tasks.py):

1. maintain a task queue with one worker per logical CPU (implicitly done when starting the celery worker via verdi daemon start)
2. periodically inject the following tasks into that queue (via the celery --beat) which then get execute by one of the workers:
   1. submit (calculation) jobs
   2. update (calculation) jobs
   3. retrieve (calculation) jobs
   4. tick the workflow engine
   5. execute steps in the legacy workflow engine (singleton task)

The new design of plumpy and apricotpy foresee the introduction of a persistent event loop to execute the Work Chain. This event loop will also listen on a RabbitMQ queue over which a users client sends a message with the WorkChain object to run on the event loop.
For this we need a new process to run this event loop. Furthermore, since we have RabbitMQ in the picture, we also need a way to bring up the broker. This can either be done via Supervisor (so PR #790 should not be pulled) or directly by writing the new process as a daemon and starting both this new daemon and the RabbitMQ broker directly instead (which would remove the burden to maintain/create supervisor configs, as well as remove supervisor as dep).

[dev-zero]

Now, for the point about writing new daemons: writing a proper daemon even with the help of a library like python-daemon is notoriously hard (think about logfile rotation, etc.). What one can do instead is to not write a daemon (meaning: do not fork, log to stdout, do not write a PID-file) and do only signal handling to re-load configuration and shut down properly. In that case you need a parent process to take over the responsibility of doing the rest (PID-file, etc.).
This can be either a supervisor daemon like supervisord (which we would like to get rid of due to missing Python 2 support) or systemd which is present on most Linux systems nowadays (for those where systemd is unavailable, one could use zdaemon instead) and which is perfectly capable of managing services under a user account, without root privileges.

[dev-zero]

As we discussed today at the developer workshop, it does not make much sense to keep Celery in the picture as it is now:

• the periodic task scheduling can be moved over to the event-loop daemon
• the message queuing can be done over the RabbitMQ queue
• execution of the long-running tasks can be implemented in a simple worker daemon (to unblock the event loop daemon)

A different option may be to leverage Celery also for the communication between the users client instead and put the event loop in a Celery function call instead.

[szoupanos]

This is very similar to issue #885

[sphuber]

No longer relevant due to merge #1468