New daemon 2) running as daemon process

[muhrin]

The daemon from step 1 (see #1054) needs to be set up to run as a OS daemon service. Either using celeryd or something else.

[sphuber]

This will also involve, address and potentially solve other issues:

• Unstable (?) Daemon Broker #885 the daemon broker
• Automatic module reload by daemon #933 automatic module reload
• IntegrityError in daemon log #833 database integrity errors
• Celeryd replacement or leveraging #820 celeryd replacement or leveraging
• Daemon - Jobs stuck in SUBMITTING state #490 jobs stuck in SUBMITTING state
• Daemon ticks a running workflow multiple times #279 old bug with multiple ticking of workflows (potentially outdated)
• Job submission fails without internet connection #270 job submission fails without internet connection
• celerybeat-schedule file getting corrupted #268 celery beat schedule file getting corrupted
• workflow stepper #257 legacy workflow stepper task locking

[muhrin]

I can think of a couple of ways to go:

1. Keep celeryd and make each celery worker just launch a Runner
2. Rip out celeryd and use another library to handle the daemonisation and use mutiprocessing to instantiate mutiple Runners

Because of the heavyweight nature of celeryd and the issues we already have with it requiring a broker with a other than SQLA in recent versions (see requirements file) I would favour ripping it out.

[muhrin]

@DropD is currently looking into daemonize as a potential demonizer library

[muhrin]

@DropD will have a look at circus. Things to check:

• Logging to file works including settings AiiDA log level
• Is there log rotation support?
• How are child process kills handled? Is there a signal we can catch to exit gracefully?
• Check how restarting of crashed processses works - is there exponential backoff?
• Check API is capable of giving us a verdi daemon status like report?
• Does API support start/stop hooks (without using their CLI tools)?

[DropD]

Restarting:

• Default behaviour is to always restart immediately (implied in docs, manually verified by sending sigkill multiple times and watching restarts happen).
• There is a builtin plugin that provides a service watching for runners that restart often (tunable frequency) and times them out for a user set amount of time. If exponential backoff is required, maybe the plugin would be easy to modify thusly.
  The plugin, which is called "Flapping" has yet to be tested.

Logging:

• python logging does it's own thing, so aiida logs still get written exactly the same regardless (not tested)
• circus deamon uses python logger too, can rotate logs
• redirecting stdout / stderr into file only works if starting from a config file, not for dynamically added watchers.

[muhrin]

@DropD good work. I'm happy without exponential backoff for now.
Let me know if you get a chance to check the other issues above and then maybe you could get a working proof of concept - I'd be very happy to have a look

[sphuber]

Fixed in #1217