Allow sampling and saving on demand using signals

[fabbarix]

This patch registers a signal handler that will sample the model or save it on demand when a signal is received.

• SIGUSR1: this signal will sample the model with the given prompts on the next step
• SIGUSR2: this signal will save model - and state, if configured - on the next step

[rockerBOO]

The library/signal_handler.py is missing

[fabbarix]

Fixed. Don't know how I missed that one! Thanks for pointing it out.

[rockerBOO]

How would a user trigger the signals? Would it be programmatically made?

[fabio-at-kumandgo]

The PID of the process is printed if training from the command line; you can then ‘kill -SIGUSR1 ’ to send the signal. GUIs can instead map UI buttons to signal sending.

Currently this works on Mac and Linux, looking on how to implement similar functionality in Windows.

[rockerBOO]

Maybe it would be possible to listen for some key command to trigger the similar behavior as well? Like ctrl+c has a similar behavior. Not sure of Kohya's impression of this to PR but that would be something I would like to be able to do.