Add config option daemon.default_workers and daemon.worker_process_slots
#3949
Conversation
Codecov Report
@@ Coverage Diff @@
## develop #3949 +/- ##
========================================
Coverage 78.31% 78.31%
========================================
Files 461 461
Lines 34079 34082 +3
========================================
+ Hits 26688 26693 +5
+ Misses 7391 7389 -2
Continue to review full report at Codecov.
|
| 'valid_type': 'int', | ||
| 'valid_values': None, | ||
| 'default': DEFAULT_DAEMON_WORKERS, | ||
| 'description': 'The default number of workers to be launched by `verdi daemon start`', |
There was a problem hiding this comment.
Just to understand - e.g. if I use incr does this value change?
In which sense is it a default rather than the actual number of daemon workers?
There was a problem hiding this comment.
verdi daemon incr is an independent command. It still takes an argument to tell how many workers should be added, verdi daemon incr N will add N workers with default being 1. The new option only applies how many workers are started automatically when invoking verdi daemon start. Already you can do verdi daemon start 10 to start 10 workers straight away, but now you can configure it to have it start 10 by default. Saves an entire three key strokes ;)
There was a problem hiding this comment.
Ok, I just wanted to make sure.
I think some users might assume that if you do verdi daemon incr 1 and then restart the daemon that you would keep the increased number of workers rather than resetting to this default.
It might make sense to mention explicitly in the docstring of verdi daemon incr that these changes are reset whenever you stop the daemon, and that you should use this new config option to make permanent changes.
There was a problem hiding this comment.
It is a bit more subtle than that even. Doing verdi daemon restart will restart the workers but not the daemon itself, i.e. you will keep how every many workers that were running before the restart command. Doing verdi daemon restart --reset literally does verdi daemon stop && verdi daemon start so there it will start whatever is the default for verdi daemon start, which can now be configured through this new option
There was a problem hiding this comment.
This was already explained in the docstring of verdi daemon restart:
By default will only reset the workers of the running daemon. After the restart the same amount of workers will be
running. If the `--reset` flag is passed, however, the full circus daemon will be stopped and restarted with just
a single worker.
I just noticed that we just have to update the last bit with respect to the new default behavior.
There was a problem hiding this comment.
I have updated the docstrings of verdi daemon start and verdi daemon restart in the first commit to reflect the new behavior and give more detail
sphuber
force-pushed
the
feature/3677/configurable-task-prefetch-count
branch
from
35393a9 to
90889c3
Compare
This option will determine the default number of workers that are started by the `verdi daemon start` command. Specifying an explicit value as argument will of course still override this.
This option is used in the construction of the RabbitMQ communicator and will determine the maximum number of process tasks that any one daemon runner can operate on concurrently. If this number is set too high, the runner can get overwhelmed as there is no logic for it to prioritize finishing active processes over taking on new processes that have arrived in the process task queue.
sphuber
force-pushed
the
feature/3677/configurable-task-prefetch-count
branch
from
90889c3 to
7fd1b81
Compare
|
Thanks for the reviews @ltalirz ! |
Fixes #3677
This option will determine the default number of workers that are
started by the
verdi daemon startcommand. Specifying an explicitvalue as argument will of course still override this.
This option is used in the construction of the RabbitMQ communicator and
will determine the maximum number of process tasks that any one daemon
runner can operate on concurrently. If this number is set too high, the
runner can get overwhelmed as there is no logic for it to prioritize
finishing active processes over taking on new processes that have
arrived in the process task queue.