Allow to inject a context manager around TaskProcess.run #2449
Conversation
ulzha
force-pushed
the
capturer
branch
2 times, most recently
from
fe52cf3 to
5c986c9
Compare
ulzha
changed the title
ulzha
changed the title
The intent with this is to enable collection of logs (stdout and stderr output) per task, as opposed to having to read through the worker's logs where all the task outputs are interspersed.
luigi/worker.py
Outdated
| @@ -258,6 +259,27 @@ def terminate(self): | |||
| return super(TaskProcess, self).terminate() | |||
|
|
|||
|
|
|||
| # TODO be composable with arbitrarily many custom context managers? | |||
| # Introduce a convention shared for extension points other than TaskProcess? | |||
| # Use https://docs.openstack.org/stevedore? | |||
There was a problem hiding this comment.
Maybe we could just link to our issue with the full debate/context on plugins?
| @@ -419,6 +441,12 @@ class worker(Config): | |||
| force_multiprocessing = BoolParameter(default=False, | |||
| description='If true, use multiprocessing also when ' | |||
| 'running with 1 worker') | |||
| task_process_context = Parameter(default=None, | |||
There was a problem hiding this comment.
Did you consider making this a "list" by making a comma-separate name of classnames?
But I guess users can make their single context-manager itself have multiple ones.
There was a problem hiding this comment.
Yes, exactly, that would be the next step I could think of; about to add in a separate PR.
There was a problem hiding this comment.
This will throw a warning because None is not allowed in string parameter:
UserWarning: Parameter "task_process_context" with value "None" is not of type string.
There was a problem hiding this comment.
This will throw a warning because
Noneis not allowed in string parameter:UserWarning: Parameter "task_process_context" with value "None" is not of type string.
I wonder why can't it be defaulted to "" instead, like in
task_process_context = Parameter(default="",
@ulzha any thoughts?
There was a problem hiding this comment.
Alternatively, you can leave the None default and make it an OptionalParameter
luigi/worker.py
Outdated
|
|
||
| def run(self): | ||
| if self.context: | ||
| logger.debug('Instantiating ' + self.context) |
There was a problem hiding this comment.
Maybe something you can grep easier like "Importing module for "?
|
Feel free to merge this. Perhaps someone else from Spotify should review too? |
Yes, tested 👍 |
Description
Adds a configurable context manager around TaskProcess.
Motivation and Context
The intent with this is to enable customizable collection of logs per task (stdout and stderr output from any Task's run() method, including any subprocesses spawned by it), as opposed to having to read through the worker's logs where outputs from all the tasks are interspersed.
Inspired by CaptureOutput from the opensource capturer package.
Such configurable context managers could be useful extension points for "plugins" made for other purposes too. (Related: #1897)
Have you tested this? If so, how?
Has unittests.