-
Notifications
You must be signed in to change notification settings - Fork 328
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Scheduling] Expand the ability of resource evaluator #2997
[Scheduling] Expand the ability of resource evaluator #2997
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
About resource evaluator, is there a plan about it? Apart from the plan, I left some comments.
It is necessary to build an external resource recommendation service that can take advantage of historical job information if we want to fully utilize resource evaluator. Maybe we could add an extension library of mars later. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
What do these changes do?
Resource evaluator is used to estimate and set resources required by subtasks. It can be an internal service or an external service. If it is an internal service, we can set default of adjustable resources for subtasks. If it is an external service, we should report the running result of the task to the external service, so that it can accurately predict the required resources of subtasks based on the historical running information, we call it HBO.
But it is not easy to implement a new resource evaluator and config it. This pr introduces an extension point of resource evaluator with which we could add a new evaluator as follows:
Inherit
ResourceEvaluator
and implementcreate
,evaluate
andreport
methods. Thecreate
method is to create a new resource evaluator instance. Theevaluate
method is to estimate and set required resources for the subtasks of a task stage. And this method must be implemented. Thereport
method is to report the running information and result of the task. And this method does not have to be implemented.Add default configs of the new evaluator needed in
base_config.xml
or its descendant files.Set the
resource_evaluator
to choose a resource evaluator inbase_config.xml
when running a mars job.Related issue number
Fixes #xxxx
Check code requirements