Create Tune API in the Katib SDK #1951
Conversation
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
|
/hold Hold until kubeflow/training-operator#1659 is merged. |
|
Thanks for implementing the new feature! |
| # Import required packages. | ||
| import time |
There was a problem hiding this comment.
I understand why this kind of logic is necesary, but we might find out more friendly way to specify packages by users.
However, as a minimal feature, this is ok if we could instruct users that they couldn't make a mistake to miss required packages in the objective function.
There was a problem hiding this comment.
I agree, but I think Kubeflow Pipelines follows the same way: https://www.kubeflow.org/docs/components/pipelines/sdk/python-function-components/#building-python-function-based-components
@zijianjoy Do you know if Pipelines SDK verifies that all function imports are defined properly in Python Lightweight component?
@anencore94 Any ideas how to verify this?
cc @tenzen-y @gaocegege @johnugeorge
There was a problem hiding this comment.
We may need to automatically validating this after this PR. How about making an issue for tracking this feature now ?
If we could make an idea for this validation in general, we may apply this for other kf components like pipelines and training-operators and so on.
There was a problem hiding this comment.
Sure, that sound good @anencore94.
|
|
||
| return result | ||
|
|
||
|
|
||
| def double(min: float, max: float, step: float = None): |
There was a problem hiding this comment.
How about gather these hyperparameter helper function to another inner module?
For example, the users could call these functions by
import kubeflow.katib as katib
hp_search_space = {
"a": katib.hp_search_space.int(min=10, max=20),
"b": katib.hp_search_space.double(min=0.1, max=0.2)
}
katib_client.tune(
name=name,
objective=objective,
hp_search_space=hp_search_space,
objective_metric_name="result",
max_trial_count=12
)IMO, importing by katib.int makes me it's too board that we couldn't see it is for hyperparameter search space type immediately.
There was a problem hiding this comment.
I was even thinking to change this API in the future to support various distributions: uniform, loguniform, categorical, etc.
FYI, hyperopt uses hp module, skop uses space module, optuna uses distributions module, Ray uses tune module. I think we can learn something from it.
search module (e.g. katib.search.int(1,5)) looks good to me, but I am not sure what is the best UX here.
What others think @terrytangyuan @gaocegege @johnugeorge @c-bata @a9p @g-votte @tenzen-y ?
There was a problem hiding this comment.
Ref #1951 (comment).
IMO it feels more natural if we could support built-in lists or common numpy arrays, etc. Is there a particular motivation for bringing a separate module in the Katib SDK just to do the same thing?
There was a problem hiding this comment.
@terrytangyuan List and numpy arrays might not be enough.
How we can identify search space for uniform or loguniform in the future?
e.g. Hyperopt utilises some numpy APIs, but it still has it's own functions for the search space: https://github.com/hyperopt/hyperopt/blob/master/hyperopt/pyll/stochastic.py#L41-L43
There was a problem hiding this comment.
The functions in that module seem to be very thin wrappers. If we could completely avoid creating additional wrappers/layers, it would be great.
There was a problem hiding this comment.
I agree with @andreyvelich.
In this PR, we can start in the minimal feature. So, we can support other distributions in other PRs.
There was a problem hiding this comment.
@terrytangyuan Do you have any ideas how we can improve UX on this ?
There was a problem hiding this comment.
I don't have a strong preference. I am ok as is if other projects are following the same convention.
| algorithm_name: str = "random", | ||
| objective_metric_name: str = None, | ||
| additional_metric_names: List[str] = [], | ||
| objective_type: str = "maximize", |
There was a problem hiding this comment.
maybe we could use "maximize" and "minimize" as an Enum class or Literal
There was a problem hiding this comment.
I was also thinking about it.
@anencore94 Is there a way to automatically generate enums for our Python Models ?
In our APIs type param is enum: https://github.com/kubeflow/katib/blob/master/pkg/apis/controller/common/v1beta1/common_types.go#L96
There was a problem hiding this comment.
By "automatically" means that automatically generate python Enum class from the golang file or openapi.json ?
There was a problem hiding this comment.
By "automatically" means that automatically generate python Enum class from the golang file or openapi.json ?
Exactly. I am not sure if kube-openapi supports enum generation. Check this: kubernetes/enhancements#2887.
Maybe we should update kube-openapi version for this feature.
There was a problem hiding this comment.
I have no many experience with it sadly. How about make an issue with this feature, and merge this PR with hardcoded for now ?
There was a problem hiding this comment.
Sounds good @anencore94 !
Modify packages_to_install doc Create validate objective function
There was a problem hiding this comment.
@andreyvelich Thanks for your contributions!
I left a few comments.
Change k8s version package
|
Thanks everyone for review! |
Co-authored-by: Yuki Iwai <yuki.iwai.tz@gmail.com>
|
Thanks for addressing my comments! |
|
/approve |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: andreyvelich, johnugeorge The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
Thanks @andreyvelich for this great feature |
|
Thanks everyone for your feedback and review! |
Fixes: #1585.
Inspired by KFP
create_component_from_func.This API will allow user to create Katib HP Tuning Experiment without building the Experiment YAML.
This is the first small step to simplify our Kubeflow SDKs and to avoid Kubernetes complexity.
Later we can extend this functionality (support more Katib features in a simple way), give more spec options via APIs.
Also, we can provide API (e.g.
katib.list_base_images) to show list of supported images (e.g.BASE_IMAGE_TENSORFLOW,BASE_IMAGE_MXNET) with their description.cc @kubeflow/wg-training-leads @tenzen-y @anencore94 Please give your feedback on the API design.