Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Validate possible operations for Grid suggestion #1205

Conversation

andreyvelich
Copy link
Member

Fixes: #1122.

I added validation for Chocolate Grid suggestion to check if max_trial_count >= all possible search space combination.
Also, I moved call_validate() test function to utils to use it in Chocolate also.

/assign @gaocegege @johnugeorge
/cc @sperlingxx

@kubeflow-bot
Copy link

This change is Reviewable

@andreyvelich
Copy link
Member Author

/retest

@elikatsis
Copy link
Member

Hello @andreyvelich,

Me and @StefanoFioravanzo have been tinkering with Katib for some time now and have a different proposal on tackling #1122. We will describe it in detail as a comment to that issue by the end of the week.

Would you mind holding this PR until then and decide afterwards whether to merge this or not?

Thanks in advance!

@andreyvelich
Copy link
Member Author

andreyvelich commented Jun 3, 2020

Hi @elikatsis, sure.
Is that different from regular Validate Algorithm Settings?

@@ -4,12 +4,15 @@
from pkg.apis.manager.v1beta1.python import api_pb2
from pkg.apis.manager.v1beta1.python import api_pb2_grpc

from pkg.suggestion.v1beta1.internal.constant import DOUBLE
from pkg.suggestion.v1beta1.internal.constant import (INTEGER, DOUBLE, CATEGORICAL, DISCRETE)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not good at py, why do we need () for it?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can remove it from here, it uses when we need to use more than 1 line for importing

Copy link
Member

@sperlingxx sperlingxx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

By the way, I think we had better to support other common scales (like log-uniform scale) in feasible space.

@gaocegege
Copy link
Member

By the way, I think we had better to support other common scales (like log-uniform scale) in feasible space.

I think so, would you mind opening an issue for it? @sperlingxx

@k8s-ci-robot k8s-ci-robot removed the lgtm label Jun 4, 2020
@andreyvelich
Copy link
Member Author

/lgtm

By the way, I think we had better to support other common scales (like log-uniform scale) in feasible space.

In skopt we use log-uniform distribution: https://github.com/kubeflow/katib/blob/master/pkg/suggestion/v1beta1/skopt/base_service.py#L44.
To define it, max and min DOUBLE value is not enough?

@sperlingxx
Copy link
Member

/lgtm
By the way, I think we had better to support other common scales (like log-uniform scale) in feasible space.

In skopt we use log-uniform distribution: https://github.com/kubeflow/katib/blob/master/pkg/suggestion/v1beta1/skopt/base_service.py#L44.
To define it, max and min DOUBLE value is not enough?

Other packages used by katib also support multiple distributions:

  • hyperopt supports loguniform, lognormal, normal, qnormal ...
  • optuna supports loguniform and uniform

So, maybe is it better to add an option to configure distribution of continuous search space in FeasibleSpace ? And we can validate whether the tuning algorithm supporting specific distribution in ValidateAlgorithmSettings method of suggestion service.

@gaocegege
Copy link
Member

/lgtm
/approve

@k8s-ci-robot
Copy link

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: gaocegege

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@andreyvelich
Copy link
Member Author

/hold

@andreyvelich
Copy link
Member Author

I believe @elikatsis wanted to hold this PR until end of this week.
We will merge it after.
/cc @gaocegege @sperlingxx

@gaocegege
Copy link
Member

Oh SGTM. Thanks for the hold in time.

@stale
Copy link

stale bot commented Jun 11, 2021

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@andreyvelich
Copy link
Member Author

/lifecycle frozen

@google-oss-robot
Copy link

@andreyvelich: The lifecycle/frozen label cannot be applied to Pull Requests.

In response to this:

/lifecycle frozen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@andreyvelich andreyvelich force-pushed the issue-1122-validate-chocolate-exhausted branch from 33a188e to 9c89dcb Compare August 9, 2021 17:16
@google-oss-robot
Copy link

New changes are detected. LGTM label has been removed.

@google-oss-robot
Copy link

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: andreyvelich, gaocegege, sperlingxx

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:
  • OWNERS [andreyvelich,gaocegege]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@andreyvelich
Copy link
Member Author

I think we can merge these changes for Grid suggestion.
WDYT @anencore94 @johnugeorge @gaocegege ?

@andreyvelich
Copy link
Member Author

/retest

@gaocegege
Copy link
Member

SGTM!

Copy link
Member

@anencore94 anencore94 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

moving call_validate function to utils looks better! Thanks :) @andreyvelich

@andreyvelich
Copy link
Member Author

/retest

2 similar comments
@andreyvelich
Copy link
Member Author

/retest

@andreyvelich
Copy link
Member Author

/retest

@aws-kf-ci-bot
Copy link
Contributor

aws-kf-ci-bot commented Aug 10, 2021

@andreyvelich: The following test failed, say /retest to rerun all failed tests:

Test name Commit Details Rerun command
kubeflow-katib-presubmit-e2e 33a188e link /test kubeflow-katib-presubmit-e2e

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@andreyvelich
Copy link
Member Author

/retest

@gaocegege
Copy link
Member

LGTM

@google-oss-robot google-oss-robot merged commit be2b26d into kubeflow:master Aug 11, 2021
@andreyvelich andreyvelich deleted the issue-1122-validate-chocolate-exhausted branch October 6, 2021 00:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Chocolate service db exhausted
10 participants