diff --git a/docs/en_US/Tuner/BuiltinTuner.md b/docs/en_US/Tuner/BuiltinTuner.md index 1825e8dcd1..c7d27fc5d1 100644 --- a/docs/en_US/Tuner/BuiltinTuner.md +++ b/docs/en_US/Tuner/BuiltinTuner.md @@ -308,8 +308,7 @@ tuner: > Built-in Tuner Name: **MetisTuner** -Note that the only acceptable types of search space are `choice`, `quniform`, `uniform` and `randint`. - +Note that the only acceptable types of search space are `quniform`, `uniform` and `randint` and numerical `choice`. Only numerical values are supported since the values will be used to evaluate the 'distance' between different points. **Suggested scenario** Similar to TPE and SMAC, Metis is a black-box tuner. If your system takes a long time to finish each trial, Metis is more favorable than other approaches such as random search. Furthermore, Metis provides guidance on the subsequent trial. Here is an [example](https://github.com/Microsoft/nni/tree/master/examples/trials/auto-gbdt/search_space_metis.json) about the use of Metis. User only need to send the final result like `accuracy` to tuner, by calling the NNI SDK. [Detailed Description](./MetisTuner.md) @@ -381,7 +380,7 @@ advisor: > Built-in Tuner Name: **GPTuner** -Note that the only acceptable types of search space are `choice`, `randint`, `uniform`, `quniform`, `loguniform`, `qloguniform`. +Note that the only acceptable types of search space are `randint`, `uniform`, `quniform`, `loguniform`, `qloguniform`, and numerical `choice`. Only numerical values are supported since the values will be used to evaluate the 'distance' between different points. **Suggested scenario** diff --git a/docs/en_US/Tuner/GPTuner.md b/docs/en_US/Tuner/GPTuner.md index 9ef49db2bb..0159c3d5c9 100644 --- a/docs/en_US/Tuner/GPTuner.md +++ b/docs/en_US/Tuner/GPTuner.md @@ -7,4 +7,6 @@ Bayesian optimization works by constructing a posterior distribution of function GP Tuner is designed to minimize/maximize the number of steps required to find a combination of parameters that are close to the optimal combination. To do so, this method uses a proxy optimization problem (finding the maximum of the acquisition function) that, albeit still a hard problem, is cheaper (in the computational sense) and common tools can be employed. Therefore Bayesian Optimization is most adequate for situations where sampling the function to be optimized is a very expensive endeavor. +Note that the only acceptable types of search space are `randint`, `uniform`, `quniform`, `loguniform`, `qloguniform`, and numerical `choice`. + This optimization approach is described in Section 3 of [Algorithms for Hyper-Parameter Optimization](https://papers.nips.cc/paper/4443-algorithms-for-hyper-parameter-optimization.pdf). diff --git a/docs/en_US/Tuner/MetisTuner.md b/docs/en_US/Tuner/MetisTuner.md index 7c0c8e3e37..d653796e01 100644 --- a/docs/en_US/Tuner/MetisTuner.md +++ b/docs/en_US/Tuner/MetisTuner.md @@ -15,6 +15,6 @@ It finds the global optimal point in the Gaussian Process space. This point repr It identifies the next hyper-parameter candidate. This is achieved by inferring the potential information gain of exploration, exploitation, and re-sampling. -Note that the only acceptable types of search space are `choice`, `quniform`, `uniform` and `randint`. +Note that the only acceptable types of search space are `quniform`, `uniform` and `randint` and numerical `choice`. More details can be found in our paper: https://www.microsoft.com/en-us/research/publication/metis-robustly-tuning-tail-latencies-cloud-systems/ \ No newline at end of file diff --git a/docs/en_US/Tutorial/SearchSpaceSpec.md b/docs/en_US/Tutorial/SearchSpaceSpec.md index ea0d6bdf9e..b892a5e1e5 100644 --- a/docs/en_US/Tutorial/SearchSpaceSpec.md +++ b/docs/en_US/Tutorial/SearchSpaceSpec.md @@ -94,7 +94,7 @@ All types of sampling strategies and their parameter are listed here: Known Limitations: -* Note that Metis Tuner only supports numerical `choice` now +* GP Tuner and Metis Tuner support only **numerical values** in search space(`choice` type values can be no-numeraical with other tuners, e.g. string values). Both GP Tuner and Metis Tuner use Gaussian Process Regressor(GPR). GPR make predictions based on a kernel function and the 'distance' between different points, it's hard to get the true distance between no-numerical values. * Note that for nested search space: diff --git a/src/sdk/pynni/nni/gp_tuner/target_space.py b/src/sdk/pynni/nni/gp_tuner/target_space.py index 831bc335df..56481fa691 100644 --- a/src/sdk/pynni/nni/gp_tuner/target_space.py +++ b/src/sdk/pynni/nni/gp_tuner/target_space.py @@ -55,6 +55,14 @@ def __init__(self, pbounds, random_state=None): [item[1] for item in sorted(pbounds.items(), key=lambda x: x[0])] ) + # check values type + for _bound in self._bounds: + if _bound['_type'] == 'choice': + try: + [float(val) for val in _bound['_value']] + except ValueError: + raise ValueError("GP Tuner supports only numerical values") + # preallocated memory for X and Y points self._params = np.empty(shape=(0, self.dim)) self._target = np.empty(shape=(0))