Unverified Commit 04d2d7cb authored by Guoxin's avatar Guoxin Committed by GitHub
Browse files

#1546 add doc for gp-tuner: support only numerical values (#1551)

* add doc & err catch for gp-tuner: support only numerical values
parent 59ce65c5
...@@ -308,8 +308,7 @@ tuner: ...@@ -308,8 +308,7 @@ tuner:
> Built-in Tuner Name: **MetisTuner** > Built-in Tuner Name: **MetisTuner**
Note that the only acceptable types of search space are `choice`, `quniform`, `uniform` and `randint`. Note that the only acceptable types of search space are `quniform`, `uniform` and `randint` and numerical `choice`. Only numerical values are supported since the values will be used to evaluate the 'distance' between different points.
**Suggested scenario** **Suggested scenario**
Similar to TPE and SMAC, Metis is a black-box tuner. If your system takes a long time to finish each trial, Metis is more favorable than other approaches such as random search. Furthermore, Metis provides guidance on the subsequent trial. Here is an [example](https://github.com/Microsoft/nni/tree/master/examples/trials/auto-gbdt/search_space_metis.json) about the use of Metis. User only need to send the final result like `accuracy` to tuner, by calling the NNI SDK. [Detailed Description](./MetisTuner.md) Similar to TPE and SMAC, Metis is a black-box tuner. If your system takes a long time to finish each trial, Metis is more favorable than other approaches such as random search. Furthermore, Metis provides guidance on the subsequent trial. Here is an [example](https://github.com/Microsoft/nni/tree/master/examples/trials/auto-gbdt/search_space_metis.json) about the use of Metis. User only need to send the final result like `accuracy` to tuner, by calling the NNI SDK. [Detailed Description](./MetisTuner.md)
...@@ -381,7 +380,7 @@ advisor: ...@@ -381,7 +380,7 @@ advisor:
> Built-in Tuner Name: **GPTuner** > Built-in Tuner Name: **GPTuner**
Note that the only acceptable types of search space are `choice`, `randint`, `uniform`, `quniform`, `loguniform`, `qloguniform`. Note that the only acceptable types of search space are `randint`, `uniform`, `quniform`, `loguniform`, `qloguniform`, and numerical `choice`. Only numerical values are supported since the values will be used to evaluate the 'distance' between different points.
**Suggested scenario** **Suggested scenario**
......
...@@ -7,4 +7,6 @@ Bayesian optimization works by constructing a posterior distribution of function ...@@ -7,4 +7,6 @@ Bayesian optimization works by constructing a posterior distribution of function
GP Tuner is designed to minimize/maximize the number of steps required to find a combination of parameters that are close to the optimal combination. To do so, this method uses a proxy optimization problem (finding the maximum of the acquisition function) that, albeit still a hard problem, is cheaper (in the computational sense) and common tools can be employed. Therefore Bayesian Optimization is most adequate for situations where sampling the function to be optimized is a very expensive endeavor. GP Tuner is designed to minimize/maximize the number of steps required to find a combination of parameters that are close to the optimal combination. To do so, this method uses a proxy optimization problem (finding the maximum of the acquisition function) that, albeit still a hard problem, is cheaper (in the computational sense) and common tools can be employed. Therefore Bayesian Optimization is most adequate for situations where sampling the function to be optimized is a very expensive endeavor.
Note that the only acceptable types of search space are `randint`, `uniform`, `quniform`, `loguniform`, `qloguniform`, and numerical `choice`.
This optimization approach is described in Section 3 of [Algorithms for Hyper-Parameter Optimization](https://papers.nips.cc/paper/4443-algorithms-for-hyper-parameter-optimization.pdf). This optimization approach is described in Section 3 of [Algorithms for Hyper-Parameter Optimization](https://papers.nips.cc/paper/4443-algorithms-for-hyper-parameter-optimization.pdf).
...@@ -15,6 +15,6 @@ It finds the global optimal point in the Gaussian Process space. This point repr ...@@ -15,6 +15,6 @@ It finds the global optimal point in the Gaussian Process space. This point repr
It identifies the next hyper-parameter candidate. This is achieved by inferring the potential information gain of exploration, exploitation, and re-sampling. It identifies the next hyper-parameter candidate. This is achieved by inferring the potential information gain of exploration, exploitation, and re-sampling.
Note that the only acceptable types of search space are `choice`, `quniform`, `uniform` and `randint`. Note that the only acceptable types of search space are `quniform`, `uniform` and `randint` and numerical `choice`.
More details can be found in our paper: https://www.microsoft.com/en-us/research/publication/metis-robustly-tuning-tail-latencies-cloud-systems/ More details can be found in our paper: https://www.microsoft.com/en-us/research/publication/metis-robustly-tuning-tail-latencies-cloud-systems/
\ No newline at end of file
...@@ -94,7 +94,7 @@ All types of sampling strategies and their parameter are listed here: ...@@ -94,7 +94,7 @@ All types of sampling strategies and their parameter are listed here:
Known Limitations: Known Limitations:
* Note that Metis Tuner only supports numerical `choice` now * GP Tuner and Metis Tuner support only **numerical values** in search space(`choice` type values can be no-numeraical with other tuners, e.g. string values). Both GP Tuner and Metis Tuner use Gaussian Process Regressor(GPR). GPR make predictions based on a kernel function and the 'distance' between different points, it's hard to get the true distance between no-numerical values.
* Note that for nested search space: * Note that for nested search space:
......
...@@ -55,6 +55,14 @@ class TargetSpace(): ...@@ -55,6 +55,14 @@ class TargetSpace():
[item[1] for item in sorted(pbounds.items(), key=lambda x: x[0])] [item[1] for item in sorted(pbounds.items(), key=lambda x: x[0])]
) )
# check values type
for _bound in self._bounds:
if _bound['_type'] == 'choice':
try:
[float(val) for val in _bound['_value']]
except ValueError:
raise ValueError("GP Tuner supports only numerical values")
# preallocated memory for X and Y points # preallocated memory for X and Y points
self._params = np.empty(shape=(0, self.dim)) self._params = np.empty(shape=(0, self.dim))
self._target = np.empty(shape=(0)) self._target = np.empty(shape=(0))
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment