@@ -14,7 +14,7 @@ Currently we support the following algorithms:
|[__Naïve Evolution__](#Evolution)|Naïve Evolution comes from Large-Scale Evolution of Image Classifiers. It randomly initializes a population-based on search space. For each generation, it chooses better ones and does some mutation (e.g., change a hyperparameter, add/remove one layer) on them to get the next generation. Naïve Evolution requires many trials to works, but it's very simple and easy to expand new features. [Reference paper](https://arxiv.org/pdf/1703.01041.pdf)|
|[__SMAC__](#SMAC)|SMAC is based on Sequential Model-Based Optimization (SMBO). It adapts the most prominent previously used model class (Gaussian stochastic process models) and introduces the model class of random forests to SMBO, in order to handle categorical parameters. The SMAC supported by NNI is a wrapper on the SMAC3 GitHub repo. Notice, SMAC need to be installed by `nnictl package` command. [Reference Paper,](https://www.cs.ubc.ca/~hutter/papers/10-TR-SMAC.pdf)[GitHub Repo](https://github.com/automl/SMAC3)|
|[__Batch tuner__](#Batch)|Batch tuner allows users to simply provide several configurations (i.e., choices of hyper-parameters) for their trial code. After finishing all the configurations, the experiment is done. Batch tuner only supports the type choice in search space spec.|
|[__Grid Search__](#GridSearch)|Grid Search performs an exhaustive searching through a manually specified subset of the hyperparameter space defined in the searchspace file. Note that the only acceptable types of search space are choice, quniform, qloguniform. |
|[__Grid Search__](#GridSearch)|Grid Search performs an exhaustive searching through a manually specified subset of the hyperparameter space defined in the searchspace file. Note that the only acceptable types of search space are choice, quniform, randint. |
|[__Hyperband__](#Hyperband)|Hyperband tries to use the limited resource to explore as many configurations as possible, and finds out the promising ones to get the final result. The basic idea is generating many configurations and to run them for the small number of trial budget to find out promising one, then further training those promising ones to select several more promising one.[Reference Paper](https://arxiv.org/pdf/1603.06560.pdf)|
|[__Network Morphism__](#NetworkMorphism)|Network Morphism provides functions to automatically search for architecture of deep learning models. Every child network inherits the knowledge from its parent network and morphs into diverse types of networks, including changes of depth, width, and skip-connection. Next, it estimates the value of a child network using the historic architecture and metric pairs. Then it selects the most promising one to train. [Reference Paper](https://arxiv.org/abs/1806.10282)|
|[__Metis Tuner__](#MetisTuner)|Metis offers the following benefits when it comes to tuning parameters: While most tools only predict the optimal configuration, Metis gives you two outputs: (a) current prediction of optimal configuration, and (b) suggestion for the next trial. No more guesswork. While most tools assume training datasets do not have noisy data, Metis actually tells you if you need to re-sample a particular hyper-parameter. [Reference Paper](https://www.microsoft.com/en-us/research/publication/metis-robustly-tuning-tail-latencies-cloud-systems/)|
...
...
@@ -211,7 +211,7 @@ The search space file including the high-level key `combine_params`. The type of
**Suggested scenario**
Note that the only acceptable types of search space are `choice`, `quniform`, `qloguniform`.
Note that the only acceptable types of search space are `choice`, `quniform`, `randint`.
It is suggested when search space is small, it is feasible to exhaustively sweeping the whole search space. [Detailed Description](./GridsearchTuner.md)
GridSearchTuner will search all the possible configures that the user define in the searchSpace.
The only acceptable types of search space are 'quniform', 'qloguniform' and 'choice'
The only acceptable types of search space are 'choice', 'quniform', 'randint'
Type 'choice' will select one of the options. Note that it can also be nested.
Type 'quniform' will receive three values [low, high, q], where [low, high] specifies a range and 'q' specifies the interval
It will be sampled in a way that the first sampled value is 'low', and each of the following values is 'interval' larger than the value in front of it.
Type 'qloguniform' behaves like 'quniform' except that it will first change the range to [log(low), log(high)]
and sample and then change the sampled value back.
Type 'randint' gives all possible intergers in range(low, high). Note that 'high' is not included.
'''
def__init__(self):
...
...
@@ -72,8 +71,12 @@ class GridSearchTuner(Tuner):
chosen_params.extend(choice)
else:
chosen_params.append(choice)
elif_type=='quniform':
chosen_params=self._parse_quniform(_value)
elif_type=='randint':
chosen_params=self._parse_randint(_value)
else:
chosen_params=self.parse_qtype(_type,_value)
raiseRuntimeError("Not supported type: %s"%_type)
else:
chosen_params=dict()
forkeyinss_spec.keys():
...
...
@@ -98,14 +101,9 @@ class GridSearchTuner(Tuner):
count=int(np.floor((high-low)/interval))+1
return[low+interval*iforiinrange(count)]
defparse_qtype(self,param_type,param_value):
'''parse type of quniform or qloguniform'''
ifparam_typein['quniform','qloguniform']:
# qloguniform has no difference from quniform,
# since grid search does not consider possibility distribution
