@@ -6,7 +6,7 @@ In NNI, tuner will sample parameters/architecture according to the search space,
...
@@ -6,7 +6,7 @@ In NNI, tuner will sample parameters/architecture according to the search space,
To define a search space, users should define the name of variable, the type of sampling strategy and its parameters.
To define a search space, users should define the name of variable, the type of sampling strategy and its parameters.
* A example of search space definition as follow:
* An example of search space definition as follow:
```yaml
```yaml
{
{
...
@@ -26,9 +26,18 @@ Take the first line as an example. `dropout_rate` is defined as a variable whose
...
@@ -26,9 +26,18 @@ Take the first line as an example. `dropout_rate` is defined as a variable whose
All types of sampling strategies and their parameter are listed here:
All types of sampling strategies and their parameter are listed here:
* {"_type":"choice","_value":options}
* {"_type":"choice","_value":options}
* Which means the variable value is one of the options, which should be a list. The elements of options can themselves be [nested] stochastic expressions. In this case, the stochastic choices that only appear in some of the options become conditional parameters.
* Which means the variable's value is one of the options. Here 'options' should be a list. Each element of options is a number of string. It could also be a nested sub-search-space, this sub-search-space takes effect only when the corresponding element is chosen. The variables in this sub-search-space could be seen as conditional variables.
* An simple [example](../../examples/trials/mnist-cascading-search-space/search_space.json) of [nested] search space definition. If an element in the options list is a dict, it is a sub-search-space, and for our built-in tuners you have to add a key '_name' in this dict, which helps you to identify which element is chosen. Accordingly, here is a [sample](../../examples/trials/mnist-cascading-search-space/sample.json) which users can get from nni with nested search space definition. Tuners which support nested search space is as follows:
- Random Search
- TPE
- Anneal
- Evolution
* {"_type":"randint","_value":[upper]}
* {"_type":"randint","_value":[upper]}
* Which means the variable value is a random integer in the range [0, upper). The semantics of this distribution is that there is no more correlation in the loss function between nearby integer values, as compared with more distant integer values. This is an appropriate distribution for describing random seeds for example. If the loss function is probably more correlated for nearby integer values, then you should probably use one of the "quantized" continuous distributions, such as either quniform, qloguniform, qnormal or qlognormal. Note that if you want to change lower bound, you can use `quniform` for now.
* Which means the variable value is a random integer in the range [0, upper). The semantics of this distribution is that there is no more correlation in the loss function between nearby integer values, as compared with more distant integer values. This is an appropriate distribution for describing random seeds for example. If the loss function is probably more correlated for nearby integer values, then you should probably use one of the "quantized" continuous distributions, such as either quniform, qloguniform, qnormal or qlognormal. Note that if you want to change lower bound, you can use `quniform` for now.
* {"_type":"uniform","_value":[low, high]}
* {"_type":"uniform","_value":[low, high]}
...
@@ -48,6 +57,7 @@ All types of sampling strategies and their parameter are listed here:
...
@@ -48,6 +57,7 @@ All types of sampling strategies and their parameter are listed here:
* Suitable for a discrete variable with respect to which the objective is "smooth" and gets smoother with the size of the value, but which should be bounded both above and below.
* Suitable for a discrete variable with respect to which the objective is "smooth" and gets smoother with the size of the value, but which should be bounded both above and below.
* {"_type":"normal","_value":[mu, sigma]}
* {"_type":"normal","_value":[mu, sigma]}
* Which means the variable value is a real value that's normally-distributed with mean mu and standard deviation sigma. When optimizing, this is an unconstrained variable.
* Which means the variable value is a real value that's normally-distributed with mean mu and standard deviation sigma. When optimizing, this is an unconstrained variable.
* {"_type":"qnormal","_value":[mu, sigma, q]}
* {"_type":"qnormal","_value":[mu, sigma, q]}
...
@@ -55,6 +65,7 @@ All types of sampling strategies and their parameter are listed here:
...
@@ -55,6 +65,7 @@ All types of sampling strategies and their parameter are listed here:
* Suitable for a discrete variable that probably takes a value around mu, but is fundamentally unbounded.
* Suitable for a discrete variable that probably takes a value around mu, but is fundamentally unbounded.
* {"_type":"lognormal","_value":[mu, sigma]}
* {"_type":"lognormal","_value":[mu, sigma]}
* Which means the variable value is a value drawn according to exp(normal(mu, sigma)) so that the logarithm of the return value is normally distributed. When optimizing, this variable is constrained to be positive.
* Which means the variable value is a value drawn according to exp(normal(mu, sigma)) so that the logarithm of the return value is normally distributed. When optimizing, this variable is constrained to be positive.
`metrics` could be any python object. If users use NNI built-in tuner/assessor, `metrics` can only have two formats: 1) a number e.g., float, int, 2) a dict object that has a key named `default` whose value is a number. This `metrics` is reported to [assessor](Builtin_Assessors.md). Usually, `metrics` could be periodically evaluated loss or accuracy.
`metrics` could be any python object. If users use NNI built-in tuner/assessor, `metrics` can only have two formats: 1) a number e.g., float, int, 2) a dict object that has a key named `default` whose value is a number. This `metrics` is reported to [assessor](BuiltinAssessors.md). Usually, `metrics` could be periodically evaluated loss or accuracy.
- Report performance of the configuration
- Report performance of the configuration
```python
```python
nni.report_final_result(metrics)
nni.report_final_result(metrics)
```
```
`metrics` also could be any python object. If users use NNI built-in tuner/assessor, `metrics` follows the same format rule as that in `report_intermediate_result`, the number indicates the model's performance, for example, the model's accuracy, loss etc. This `metrics` is reported to [tuner](Builtin_Tuner.md).
`metrics` also could be any python object. If users use NNI built-in tuner/assessor, `metrics` follows the same format rule as that in `report_intermediate_result`, the number indicates the model's performance, for example, the model's accuracy, loss etc. This `metrics` is reported to [tuner](BuiltinTuner.md).
### Step 3 - Enable NNI API
### Step 3 - Enable NNI API
...
@@ -156,8 +156,8 @@ For more information, please refer to [HowToDebug](HowToDebug.md)
...
@@ -156,8 +156,8 @@ For more information, please refer to [HowToDebug](HowToDebug.md)
<aname="more-examples"></a>
<aname="more-examples"></a>
## More Trial Examples
## More Trial Examples
*[MNIST examples](mnist_examples.md)
*[MNIST examples](MnistExamples.md)
*[Finding out best optimizer for Cifar10 classification](cifar10_examples.md)
*[Finding out best optimizer for Cifar10 classification](Cifar10Examples.md)
*[How to tune Scikit-learn on NNI](sklearn_examples.md)
*[How to tune Scikit-learn on NNI](SklearnExamples.md)
*[Automatic Model Architecture Search for Reading Comprehension.](SQuAD_evolution_examples.md)
*[Automatic Model Architecture Search for Reading Comprehension.](SquadEvolutionExamples.md)
