@@ -4,10 +4,8 @@ NNI provides state-of-the-art tuning algorithm in our builtin-assessors and make
|Assessor|Brief Introduction of Algorithm|
|---|---|
|**Medianstop**<br>[(Usage)](#MedianStop)|Medianstop is a simple early stopping rule mentioned in the [paper](https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/46180.pdf). It stops a pending trial X at step S if the trial’s best objective value by step S is strictly worse than the median value of the running averages of all completed trials’ objectives reported up to step S.|
|[Curvefitting](https://github.com/Microsoft/nni/blob/master/src/sdk/pynni/nni/curvefitting_assessor/README.md)<br>[(Usage)](#Curvefitting)|Curve Fitting Assessor is a LPA(learning, predicting, assessing) algorithm. It stops a pending trial X at step S if the prediction of final epoch's performance worse than the best final performance in the trial history. In this algorithm, we use 12 curves to fit the accuracy curve|
<br>
|**Medianstop**[(Usage)](#MedianStop)|Medianstop is a simple early stopping rule mentioned in the [paper](https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/46180.pdf). It stops a pending trial X at step S if the trial’s best objective value by step S is strictly worse than the median value of the running averages of all completed trials’ objectives reported up to step S.|
|[Curvefitting](https://github.com/Microsoft/nni/blob/master/src/sdk/pynni/nni/curvefitting_assessor/README.md)[(Usage)](#Curvefitting)|Curve Fitting Assessor is a LPA(learning, predicting, assessing) algorithm. It stops a pending trial X at step S if the prediction of final epoch's performance worse than the best final performance in the trial history. In this algorithm, we use 12 curves to fit the accuracy curve|
@@ -4,16 +4,16 @@ NNI provides state-of-the-art tuning algorithm as our builtin-tuners and makes t
|Tuner|Brief Introduction of Algorithm|
|---|---|
|**TPE**<br>[(Usage)](#TPE)|The Tree-structured Parzen Estimator (TPE) is a sequential model-based optimization (SMBO) approach. SMBO methods sequentially construct models to approximate the performance of hyperparameters based on historical measurements, and then subsequently choose new hyperparameters to test based on this model.|
|**Random Search**<br>[(Usage)](#Random)|In Random Search for Hyper-Parameter Optimization show that Random Search might be surprisingly simple and effective. We suggest that we could use Random Search as the baseline when we have no knowledge about the prior distribution of hyper-parameters.|
|**Anneal**<br>[(Usage)](#Anneal)|This simple annealing algorithm begins by sampling from the prior, but tends over time to sample from points closer and closer to the best ones observed. This algorithm is a simple variation on the random search that leverages smoothness in the response surface. The annealing rate is not adaptive.|
|**Naive Evolution**<br>[(Usage)](#Evolution)|Naive Evolution comes from Large-Scale Evolution of Image Classifiers. It randomly initializes a population-based on search space. For each generation, it chooses better ones and does some mutation (e.g., change a hyperparameter, add/remove one layer) on them to get the next generation. Naive Evolution requires many trials to works, but it's very simple and easy to expand new features.|
|**SMAC**<br>[(Usage)](#SMAC)|SMAC is based on Sequential Model-Based Optimization (SMBO). It adapts the most prominent previously used model class (Gaussian stochastic process models) and introduces the model class of random forests to SMBO, in order to handle categorical parameters. The SMAC supported by nni is a wrapper on the SMAC3 Github repo.|
|**Batch tuner**<br>[(Usage)](#Batch)|Batch tuner allows users to simply provide several configurations (i.e., choices of hyper-parameters) for their trial code. After finishing all the configurations, the experiment is done. Batch tuner only supports the type choice in search space spec.|
|**Grid Search**<br>[(Usage)](#GridSearch)|Grid Search performs an exhaustive searching through a manually specified subset of the hyperparameter space defined in the searchspace file. Note that the only acceptable types of search space are choice, quniform, qloguniform. The number q in quniform and qloguniform has special meaning (different from the spec in search space spec). It means the number of values that will be sampled evenly from the range low and high.|
|[Hyperband](https://github.com/Microsoft/nni/tree/master/src/sdk/pynni/nni/hyperband_advisor)<br>[(Usage)](#Hyperband)|Hyperband tries to use the limited resource to explore as many configurations as possible, and finds out the promising ones to get the final result. The basic idea is generating many configurations and to run them for the small number of STEPs to find out promising one, then further training those promising ones to select several more promising one.|
|[Network Morphism](https://github.com/Microsoft/nni/blob/master/src/sdk/pynni/nni/networkmorphism_tuner/README.md)<br>[(Usage)](#NetworkMorphism)|Network Morphism provides functions to automatically search for architecture of deep learning models. Every child network inherits the knowledge from its parent network and morphs into diverse types of networks, including changes of depth, width, and skip-connection. Next, it estimates the value of a child network using the historic architecture and metric pairs. Then it selects the most promising one to train.|
|**Metis Tuner**<br>[(Usage)](#MetisTuner)|Metis offers the following benefits when it comes to tuning parameters: While most tools only predict the optimal configuration, Metis gives you two outputs: (a) current prediction of optimal configuration, and (b) suggestion for the next trial. No more guesswork. While most tools assume training datasets do not have noisy data, Metis actually tells you if you need to re-sample a particular hyper-parameter.|
|**TPE**[(Usage)](#TPE)|The Tree-structured Parzen Estimator (TPE) is a sequential model-based optimization (SMBO) approach. SMBO methods sequentially construct models to approximate the performance of hyperparameters based on historical measurements, and then subsequently choose new hyperparameters to test based on this model.|
|**Random Search**[(Usage)](#Random)|In Random Search for Hyper-Parameter Optimization show that Random Search might be surprisingly simple and effective. We suggest that we could use Random Search as the baseline when we have no knowledge about the prior distribution of hyper-parameters.|
|**Anneal**[(Usage)](#Anneal)|This simple annealing algorithm begins by sampling from the prior, but tends over time to sample from points closer and closer to the best ones observed. This algorithm is a simple variation on the random search that leverages smoothness in the response surface. The annealing rate is not adaptive.|
|**Naive Evolution**[(Usage)](#Evolution)|Naive Evolution comes from Large-Scale Evolution of Image Classifiers. It randomly initializes a population-based on search space. For each generation, it chooses better ones and does some mutation (e.g., change a hyperparameter, add/remove one layer) on them to get the next generation. Naive Evolution requires many trials to works, but it's very simple and easy to expand new features.|
|**SMAC**[(Usage)](#SMAC)|SMAC is based on Sequential Model-Based Optimization (SMBO). It adapts the most prominent previously used model class (Gaussian stochastic process models) and introduces the model class of random forests to SMBO, in order to handle categorical parameters. The SMAC supported by nni is a wrapper on the SMAC3 Github repo.|
|**Batch tuner**[(Usage)](#Batch)|Batch tuner allows users to simply provide several configurations (i.e., choices of hyper-parameters) for their trial code. After finishing all the configurations, the experiment is done. Batch tuner only supports the type choice in search space spec.|
|**Grid Search**[(Usage)](#GridSearch)|Grid Search performs an exhaustive searching through a manually specified subset of the hyperparameter space defined in the searchspace file. Note that the only acceptable types of search space are choice, quniform, qloguniform. The number q in quniform and qloguniform has special meaning (different from the spec in search space spec). It means the number of values that will be sampled evenly from the range low and high.|
|[Hyperband](https://github.com/Microsoft/nni/tree/master/src/sdk/pynni/nni/hyperband_advisor)[(Usage)](#Hyperband)|Hyperband tries to use the limited resource to explore as many configurations as possible, and finds out the promising ones to get the final result. The basic idea is generating many configurations and to run them for the small number of STEPs to find out promising one, then further training those promising ones to select several more promising one.|
|[Network Morphism](https://github.com/Microsoft/nni/blob/master/src/sdk/pynni/nni/networkmorphism_tuner/README.md)[(Usage)](#NetworkMorphism)|Network Morphism provides functions to automatically search for architecture of deep learning models. Every child network inherits the knowledge from its parent network and morphs into diverse types of networks, including changes of depth, width, and skip-connection. Next, it estimates the value of a child network using the historic architecture and metric pairs. Then it selects the most promising one to train.|
|**Metis Tuner**[(Usage)](#MetisTuner)|Metis offers the following benefits when it comes to tuning parameters: While most tools only predict the optimal configuration, Metis gives you two outputs: (a) current prediction of optimal configuration, and (b) suggestion for the next trial. No more guesswork. While most tools assume training datasets do not have noisy data, Metis actually tells you if you need to re-sample a particular hyper-parameter.|
NNI also support building an assessor by yourself to adjust your tuning demand.
NNI supports to build an assessor by yourself for tuning demand.
If you want to implement a customized Assessor, there are three things for you to do:
If you want to implement a customized Assessor, there are three things to do:
1) Inherit an assessor of a base Assessor class
2) Implement assess_trial function
3) Configure your customized Assessor in experiment YAML config file
1. Inherit the base Assessor class
1. Implement assess_trial function
1. Configure your customized Assessor in experiment YAML config file
**1. Inherit an assessor of a base Assessor class**
**1. Inherit the base Assessor class**
```python
fromnni.assessorimportAssessor
...
...
@@ -19,6 +19,7 @@ class CustomizedAssessor(Assessor):
```
**2. Implement assess trial function**
```python
fromnni.assessorimportAssessor,AssessResult
...
...
@@ -41,7 +42,6 @@ class CustomizedAssessor(Assessor):
NNI needs to locate your customized Assessor class and instantiate the class, so you need to specify the location of the customized Assessor class and pass literal values as parameters to the \_\_init__ constructor.
```yaml
assessor:
codeDir:/home/abc/myassessor
classFileName:my_customized_assessor.py
...
...
@@ -50,7 +50,6 @@ assessor:
# can be specified in this optional classArgs field, for example
classArgs:
arg1:value1
```
Please noted in **2**. The object `trial_history` are exact the object that Trial send to Assessor by using SDK `report_intermediate_result` function.
It means your Tuner will always generate parameters `{"dropout": 0.3, "learning_rate": 0.4}`. Then Trial will receive `{"dropout": 0.3, "learning_rate": 0.4}` by calling API `nni.get_next_parameter()`. Once the trial ends with a result (normally some kind of metrics), it can send the result to Tuner by calling API `nni.report_final_result()`, for example `nni.report_final_result(0.93)`. Then your Tuner's `receive_trial_result` function will receied the result like：
```
```python
parameter_id=82347
parameters={"dropout":0.3,"learning_rate":0.4}
value=0.93
```
**Note that** if you want to access a file (e.g., `data.txt`) in the directory of your own tuner, you cannot use `open('data.txt', 'r')`. Instead, you should use the following:
```
```python
_pwd=os.path.dirname(__file__)
_fd=open(os.path.join(_pwd,'data.txt'),'r')
```
This is because your tuner is not executed in the directory of your tuner (i.e., `pwd` is not the directory of your own tuner).
...
...
@@ -96,7 +91,6 @@ This is because your tuner is not executed in the directory of your tuner (i.e.,
NNI needs to locate your customized tuner class and instantiate the class, so you need to specify the location of the customized tuner class and pass literal values as parameters to the \_\_init__ constructor.
@@ -4,184 +4,186 @@ A config file is needed when create an experiment, the path of the config file i
The config file is written in YAML format, and need to be written correctly.
This document describes the rule to write config file, and will provide some examples and templates.
-[Template](#Template)(the templates of an config file)
-[Configuration spec](#Configuration)(the configuration specification of every attribute in config file)
-[Examples](#Examples)(the examples of config file)
*[Template](#Template)(the templates of an config file)
*[Configuration spec](#Configuration)(the configuration specification of every attribute in config file)
*[Examples](#Examples)(the examples of config file)
<aname="Template"></a>
## Template
* __light weight(without Annotation and Assessor)__
```
authorName:
experimentName:
trialConcurrency:
maxExecDuration:
maxTrialNum:
* __light weight(without Annotation and Assessor)__
```yaml
authorName:
experimentName:
trialConcurrency:
maxExecDuration:
maxTrialNum:
#choice: local, remote, pai, kubeflow
trainingServicePlatform:
searchSpacePath:
trainingServicePlatform:
searchSpacePath:
#choice: true, false
useAnnotation:
useAnnotation:
tuner:
#choice: TPE, Random, Anneal, Evolution
builtinTunerName:
classArgs:
#choice: maximize, minimize
optimize_mode:
gpuNum:
gpuNum:
trial:
command:
codeDir:
gpuNum:
command:
codeDir:
gpuNum:
#machineList can be empty if the platform is local
machineList:
- ip:
port:
username:
passwd:
-ip:
port:
username:
passwd:
```
* __Use Assessor__
```
authorName:
experimentName:
trialConcurrency:
maxExecDuration:
maxTrialNum:
```yaml
authorName:
experimentName:
trialConcurrency:
maxExecDuration:
maxTrialNum:
#choice: local, remote, pai, kubeflow
trainingServicePlatform:
searchSpacePath:
trainingServicePlatform:
searchSpacePath:
#choice: true, false
useAnnotation:
useAnnotation:
tuner:
#choice: TPE, Random, Anneal, Evolution
builtinTunerName:
classArgs:
#choice: maximize, minimize
optimize_mode:
gpuNum:
gpuNum:
assessor:
#choice: Medianstop
builtinAssessorName:
classArgs:
#choice: maximize, minimize
optimize_mode:
gpuNum:
gpuNum:
trial:
command:
codeDir:
gpuNum:
command:
codeDir:
gpuNum:
#machineList can be empty if the platform is local
machineList:
- ip:
port:
username:
passwd:
-ip:
port:
username:
passwd:
```
* __Use Annotation__
```
authorName:
experimentName:
trialConcurrency:
maxExecDuration:
maxTrialNum:
```yaml
authorName:
experimentName:
trialConcurrency:
maxExecDuration:
maxTrialNum:
#choice: local, remote, pai, kubeflow
trainingServicePlatform:
trainingServicePlatform:
#choice: true, false
useAnnotation:
useAnnotation:
tuner:
#choice: TPE, Random, Anneal, Evolution
builtinTunerName:
classArgs:
#choice: maximize, minimize
optimize_mode:
gpuNum:
gpuNum:
assessor:
#choice: Medianstop
builtinAssessorName:
classArgs:
#choice: maximize, minimize
optimize_mode:
gpuNum:
gpuNum:
trial:
command:
codeDir:
gpuNum:
command:
codeDir:
gpuNum:
#machineList can be empty if the platform is local
machineList:
- ip:
port:
username:
passwd:
-ip:
port:
username:
passwd:
```
<aname="Configuration"></a>
## Configuration spec
* __authorName__
* Description
__authorName__ is the name of the author who create the experiment.
__authorName__ is the name of the author who create the experiment.
TBD: add default value
* __experimentName__
* Description
__experimentName__ is the name of the experiment created.
TBD: add default value
* __trialConcurrency__
* Description
__trialConcurrency__ specifies the max num of trial jobs run simultaneously.
Note: if trialGpuNum is bigger than the free gpu numbers, and the trial jobs running simultaneously can not reach trialConcurrency number, some trial jobs will be put into a queue to wait for gpu allocation.
__trialConcurrency__ specifies the max num of trial jobs run simultaneously.
Note: if trialGpuNum is bigger than the free gpu numbers, and the trial jobs running simultaneously can not reach trialConcurrency number, some trial jobs will be put into a queue to wait for gpu allocation.
* __maxExecDuration__
* Description
__maxExecDuration__ specifies the max duration time of an experiment.The unit of the time is {__s__, __m__, __h__, __d__}, which means {_seconds_, _minutes_, _hours_, _days_}.
Note: The maxExecDuration spec set the time of an experiment, not a trial job. If the experiment reach the max duration time, the experiment will not stop, but could not submit new trial jobs any more.
__maxExecDuration__ specifies the max duration time of an experiment.The unit of the time is {__s__, __m__, __h__, __d__}, which means {_seconds_, _minutes_, _hours_, _days_}.
Note: The maxExecDuration spec set the time of an experiment, not a trial job. If the experiment reach the max duration time, the experiment will not stop, but could not submit new trial jobs any more.
* __maxTrialNum__
*Description
__maxTrialNum__ specifies the max number of trial jobs created by NNI, including succeeded and failed jobs.
* Description
__maxTrialNum__ specifies the max number of trial jobs created by NNI, including succeeded and failed jobs.
* __trainingServicePlatform__
* Description
__trainingServicePlatform__ specifies the platform to run the experiment, including {__local__, __remote__, __pai__, __kubeflow__}.
__trainingServicePlatform__ specifies the platform to run the experiment, including {__local__, __remote__, __pai__, __kubeflow__}.
* __local__ run an experiment on local ubuntu machine.
* __remote__ submit trial jobs to remote ubuntu machines, and __machineList__ field should be filed in order to set up SSH connection to remote machine.
* __pai__ submit trial jobs to [OpenPai](https://github.com/Microsoft/pai) of Microsoft. For more details of pai configuration, please reference [PAIMOdeDoc](./PAIMode.md)
* __kubeflow__ submit trial jobs to [kubeflow](https://www.kubeflow.org/docs/about/kubeflow/), NNI support kubeflow based on normal kubernetes and [azure kubernetes](https://azure.microsoft.com/en-us/services/kubernetes-service/).
* __searchSpacePath__
* Description
__searchSpacePath__ specifies the path of search space file, which should be a valid path in the local linux machine.
Note: if set useAnnotation=True, the searchSpacePath field should be removed.
__searchSpacePath__ specifies the path of search space file, which should be a valid path in the local linux machine.
Note: if set useAnnotation=True, the searchSpacePath field should be removed.
* __useAnnotation__
* Description
__useAnnotation__ use annotation to analysis trial code and generate search space.
__useAnnotation__ use annotation to analysis trial code and generate search space.
Note: if set useAnnotation=True, the searchSpacePath field should be removed.
* __nniManagerIp__
* Description
__nniManagerIp__ set the IP address of the machine on which NNI manager process runs. This field is optional, and if it's not set, eth0 device IP will be used instead.
Note: run ifconfig on NNI manager's machine to check if eth0 device exists. If not, we recommend to set nnimanagerIp explicitly.
...
...
@@ -196,95 +198,109 @@ machineList:
__logLevel__ sets log level for the experiment, available log levels are: `trace, debug, info, warning, error, fatal`. The default value is `info`.
* __tuner__
* Description
__tuner__ specifies the tuner algorithm in the experiment, there are two kinds of ways to set tuner. One way is to use tuner provided by NNI sdk, need to set __builtinTunerName__ and __classArgs__. Another way is to use users' own tuner file, and need to set __codeDirectory__, __classFileName__, __className__ and __classArgs__.
* __builtinTunerName__ and __classArgs__
* __builtinTunerName__
__builtinTunerName__ specifies the name of system tuner, NNI sdk provides four kinds of tuner, including {__TPE__, __Random__, __Anneal__, __Evolution__, __BatchTuner__, __GridSearch__}
__builtinTunerName__ specifies the name of system tuner, NNI sdk provides four kinds of tuner, including {__TPE__, __Random__, __Anneal__, __Evolution__, __BatchTuner__, __GridSearch__}
* __classArgs__
__classArgs__ specifies the arguments of tuner algorithm. If the __builtinTunerName__ is in {__TPE__, __Random__, __Anneal__, __Evolution__}, user should set __optimize_mode__.
__classArgs__ specifies the arguments of tuner algorithm. If the __builtinTunerName__ is in {__TPE__, __Random__, __Anneal__, __Evolution__}, user should set __optimize_mode__.
* __codeDir__, __classFileName__, __className__ and __classArgs__
* __codeDir__
__codeDir__ specifies the directory of tuner code.
* __classFileName__
__classFileName__ specifies the name of tuner file.
* __className__
__className__ specifies the name of tuner class.
* __classArgs__
__classArgs__ specifies the arguments of tuner algorithm.
* __gpuNum__
* __codeDir__
__codeDir__ specifies the directory of tuner code.
* __classFileName__
__classFileName__ specifies the name of tuner file.
* __className__
__className__ specifies the name of tuner class.
* __classArgs__
__classArgs__ specifies the arguments of tuner algorithm.
* __gpuNum__
__gpuNum__ specifies the gpu number to run the tuner process. The value of this field should be a positive number.
Note: users could only specify one way to set tuner, for example, set {tunerName, optimizationMode} or {tunerCommand, tunerCwd}, and could not set them both.
Note: users could only specify one way to set tuner, for example, set {tunerName, optimizationMode} or {tunerCommand, tunerCwd}, and could not set them both.
* __assessor__
* Description
__assessor__ specifies the assessor algorithm to run an experiment, there are two kinds of ways to set assessor. One way is to use assessor provided by NNI sdk, users need to set __builtinAssessorName__ and __classArgs__. Another way is to use users' own assessor file, and need to set __codeDirectory__, __classFileName__, __className__ and __classArgs__.
* __builtinAssessorName__ and __classArgs__
* __builtinAssessorName__
__builtinAssessorName__ specifies the name of system assessor, NNI sdk provides one kind of assessor {__Medianstop__}
__builtinAssessorName__ specifies the name of system assessor, NNI sdk provides one kind of assessor {__Medianstop__}
* __classArgs__
__classArgs__ specifies the arguments of assessor algorithm
__classArgs__ specifies the arguments of assessor algorithm
* __codeDir__, __classFileName__, __className__ and __classArgs__
* __codeDir__
__codeDir__ specifies the directory of assessor code.
__codeDir__ specifies the directory of assessor code.
* __classFileName__
__classFileName__ specifies the name of assessor file.
__classFileName__ specifies the name of assessor file.
* __className__
__className__ specifies the name of assessor class.
__className__ specifies the name of assessor class.
* __classArgs__
__classArgs__ specifies the arguments of assessor algorithm.
__classArgs__ specifies the arguments of assessor algorithm.
* __gpuNum__
__gpuNum__ specifies the gpu number to run the assessor process. The value of this field should be a positive number.
Note: users' could only specify one way to set assessor, for example,set {assessorName, optimizationMode} or {assessorCommand, assessorCwd}, and users could not set them both.If users do not want to use assessor, assessor fileld should leave to empty.
__gpuNum__ specifies the gpu number to run the assessor process. The value of this field should be a positive number.
Note: users' could only specify one way to set assessor, for example,set {assessorName, optimizationMode} or {assessorCommand, assessorCwd}, and users could not set them both.If users do not want to use assessor, assessor fileld should leave to empty.
* __trial(local, remote)__
* __command__
__command__ specifies the command to run trial process.
__command__ specifies the command to run trial process.
* __codeDir__
__codeDir__ specifies the directory of your own trial file.
__codeDir__ specifies the directory of your own trial file.
* __gpuNum__
__gpuNum__ specifies the num of gpu to run the trial process. Default value is 0.
__gpuNum__ specifies the num of gpu to run the trial process. Default value is 0.
* __trial(pai)__
* __command__
__command__ specifies the command to run trial process.
__command__ specifies the command to run trial process.
* __codeDir__
__codeDir__ specifies the directory of the own trial file.
__codeDir__ specifies the directory of the own trial file.
* __gpuNum__
__gpuNum__ specifies the num of gpu to run the trial process. Default value is 0.
__gpuNum__ specifies the num of gpu to run the trial process. Default value is 0.
* __cpuNum__
__cpuNum__ is the cpu number of cpu to be used in pai container.
* __memoryMB__
__memoryMB__ set the momory size to be used in pai's container.
* __image__
__image__ set the image to be used in pai.
...
...
@@ -292,417 +308,416 @@ machineList:
* __dataDir__
__dataDir__ is the data directory in hdfs to be used.
* __outputDir__
__outputDir__ is the output directory in hdfs to be used in pai, the stdout and stderr files are stored in the directory after job finished.
__outputDir__ is the output directory in hdfs to be used in pai, the stdout and stderr files are stored in the directory after job finished.
* __trial(kubeflow)__
* __codeDir__
__codeDir__ is the local directory where the code files in.
* __ps(optional)__
__ps__ is the configuration for kubeflow's tensorflow-operator.
__ps__ is the configuration for kubeflow's tensorflow-operator.
* __replicas__
__replicas__ is the replica number of __ps__ role.
* __command__
__command__ is the run script in __ps__'s container.
* __gpuNum__
__gpuNum__ set the gpu number to be used in __ps__ container.
* __cpuNum__
__cpuNum__ set the cpu number to be used in __ps__ container.
* __memoryMB__
__memoryMB__ set the memory size of the container.
* __image__
__image__ set the image to be used in __ps__.
* __worker__
__worker__ is the configuration for kubeflow's tensorflow-operator.
__worker__ is the configuration for kubeflow's tensorflow-operator.
* __replicas__
__replicas__ is the replica number of __worker__ role.
* __command__
__command__ is the run script in __worker__'s container.
* __gpuNum__
__gpuNum__ set the gpu number to be used in __worker__ container.
* __cpuNum__
__cpuNum__ set the cpu number to be used in __worker__ container.
* __memoryMB__
__memoryMB__ set the memory size of the container.
* __image__
__image__ set the image to be used in __worker__.
* __machineList__
__machineList__ should be set if __trainingServicePlatform__ is set to remote, or it should be empty.
* __machineList__
__machineList__ should be set if users set __trainingServicePlatform__=remote, or it could be empty.
* __ip__
__ip__ is the ip address of remote machine.
__ip__ is the ip address of remote machine.
* __port__
__port__ is the ssh port to be used to connect machine.
Note: if users set port empty, the default value will be 22.
__port__ is the ssh port to be used to connect machine.
Note: if users set port empty, the default value will be 22.
* __username__
__username__ is the account of remote machine.
__username__ is the account of remote machine.
* __passwd__
__passwd__ specifies the password of the account.
__passwd__ specifies the password of the account.
* __sshKeyPath__
If users use ssh key to login remote machine, could set __sshKeyPath__ in config file. __sshKeyPath__ is the path of ssh key file, which should be valid.
Note: if users set passwd and sshKeyPath simultaneously, NNI will try passwd.
Note: if users set passwd and sshKeyPath simultaneously, NNI will try passwd.
* __passphrase__
__passphrase__ is used to protect ssh key, which could be empty if users don't have passphrase.
* __kubeflowConfig__:
* __operator__
__operator__ specify the kubeflow's operator to be used, NNI support __tf-operator__ in current version.
* __storage__
__storage__ specify the storage type of kubeflow, including {__nfs__, __azureStorage__}. This field is optional, and the default value is __nfs__. If the config use azureStorage, this field must be completed.
* __nfs__
__server__ is the host of nfs server
__path__ is the mounted path of nfs
* __keyVault__
If users want to use azure kubernetes service, they should set keyVault to storage the private key of your azure storage account. Refer: https://docs.microsoft.com/en-us/azure/key-vault/key-vault-manage-with-cli2
* __vaultName__
__vaultName__ is the value of ```--vault-name``` used in az command.
__vaultName__ is the value of `--vault-name` used in az command.
* __name__
__name__ is the value of ```--name``` used in az command.
__name__ is the value of `--name` used in az command.
* __azureStorage__
If users use azure kubernetes service, they should set azure storage account to store code files.
* __accountName__
__accountName__ is the name of azure storage account.
* __azureShare__
__azureShare__ is the share of the azure file storage.
* __paiConfig__
* __userName__
__userName__ is the user name of your pai account.
* __password__
__password__ is the password of the pai account.
* __host__
__host__ is the host of pai.
<aname="Examples"></a>
<aname="Examples"></a>
## Examples
* __local mode__
If users want to run trial jobs in local machine, and use annotation to generate search space, could use the following config:
```
authorName: test
experimentName: test_experiment
trialConcurrency: 3
maxExecDuration: 1h
maxTrialNum: 10
#choice: local, remote, pai, kubeflow
trainingServicePlatform: local
#choice: true, false
useAnnotation: true
tuner:
#choice: TPE, Random, Anneal, Evolution
builtinTunerName: TPE
classArgs:
#choice: maximize, minimize
optimize_mode: maximize
gpuNum: 0
trial:
command: python3 mnist.py
codeDir: /nni/mnist
gpuNum: 0
```
Could add assessor configuration in config file if set assessor.
```
authorName: test
experimentName: test_experiment
trialConcurrency: 3
maxExecDuration: 1h
maxTrialNum: 10
#choice: local, remote, pai, kubeflow
trainingServicePlatform: local
searchSpacePath: /nni/search_space.json
#choice: true, false
useAnnotation: false
tuner:
#choice: TPE, Random, Anneal, Evolution
builtinTunerName: TPE
classArgs:
#choice: maximize, minimize
optimize_mode: maximize
gpuNum: 0
assessor:
#choice: Medianstop
builtinAssessorName: Medianstop
classArgs:
#choice: maximize, minimize
optimize_mode: maximize
gpuNum: 0
trial:
command: python3 mnist.py
codeDir: /nni/mnist
gpuNum: 0
```
Or you could specify your own tuner and assessor file as following:
```
authorName: test
experimentName: test_experiment
trialConcurrency: 3
maxExecDuration: 1h
maxTrialNum: 10
#choice: local, remote, pai, kubeflow
trainingServicePlatform: local
searchSpacePath: /nni/search_space.json
#choice: true, false
useAnnotation: false
tuner:
codeDir: /nni/tuner
classFileName: mytuner.py
className: MyTuner
classArgs:
#choice: maximize, minimize
optimize_mode: maximize
gpuNum: 0
assessor:
codeDir: /nni/assessor
classFileName: myassessor.py
className: MyAssessor
classArgs:
#choice: maximize, minimize
optimize_mode: maximize
gpuNum: 0
trial:
command: python3 mnist.py
codeDir: /nni/mnist
gpuNum: 0
```
```yaml
authorName:test
experimentName:test_experiment
trialConcurrency:3
maxExecDuration:1h
maxTrialNum:10
#choice: local, remote, pai, kubeflow
trainingServicePlatform:local
#choice: true, false
useAnnotation:true
tuner:
#choice: TPE, Random, Anneal, Evolution
builtinTunerName:TPE
classArgs:
#choice: maximize, minimize
optimize_mode:maximize
gpuNum:0
trial:
command:python3 mnist.py
codeDir:/nni/mnist
gpuNum:0
```
You can add assessor configuration.
```yaml
authorName:test
experimentName:test_experiment
trialConcurrency:3
maxExecDuration:1h
maxTrialNum:10
#choice: local, remote, pai, kubeflow
trainingServicePlatform:local
searchSpacePath:/nni/search_space.json
#choice: true, false
useAnnotation:false
tuner:
#choice: TPE, Random, Anneal, Evolution
builtinTunerName:TPE
classArgs:
#choice: maximize, minimize
optimize_mode:maximize
gpuNum:0
assessor:
#choice: Medianstop
builtinAssessorName:Medianstop
classArgs:
#choice: maximize, minimize
optimize_mode:maximize
gpuNum:0
trial:
command:python3 mnist.py
codeDir:/nni/mnist
gpuNum:0
```
Or you could specify your own tuner and assessor file as following,
```yaml
authorName:test
experimentName:test_experiment
trialConcurrency:3
maxExecDuration:1h
maxTrialNum:10
#choice: local, remote, pai, kubeflow
trainingServicePlatform:local
searchSpacePath:/nni/search_space.json
#choice: true, false
useAnnotation:false
tuner:
codeDir:/nni/tuner
classFileName:mytuner.py
className:MyTuner
classArgs:
#choice: maximize, minimize
optimize_mode:maximize
gpuNum:0
assessor:
codeDir:/nni/assessor
classFileName:myassessor.py
className:MyAssessor
classArgs:
#choice: maximize, minimize
optimize_mode:maximize
gpuNum:0
trial:
command:python3 mnist.py
codeDir:/nni/mnist
gpuNum:0
```
* __remote mode__
If run trial jobs in remote machine, users could specify the remote mahcine information as fllowing format:
```
authorName: test
experimentName: test_experiment
trialConcurrency: 3
maxExecDuration: 1h
maxTrialNum: 10
#choice: local, remote, pai, kubeflow
trainingServicePlatform: remote
searchSpacePath: /nni/search_space.json
#choice: true, false
useAnnotation: false
tuner:
#choice: TPE, Random, Anneal, Evolution
builtinTunerName: TPE
classArgs:
#choice: maximize, minimize
optimize_mode: maximize
gpuNum: 0
trial:
command: python3 mnist.py
codeDir: /nni/mnist
gpuNum: 0
#machineList can be empty if the platform is local
machineList:
- ip: 10.10.10.10
port: 22
username: test
passwd: test
- ip: 10.10.10.11
port: 22
username: test
passwd: test
- ip: 10.10.10.12
port: 22
username: test
sshKeyPath: /nni/sshkey
passphrase: qwert
```
If run trial jobs in remote machine, users could specify the remote mahcine information as fllowing format:
```yaml
authorName:test
experimentName:test_experiment
trialConcurrency:3
maxExecDuration:1h
maxTrialNum:10
#choice: local, remote, pai, kubeflow
trainingServicePlatform:remote
searchSpacePath:/nni/search_space.json
#choice: true, false
useAnnotation:false
tuner:
#choice: TPE, Random, Anneal, Evolution
builtinTunerName:TPE
classArgs:
#choice: maximize, minimize
optimize_mode:maximize
gpuNum:0
trial:
command:python3 mnist.py
codeDir:/nni/mnist
gpuNum:0
#machineList can be empty if the platform is local
As we all know, the choice of model optimizer is directly affects the performance of the final matrix. The goal of this tutorial is to **tune a better performace optimizer** to train a relatively small convolutional neural network (CNN) for recognizing images.
As we all know, the choice of model optimizer is directly affects the performance of the final metrics. The goal of this tutorial is to **tune a better performace optimizer** to train a relatively small convolutional neural network (CNN) for recognizing images.
In this example, we have selected the following common deep learning optimizer:
NNI provides an easy way to adopt an approach to set up parameter tuning algorithms, we call them **Tuner**.
Tuner receives the result from `Trial` as a matrix to evaluate the performance of a specific parameters/architecture configures. And tuner sends next hyper-parameter or architecture configure to Trial.
Tuner receives metrics from `Trial` to evaluate the performance of a specific parameters/architecture configures. And tuner sends next hyper-parameter or architecture configure to Trial.
In NNI, we support two approaches to set the tuner: first is directly use builtin tuner provided by nni sdk, second is customize a tuner file by yourself. We also have Advisor that combines the functinality of Tuner & Assessor.
