Unverified Commit 4539b4d3 authored by Yan Ni's avatar Yan Ni Committed by GitHub
Browse files

fix warning for homepage build (#680)

* fix lex format for code

* fix doc links

* hide api reference

* delete orphan files

* fix deadlink in FAQ

* remove email address

* fix deadlinks for deleted orphan files
parent d76d379b
...@@ -49,30 +49,30 @@ The tool dispatches and runs trial jobs generated by tuning algorithms to search ...@@ -49,30 +49,30 @@ The tool dispatches and runs trial jobs generated by tuning algorithms to search
</ul> </ul>
</td> </td>
<td> <td>
<a href="docs/HowToChooseTuner.md">Tuner</a> <a href="docs/Builtin_Tuner.md">Tuner</a>
<ul> <ul>
<li><a href="docs/HowToChooseTuner.md#TPE">TPE</a></li> <li><a href="docs/Builtin_Tuner.md#TPE">TPE</a></li>
<li><a href="docs/HowToChooseTuner.md#Random">Random Search</a></li> <li><a href="docs/Builtin_Tuner.md#Random">Random Search</a></li>
<li><a href="docs/HowToChooseTuner.md#Anneal">Anneal</a></li> <li><a href="docs/Builtin_Tuner.md#Anneal">Anneal</a></li>
<li><a href="docs/HowToChooseTuner.md#Evolution">Naive Evolution</a></li> <li><a href="docs/Builtin_Tuner.md#Evolution">Naive Evolution</a></li>
<li><a href="docs/HowToChooseTuner.md#SMAC">SMAC</a></li> <li><a href="docs/Builtin_Tuner.md#SMAC">SMAC</a></li>
<li><a href="docs/HowToChooseTuner.md#Batch">Batch</a></li> <li><a href="docs/Builtin_Tuner.md#Batch">Batch</a></li>
<li><a href="docs/HowToChooseTuner.md#Grid">Grid Search</a></li> <li><a href="docs/Builtin_Tuner.md#Grid">Grid Search</a></li>
<li><a href="docs/HowToChooseTuner.md#Hyperband">Hyperband</a></li> <li><a href="docs/Builtin_Tuner.md#Hyperband">Hyperband</a></li>
<li><a href="docs/HowToChooseTuner.md#NetworkMorphism">Network Morphism</a></li> <li><a href="docs/Builtin_Tuner.md#NetworkMorphism">Network Morphism</a></li>
<li><a href="examples/tuners/enas_nni/README.md">ENAS</a></li> <li><a href="examples/tuners/enas_nni/README.md">ENAS</a></li>
<li><a href="docs/HowToChooseTuner.md#NetworkMorphism#MetisTuner">Metis Tuner</a></li> <li><a href="docs/Builtin_Tuner.md#NetworkMorphism#MetisTuner">Metis Tuner</a></li>
</ul> </ul>
<a href="docs/HowToChooseTuner.md#assessor">Assessor</a> <a href="docs/Builtin_Tuner.md#assessor">Assessor</a>
<ul> <ul>
<li><a href="docs/HowToChooseTuner.md#Medianstop">Median Stop</a></li> <li><a href="docs/Builtin_Tuner.md#Medianstop">Median Stop</a></li>
<li><a href="docs/HowToChooseTuner.md#Curvefitting">Curve Fitting</a></li> <li><a href="docs/Builtin_Tuner.md#Curvefitting">Curve Fitting</a></li>
</ul> </ul>
</td> </td>
<td> <td>
<ul> <ul>
<li><a href="docs/tutorial_1_CR_exp_local_api.md">Local Machine</a></li> <li><a href="docs/tutorial_1_CR_exp_local_api.md">Local Machine</a></li>
<li><a href="docs/tutorial_2_RemoteMachineMode.md">Remote Servers</a></li> <li><a href="docs/RemoteMachineMode.md">Remote Servers</a></li>
<li><a href="docs/PAIMode.md">OpenPAI</a></li> <li><a href="docs/PAIMode.md">OpenPAI</a></li>
<li><a href="docs/KubeflowMode.md">Kubeflow</a></li> <li><a href="docs/KubeflowMode.md">Kubeflow</a></li>
<li><a href="docs/KubeflowMode.md">FrameworkController on K8S (AKS etc.)</a></li> <li><a href="docs/KubeflowMode.md">FrameworkController on K8S (AKS etc.)</a></li>
...@@ -168,24 +168,25 @@ You can use these commands to get more information about the experiment ...@@ -168,24 +168,25 @@ You can use these commands to get more information about the experiment
## **Documentation** ## **Documentation**
* [NNI overview](docs/Overview.md) * [NNI overview](docs/Overview.md)
* [Quick start](docs/GetStarted.md) * [Quick start](docs/QuickStart.md)
## **How to** ## **How to**
* [Install NNI](docs/Installation.md) * [Install NNI](docs/Installation.md)
* [Use command line tool nnictl](docs/NNICTLDOC.md) * [Use command line tool nnictl](docs/NNICTLDOC.md)
* [Use NNIBoard](docs/WebUI.md) * [Use NNIBoard](docs/WebUI.md)
* [How to define search space](docs/SearchSpaceSpec.md) * [How to define search space](docs/SearchSpaceSpec.md)
* [How to define a trial](docs/howto_1_WriteTrial.md) * [How to define a trial](docs/Trials.md)
* [How to choose tuner/search-algorithm](docs/HowToChooseTuner.md) * [How to choose tuner/search-algorithm](docs/Builtin_Tuner.md)
* [Config an experiment](docs/ExperimentConfig.md) * [Config an experiment](docs/ExperimentConfig.md)
* [How to use annotation](docs/howto_1_WriteTrial.md#nni-python-annotation) * [How to use annotation](docs/Trials.md#nni-python-annotation)
## **Tutorials** ## **Tutorials**
* [Run an experiment on local (with multiple GPUs)?](docs/tutorial_1_CR_exp_local_api.md) * [Run an experiment on local (with multiple GPUs)?](docs/tutorial_1_CR_exp_local_api.md)
* [Run an experiment on multiple machines?](docs/tutorial_2_RemoteMachineMode.md) * [Run an experiment on multiple machines?](docs/RemoteMachineMode.md)
* [Run an experiment on OpenPAI?](docs/PAIMode.md) * [Run an experiment on OpenPAI?](docs/PAIMode.md)
* [Run an experiment on Kubeflow?](docs/KubeflowMode.md) * [Run an experiment on Kubeflow?](docs/KubeflowMode.md)
* [Try different tuners and assessors](docs/tutorial_3_tryTunersAndAssessors.md) * [Try different tuners](docs/tuners.rst)
* [Implement a customized tuner](docs/howto_2_CustomizedTuner.md) * [Try different assessors](docs/assessors.rst)
* [Implement a customized tuner](docs/Customize_Tuner.md)
* [Implement a customized assessor](examples/assessors/README.md) * [Implement a customized assessor](examples/assessors/README.md)
* [Use Genetic Algorithm to find good model architectures for Reading Comprehension task](examples/trials/ga_squad/README.md) * [Use Genetic Algorithm to find good model architectures for Reading Comprehension task](examples/trials/ga_squad/README.md)
......
...@@ -9,7 +9,7 @@ Currently we recommend sharing weights through NFS (Network File System), which ...@@ -9,7 +9,7 @@ Currently we recommend sharing weights through NFS (Network File System), which
### Weight Sharing through NFS file ### Weight Sharing through NFS file
With the NFS setup (see below), trial code can share model weight through loading & saving files. Here we recommend that user feed the tuner with the storage path: With the NFS setup (see below), trial code can share model weight through loading & saving files. Here we recommend that user feed the tuner with the storage path:
```yml ```yaml
tuner: tuner:
codeDir: path/to/customer_tuner codeDir: path/to/customer_tuner
classFileName: customer_tuner.py classFileName: customer_tuner.py
...@@ -82,7 +82,7 @@ The feature of weight sharing enables trials from different machines, in which m ...@@ -82,7 +82,7 @@ The feature of weight sharing enables trials from different machines, in which m
``` ```
## Examples ## Examples
For details, please refer to this [simple weight sharing example](../test/async_sharing_test). We also provided a [practice example](../examples/trials/weight_sharing/ga_squad) for reading comprehension, based on previous [ga_squad](../examples/trials/ga_squad) example. For details, please refer to this [simple weight sharing example](https://github.com/Microsoft/nni/tree/master/test/async_sharing_test). We also provided a [practice example](https://github.com/Microsoft/nni/tree/master/examples/trials/weight_sharing/ga_squad) for reading comprehension, based on previous [ga_squad](https://github.com/Microsoft/nni/tree/master/examples/trials/ga_squad) example.
[1]: https://arxiv.org/abs/1802.03268 [1]: https://arxiv.org/abs/1802.03268
[2]: https://arxiv.org/abs/1707.07012 [2]: https://arxiv.org/abs/1707.07012
......
...@@ -32,7 +32,7 @@ It is applicable in a wide range of performance curves, thus, can be used in var ...@@ -32,7 +32,7 @@ It is applicable in a wide range of performance curves, thus, can be used in var
**Usage example:** **Usage example:**
```yml ```yaml
# config.yml # config.yml
assessor: assessor:
builtinAssessorName: Medianstop builtinAssessorName: Medianstop
...@@ -62,7 +62,7 @@ It is applicable in a wide range of performance curves, thus, can be used in var ...@@ -62,7 +62,7 @@ It is applicable in a wide range of performance curves, thus, can be used in var
**Usage example:** **Usage example:**
```yml ```yaml
# config.yml # config.yml
assessor: assessor:
builtinAssessorName: Curvefitting builtinAssessorName: Curvefitting
......
...@@ -39,7 +39,7 @@ TPE, as a black-box optimization, can be used in various scenarios and shows goo ...@@ -39,7 +39,7 @@ TPE, as a black-box optimization, can be used in various scenarios and shows goo
**Usage example:** **Usage example:**
```yml ```yaml
# config.yml # config.yml
tuner: tuner:
builtinTunerName: TPE builtinTunerName: TPE
...@@ -65,7 +65,7 @@ Random search is suggested when each trial does not take too long (e.g., each tr ...@@ -65,7 +65,7 @@ Random search is suggested when each trial does not take too long (e.g., each tr
**Usage example** **Usage example**
```yml ```yaml
# config.yml # config.yml
tuner: tuner:
builtinTunerName: Random builtinTunerName: Random
...@@ -91,7 +91,7 @@ Anneal is suggested when each trial does not take too long, and you have enough ...@@ -91,7 +91,7 @@ Anneal is suggested when each trial does not take too long, and you have enough
**Usage example** **Usage example**
```yml ```yaml
# config.yml # config.yml
tuner: tuner:
builtinTunerName: Anneal builtinTunerName: Anneal
...@@ -117,7 +117,7 @@ Its requirement of computation resource is relatively high. Specifically, it req ...@@ -117,7 +117,7 @@ Its requirement of computation resource is relatively high. Specifically, it req
**Usage example** **Usage example**
```yml ```yaml
# config.yml # config.yml
tuner: tuner:
builtinTunerName: Evolution builtinTunerName: Evolution
...@@ -143,7 +143,7 @@ Similar to TPE, SMAC is also a black-box tuner which can be tried in various sce ...@@ -143,7 +143,7 @@ Similar to TPE, SMAC is also a black-box tuner which can be tried in various sce
**Usage example** **Usage example**
```yml ```yaml
# config.yml # config.yml
tuner: tuner:
builtinTunerName: SMAC builtinTunerName: SMAC
...@@ -165,7 +165,7 @@ If the configurations you want to try have been decided, you can list them in se ...@@ -165,7 +165,7 @@ If the configurations you want to try have been decided, you can list them in se
**Usage example** **Usage example**
```yml ```yaml
# config.yml # config.yml
tuner: tuner:
builtinTunerName: BatchTuner builtinTunerName: BatchTuner
...@@ -206,7 +206,7 @@ It is suggested when search space is small, it is feasible to exhaustively sweep ...@@ -206,7 +206,7 @@ It is suggested when search space is small, it is feasible to exhaustively sweep
**Usage example** **Usage example**
```yml ```yaml
# config.yml # config.yml
tuner: tuner:
builtinTunerName: GridSearch builtinTunerName: GridSearch
...@@ -232,7 +232,7 @@ It is suggested when you have limited computation resource but have relatively l ...@@ -232,7 +232,7 @@ It is suggested when you have limited computation resource but have relatively l
**Usage example** **Usage example**
```yml ```yaml
# config.yml # config.yml
advisor: advisor:
builtinAdvisorName: Hyperband builtinAdvisorName: Hyperband
...@@ -256,7 +256,7 @@ NetworkMorphism requires [pyTorch](https://pytorch.org/get-started/locally), so ...@@ -256,7 +256,7 @@ NetworkMorphism requires [pyTorch](https://pytorch.org/get-started/locally), so
**Suggested scenario** **Suggested scenario**
It is suggested that you want to apply deep learning methods to your task (your own dataset) but you have no idea of how to choose or design a network. You modify the [example](../examples/trials/network_morphism/cifar10/cifar10_keras.py) to fit your own dataset and your own data augmentation method. Also you can change the batch size, learning rate or optimizer. It is feasible for different tasks to find a good network architecture. Now this tuner only supports the computer vision domain. It is suggested that you want to apply deep learning methods to your task (your own dataset) but you have no idea of how to choose or design a network. You modify the [example](https://github.com/Microsoft/nni/tree/master/examples/trials/network_morphism/cifar10/cifar10_keras.py) to fit your own dataset and your own data augmentation method. Also you can change the batch size, learning rate or optimizer. It is feasible for different tasks to find a good network architecture. Now this tuner only supports the computer vision domain.
**Requirement of classArg** **Requirement of classArg**
...@@ -268,7 +268,7 @@ It is suggested that you want to apply deep learning methods to your task (your ...@@ -268,7 +268,7 @@ It is suggested that you want to apply deep learning methods to your task (your
**Usage example** **Usage example**
```yml ```yaml
# config.yml # config.yml
tuner: tuner:
builtinTunerName: NetworkMorphism builtinTunerName: NetworkMorphism
...@@ -296,7 +296,7 @@ Metis Tuner requires [sklearn](https://scikit-learn.org/), so users should insta ...@@ -296,7 +296,7 @@ Metis Tuner requires [sklearn](https://scikit-learn.org/), so users should insta
**Suggested scenario** **Suggested scenario**
Similar to TPE and SMAC, Metis is a black-box tuner. If your system takes a long time to finish each trial, Metis is more favorable than other approaches such as random search. Furthermore, Metis provides guidance on the subsequent trial. Here is an [example](../examples/trials/auto-gbdt/search_space_metis.json) about the use of Metis. User only need to send the final result like `accuracy` to tuner, by calling the nni SDK. Similar to TPE and SMAC, Metis is a black-box tuner. If your system takes a long time to finish each trial, Metis is more favorable than other approaches such as random search. Furthermore, Metis provides guidance on the subsequent trial. Here is an [example](https://github.com/Microsoft/nni/tree/master/examples/trials/auto-gbdt/search_space_metis.json) about the use of Metis. User only need to send the final result like `accuracy` to tuner, by calling the nni SDK.
**Requirement of classArg** **Requirement of classArg**
...@@ -304,7 +304,7 @@ Similar to TPE and SMAC, Metis is a black-box tuner. If your system takes a long ...@@ -304,7 +304,7 @@ Similar to TPE and SMAC, Metis is a black-box tuner. If your system takes a long
**Usage example** **Usage example**
```yml ```yaml
# config.yml # config.yml
tuner: tuner:
builtinTunerName: MetisTuner builtinTunerName: MetisTuner
......
...@@ -6,7 +6,7 @@ Firstly, if you are unsure or afraid of anything, just ask or submit the issue o ...@@ -6,7 +6,7 @@ Firstly, if you are unsure or afraid of anything, just ask or submit the issue o
However, for those individuals who want a bit more guidance on the best way to contribute to the project, read on. This document will cover all the points we're looking for in your contributions, raising your chances of quickly merging or addressing your contributions. However, for those individuals who want a bit more guidance on the best way to contribute to the project, read on. This document will cover all the points we're looking for in your contributions, raising your chances of quickly merging or addressing your contributions.
Looking for a quickstart, get acquainted with our [Get Started](./GetStarted.md) guide. Looking for a quickstart, get acquainted with our [Get Started](./QuickStart.md) guide.
There are a few simple guidelines that you need to follow before providing your hacks. There are a few simple guidelines that you need to follow before providing your hacks.
...@@ -30,7 +30,7 @@ Provide PRs with appropriate tags for bug fixes or enhancements to the source co ...@@ -30,7 +30,7 @@ Provide PRs with appropriate tags for bug fixes or enhancements to the source co
If you are looking for How to develop and debug the NNI source code, you can refer to [How to set up NNI developer environment doc](./SetupNNIDeveloperEnvironment.md) file in the `docs` folder. If you are looking for How to develop and debug the NNI source code, you can refer to [How to set up NNI developer environment doc](./SetupNNIDeveloperEnvironment.md) file in the `docs` folder.
Similarly for [writing trials](./WriteYourTrial.md) or [starting experiments](StartExperiment.md). For everything else, refer [here](https://github.com/Microsoft/nni/tree/master/docs). Similarly for [Quick Start](QuickStart.md). For everything else, refer to [NNI Home page](http://nni.readthedocs.io).
## Solve Existing Issues ## Solve Existing Issues
Head over to [issues](https://github.com/Microsoft/nni/issues) to find issues where help is needed from contributors. You can find issues tagged with 'good-first-issue' or 'help-wanted' to contribute in. Head over to [issues](https://github.com/Microsoft/nni/issues) to find issues where help is needed from contributors. You can find issues tagged with 'good-first-issue' or 'help-wanted' to contribute in.
......
...@@ -22,13 +22,13 @@ class CustomizedAdvisor(MsgDispatcherBase): ...@@ -22,13 +22,13 @@ class CustomizedAdvisor(MsgDispatcherBase):
**2) Implement the methods with prefix `handle_` except `handle_request`** **2) Implement the methods with prefix `handle_` except `handle_request`**
Please refer to the implementation of Hyperband ([src/sdk/pynni/nni/hyperband_advisor/hyperband_advisor.py](../src/sdk/pynni/nni/hyperband_advisor/hyperband_advisor.py)) for how to implement the methods. Please refer to the implementation of Hyperband ([src/sdk/pynni/nni/hyperband_advisor/hyperband_advisor.py](https://github.com/Microsoft/nni/tree/master/src/sdk/pynni/nni/hyperband_advisor/hyperband_advisor.py)) for how to implement the methods.
**3) Configure your customized Advisor in experiment YAML config file** **3) Configure your customized Advisor in experiment YAML config file**
Similar to tuner and assessor. NNI needs to locate your customized Advisor class and instantiate the class, so you need to specify the location of the customized Advisor class and pass literal values as parameters to the \_\_init__ constructor. Similar to tuner and assessor. NNI needs to locate your customized Advisor class and instantiate the class, so you need to specify the location of the customized Advisor class and pass literal values as parameters to the \_\_init__ constructor.
```yml ```yaml
advisor: advisor:
codeDir: /home/abc/myadvisor codeDir: /home/abc/myadvisor
classFileName: my_customized_advisor.py classFileName: my_customized_advisor.py
......
...@@ -40,7 +40,7 @@ class CustomizedAssessor(Assessor): ...@@ -40,7 +40,7 @@ class CustomizedAssessor(Assessor):
NNI needs to locate your customized Assessor class and instantiate the class, so you need to specify the location of the customized Assessor class and pass literal values as parameters to the \_\_init__ constructor. NNI needs to locate your customized Assessor class and instantiate the class, so you need to specify the location of the customized Assessor class and pass literal values as parameters to the \_\_init__ constructor.
```yml ```yaml
assessor: assessor:
codeDir: /home/abc/myassessor codeDir: /home/abc/myassessor
...@@ -56,5 +56,5 @@ assessor: ...@@ -56,5 +56,5 @@ assessor:
Please noted in **2**. The object `trial_history` are exact the object that Trial send to Assessor by using SDK `report_intermediate_result` function. Please noted in **2**. The object `trial_history` are exact the object that Trial send to Assessor by using SDK `report_intermediate_result` function.
More detail example you could see: More detail example you could see:
> * [medianstop-assessor](../src/sdk/pynni/nni/medianstop_assessor) > * [medianstop-assessor](https://github.com/Microsoft/nni/tree/master/src/sdk/pynni/nni/medianstop_assessor)
> * [curvefitting-assessor](../src/sdk/pynni/nni/curvefitting_assessor) > * [curvefitting-assessor](https://github.com/Microsoft/nni/tree/master/src/sdk/pynni/nni/curvefitting_assessor)
\ No newline at end of file \ No newline at end of file
...@@ -95,7 +95,7 @@ This is because your tuner is not executed in the directory of your tuner (i.e., ...@@ -95,7 +95,7 @@ This is because your tuner is not executed in the directory of your tuner (i.e.,
NNI needs to locate your customized tuner class and instantiate the class, so you need to specify the location of the customized tuner class and pass literal values as parameters to the \_\_init__ constructor. NNI needs to locate your customized tuner class and instantiate the class, so you need to specify the location of the customized tuner class and pass literal values as parameters to the \_\_init__ constructor.
```yml ```yaml
tuner: tuner:
codeDir: /home/abc/mytuner codeDir: /home/abc/mytuner
...@@ -109,10 +109,10 @@ tuner: ...@@ -109,10 +109,10 @@ tuner:
``` ```
More detail example you could see: More detail example you could see:
> * [evolution-tuner](../src/sdk/pynni/nni/evolution_tuner) > * [evolution-tuner](https://github.com/Microsoft/nni/tree/master/src/sdk/pynni/nni/evolution_tuner)
> * [hyperopt-tuner](../src/sdk/pynni/nni/hyperopt_tuner) > * [hyperopt-tuner](https://github.com/Microsoft/nni/tree/master/src/sdk/pynni/nni/hyperopt_tuner)
> * [evolution-based-customized-tuner](../examples/tuners/ga_customer_tuner) > * [evolution-based-customized-tuner](https://github.com/Microsoft/nni/tree/master/examples/tuners/ga_customer_tuner)
### Write a more advanced automl algorithm ### Write a more advanced automl algorithm
The methods above are usually enough to write a general tuner. However, users may also want more methods, for example, intermediate results, trials' state (e.g., the methods in assessor), in order to have a more powerful automl algorithm. Therefore, we have another concept called `advisor` which directly inherits from `MsgDispatcherBase` in [`src/sdk/pynni/nni/msg_dispatcher_base.py`](../src/sdk/pynni/nni/msg_dispatcher_base.py). Please refer to [here](./howto_3_CustomizedAdvisor.md) for how to write a customized advisor. The methods above are usually enough to write a general tuner. However, users may also want more methods, for example, intermediate results, trials' state (e.g., the methods in assessor), in order to have a more powerful automl algorithm. Therefore, we have another concept called `advisor` which directly inherits from `MsgDispatcherBase` in [`src/sdk/pynni/nni/msg_dispatcher_base.py`](https://github.com/Microsoft/nni/tree/master/src/sdk/pynni/nni/msg_dispatcher_base.py). Please refer to [here](Customize_Advisor.md) for how to write a customized advisor.
\ No newline at end of file \ No newline at end of file
**Enable Assessor in your expeirment**
===
Assessor module is for assessing running trials. One common use case is early stopping, which terminates unpromising trial jobs based on their intermediate results.
## Using NNI built-in Assessor
Here we use the same example `examples/trials/mnist-annotation`. We use `Medianstop` assessor for this experiment. The yml configure file is shown below:
```
authorName: your_name
experimentName: auto_mnist
# how many trials could be concurrently running
trialConcurrency: 2
# maximum experiment running duration
maxExecDuration: 3h
# empty means never stop
maxTrialNum: 100
# choice: local, remote
trainingServicePlatform: local
# choice: true, false
useAnnotation: true
tuner:
builtinTunerName: TPE
classArgs:
optimize_mode: maximize
assessor:
builtinAssessorName: Medianstop
classArgs:
optimize_mode: maximize
trial:
command: python mnist.py
codeDir: /usr/share/nni/examples/trials/mnist-annotation
gpuNum: 0
```
For our built-in assessors, you need to fill two fields: `builtinAssessorName` which chooses NNI provided assessors (refer to [here]() for built-in assessors), `optimize_mode` which includes maximize and minimize (you want to maximize or minimize your trial result).
## Using user customized Assessor
You can also write your own assessor following the guidance [here](). For example, you wrote an assessor for `examples/trials/mnist-annotation`. You should prepare the yml configure below:
```
authorName: your_name
experimentName: auto_mnist
# how many trials could be concurrently running
trialConcurrency: 2
# maximum experiment running duration
maxExecDuration: 3h
# empty means never stop
maxTrialNum: 100
# choice: local, remote
trainingServicePlatform: local
# choice: true, false
useAnnotation: true
tuner:
# Possible values: TPE, Random, Anneal, Evolution
builtinTunerName: TPE
classArgs:
optimize_mode: maximize
assessor:
# Your assessor code directory
codeDir:
# Name of the file which contains your assessor class
classFileName:
# Your assessor class name, must be a subclass of nni.Assessor
className:
# Parameter names and literal values you want to pass to
# the __init__ constructor of your assessor class
classArgs:
arg1: value1
gpuNum: 0
trial:
command: python mnist.py
codeDir: /usr/share/nni/examples/trials/mnist-annotation
gpuNum: 0
```
You need to fill: `codeDir`, `classFileName`, `className`, and pass parameters to \_\_init__ constructor through `classArgs` field if the \_\_init__ constructor of your assessor class has required parameters.
**Note that** if you want to access a file (e.g., ```data.txt```) in the directory of your own assessor, you cannot use ```open('data.txt', 'r')```. Instead, you should use the following:
```
_pwd = os.path.dirname(__file__)
_fd = open(os.path.join(_pwd, 'data.txt'), 'r')
```
This is because your assessor is not executed in the directory of your assessor (i.e., ```pwd``` is not the directory of your own assessor).
\ No newline at end of file
...@@ -19,7 +19,7 @@ Please try the following solutions in turn: ...@@ -19,7 +19,7 @@ Please try the following solutions in turn:
* Install NNI with `--no-cache-dir` flag like `python3 -m pip install nni --no-cache-dir` * Install NNI with `--no-cache-dir` flag like `python3 -m pip install nni --no-cache-dir`
### Job management error: getIPV4Address() failed because os.networkInterfaces().eth0 is undefined. ### Job management error: getIPV4Address() failed because os.networkInterfaces().eth0 is undefined.
Your machine don't have eth0 device, please set nniManagerIp in your config file manually. [refer](https://github.com/Microsoft/nni/blob/master/docs/ExperimentConfig.md) Your machine don't have eth0 device, please set [nniManagerIp](ExperimentConfig.md) in your config file manually.
### Exceed the MaxDuration but didn't stop ### Exceed the MaxDuration but didn't stop
When the duration of experiment reaches the maximum duration, nniManager will not create new trials, but the existing trials will continue unless user manually stop the experiment. When the duration of experiment reaches the maximum duration, nniManager will not create new trials, but the existing trials will continue unless user manually stop the experiment.
......
...@@ -12,7 +12,7 @@ NNI supports running experiment using [FrameworkController](https://github.com/M ...@@ -12,7 +12,7 @@ NNI supports running experiment using [FrameworkController](https://github.com/M
apt-get install nfs-common apt-get install nfs-common
``` ```
6. Install **NNI**, follow the install guide [here](GetStarted.md). 6. Install **NNI**, follow the install guide [here](QuickStart.md).
## Prerequisite for Azure Kubernetes Service ## Prerequisite for Azure Kubernetes Service
1. NNI support kubeflow based on Azure Kubernetes Service, follow the [guideline](https://azure.microsoft.com/en-us/services/kubernetes-service/) to set up Azure Kubernetes Service. 1. NNI support kubeflow based on Azure Kubernetes Service, follow the [guideline](https://azure.microsoft.com/en-us/services/kubernetes-service/) to set up Azure Kubernetes Service.
......
# Get Started with NNI
## **Installation**
* __Dependencies__
```bash
python >= 3.5
git
wget
```
python pip should also be correctly installed. You could use "python3 -m pip -v" to check in Linux.
* Note: we don't support virtual environment in current releases.
* __Install NNI through pip__
```bash
python3 -m pip install --user --upgrade nni
```
* __Install NNI through source code__
```bash
git clone -b v0.5 https://github.com/Microsoft/nni.git
cd nni
source install.sh
```
## **Quick start: run a customized experiment**
An experiment is to run multiple trial jobs, each trial job tries a configuration which includes a specific neural architecture (or model) and hyper-parameter values. To run an experiment through NNI, you should:
* Provide a runnable trial
* Provide or choose a tuner
* Provide a YAML experiment configure file
* (optional) Provide or choose an assessor
**Prepare trial**: Let's use a simple trial example, e.g. mnist, provided by NNI. After you installed NNI, NNI examples have been put in ~/nni/examples, run `ls ~/nni/examples/trials` to see all the trial examples. You can simply execute the following command to run the NNI mnist example:
```bash
python3 ~/nni/examples/trials/mnist-annotation/mnist.py
```
This command will be filled in the YAML configure file below. Please refer to [here](howto_1_WriteTrial.md) for how to write your own trial.
**Prepare tuner**: NNI supports several popular automl algorithms, including Random Search, Tree of Parzen Estimators (TPE), Evolution algorithm etc. Users can write their own tuner (refer to [here](howto_2_CustomizedTuner.md), but for simplicity, here we choose a tuner provided by NNI as below:
```yml
tuner:
builtinTunerName: TPE
classArgs:
optimize_mode: maximize
```
*builtinTunerName* is used to specify a tuner in NNI, *classArgs* are the arguments pass to the tuner, *optimization_mode* is to indicate whether you want to maximize or minimize your trial's result.
**Prepare configure file**: Since you have already known which trial code you are going to run and which tuner you are going to use, it is time to prepare the YAML configure file. NNI provides a demo configure file for each trial example, `cat ~/nni/examples/trials/mnist-annotation/config.yml` to see it. Its content is basically shown below:
```yml
authorName: your_name
experimentName: auto_mnist
# how many trials could be concurrently running
trialConcurrency: 2
# maximum experiment running duration
maxExecDuration: 3h
# empty means never stop
maxTrialNum: 100
# choice: local, remote, pai
trainingServicePlatform: local
# choice: true, false
useAnnotation: true
tuner:
builtinTunerName: TPE
classArgs:
optimize_mode: maximize
trial:
command: python mnist.py
codeDir: ~/nni/examples/trials/mnist-annotation
gpuNum: 0
```
Here *useAnnotation* is true because this trial example uses our python annotation (refer to [here](../tools/annotation/README.md) for details). For trial, we should provide *trialCommand* which is the command to run the trial, provide *trialCodeDir* where the trial code is. The command will be executed in this directory. We should also provide how many GPUs a trial requires.
With all these steps done, we can run the experiment with the following command:
nnictl create --config ~/nni/examples/trials/mnist-annotation/config.yml
You can refer to [here](NNICTLDOC.md) for more usage guide of *nnictl* command line tool.
## View experiment results
The experiment has been running now, NNI provides WebUI for you to view experiment progress, to control your experiment, and some other appealing features. The WebUI is opened by default by `nnictl create`.
## Read more
* [Tuners supported in the latest NNI release](./HowToChooseTuner.md)
* [Overview](Overview.md)
* [Installation](Installation.md)
* [Use command line tool nnictl](NNICTLDOC.md)
* [Use NNIBoard](WebUI.md)
* [Define search space](SearchSpaceSpec.md)
* [Config an experiment](ExperimentConfig.md)
* [How to run an experiment on local (with multiple GPUs)?](tutorial_1_CR_exp_local_api.md)
* [How to run an experiment on multiple machines?](tutorial_2_RemoteMachineMode.md)
* [How to run an experiment on OpenPAI?](PAIMode.md)
* [How to create a multi-phase experiment](multiPhase.md)
Grid Search on nni
===
## 1. Introduction
Grid Search performs an exhaustive searching through a manually specified subset of the hyperparameter space defined in the searchspace file.
Note that the only acceptable types of search space are `choice`, `quniform`, `qloguniform` since only these three types of search space can be exhausted.
Moreover, in GridSearch Tuner, for users' convenience, the definition of `quniform` and `qloguniform` change, where q here specifies the number of values that will be sampled. Details about them are listed as follows:
* Type 'quniform' will receive three values [low, high, q], where [low, high] specifies a range and 'q' specifies the number of values that will be sampled evenly. Note that q should be at least 2. It will be sampled in a way that the first sampled value is 'low', and each of the following values is (high-low)/q larger that the value in front of it.
* Type 'qloguniform' behaves like 'quniform' except that it will first change the range to [log(low), log(high)] and sample and then change the sampled value back.
## 2. Usage
Since Grid Search Tuner will exhaust all possible hyper-parameter combination according to the search space file without any hyper-parameter for tuner itself, all you need to do is to specify tuner name in your experiment's YAML config file:
```
tuner:
builtinTunerName: GridSearch
```
# How to use Tuner that NNI supports?
For now, NNI has supported the following tuner algorithms. Note that NNI installation only installs a subset of those algorithms, other algorithms should be installed through `nnictl package install` before you use them. For example, for SMAC the installation command is `nnictl package install --name=SMAC`.
- [TPE](#TPE)
- [Random Search](#Random)
- [Anneal](#Anneal)
- [Naive Evolution](#Evolution)
- [SMAC](#SMAC) (to install through `nnictl`)
- [Batch Tuner](#Batch)
- [Grid Search](#Grid)
- [Hyperband](#Hyperband)
- [Network Morphism](#NetworkMorphism) (require pyTorch)
- [Metis Tuner](#MetisTuner) (require sklearn)
## Supported tuner algorithms
We will introduce some basic knowledge about the tuner algorithms, suggested scenarios for each tuner, and their example usage (for complete usage spec, please refer to [here]()).
<a name="TPE"></a>
**TPE**
The Tree-structured Parzen Estimator (TPE) is a sequential model-based optimization (SMBO) approach. SMBO methods sequentially construct models to approximate the performance of hyperparameters based on historical measurements, and then subsequently choose new hyperparameters to test based on this model.
The TPE approach models P(x|y) and P(y) where x represents hyperparameters and y the associated evaluate matric. P(x|y) is modeled by transforming the generative process of hyperparameters, replacing the distributions of the configuration prior with non-parametric densities.
This optimization approach is described in detail in [Algorithms for Hyper-Parameter Optimization][1].
_Suggested scenario_: TPE, as a black-box optimization, can be used in various scenarios, and shows good performance in general. Especially when you have limited computation resource and can only try a small number of trials. From a large amount of experiments, we could found that TPE is far better than Random Search.
_Usage_:
```yml
# config.yml
tuner:
builtinTunerName: TPE
classArgs:
# choice: maximize, minimize
optimize_mode: maximize
```
<a name="Random"></a>
**Random Search**
In [Random Search for Hyper-Parameter Optimization][2] show that Random Search might be surprisingly simple and effective. We suggests that we could use Random Search as baseline when we have no knowledge about the prior distribution of hyper-parameters.
_Suggested scenario_: Random search is suggested when each trial does not take too long (e.g., each trial can be completed very soon, or early stopped by assessor quickly), and you have enough computation resource. Or you want to uniformly explore the search space. Random Search could be considered as baseline of search algorithm.
_Usage_:
```yml
# config.yml
tuner:
builtinTunerName: Random
```
<a name="Anneal"></a>
**Anneal**
This simple annealing algorithm begins by sampling from the prior, but tends over time to sample from points closer and closer to the best ones observed. This algorithm is a simple variation on random search that leverages smoothness in the response surface. The annealing rate is not adaptive.
_Suggested scenario_: Anneal is suggested when each trial does not take too long, and you have enough computation resource(almost same with Random Search). Or the variables in search space could be sample from some prior distribution.
_Usage_:
```yml
# config.yml
tuner:
builtinTunerName: Anneal
classArgs:
# choice: maximize, minimize
optimize_mode: maximize
```
<a name="Evolution"></a>
**Naive Evolution**
Naive Evolution comes from [Large-Scale Evolution of Image Classifiers][3]. It randomly initializes a population based on search space. For each generation, it chooses better ones and do some mutation (e.g., change a hyperparameter, add/remove one layer) on them to get the next generation. Naive Evolution requires many trials to works, but it's very simple and easily to expand new features.
_Suggested scenario_: Its requirement of computation resource is relatively high. Specifically, it requires large inital population to avoid falling into local optimum. If your trial is short or leverages assessor, this tuner is a good choice. And, it is more suggested when your trial code supports weight transfer, that is, the trial could inherit the converged weights from its parent(s). This can greatly speed up the training progress.
_Usage_:
```yml
# config.yml
tuner:
builtinTunerName: Evolution
classArgs:
# choice: maximize, minimize
optimize_mode: maximize
```
<a name="SMAC"></a>
**SMAC**
[SMAC][4] is based on Sequential Model-Based Optimization (SMBO). It adapts the most prominent previously used model class (Gaussian stochastic process models) and introduces the model class of random forests to SMBO, in order to handle categorical parameters. The SMAC supported by NNI is a wrapper on [the SMAC3 github repo][5].
Note that SMAC on NNI only supports a subset of the types in [search space spec](./SearchSpaceSpec.md), including `choice`, `randint`, `uniform`, `loguniform`, `quniform(q=1)`.
_Installation_:
* Install swig first. (`sudo apt-get install swig` for Ubuntu users)
* Run `nnictl package install --name=SMAC`
_Suggested scenario_: Similar to TPE, SMAC is also a black-box tuner which can be tried in various scenarios, and is suggested when computation resource is limited. It is optimized for discrete hyperparameters, thus, suggested when most of your hyperparameters are discrete.
_Usage_:
```yml
# config.yml
tuner:
builtinTunerName: SMAC
classArgs:
# choice: maximize, minimize
optimize_mode: maximize
```
<a name="Batch"></a>
**Batch tuner**
Batch tuner allows users to simply provide several configurations (i.e., choices of hyper-parameters) for their trial code. After finishing all the configurations, the experiment is done. Batch tuner only supports the type `choice` in [search space spec](./SearchSpaceSpec.md).
_Suggested sceanrio_: If the configurations you want to try have been decided, you can list them in searchspace file (using `choice`) and run them using batch tuner.
_Usage_:
```yml
# config.yml
tuner:
builtinTunerName: BatchTuner
```
Note that the search space that BatchTuner supported like:
```json
{
"combine_params":
{
"_type" : "choice",
"_value" : [{"optimizer": "Adam", "learning_rate": 0.00001},
{"optimizer": "Adam", "learning_rate": 0.0001},
{"optimizer": "Adam", "learning_rate": 0.001},
{"optimizer": "SGD", "learning_rate": 0.01},
{"optimizer": "SGD", "learning_rate": 0.005},
{"optimizer": "SGD", "learning_rate": 0.0002}]
}
}
```
The search space file including the high-level key `combine_params`. The type of params in search space must be `choice` and the `values` including all the combined-params value.
<a name="Grid"></a>
**Grid Search**
Grid Search performs an exhaustive searching through a manually specified subset of the hyperparameter space defined in the searchspace file.
Note that the only acceptable types of search space are `choice`, `quniform`, `qloguniform`. **The number `q` in `quniform` and `qloguniform` has special meaning (different from the spec in [search space spec](./SearchSpaceSpec.md)). It means the number of values that will be sampled evenly from the range `low` and `high`.**
_Suggested scenario_: It is suggested when search space is small, it is feasible to exhaustively sweeping the whole search space.
_Usage_:
```yml
# config.yml
tuner:
builtinTunerName: GridSearch
```
<a name="Hyperband"></a>
**Hyperband**
[Hyperband][6] tries to use limited resource to explore as many configurations as possible, and finds out the promising ones to get the final result. The basic idea is generating many configurations and to run them for small number of STEPs to find out promising one, then further training those promising ones to select several more promising one. More detail can be referred to [here](../src/sdk/pynni/nni/hyperband_advisor/README.md).
_Suggested scenario_: It is suggested when you have limited computation resource but have relatively large search space. It performs good in the scenario that intermediate result (e.g., accuracy) can reflect good or bad of final result (e.g., accuracy) to some extent.
_Usage_:
```yml
# config.yml
advisor:
builtinAdvisorName: Hyperband
classArgs:
# choice: maximize, minimize
optimize_mode: maximize
# R: the maximum STEPS (could be the number of mini-batches or epochs) can be
# allocated to a trial. Each trial should use STEPS to control how long it runs.
R: 60
# eta: proportion of discarded trials
eta: 3
```
<a name="NetworkMorphism"></a>
**Network Morphism**
[Network Morphism][7] provides functions to automatically search for architecture of deep learning models. Every child network inherits the knowledge from its parent network and morphs into diverse types of networks, including changes of depth, width and skip-connection. Next, it estimates the value of child network using the history architecture and metric pairs. Then it selects the most promising one to train. More detail can be referred to [here](../src/sdk/pynni/nni/networkmorphism_tuner/README.md).
_Installation_:
NetworkMorphism requires [pyTorch](https://pytorch.org/get-started/locally), so users should install it first.
_Suggested scenario_: It is suggested that you want to apply deep learning methods to your task (your own dataset) but you have no idea of how to choose or design a network. You modify the [example](../examples/trials/network_morphism/cifar10/cifar10_keras.py) to fit your own dataset and your own data augmentation method. Also you can change the batch size, learning rate or optimizer. It is feasible for different tasks to find a good network architecture. Now this tuner only supports the cv domain.
_Usage_:
```yml
# config.yml
tuner:
builtinTunerName: NetworkMorphism
classArgs:
#choice: maximize, minimize
optimize_mode: maximize
#for now, this tuner only supports cv domain
task: cv
#input image width
input_width: 32
#input image channel
input_channel: 3
#number of classes
n_output_node: 10
```
<a name="MetisTuner"></a>
**Metis Tuner**
[Metis][10] offers the following benefits when it comes to tuning parameters:
While most tools only predicts the optimal configuration, Metis gives you two outputs: (a) current prediction of optimal configuration, and (b) suggestion for the next trial. No more guess work!
While most tools assume training datasets do not have noisy data, Metis actually tells you if you need to re-sample a particular hyper-parameter.
While most tools have problems of being exploitation-heavy, Metis' search strategy balances exploration, exploitation, and (optional) re-sampling.
Metis belongs to the class of sequential model-based optimization (SMBO), and it is based on the Bayesian Optimization framework. To model the parameter-vs-performance space, Metis uses both Gaussian Process and GMM. Since each trial can impose a high time cost, Metis heavily trades inference computations with naive trial. At each iteration, Metis does two tasks:
* It finds the global optimal point in the Gaussian Process space. This point represents the optimal configuration.
* It identifies the next hyper-parameter candidate. This is achieved by inferring the potential information gain of exploration, exploitation, and re-sampling.
Note that the only acceptable types of search space are `choice`, `quniform`, `uniform` and `randint`. We only support
numerical `choice` now. More features will support later.
More details can be found in our paper: https://www.microsoft.com/en-us/research/publication/metis-robustly-tuning-tail-latencies-cloud-systems/
_Installation_:
Metis Tuner requires [sklearn](https://scikit-learn.org/), so users should install it first. User could use `pip3 install sklearn` to install it.
_Suggested scenario_:
Similar to TPE and SMAC, Metis is a black-box tuner. If your system takes a long time to finish each trial, Metis is more favorable than other approaches such as random search. Furthermore, Metis provides guidance on the subsequent trial. Here is an [example](../examples/trials/auto-gbdt/search_space_metis.json) about the use of Metis. User only need to send the final result like `accuracy` to tuner, by calling the NNI SDK.
_Usage_:
```yml
# config.yml
tuner:
builtinTunerName: MetisTuner
classArgs:
#choice: maximize, minimize
optimize_mode: maximize
```
<a name="assessor"></a>
# How to use Assessor that NNI supports?
For now, NNI has supported the following assessor algorithms.
- [Medianstop](#Medianstop)
- [Curvefitting](#Curvefitting)
## Supported Assessor Algorithms
<a name="Medianstop"></a>
**Medianstop**
Medianstop is a simple early stopping rule mentioned in the [paper][8]. It stops a pending trial X at step S if the trial’s best objective value by step S is strictly worse than the median value of the running averages of all completed trials’ objectives reported up to step S.
_Suggested scenario_: It is applicable in a wide range of performance curves, thus, can be used in various scenarios to speed up the tuning progress.
_Usage_:
```yml
assessor:
builtinAssessorName: Medianstop
classArgs:
#choice: maximize, minimize
optimize_mode: maximize
# (optional) A trial is determined to be stopped or not,
* only after receiving start_step number of reported intermediate results.
* The default value of start_step is 0.
start_step: 5
```
<a name="Curvefitting"></a>
**Curvefitting**
Curve Fitting Assessor is a LPA(learning, predicting, assessing) algorithm. It stops a pending trial X at step S if the prediction of final epoch's performance worse than the best final performance in the trial history. In this algorithm, we use 12 curves to fit the accuracy curve, the large set of parametric curve models are chosen from [reference paper][9]. The learning curves' shape coincides with our prior knowlwdge about the form of learning curves: They are typically increasing, saturating functions.
_Suggested scenario_: It is applicable in a wide range of performance curves, thus, can be used in various scenarios to speed up the tuning progress. Even better, it's able to handle and assess curves with similar performance.
_Usage_:
```yml
assessor:
builtinAssessorName: Curvefitting
classArgs:
# (required)The total number of epoch.
# We need to know the number of epoch to determine which point we need to predict.
epoch_num: 20
# (optional) choice: maximize, minimize
# Kindly reminds that if you choose minimize mode, please adjust the value of threshold >= 1.0 (e.g threshold=1.1)
* The default value of optimize_mode is maximize
optimize_mode: maximize
# (optional) A trial is determined to be stopped or not
# In order to save our computing resource, we start to predict when we have more than start_step(default=6) accuracy points.
# only after receiving start_step number of reported intermediate results.
* The default value of start_step is 6.
start_step: 6
# (optional) The threshold that we decide to early stop the worse performance curve.
# For example: if threshold = 0.95, optimize_mode = maximize, best performance in the history is 0.9, then we will stop the trial which predict value is lower than 0.95 * 0.9 = 0.855.
* The default value of threshold is 0.95.
threshold: 0.95
```
[1]: https://papers.nips.cc/paper/4443-algorithms-for-hyper-parameter-optimization.pdf
[2]: http://www.jmlr.org/papers/volume13/bergstra12a/bergstra12a.pdf
[3]: https://arxiv.org/pdf/1703.01041.pdf
[4]: https://www.cs.ubc.ca/~hutter/papers/10-TR-SMAC.pdf
[5]: https://github.com/automl/SMAC3
[6]: https://arxiv.org/pdf/1603.06560.pdf
[7]: https://arxiv.org/abs/1806.10282
[8]: https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/46180.pdf
[9]: http://aad.informatik.uni-freiburg.de/papers/15-IJCAI-Extrapolation_of_Learning_Curves.pdf
[10]:https://www.microsoft.com/en-us/research/publication/metis-robustly-tuning-tail-latencies-cloud-systems/
...@@ -22,7 +22,7 @@ Currently we only support installation on Linux & Mac. ...@@ -22,7 +22,7 @@ Currently we only support installation on Linux & Mac.
* __Install NNI in docker image__ * __Install NNI in docker image__
You can also install NNI in a docker image. Please follow the instructions [here](../deployment/docker/README.md) to build NNI docker image. The NNI docker image can also be retrieved from Docker Hub through the command `docker pull msranni/nni:latest`. You can also install NNI in a docker image. Please follow the instructions [here](https://github.com/Microsoft/nni/tree/master/deployment/docker/README.md) to build NNI docker image. The NNI docker image can also be retrieved from Docker Hub through the command `docker pull msranni/nni:latest`.
## **System requirements** ## **System requirements**
......
...@@ -13,7 +13,7 @@ Now NNI supports running experiment on [Kubeflow](https://github.com/kubeflow/ku ...@@ -13,7 +13,7 @@ Now NNI supports running experiment on [Kubeflow](https://github.com/kubeflow/ku
apt-get install nfs-common apt-get install nfs-common
``` ```
7. Install **NNI**, follow the install guide [here](GetStarted.md). 7. Install **NNI**, follow the install guide [here](QuickStart.md).
## Prerequisite for Azure Kubernetes Service ## Prerequisite for Azure Kubernetes Service
1. NNI support kubeflow based on Azure Kubernetes Service, follow the [guideline](https://azure.microsoft.com/en-us/services/kubernetes-service/) to set up Azure Kubernetes Service. 1. NNI support kubeflow based on Azure Kubernetes Service, follow the [guideline](https://azure.microsoft.com/en-us/services/kubernetes-service/) to set up Azure Kubernetes Service.
...@@ -179,7 +179,7 @@ Trial configuration in kubeflow mode have the following configuration keys: ...@@ -179,7 +179,7 @@ Trial configuration in kubeflow mode have the following configuration keys:
* gpuNum * gpuNum
* image * image
* Required key. In kubeflow mode, your trial program will be scheduled by Kubernetes to run in [Pod](https://kubernetes.io/docs/concepts/workloads/pods/pod/). This key is used to specify the Docker image used to create the pod where your trail program will run. * Required key. In kubeflow mode, your trial program will be scheduled by Kubernetes to run in [Pod](https://kubernetes.io/docs/concepts/workloads/pods/pod/). This key is used to specify the Docker image used to create the pod where your trail program will run.
* We already build a docker image [msranni/nni](https://hub.docker.com/r/msranni/nni/) on [Docker Hub](https://hub.docker.com/). It contains NNI python packages, Node modules and javascript artifact files required to start experiment, and all of NNI dependencies. The docker file used to build this image can be found at [here](../deployment/Dockerfile.build.base). You can either use this image directly in your config file, or build your own image based on it. * We already build a docker image [msranni/nni](https://hub.docker.com/r/msranni/nni/) on [Docker Hub](https://hub.docker.com/). It contains NNI python packages, Node modules and javascript artifact files required to start experiment, and all of NNI dependencies. The docker file used to build this image can be found at [here](https://github.com/Microsoft/nni/tree/master/deployment/Dockerfile.build.base). You can either use this image directly in your config file, or build your own image based on it.
* apiVersion * apiVersion
* Required key. The API version of your kubeflow. * Required key. The API version of your kubeflow.
* ps (optional). This config section is used to configure tensorflow parameter server role. * ps (optional). This config section is used to configure tensorflow parameter server role.
...@@ -196,4 +196,4 @@ Notice: In kubeflow mode, NNIManager will start a rest server and listen on a po ...@@ -196,4 +196,4 @@ Notice: In kubeflow mode, NNIManager will start a rest server and listen on a po
Once a trial job is completed, you can goto NNI WebUI's overview page (like http://localhost:8080/oview) to check trial's information. Once a trial job is completed, you can goto NNI WebUI's overview page (like http://localhost:8080/oview) to check trial's information.
Any problems when using NNI in kubeflow mode, plesae create issues on [NNI github repo](https://github.com/Microsoft/nni), or send mail to nni@microsoft.com Any problems when using NNI in kubeflow mode, plesae create issues on [NNI Github repo](https://github.com/Microsoft/nni).
\ No newline at end of file
...@@ -54,6 +54,6 @@ More details about how to run an experiment, please refer to [Get Started](Quick ...@@ -54,6 +54,6 @@ More details about how to run an experiment, please refer to [Get Started](Quick
* [What are assessors supported by NNI?](Builtin_Assessors.md) * [What are assessors supported by NNI?](Builtin_Assessors.md)
* [How to customize your own assessor?](Customize_Assessor.md) * [How to customize your own assessor?](Customize_Assessor.md)
* [How to run an experiment on local?](tutorial_1_CR_exp_local_api.md) * [How to run an experiment on local?](tutorial_1_CR_exp_local_api.md)
* [How to run an experiment on multiple machines?](tutorial_2_RemoteMachineMode.md) * [How to run an experiment on multiple machines?](RemoteMachineMode.md)
* [How to run an experiment on OpenPAI?](PAIMode.md) * [How to run an experiment on OpenPAI?](PAIMode.md)
* [Examples](mnist_examples.md) * [Examples](mnist_examples.md)
\ No newline at end of file
...@@ -3,12 +3,12 @@ ...@@ -3,12 +3,12 @@
NNI supports running an experiment on [OpenPAI](https://github.com/Microsoft/pai) (aka pai), called pai mode. Before starting to use NNI pai mode, you should have an account to access an [OpenPAI](https://github.com/Microsoft/pai) cluster. See [here](https://github.com/Microsoft/pai#how-to-deploy) if you don't have any OpenPAI account and want to deploy an OpenPAI cluster. In pai mode, your trial program will run in pai's container created by Docker. NNI supports running an experiment on [OpenPAI](https://github.com/Microsoft/pai) (aka pai), called pai mode. Before starting to use NNI pai mode, you should have an account to access an [OpenPAI](https://github.com/Microsoft/pai) cluster. See [here](https://github.com/Microsoft/pai#how-to-deploy) if you don't have any OpenPAI account and want to deploy an OpenPAI cluster. In pai mode, your trial program will run in pai's container created by Docker.
## Setup environment ## Setup environment
Install NNI, follow the install guide [here](GetStarted.md). Install NNI, follow the install guide [here](QuickStart.md).
## Run an experiment ## Run an experiment
Use `examples/trials/mnist-annotation` as an example. The NNI config YAML file's content is like: Use `examples/trials/mnist-annotation` as an example. The NNI config YAML file's content is like:
```yml ```yaml
authorName: your_name authorName: your_name
experimentName: auto_mnist experimentName: auto_mnist
# how many trials could be concurrently running # how many trials could be concurrently running
...@@ -50,7 +50,7 @@ Compared with LocalMode and [RemoteMachineMode](RemoteMachineMode.md), trial con ...@@ -50,7 +50,7 @@ Compared with LocalMode and [RemoteMachineMode](RemoteMachineMode.md), trial con
* Required key. Should be positive number based on your trial program's memory requirement * Required key. Should be positive number based on your trial program's memory requirement
* image * image
* Required key. In pai mode, your trial program will be scheduled by OpenPAI to run in [Docker container](https://www.docker.com/). This key is used to specify the Docker image used to create the container in which your trial will run. * Required key. In pai mode, your trial program will be scheduled by OpenPAI to run in [Docker container](https://www.docker.com/). This key is used to specify the Docker image used to create the container in which your trial will run.
* We already build a docker image [nnimsra/nni](https://hub.docker.com/r/msranni/nni/) on [Docker Hub](https://hub.docker.com/). It contains NNI python packages, Node modules and javascript artifact files required to start experiment, and all of NNI dependencies. The docker file used to build this image can be found at [here](../deployment/Dockerfile.build.base). You can either use this image directly in your config file, or build your own image based on it. * We already build a docker image [nnimsra/nni](https://hub.docker.com/r/msranni/nni/) on [Docker Hub](https://hub.docker.com/). It contains NNI python packages, Node modules and javascript artifact files required to start experiment, and all of NNI dependencies. The docker file used to build this image can be found at [here](https://github.com/Microsoft/nni/tree/master/deployment/Dockerfile.build.base). You can either use this image directly in your config file, or build your own image based on it.
* dataDir * dataDir
* Optional key. It specifies the HDFS data direcotry for trial to download data. The format should be something like hdfs://{your HDFS host}:9000/{your data directory} * Optional key. It specifies the HDFS data direcotry for trial to download data. The format should be something like hdfs://{your HDFS host}:9000/{your data directory}
* outputDir * outputDir
......
...@@ -40,7 +40,7 @@ if __name__ == '__main__': ...@@ -40,7 +40,7 @@ if __name__ == '__main__':
run_trial(params) run_trial(params)
``` ```
Note: If you want to see the full implementation, please refer to [examples/trials/mnist/mnist_before.py](../examples/trials/mnist/mnist_before.py) Note: If you want to see the full implementation, please refer to [examples/trials/mnist/mnist_before.py](https://github.com/Microsoft/nni/tree/master/examples/trials/mnist/mnist_before.py)
The above code can only try one set of parameters at a time, if we want to tune learning rate, we need to manually modify the hyperparameter and start the trial again and again. The above code can only try one set of parameters at a time, if we want to tune learning rate, we need to manually modify the hyperparameter and start the trial again and again.
...@@ -77,7 +77,7 @@ If you want to use NNI to automatically train your model and find the optimal hy ...@@ -77,7 +77,7 @@ If you want to use NNI to automatically train your model and find the optimal hy
+ } + }
``` ```
*Implemented code directory: [search_space.json](../examples/trials/mnist/search_space.json)* *Implemented code directory: [search_space.json](https://github.com/Microsoft/nni/tree/master/examples/trials/mnist/search_space.json)*
**Step 2**: Modified your `Trial` file to get the hyperparameter set from NNI and report the final result to NNI. **Step 2**: Modified your `Trial` file to get the hyperparameter set from NNI and report the final result to NNI.
...@@ -102,11 +102,11 @@ If you want to use NNI to automatically train your model and find the optimal hy ...@@ -102,11 +102,11 @@ If you want to use NNI to automatically train your model and find the optimal hy
run_trial(params) run_trial(params)
``` ```
*Implemented code directory: [mnist.py](../examples/trials/mnist/mnist.py)* *Implemented code directory: [mnist.py](https://github.com/Microsoft/nni/tree/master/examples/trials/mnist/mnist.py)*
**Step 3**: Define a `config` file in YAML, which declare the `path` to search space and trial, also give `other information` such as tuning algorithm, max trial number and max runtime arguments. **Step 3**: Define a `config` file in YAML, which declare the `path` to search space and trial, also give `other information` such as tuning algorithm, max trial number and max runtime arguments.
```yml ```yaml
authorName: default authorName: default
experimentName: example_mnist experimentName: example_mnist
trialConcurrency: 1 trialConcurrency: 1
...@@ -125,9 +125,9 @@ trial: ...@@ -125,9 +125,9 @@ trial:
gpuNum: 0 gpuNum: 0
``` ```
*Implemented code directory: [config.yml](../examples/trials/mnist/config.yml)* *Implemented code directory: [config.yml](https://github.com/Microsoft/nni/tree/master/examples/trials/mnist/config.yml)*
All the codes above are already prepared and stored in [examples/trials/mnist/](../examples/trials/mnist). All the codes above are already prepared and stored in [examples/trials/mnist/](https://github.com/Microsoft/nni/tree/master/examples/trials/mnist).
When these things are done, **run the config.yml file from your command line to start the experiment**. When these things are done, **run the config.yml file from your command line to start the experiment**.
......
# Overview
<p align="center">
<img src="https://raw.githubusercontent.com/Microsoft/nni/master/docs/img/nni_logo.png" alt="drawing" width="300"/>
</p>
-----------
[![](https://img.shields.io/badge/license-MIT-yellow.svg)](https://github.com/Microsoft/nni/blob/master/LICENSE)
[![](https://msrasrg.visualstudio.com/NNIOpenSource/_apis/build/status/Microsoft.nni)](https://msrasrg.visualstudio.com/NNIOpenSource/_build/latest?definitionId=6)
[![](https://img.shields.io/github/issues-raw/Microsoft/nni.svg)](https://github.com/Microsoft/nni/issues?q=is%3Aissue+is%3Aopen)
[![](https://img.shields.io/github/issues/Microsoft/nni/bug.svg)](https://github.com/Microsoft/nni/issues?q=is%3Aissue+is%3Aopen+label%3Abug)
[![](https://img.shields.io/github/issues-pr-raw/Microsoft/nni.svg)](https://github.com/Microsoft/nni/pulls?q=is%3Apr+is%3Aopen)
[![](https://img.shields.io/github/release/Microsoft/nni.svg)](https://github.com/Microsoft/nni/releases)
[![](https://badges.gitter.im/Microsoft/nni.svg)](https://gitter.im/Microsoft/nni?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
NNI (Neural Network Intelligence) is a toolkit to help users run automated machine learning (AutoML) experiments.
The tool dispatches and runs trial jobs generated by tuning algorithms to search the best neural architecture and/or hyper-parameters in different environments like local machine, remote servers and cloud.
![](img/nni_arch_overview.png)
## **Who should consider using NNI**
* Those who want to try different AutoML algorithms in their training code (model) at their local machine.
* Those who want to run AutoML trial jobs in different environments to speed up search (e.g. remote servers and cloud).
* Researchers and data scientists who want to implement their own AutoML algorithms and compare it with other algorithms.
* ML Platform owners who want to support AutoML in their platform.
## **Install & Verify**
**Install through pip**
* We support Linux and MacOS in current stage, Ubuntu 16.04 or higher, along with MacOS 10.14.1 are tested and supported. Simply run the following `pip install` in an environment that has `python >= 3.5`.
```bash
python3 -m pip install --user --upgrade nni
```
* Note:
* If you are in docker container (as root), please remove `--user` from the installation command.
* If there is any error like `Segmentation fault`, please refer to [FAQ](FAQ.md)
**Install through source code**
* We support Linux (Ubuntu 16.04 or higher), MacOS (10.14.1) in our current stage.
* Run the following commands in an environment that has `python >= 3.5`, `git` and `wget`.
```bash
git clone -b v0.4.1 https://github.com/Microsoft/nni.git
cd nni
source install.sh
```
For the system requirements of NNI, please refer to [Install NNI](Installation.md)
**Verify install**
The following example is an experiment built on TensorFlow. Make sure you have **TensorFlow installed** before running it.
* Download the examples via clone the source code.
```bash
git clone -b v0.4.1 https://github.com/Microsoft/nni.git
```
* Run the mnist example.
```bash
nnictl create --config nni/examples/trials/mnist/config.yml
```
* Wait for the message `INFO: Successfully started experiment!` in the command line. This message indicates that your experiment has been successfully started. You can explore the experiment using the `Web UI url`.
```
INFO: Starting restful server...
INFO: Successfully started Restful server!
INFO: Setting local config...
INFO: Successfully set local config!
INFO: Starting experiment...
INFO: Successfully started experiment!
-----------------------------------------------------------------------
The experiment id is egchD4qy
The Web UI urls are: http://223.255.255.1:8080 http://127.0.0.1:8080
-----------------------------------------------------------------------
You can use these commands to get more information about the experiment
-----------------------------------------------------------------------
commands description
1. nnictl experiment show show the information of experiments
2. nnictl trial ls list all of trial jobs
3. nnictl log stderr show stderr log content
4. nnictl log stdout show stdout log content
5. nnictl stop stop an experiment
6. nnictl trial kill kill a trial job by id
7. nnictl --help get help information about nnictl
-----------------------------------------------------------------------
```
* Open the `Web UI url` in your browser, you can view detail information of the experiment and all the submitted trial jobs as shown below. [Here](WebUI.md) are more Web UI pages.
<table style="border: none">
<th><img src="https://raw.githubusercontent.com/Microsoft/nni/dev-doc/docs/img/webui_overview_page.png" alt="drawing" width="395"/></th>
<th><img src="https://raw.githubusercontent.com/Microsoft/nni/dev-doc/docs/img/webui_trialdetail_page.png" alt="drawing" width="410"/></th>
</table>
## **Documentation**
* [NNI overview](Overview.md)
* [Quick start](GetStarted.md)
## **How to**
* [Install NNI](Installation.md)
* [Use command line tool nnictl](NNICTLDOC.md)
* [Use NNIBoard](WebUI.md)
* [How to define search space](SearchSpaceSpec.md)
* [How to define a trial](howto_1_WriteTrial.md)
* [Config an experiment](ExperimentConfig.md)
* [How to use annotation](howto_1_WriteTrial.md#nni-python-annotation)
## **Tutorials**
* [Run an experiment on local (with multiple GPUs)?](tutorial_1_CR_exp_local_api.md)
* [Run an experiment on multiple machines?](tutorial_2_RemoteMachineMode.md)
* [Run an experiment on OpenPAI?](PAIMode.md)
* [Run an experiment on Kubeflow?](KubeflowMode.md)
* [Try different tuners and assessors](tutorial_3_tryTunersAndAssessors.md)
* [Implement a customized tuner](howto_2_CustomizedTuner.md)
* [Implement a customized assessor](../examples/assessors/README.md)
* [Use Genetic Algorithm to find good model architectures for Reading Comprehension task](../examples/trials/ga_squad/README.md)
## **Contribute**
This project welcomes contributions and suggestions, we use [GitHub issues](https://github.com/Microsoft/nni/issues) for tracking requests and bugs.
Issues with the **good first issue** label are simple and easy-to-start ones that we recommend new contributors to start with.
To set up environment for NNI development, refer to the instruction: [Set up NNI developer environment](SetupNNIDeveloperEnvironment.md)
Before start coding, review and get familiar with the NNI Code Contribution Guideline: [Contributing](CONTRIBUTING.md)
We are in construction of the instruction for [How to Debug](HowToDebug.md), you are also welcome to contribute questions or suggestions on this area.
## **License**
The entire codebase is under [MIT license](https://github.com/Microsoft/nni/blob/master/LICENSE)
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment