Skip to content

GitLab

"vscode:/vscode.git/clone" did not exist on "51df846598544471bdc2126d39c13dca66884f90"
Unverified Commit 035d58bc authored by SparkSnail's avatar SparkSnail Committed by GitHub
Browse files

Merge pull request #121 from Microsoft/master 

merge master
parents b633c265 8e732f2c
Show whitespace changes
Inline Side-by-side
Showing with 1480 additions and 420 deletions
docs/Installation.md
View file @ 035d58bc
**Installation of NNI** # Installation of NNI
===
Currently we only support installation on Linux & Mac. Currently we only support installation on Linux & Mac.
## **Installation** ## **Installation**
* __Dependencies__
```bash
python >= 3.5
git
wget
```
python pip should also be correctly installed. You could use "python3 -m pip -v" to check pip version.
* __Install NNI through pip__ * __Install NNI through pip__
Prerequisite: `python >= 3.5`
```bash ```bash
python3 -m pip install --user --upgrade nni python3 -m pip install --upgrade nni
``` ```
* __Install NNI through source code__ * __Install NNI through source code__
Prerequisite: `python >=3.5, git, wget`
```bash ```bash
git clone -b v0.5 https://github.com/Microsoft/nni.git git clone -b v0.5.1 https://github.com/Microsoft/nni.git
cd nni cd nni
source install.sh ./install.sh
``` ```
* __Install NNI in docker image__ * __Install NNI in docker image__
You can also install NNI in a docker image. Please follow the instructions [here](../deployment/docker/README.md) to build NNI docker image. The NNI docker image can also be retrieved from Docker Hub through the command `docker pull msranni/nni:latest`. You can also install NNI in a docker image. Please follow the instructions [here](https://github.com/Microsoft/nni/tree/master/deployment/docker/README.md) to build NNI docker image. The NNI docker image can also be retrieved from Docker Hub through the command `docker pull msranni/nni:latest`.
## **System requirements** ## **System requirements**
...@@ -66,5 +58,7 @@ Below are the minimum system requirements for NNI on macOS. Due to potential pro ...@@ -66,5 +58,7 @@ Below are the minimum system requirements for NNI on macOS. Due to potential pro
* [Define search space](SearchSpaceSpec.md) * [Define search space](SearchSpaceSpec.md)
* [Config an experiment](ExperimentConfig.md) * [Config an experiment](ExperimentConfig.md)
* [How to run an experiment on local (with multiple GPUs)?](tutorial_1_CR_exp_local_api.md) * [How to run an experiment on local (with multiple GPUs)?](tutorial_1_CR_exp_local_api.md)
* [How to run an experiment on multiple machines?](tutorial_2_RemoteMachineMode.md) * [How to run an experiment on multiple machines?](RemoteMachineMode.md)
* [How to run an experiment on OpenPAI?](PAIMode.md) * [How to run an experiment on OpenPAI?](PAIMode.md)
* [How to run an experiment on Kubernetes through Kubeflow?](KubeflowMode.md)
* [How to run an experiment on Kubernetes through FrameworkController?](FrameworkControllerMode.md)
\ No newline at end of file
docs/KubeflowMode.md
View file @ 035d58bc
...@@ -7,26 +7,26 @@ Now NNI supports running experiment on [Kubeflow](https://github.com/kubeflow/ku ...@@ -7,26 +7,26 @@ Now NNI supports running experiment on [Kubeflow](https://github.com/kubeflow/ku
2. Download, set up, and deploy **Kubelow** to your Kubernetes cluster. Follow this [guideline](https://www.kubeflow.org/docs/started/getting-started/) to set up Kubeflow 2. Download, set up, and deploy **Kubelow** to your Kubernetes cluster. Follow this [guideline](https://www.kubeflow.org/docs/started/getting-started/) to set up Kubeflow
3. Prepare a **kubeconfig** file, which will be used by NNI to interact with your kubernetes API server. By default, NNI manager will use $(HOME)/.kube/config as kubeconfig file's path. You can also specify other kubeconfig files by setting the **KUBECONFIG** environment variable. Refer this [guideline]( https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig) to learn more about kubeconfig. 3. Prepare a **kubeconfig** file, which will be used by NNI to interact with your kubernetes API server. By default, NNI manager will use $(HOME)/.kube/config as kubeconfig file's path. You can also specify other kubeconfig files by setting the **KUBECONFIG** environment variable. Refer this [guideline]( https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig) to learn more about kubeconfig.
4. If your NNI trial job needs GPU resource, you should follow this [guideline](https://github.com/NVIDIA/k8s-device-plugin) to configure **Nvidia device plugin for Kubernetes**. 4. If your NNI trial job needs GPU resource, you should follow this [guideline](https://github.com/NVIDIA/k8s-device-plugin) to configure **Nvidia device plugin for Kubernetes**.
5. Prepare a **NFS server** and export a general purpose mount (we recommend to map your NFS server path in `root_squash option`, otherwise permission issue may raise when nni copy files to NFS. Refer this [page](https://linux.die.net/man/5/exports) to learn what root_squash option is), or **Azure File Storage**. 5. Prepare a **NFS server** and export a general purpose mount (we recommend to map your NFS server path in `root_squash option`, otherwise permission issue may raise when NNI copy files to NFS. Refer this [page](https://linux.die.net/man/5/exports) to learn what root_squash option is), or **Azure File Storage**.
6. Install **NFS client** on the machine where you install NNI and run nnictl to create experiment. Run this command to install NFSv4 client: 6. Install **NFS client** on the machine where you install NNI and run nnictl to create experiment. Run this command to install NFSv4 client:
``` ```
apt-get install nfs-common apt-get install nfs-common
``` ```
7. Install **NNI**, follow the install guide [here](GetStarted.md). 7. Install **NNI**, follow the install guide [here](QuickStart.md).
## Prerequisite for Azure Kubernetes Service ## Prerequisite for Azure Kubernetes Service
1. NNI support kubeflow based on Azure Kubernetes Service, follow the [guideline](https://azure.microsoft.com/en-us/services/kubernetes-service/) to set up Azure Kubernetes Service. 1. NNI support kubeflow based on Azure Kubernetes Service, follow the [guideline](https://azure.microsoft.com/en-us/services/kubernetes-service/) to set up Azure Kubernetes Service.
2. Install [Azure CLI](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest) and __kubectl__. Use `az login` to set azure account, and connect kubectl client to AKS, refer this [guideline](https://docs.microsoft.com/en-us/azure/aks/kubernetes-walkthrough#connect-to-the-cluster). 2. Install [Azure CLI](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest) and __kubectl__. Use `az login` to set azure account, and connect kubectl client to AKS, refer this [guideline](https://docs.microsoft.com/en-us/azure/aks/kubernetes-walkthrough#connect-to-the-cluster).
3. Deploy kubeflow on Azure Kubernetes Service, follow the [guideline](https://www.kubeflow.org/docs/started/getting-started/). 3. Deploy kubeflow on Azure Kubernetes Service, follow the [guideline](https://www.kubeflow.org/docs/started/getting-started/).
4. Follow the [guideline](https://docs.microsoft.com/en-us/azure/storage/common/storage-quickstart-create-account?tabs=portal) to create azure file storage account. If you use Azure Kubernetes Service, nni need Azure Storage Service to store code files and the output files. 4. Follow the [guideline](https://docs.microsoft.com/en-us/azure/storage/common/storage-quickstart-create-account?tabs=portal) to create azure file storage account. If you use Azure Kubernetes Service, NNI need Azure Storage Service to store code files and the output files.
5. To access Azure storage service, nni need the access key of the storage account, and nni use [Azure Key Vault](https://azure.microsoft.com/en-us/services/key-vault/) Service to protect your private key. Set up Azure Key Vault Service, add a secret to Key Vault to store the access key of Azure storage account. Follow this [guideline](https://docs.microsoft.com/en-us/azure/key-vault/quick-create-cli) to store the access key. 5. To access Azure storage service, NNI need the access key of the storage account, and NNI use [Azure Key Vault](https://azure.microsoft.com/en-us/services/key-vault/) Service to protect your private key. Set up Azure Key Vault Service, add a secret to Key Vault to store the access key of Azure storage account. Follow this [guideline](https://docs.microsoft.com/en-us/azure/key-vault/quick-create-cli) to store the access key.
## Design ## Design
![](./img/kubeflow_training_design.png) ![](./img/kubeflow_training_design.png)
Kubeflow training service instantiates a kubernetes rest client to interact with your K8s cluster's API server. Kubeflow training service instantiates a kubernetes rest client to interact with your K8s cluster's API server.
For each trial, we will upload all the files in your local codeDir path (configured in nni_config.yaml) together with NNI generated files like parameter.cfg into a storage volumn. Right now we support two kinds of storage volumns: [nfs](https://en.wikipedia.org/wiki/Network_File_System) and [azure file storage](https://azure.microsoft.com/en-us/services/storage/files/), you should configure the storage volumn in nni config yaml file. After files are prepared, Kubeflow training service will call K8S rest API to create kubeflow jobs ([tf-operator](https://github.com/kubeflow/tf-operator) job or [pytorch-operator](https://github.com/kubeflow/pytorch-operator) job) in K8S, and mount your storage volumn into the job's pod. Output files of kubeflow job, like stdout, stderr, trial.log or model files, will also be copied back to the storage volumn. NNI will show the storage volumn's URL for each trial in WebUI, to allow user browse the log files and job's output files. For each trial, we will upload all the files in your local codeDir path (configured in nni_config.yml) together with NNI generated files like parameter.cfg into a storage volumn. Right now we support two kinds of storage volumns: [nfs](https://en.wikipedia.org/wiki/Network_File_System) and [azure file storage](https://azure.microsoft.com/en-us/services/storage/files/), you should configure the storage volumn in NNI config YAML file. After files are prepared, Kubeflow training service will call K8S rest API to create kubeflow jobs ([tf-operator](https://github.com/kubeflow/tf-operator) job or [pytorch-operator](https://github.com/kubeflow/pytorch-operator) job) in K8S, and mount your storage volumn into the job's pod. Output files of kubeflow job, like stdout, stderr, trial.log or model files, will also be copied back to the storage volumn. NNI will show the storage volumn's URL for each trial in WebUI, to allow user browse the log files and job's output files.
## Supported operator ## Supported operator
NNI only support tf-operator and pytorch-operator of kubeflow, other operators is not tested. NNI only support tf-operator and pytorch-operator of kubeflow, other operators is not tested.
...@@ -55,7 +55,7 @@ kubeflowConfig: ...@@ -55,7 +55,7 @@ kubeflowConfig:
# Your NFS server export path, like /var/nfs/nni # Your NFS server export path, like /var/nfs/nni
path: {your_nfs_server_export_path} path: {your_nfs_server_export_path}
``` ```
If you use Azure storage, you should set `kubeflowConfig` in your config yaml file as follows: If you use Azure storage, you should set `kubeflowConfig` in your config YAML file as follows:
``` ```
kubeflowConfig: kubeflowConfig:
storage: azureStorage storage: azureStorage
...@@ -69,7 +69,7 @@ kubeflowConfig: ...@@ -69,7 +69,7 @@ kubeflowConfig:
## Run an experiment ## Run an experiment
Use `examples/trials/mnist` as an example. This is a tensorflow job, and use tf-operator of kubeflow. The nni config yaml file's content is like: Use `examples/trials/mnist` as an example. This is a tensorflow job, and use tf-operator of kubeflow. The NNI config yml file's content is like:
``` ```
authorName: default authorName: default
experimentName: example_mnist experimentName: example_mnist
...@@ -119,7 +119,7 @@ kubeflowConfig: ...@@ -119,7 +119,7 @@ kubeflowConfig:
path: {your_nfs_server_export_path} path: {your_nfs_server_export_path}
``` ```
Note: You should explicitly set `trainingServicePlatform: kubeflow` in nni config yaml file if you want to start experiment in kubeflow mode. Note: You should explicitly set `trainingServicePlatform: kubeflow` in NNI config yml file if you want to start experiment in kubeflow mode.
If you want to run Pytorch jobs, you could set your config files as follow: If you want to run Pytorch jobs, you could set your config files as follow:
``` ```
...@@ -179,15 +179,15 @@ Trial configuration in kubeflow mode have the following configuration keys: ...@@ -179,15 +179,15 @@ Trial configuration in kubeflow mode have the following configuration keys:
* gpuNum * gpuNum
* image * image
* Required key. In kubeflow mode, your trial program will be scheduled by Kubernetes to run in [Pod](https://kubernetes.io/docs/concepts/workloads/pods/pod/). This key is used to specify the Docker image used to create the pod where your trail program will run. * Required key. In kubeflow mode, your trial program will be scheduled by Kubernetes to run in [Pod](https://kubernetes.io/docs/concepts/workloads/pods/pod/). This key is used to specify the Docker image used to create the pod where your trail program will run.
* We already build a docker image [msranni/nni](https://hub.docker.com/r/msranni/nni/) on [Docker Hub](https://hub.docker.com/). It contains NNI python packages, Node modules and javascript artifact files required to start experiment, and all of NNI dependencies. The docker file used to build this image can be found at [here](../deployment/Dockerfile.build.base). You can either use this image directly in your config file, or build your own image based on it. * We already build a docker image [msranni/nni](https://hub.docker.com/r/msranni/nni/) on [Docker Hub](https://hub.docker.com/). It contains NNI python packages, Node modules and javascript artifact files required to start experiment, and all of NNI dependencies. The docker file used to build this image can be found at [here](https://github.com/Microsoft/nni/tree/master/deployment/Dockerfile.build.base). You can either use this image directly in your config file, or build your own image based on it.
* apiVersion * apiVersion
* Required key. The API version of your kubeflow. * Required key. The API version of your kubeflow.
* ps (optional). This config section is used to configure tensorflow parameter server role. * ps (optional). This config section is used to configure tensorflow parameter server role.
* master(optional). This config section is used to configure pytorch parameter server role. * master(optional). This config section is used to configure pytorch parameter server role.
Once complete to fill nni experiment config file and save (for example, save as exp_kubeflow.yaml), then run the following command Once complete to fill NNI experiment config file and save (for example, save as exp_kubeflow.yml), then run the following command
``` ```
nnictl create --config exp_kubeflow.yaml nnictl create --config exp_kubeflow.yml
``` ```
to start the experiment in kubeflow mode. NNI will create Kubeflow tfjob or pytorchjob for each trial, and the job name format is something like `nni_exp_{experiment_id}_trial_{trial_id}`. to start the experiment in kubeflow mode. NNI will create Kubeflow tfjob or pytorchjob for each trial, and the job name format is something like `nni_exp_{experiment_id}_trial_{trial_id}`.
You can see the kubeflow tfjob created by NNI in your Kubernetes dashboard. You can see the kubeflow tfjob created by NNI in your Kubernetes dashboard.
...@@ -196,4 +196,4 @@ Notice: In kubeflow mode, NNIManager will start a rest server and listen on a po ...@@ -196,4 +196,4 @@ Notice: In kubeflow mode, NNIManager will start a rest server and listen on a po
Once a trial job is completed, you can goto NNI WebUI's overview page (like http://localhost:8080/oview) to check trial's information. Once a trial job is completed, you can goto NNI WebUI's overview page (like http://localhost:8080/oview) to check trial's information.
Any problems when using NNI in kubeflow mode, plesae create issues on [NNI github repo](https://github.com/Microsoft/nni), or send mail to nni@microsoft.com Any problems when using NNI in kubeflow mode, plesae create issues on [NNI Github repo](https://github.com/Microsoft/nni).
\ No newline at end of file
docs/Makefile 0 → 100644
View file @ 035d58bc
# Minimal makefile for Sphinx documentation
#
# You can set these variables from the command line.
SPHINXOPTS =
SPHINXBUILD = sphinx-build
SOURCEDIR = .
BUILDDIR = _build
# Put it first so that "make" without argument is like "make help".
help:
@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
.PHONY: help Makefile
# Catch-all target: route all unknown targets to Sphinx using the new
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
%: Makefile
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
\ No newline at end of file
docs/NNICTLDOC.md
View file @ 035d58bc
# nnictl # nnictl
## Introduction ## Introduction
__nnictl__ is a command line tool, which can be used to control experiments, such as start/stop/resume an experiment, start/stop NNIBoard, etc. __nnictl__ is a command line tool, which can be used to control experiments, such as start/stop/resume an experiment, start/stop NNIBoard, etc.
...@@ -9,28 +8,30 @@ __nnictl__ is a command line tool, which can be used to control experiments, suc ...@@ -9,28 +8,30 @@ __nnictl__ is a command line tool, which can be used to control experiments, suc
nnictl support commands: nnictl support commands:
```bash * [nnictl create](#create)
nnictl create * [nnictl resume](#resume)
nnictl stop * [nnictl stop](#stop)
nnictl update * [nnictl update](#update)
nnictl resume * [nnictl trial](#trial)
nnictl trial * [nnictl top](#top)
nnictl experiment * [nnictl experiment](#experiment)
nnictl config * [nnictl config](#config)
nnictl log * [nnictl log](#log)
nnictl webui * [nnictl webui](#webui)
nnictl tensorboard * [nnictl tensorboard](#tensorboard)
nnictl top * [nnictl package](#package)
nnictl --version * [nnictl --version](#version)
```
### Manage an experiment ### Manage an experiment
<a name="create"></a>
* __nnictl create__ * __nnictl create__
* Description * Description
You can use this command to create a new experiment, using the configuration specified in config file. After this command is successfully done, the context will be set as this experiment, which means the following command you issued is associated with this experiment, unless you explicitly changes the context(not supported yet). You can use this command to create a new experiment, using the configuration specified in config file.
After this command is successfully done, the context will be set as this experiment, which means the following command you issued is associated with this experiment, unless you explicitly changes the context(not supported yet).
* Usage * Usage
...@@ -38,13 +39,14 @@ nnictl --version ...@@ -38,13 +39,14 @@ nnictl --version
nnictl create [OPTIONS] nnictl create [OPTIONS]
``` ```
Options: * Options
| Name, shorthand | Required|Default | Description |
| ------ | ------ | ------ |------ |
| --config, -c| True| |yaml configure file of the experiment|
| --port, -p | False| |the port of restful server|
| --debug, -d | False| |Set log level to debug|
|Name, shorthand|Required|Default|Description|
|------|------|------|------|
|--config, -c| True| |YAML configure file of the experiment|
|--port, -p|False| |the port of restful server|
<a name="resume"></a>
* __nnictl resume__ * __nnictl resume__
* Description * Description
...@@ -57,15 +59,16 @@ nnictl --version ...@@ -57,15 +59,16 @@ nnictl --version
nnictl resume [OPTIONS] nnictl resume [OPTIONS]
``` ```
Options: * Options
| Name, shorthand | Required|Default | Description | |Name, shorthand|Required|Default|Description|
| ------ | ------ | ------ |------ | |------|------|------ |------|
| id| False| |The id of the experiment you want to resume| |id| False| |The id of the experiment you want to resume|
| --port, -p| False| |Rest port of the experiment you want to resume| |--port, -p| False| |Rest port of the experiment you want to resume|
| --debug, -d | False| |Set log level to debug|
<a name="stop"></a>
* __nnictl stop__ * __nnictl stop__
* Description * Description
You can use this command to stop a running experiment or multiple experiments. You can use this command to stop a running experiment or multiple experiments.
...@@ -79,17 +82,13 @@ nnictl --version ...@@ -79,17 +82,13 @@ nnictl --version
* Detail * Detail
1. If there is an id specified, and the id matches the running experiment, nnictl will stop the corresponding experiment, or will print error message. 1. If there is an id specified, and the id matches the running experiment, nnictl will stop the corresponding experiment, or will print error message.
2. If there is no id specified, and there is an experiment running, stop the running experiment, or print error message. 2. If there is no id specified, and there is an experiment running, stop the running experiment, or print error message.
3. If the id ends with *, nnictl will stop all experiments whose ids matchs the regular. 3. If the id ends with *, nnictl will stop all experiments whose ids matchs the regular.
4. If the id does not exist but match the prefix of an experiment id, nnictl will stop the matched experiment. 4. If the id does not exist but match the prefix of an experiment id, nnictl will stop the matched experiment.
5. If the id does not exist but match multiple prefix of the experiment ids, nnictl will give id information. 5. If the id does not exist but match multiple prefix of the experiment ids, nnictl will give id information.
6. Users could use 'nnictl stop all' to stop all experiments.
6. Users could use 'nnictl stop all' to stop all experiments <a name="update"></a>
* __nnictl update__ * __nnictl update__
* __nnictl update searchspace__ * __nnictl update searchspace__
...@@ -103,12 +102,12 @@ nnictl --version ...@@ -103,12 +102,12 @@ nnictl --version
nnictl update searchspace [OPTIONS] nnictl update searchspace [OPTIONS]
``` ```
Options: * Options
| Name, shorthand | Required|Default | Description | |Name, shorthand|Required|Default|Description|
| ------ | ------ | ------ |------ | |------|------|------ |------|
| id| False| |ID of the experiment you want to set| |id| False| |ID of the experiment you want to set|
| --filename, -f| True| |the file storing your new search space| |--filename, -f| True| |the file storing your new search space|
* __nnictl update concurrency__ * __nnictl update concurrency__
* Description * Description
...@@ -121,14 +120,15 @@ nnictl --version ...@@ -121,14 +120,15 @@ nnictl --version
nnictl update concurrency [OPTIONS] nnictl update concurrency [OPTIONS]
``` ```
Options: * Options
| Name, shorthand | Required|Default | Description | |Name, shorthand|Required|Default|Description|
| ------ | ------ | ------ |------ | |------|------|------ |------|
| id| False| |ID of the experiment you want to set| |id| False| |ID of the experiment you want to set|
| --value, -v| True| |the number of allowed concurrent trials| |--value, -v| True| |the number of allowed concurrent trials|
* __nnictl update duration__ * __nnictl update duration__
* Description * Description
You can use this command to update an experiment's duration. You can use this command to update an experiment's duration.
...@@ -138,13 +138,12 @@ nnictl --version ...@@ -138,13 +138,12 @@ nnictl --version
```bash ```bash
nnictl update duration [OPTIONS] nnictl update duration [OPTIONS]
``` ```
* Options
Options: |Name, shorthand|Required|Default|Description|
|------|------|------ |------|
| Name, shorthand | Required|Default | Description | |id| False| |ID of the experiment you want to set|
| ------ | ------ | ------ |------ | |--value, -v| True| |the experiment duration will be NUMBER seconds. SUFFIX may be 's' for seconds (the default), 'm' for minutes, 'h' for hours or 'd' for days.|
| id| False| |ID of the experiment you want to set|
| --value, -v| True| |the experiment duration will be NUMBER seconds. SUFFIX may be 's' for seconds (the default), 'm' for minutes, 'h' for hours or 'd' for days.|
* __nnictl update trialnum__ * __nnictl update trialnum__
* Description * Description
...@@ -156,13 +155,15 @@ nnictl --version ...@@ -156,13 +155,15 @@ nnictl --version
```bash ```bash
nnictl update trialnum [OPTIONS] nnictl update trialnum [OPTIONS]
``` ```
Options:
| Name, shorthand | Required|Default | Description | * Options
| ------ | ------ | ------ |------ |
| id| False| |ID of the experiment you want to set| |Name, shorthand|Required|Default|Description|
| --value, -v| True| |the new number of maxtrialnum you want to set| |------|------|------ |------|
|id| False| |ID of the experiment you want to set|
|--value, -v| True| |the new number of maxtrialnum you want to set|
<a name="trial"></a>
* __nnictl trial__ * __nnictl trial__
* __nnictl trial ls__ * __nnictl trial ls__
...@@ -177,13 +178,14 @@ nnictl --version ...@@ -177,13 +178,14 @@ nnictl --version
nnictl trial ls nnictl trial ls
``` ```
Options: * Options
| Name, shorthand | Required|Default | Description | |Name, shorthand|Required|Default|Description|
| ------ | ------ | ------ |------ | |------|------|------ |------|
| id| False| |ID of the experiment you want to set| |id| False| |ID of the experiment you want to set|
* __nnictl trial kill__ * __nnictl trial kill__
* Description * Description
You can use this command to kill a trial job. You can use this command to kill a trial job.
...@@ -193,14 +195,16 @@ nnictl --version ...@@ -193,14 +195,16 @@ nnictl --version
```bash ```bash
nnictl trial kill [OPTIONS] nnictl trial kill [OPTIONS]
``` ```
Options:
| Name, shorthand | Required|Default | Description | * Options
| ------ | ------ | ------ |------ |
| id| False| |ID of the experiment you want to set|
| --trialid, -t| True| |ID of the trial you want to kill.|
* __nnictl top__ |Name, shorthand|Required|Default|Description|
|------|------|------ |------|
|id| False| |ID of the experiment you want to set|
|--trialid, -t| True| |ID of the trial you want to kill.|
<a name="top"></a>
* __nnictl top__
* Description * Description
...@@ -212,13 +216,14 @@ nnictl --version ...@@ -212,13 +216,14 @@ nnictl --version
nnictl top nnictl top
``` ```
Options: * Options
| Name, shorthand | Required|Default | Description | |Name, shorthand|Required|Default|Description|
| ------ | ------ | ------ |------ | |------|------|------ |------|
| id| False| |ID of the experiment you want to set| |id| False| |ID of the experiment you want to set|
| --time, -t| False| |The interval to update the experiment status, the unit of time is second, and the default value is 3 second.| |--time, -t| False| |The interval to update the experiment status, the unit of time is second, and the default value is 3 second.|
<a name="experiment"></a>
### Manage experiment information ### Manage experiment information
* __nnictl experiment show__ * __nnictl experiment show__
...@@ -233,11 +238,11 @@ nnictl --version ...@@ -233,11 +238,11 @@ nnictl --version
nnictl experiment show nnictl experiment show
``` ```
Options: * Options
| Name, shorthand | Required|Default | Description | |Name, shorthand|Required|Default|Description|
| ------ | ------ | ------ |------ | |------|------|------ |------|
| id| False| |ID of the experiment you want to set| |id| False| |ID of the experiment you want to set|
* __nnictl experiment status__ * __nnictl experiment status__
...@@ -251,13 +256,14 @@ nnictl --version ...@@ -251,13 +256,14 @@ nnictl --version
nnictl experiment status nnictl experiment status
``` ```
Options: * Options
| Name, shorthand | Required|Default | Description | |Name, shorthand|Required|Default|Description|
| ------ | ------ | ------ |------ | |------|------|------ |------|
| id| False| |ID of the experiment you want to set| |id| False| |ID of the experiment you want to set|
* __nnictl experiment list__ * __nnictl experiment list__
* Description * Description
Show the information of all the (running) experiments. Show the information of all the (running) experiments.
...@@ -268,13 +274,9 @@ nnictl --version ...@@ -268,13 +274,9 @@ nnictl --version
nnictl experiment list nnictl experiment list
``` ```
Options: <a name="config"></a>
| Name, shorthand | Required|Default | Description |
| ------ | ------ | ------ |------ |
| all| False| False|Show all of experiments, including stopped experiments.|
* __nnictl config show__ * __nnictl config show__
* Description * Description
Display the current context information. Display the current context information.
...@@ -285,6 +287,7 @@ nnictl --version ...@@ -285,6 +287,7 @@ nnictl --version
nnictl config show nnictl config show
``` ```
<a name="log"></a>
### Manage log ### Manage log
* __nnictl log stdout__ * __nnictl log stdout__
...@@ -299,14 +302,14 @@ nnictl --version ...@@ -299,14 +302,14 @@ nnictl --version
nnictl log stdout [options] nnictl log stdout [options]
``` ```
Options: * Options
| Name, shorthand | Required|Default | Description | |Name, shorthand|Required|Default|Description|
| ------ | ------ | ------ |------ | |------|------|------ |------|
| id| False| |ID of the experiment you want to set| |id| False| |ID of the experiment you want to set|
| --head, -h| False| |show head lines of stdout| |--head, -h| False| |show head lines of stdout|
| --tail, -t| False| |show tail lines of stdout| |--tail, -t| False| |show tail lines of stdout|
| --path, -p| False| |show the path of stdout file| |--path, -p| False| |show the path of stdout file|
* __nnictl log stderr__ * __nnictl log stderr__
* Description * Description
...@@ -319,16 +322,17 @@ nnictl --version ...@@ -319,16 +322,17 @@ nnictl --version
nnictl log stderr [options] nnictl log stderr [options]
``` ```
Options: * Options
| Name, shorthand | Required|Default | Description | |Name, shorthand|Required|Default|Description|
| ------ | ------ | ------ |------ | |------|------|------ |------|
| id| False| |ID of the experiment you want to set| |id| False| |ID of the experiment you want to set|
| --head, -h| False| |show head lines of stderr| |--head, -h| False| |show head lines of stderr|
| --tail, -t| False| |show tail lines of stderr| |--tail, -t| False| |show tail lines of stderr|
| --path, -p| False| |show the path of stderr file| |--path, -p| False| |show the path of stderr file|
* __nnictl log trial__ * __nnictl log trial__
* Description * Description
Show trial log path. Show trial log path.
...@@ -339,32 +343,18 @@ nnictl --version ...@@ -339,32 +343,18 @@ nnictl --version
nnictl log trial [options] nnictl log trial [options]
``` ```
Options: * Options
| Name, shorthand | Required|Default | Description | |Name, shorthand|Required|Default|Description|
| ------ | ------ | ------ |------ | |------|------|------ |------|
| id| False| |the id of trial| |id| False| |the id of trial|
<a name="webui"></a>
### Manage webui ### Manage webui
* __nnictl webui url__ * __nnictl webui url__
* Description <a name="tensorboard"></a>
Show the urls of the experiment.
* Usage
```bash
nnictl webui url
```
Options:
| Name, shorthand | Required|Default | Description |
| ------ | ------ | ------ |------ |
| id| False| |ID of the experiment you want to set|
### Manage tensorboard ### Manage tensorboard
* __nnictl tensorboard start__ * __nnictl tensorboard start__
...@@ -379,13 +369,13 @@ nnictl --version ...@@ -379,13 +369,13 @@ nnictl --version
nnictl tensorboard start nnictl tensorboard start
``` ```
Options: * Options
| Name, shorthand | Required|Default | Description | |Name, shorthand|Required|Default|Description|
| ------ | ------ | ------ |------ | |------|------|------ |------|
| id| False| |ID of the experiment you want to set| |id| False| |ID of the experiment you want to set|
| --trialid| False| |ID of the trial| |--trialid| False| |ID of the trial|
| --port| False| 6006|The port of the tensorboard process| |--port| False| 6006|The port of the tensorboard process|
* Detail * Detail
...@@ -406,19 +396,52 @@ nnictl --version ...@@ -406,19 +396,52 @@ nnictl --version
nnictl tensorboard stop nnictl tensorboard stop
``` ```
Options: * Options
|Name, shorthand|Required|Default|Description|
|------|------|------ |------|
|id| False| |ID of the experiment you want to set|
<a name="package"></a>
### Manage package
* __nnictl package install__
* Description
Install the packages needed in nni experiments.
* Usage
```bash
nnictl package install [OPTIONS]
```
* Options
|Name, shorthand|Required|Default|Description|
|------|------|------ |------|
|--name| True| |The name of package to be installed|
* __nnictl package show__
* Description
List the packages supported.
* Usage
| Name, shorthand | Required|Default | Description | ```bash
| ------ | ------ | ------ |------ | nnictl package show
| id| False| |ID of the experiment you want to set| ```
### Check nni version <a name="version"></a>
### Check NNI version
* __nnictl --version__ * __nnictl --version__
* Description * Description
Describe the current version of nni installed. Describe the current version of NNI installed.
* Usage * Usage
......
docs/Overview.md
View file @ 035d58bc
# NNI Overview # Overview
NNI (Neural Network Intelligence) is a toolkit to help users run automated machine learning experiments. For each experiment, user only need to define a search space and update a few lines of code, and then leverage NNI build-in algorithms and training services to search the best hyper parameters and/or neural architecture. NNI (Neural Network Intelligence) is a toolkit to help users design and tune machine learning models (e.g., hyperparameters), neural network architectures, or complex system's parameters, in an efficient and automatic way. NNI has several appealing properties: easy-to-use, scalability, flexibility, and efficiency.
>Step 1: [Define search space](SearchSpaceSpec.md) * **Easy-to-use**: NNI can be easily installed through python pip. Only several lines need to be added to your code in order to use NNI's power. You can use both commandline tool and WebUI to work with your experiments.
* **Scalability**: Tuning hyperparameters or neural architecture often demands large amount of computation resource, while NNI is designed to fully leverage different computation resources, such as remote machines, training platforms (e.g., OpenPAI, Kubernetes). Hundreds of trials could run in parallel by depending on the capacity of your configured training platforms.
>Step 2: [Update model codes](howto_1_WriteTrial.md) * **Flexibility**: Besides rich built-in algorithms, NNI allows users to customize various hyperparameter tuning algorithms, neural architecture search algorithms, early stopping algorithms, etc. Users could also extend NNI with more training platforms, such as virtual machines, kubernetes service on the cloud. Moreover, NNI can connect to external environments to tune special applications/models on them.
* **Efficiency**: We are intensively working on more efficient model tuning from both system level and algorithm level. For example, leveraging early feedback to speedup tuning procedure.
>Step 3: [Define Experiment](ExperimentConfig.md)
The figure below shows high-level architecture of NNI.
<p align="center"> <p align="center">
<img src="./img/3_steps.jpg" alt="drawing"/> <img src="https://user-images.githubusercontent.com/23273522/51816536-ed055580-2301-11e9-8ad8-605a79ee1b9a.png" alt="drawing" width="700"/>
</p> </p>
After user submits the experiment through a command line tool [nnictl](../tools/README.md), a demon process (NNI manager) take care of search process. NNI manager continuously get search settings that generated by tuning algorithms, then NNI manager asks the training service component to dispatch and run trial jobs in a targeted training environment (e.g. local machine, remote servers and cloud). The results of trials jobs such as model accurate will send back to tuning algorithms for generating more meaningful search settings. NNI manager stops the search process after it find the best models. ## Key Concepts
* *Experiment*: An experiment is one task of, for example, finding out the best hyperparameters of a model, finding out the best neural network architecture. It consists of trials and AutoML algorithms.
## Architecture Overview * *Search Space*: It means the feasible region for tuning the model. For example, the value range of each hyperparameters.
<p align="center">
<img src="./img/nni_arch_overview.png" alt="drawing"/>
</p>
User can use the nnictl and/or a visualized Web UI nniboard to monitor and debug a given experiment. * *Configuration*: A configuration is an instance from the search space, that is, each hyperparameter has a specific value.
NNI provides a set of examples in the package to get you familiar with the above process. * *Trial*: Trial is an individual attempt at applying a new configuration (e.g., a set of hyperparameter values, a specific nerual architecture). Trial code should be able to run with the provided configuration.
## Key Concepts * *Tuner*: Tuner is an AutoML algorithm, which generates a new configuration for the next try. A new trial will run with this configuration.
* *Assessor*: Assessor analyzes trial's intermediate results (e.g., periodically evaluated accuracy on test dataset) to tell whether this trial can be early stopped or not.
**Experiment** in NNI is a method for testing different assumptions (hypotheses) by Trials under conditions constructed and controlled by NNI. During the experiment, one or more conditions are allowed to change in an organized manner and effects of these changes on associated conditions. * *Training Platform*: It means where trials are executed. Depending on your experiment's configuration, it could be your local machine, or remote servers, or large-scale training platform (e.g., OpenPAI, Kubernetes).
### **Trial** Basically, an experiment runs as follows: Tuner receives search space and generates configurations. These configurations will be submitted to training platforms, such as local machine, remote machines, or training clusters. Their performances are reported back to Tuner. Then, new configurations are generated and submitted.
**Trial** in NNI is an individual attempt at applying a set of parameters on a model.
### **Tuner** For each experiment, user only needs to define a search space and update a few lines of code, and then leverage NNI built-in Tuner/Assessor and training platforms to search the best hyperparameters and/or neural architecture. There are basically 3 steps:
**Tuner** in NNI is an implementation of Tuner API for a special tuning algorithm. [Read more about the Tuners supported in the latest NNI release](HowToChooseTuner.md)
>Step 1: [Define search space](SearchSpaceSpec.md)
>Step 2: [Update model codes](Trials.md)
>Step 3: [Define Experiment](ExperimentConfig.md)
<p align="center">
<img src="https://user-images.githubusercontent.com/23273522/51816627-5d13db80-2302-11e9-8f3e-627e260203d5.jpg" alt="drawing"/>
</p>
### **Assessor** More details about how to run an experiment, please refer to [Get Started](QuickStart.md).
**Assessor** in NNI is an implementation of Assessor API for optimizing the execution of experiment.
## Learn More ## Learn More
* [Get started](GetStarted.md) * [Get started](QuickStart.md)
* [Install NNI](Installation.md) * [How to adapt your trial code on NNI?](Trials.md)
* [Use command line tool nnictl](NNICTLDOC.md) * [What are tuners supported by NNI?](Builtin_Tuner.md)
* [Use NNIBoard](WebUI.md) * [How to customize your own tuner?](Customize_Tuner.md)
* [Use annotation](howto_1_WriteTrial.md#nni-python-annotation) * [What are assessors supported by NNI?](Builtin_Assessors.md)
### **Tutorials** * [How to customize your own assessor?](Customize_Assessor.md)
* [How to run an experiment on local (with multiple GPUs)?](tutorial_1_CR_exp_local_api.md) * [How to run an experiment on local?](tutorial_1_CR_exp_local_api.md)
* [How to run an experiment on multiple machines?](tutorial_2_RemoteMachineMode.md) * [How to run an experiment on multiple machines?](RemoteMachineMode.md)
* [How to run an experiment on OpenPAI?](PAIMode.md) * [How to run an experiment on OpenPAI?](PAIMode.md)
* [Examples](mnist_examples.md)
\ No newline at end of file
docs/PAIMode.md
View file @ 035d58bc
...@@ -3,10 +3,10 @@ ...@@ -3,10 +3,10 @@
NNI supports running an experiment on [OpenPAI](https://github.com/Microsoft/pai) (aka pai), called pai mode. Before starting to use NNI pai mode, you should have an account to access an [OpenPAI](https://github.com/Microsoft/pai) cluster. See [here](https://github.com/Microsoft/pai#how-to-deploy) if you don't have any OpenPAI account and want to deploy an OpenPAI cluster. In pai mode, your trial program will run in pai's container created by Docker. NNI supports running an experiment on [OpenPAI](https://github.com/Microsoft/pai) (aka pai), called pai mode. Before starting to use NNI pai mode, you should have an account to access an [OpenPAI](https://github.com/Microsoft/pai) cluster. See [here](https://github.com/Microsoft/pai#how-to-deploy) if you don't have any OpenPAI account and want to deploy an OpenPAI cluster. In pai mode, your trial program will run in pai's container created by Docker.
## Setup environment ## Setup environment
Install NNI, follow the install guide [here](GetStarted.md). Install NNI, follow the install guide [here](QuickStart.md).
## Run an experiment ## Run an experiment
Use `examples/trials/mnist-annotation` as an example. The nni config yaml file's content is like: Use `examples/trials/mnist-annotation` as an example. The NNI config YAML file's content is like:
```yaml ```yaml
authorName: your_name authorName: your_name
...@@ -41,7 +41,7 @@ paiConfig: ...@@ -41,7 +41,7 @@ paiConfig:
host: 10.1.1.1 host: 10.1.1.1
``` ```
Note: You should set `trainingServicePlatform: pai` in nni config yaml file if you want to start experiment in pai mode. Note: You should set `trainingServicePlatform: pai` in NNI config YAML file if you want to start experiment in pai mode.
Compared with LocalMode and [RemoteMachineMode](RemoteMachineMode.md), trial configuration in pai mode have five additional keys: Compared with LocalMode and [RemoteMachineMode](RemoteMachineMode.md), trial configuration in pai mode have five additional keys:
* cpuNum * cpuNum
...@@ -49,19 +49,19 @@ Compared with LocalMode and [RemoteMachineMode](RemoteMachineMode.md), trial con ...@@ -49,19 +49,19 @@ Compared with LocalMode and [RemoteMachineMode](RemoteMachineMode.md), trial con
* memoryMB * memoryMB
* Required key. Should be positive number based on your trial program's memory requirement * Required key. Should be positive number based on your trial program's memory requirement
* image * image
* Required key. In pai mode, your trial program will be scheduled by OpenPAI to run in [Docker container](https://www.docker.com/). This key is used to specify the Docker image used to create the container in which your traill will run. * Required key. In pai mode, your trial program will be scheduled by OpenPAI to run in [Docker container](https://www.docker.com/). This key is used to specify the Docker image used to create the container in which your trial will run.
* We already build a docker image [nnimsra/nni](https://hub.docker.com/r/msranni/nni/) on [Docker Hub](https://hub.docker.com/). It contains NNI python packages, Node modules and javascript artifact files required to start experiment, and all of NNI dependencies. The docker file used to build this image can be found at [here](../deployment/Dockerfile.build.base). You can either use this image directly in your config file, or build your own image based on it. * We already build a docker image [nnimsra/nni](https://hub.docker.com/r/msranni/nni/) on [Docker Hub](https://hub.docker.com/). It contains NNI python packages, Node modules and javascript artifact files required to start experiment, and all of NNI dependencies. The docker file used to build this image can be found at [here](https://github.com/Microsoft/nni/tree/master/deployment/Dockerfile.build.base). You can either use this image directly in your config file, or build your own image based on it.
* dataDir * dataDir
* Optional key. It specifies the HDFS data direcotry for trial to download data. The format should be something like hdfs://{your HDFS host}:9000/{your data directory} * Optional key. It specifies the HDFS data direcotry for trial to download data. The format should be something like hdfs://{your HDFS host}:9000/{your data directory}
* outputDir * outputDir
* Optional key. It specifies the HDFS output direcotry for trial. Once the trial is completed (either succeed or fail), trial's stdout, stderr will be copied to this directory by NNI sdk automatically. The format should be something like hdfs://{your HDFS host}:9000/{your output directory} * Optional key. It specifies the HDFS output direcotry for trial. Once the trial is completed (either succeed or fail), trial's stdout, stderr will be copied to this directory by NNI sdk automatically. The format should be something like hdfs://{your HDFS host}:9000/{your output directory}
Once complete to fill nni experiment config file and save (for example, save as exp_pai.yaml), then run the following command Once complete to fill NNI experiment config file and save (for example, save as exp_pai.yml), then run the following command
``` ```
nnictl create --config exp_pai.yaml nnictl create --config exp_pai.yml
``` ```
to start the experiment in pai mode. NNI will create OpenPAI job for each trial, and the job name format is something like `nni_exp_{experiment_id}_trial_{trial_id}`. to start the experiment in pai mode. NNI will create OpenPAI job for each trial, and the job name format is something like `nni_exp_{experiment_id}_trial_{trial_id}`.
You can see the pai jobs created by NNI in your OpenPAI cluster's web portal, like: You can see jobs created by NNI in the OpenPAI cluster's web portal, like:
![](./img/nni_pai_joblist.jpg) ![](./img/nni_pai_joblist.jpg)
Notice: In pai mode, NNIManager will start a rest server and listen on a port which is your NNI WebUI's port plus 1. For example, if your WebUI port is `8080`, the rest server will listen on `8081`, to receive metrics from trial job running in Kubernetes. So you should `enable 8081` TCP port in your firewall rule to allow incoming traffic. Notice: In pai mode, NNIManager will start a rest server and listen on a port which is your NNI WebUI's port plus 1. For example, if your WebUI port is `8080`, the rest server will listen on `8081`, to receive metrics from trial job running in Kubernetes. So you should `enable 8081` TCP port in your firewall rule to allow incoming traffic.
...@@ -78,4 +78,4 @@ You can see there're three fils in output folder: stderr, stdout, and trial.log ...@@ -78,4 +78,4 @@ You can see there're three fils in output folder: stderr, stdout, and trial.log
If you also want to save trial's other output into HDFS, like model files, you can use environment variable `NNI_OUTPUT_DIR` in your trial code to save your own output files, and NNI SDK will copy all the files in `NNI_OUTPUT_DIR` from trial's container to HDFS. If you also want to save trial's other output into HDFS, like model files, you can use environment variable `NNI_OUTPUT_DIR` in your trial code to save your own output files, and NNI SDK will copy all the files in `NNI_OUTPUT_DIR` from trial's container to HDFS.
Any problems when using NNI in pai mode, plesae create issues on [NNI github repo](https://github.com/Microsoft/nni), or send mail to nni@microsoft.com Any problems when using NNI in pai mode, plesae create issues on [NNI github repo](https://github.com/Microsoft/nni).
docs/QuickStart.md 0 → 100644
View file @ 035d58bc
# QuickStart
## Installation
We support Linux and MacOS in current stage, Ubuntu 16.04 or higher and MacOS 10.14.1 are tested and supported. Simply run the following `pip install` in an environment that has `python >= 3.5`.
```bash
python3 -m pip install --upgrade nni
```
Note:
* `--user` can be added if you want to install NNI in your home directory, which does not require any special privileges.
* If there is any error like `Segmentation fault`, please refer to [FAQ](FAQ.md)
* For the `system requirements` of NNI, please refer to [Install NNI](Installation.md)
## "Hello World" example on MNIST
NNI is a toolkit to help users run automated machine learning experiments. It can automatically do the cyclic process of getting hyperparameters, running trials, testing results, tuning hyperparameters. Now, we show how to use NNI to help you find the optimal hyperparameters.
Here is an example script to train a CNN on MNIST dataset **without NNI**:
```python
def run_trial(params):
# Input data
mnist = input_data.read_data_sets(params['data_dir'], one_hot=True)
# Build network
mnist_network = MnistNetwork(channel_1_num=params['channel_1_num'], channel_2_num=params['channel_2_num'], conv_size=params['conv_size'], hidden_size=params['hidden_size'], pool_size=params['pool_size'], learning_rate=params['learning_rate'])
mnist_network.build_network()
test_acc = 0.0
with tf.Session() as sess:
# Train network
mnist_network.train(sess, mnist)
# Evaluate network
test_acc = mnist_network.evaluate(mnist)
if __name__ == '__main__':
params = {'data_dir': '/tmp/tensorflow/mnist/input_data', 'dropout_rate': 0.5, 'channel_1_num': 32, 'channel_2_num': 64, 'conv_size': 5, 'pool_size': 2, 'hidden_size': 1024, 'learning_rate': 1e-4, 'batch_num': 2000, 'batch_size': 32}
run_trial(params)
```
Note: If you want to see the full implementation, please refer to [examples/trials/mnist/mnist_before.py](https://github.com/Microsoft/nni/tree/master/examples/trials/mnist/mnist_before.py)
The above code can only try one set of parameters at a time, if we want to tune learning rate, we need to manually modify the hyperparameter and start the trial again and again.
NNI is born for helping user do the tuning jobs, the NNI working process is presented below:
```
input: search space, trial code, config file
output: one optimal hyperparameter configuration
1: For t = 0, 1, 2, ..., maxTrialNum,
2: hyperparameter = chose a set of parameter from search space
3: final result = run_trial_and_evaluate(hyperparameter)
4: report final result to NNI
5: If reach the upper limit time,
6: Stop the experiment
7: return hyperparameter value with best final result
```
If you want to use NNI to automatically train your model and find the optimal hyper-parameters, you need to do three changes base on your code:
**Three things required to do when using NNI**
**Step 1**: Give a `Search Space` file in json, includes the `name` and the `distribution` (discrete valued or continuous valued) of all the hyperparameters you need to search.
```diff
- params = {'data_dir': '/tmp/tensorflow/mnist/input_data', 'dropout_rate': 0.5, 'channel_1_num': 32, 'channel_2_num': 64,
- 'conv_size': 5, 'pool_size': 2, 'hidden_size': 1024, 'learning_rate': 1e-4, 'batch_num': 2000, 'batch_size': 32}
+ {
+ "dropout_rate":{"_type":"uniform","_value":[0.5, 0.9]},
+ "conv_size":{"_type":"choice","_value":[2,3,5,7]},
+ "hidden_size":{"_type":"choice","_value":[124, 512, 1024]},
+ "batch_size": {"_type":"choice", "_value": [1, 4, 8, 16, 32]},
+ "learning_rate":{"_type":"choice","_value":[0.0001, 0.001, 0.01, 0.1]}
+ }
```
*Implemented code directory: [search_space.json](https://github.com/Microsoft/nni/tree/master/examples/trials/mnist/search_space.json)*
**Step 2**: Modified your `Trial` file to get the hyperparameter set from NNI and report the final result to NNI.
```diff
+ import nni
def run_trial(params):
mnist = input_data.read_data_sets(params['data_dir'], one_hot=True)
mnist_network = MnistNetwork(channel_1_num=params['channel_1_num'], channel_2_num=params['channel_2_num'], conv_size=params['conv_size'], hidden_size=params['hidden_size'], pool_size=params['pool_size'], learning_rate=params['learning_rate'])
mnist_network.build_network()
with tf.Session() as sess:
mnist_network.train(sess, mnist)
test_acc = mnist_network.evaluate(mnist)
+ nni.report_final_result(acc)
if __name__ == '__main__':
- params = {'data_dir': '/tmp/tensorflow/mnist/input_data', 'dropout_rate': 0.5, 'channel_1_num': 32, 'channel_2_num': 64,
- 'conv_size': 5, 'pool_size': 2, 'hidden_size': 1024, 'learning_rate': 1e-4, 'batch_num': 2000, 'batch_size': 32}
+ params = nni.get_next_parameter()
run_trial(params)
```
*Implemented code directory: [mnist.py](https://github.com/Microsoft/nni/tree/master/examples/trials/mnist/mnist.py)*
**Step 3**: Define a `config` file in YAML, which declare the `path` to search space and trial, also give `other information` such as tuning algorithm, max trial number and max duration arguments.
```yaml
authorName: default
experimentName: example_mnist
trialConcurrency: 1
maxExecDuration: 1h
maxTrialNum: 10
trainingServicePlatform: local
# The path to Search Space
searchSpacePath: search_space.json
useAnnotation: false
tuner:
builtinTunerName: TPE
# The path and the running command of trial
trial:
command: python3 mnist.py
codeDir: .
gpuNum: 0
```
*Implemented code directory: [config.yml](https://github.com/Microsoft/nni/tree/master/examples/trials/mnist/config.yml)*
All the codes above are already prepared and stored in [examples/trials/mnist/](https://github.com/Microsoft/nni/tree/master/examples/trials/mnist).
When these things are done, **run the config.yml file from your command line to start the experiment**.
```bash
nnictl create --config nni/examples/trials/mnist/config.yml
```
Note: **nnictl** is a command line tool, which can be used to control experiments, such as start/stop/resume an experiment, start/stop NNIBoard, etc. Click [here](NNICTLDOC.md) for more usage of `nnictl`
Wait for the message `INFO: Successfully started experiment!` in the command line. This message indicates that your experiment has been successfully started. And this is what we expected to get:
```
INFO: Starting restful server...
INFO: Successfully started Restful server!
INFO: Setting local config...
INFO: Successfully set local config!
INFO: Starting experiment...
INFO: Successfully started experiment!
-----------------------------------------------------------------------
The experiment id is egchD4qy
The Web UI urls are: [Your IP]:8080
-----------------------------------------------------------------------
You can use these commands to get more information about the experiment
-----------------------------------------------------------------------
commands description
1. nnictl experiment show show the information of experiments
2. nnictl trial ls list all of trial jobs
3. nnictl top monitor the status of running experiments
4. nnictl log stderr show stderr log content
5. nnictl log stdout show stdout log content
6. nnictl stop stop an experiment
7. nnictl trial kill kill a trial job by id
8. nnictl --help get help information about nnictl
-----------------------------------------------------------------------
```
If you prepare `trial`, `search space` and `config` according to the above steps and successfully create a NNI job, NNI will automatically tune the optimal hyper-parameters and run different hyper-parameters sets for each trial according to the requirements you set. You can clearly sees its progress by NNI WebUI.
## WebUI
After you start your experiment in NNI successfully, you can find a message in the command-line interface to tell you `Web UI url` like this:
```
The Web UI urls are: [Your IP]:8080
```
Open the `Web UI url`(In this information is: `[Your IP]:8080`) in your browser, you can view detail information of the experiment and all the submitted trial jobs as shown below.
#### View summary page
Click the tab "Overview".
Information about this experiment will be shown in the WebUI, including the experiment trial profile and search space message. NNI also support `download these information and parameters` through the **Download** button. You can download the experiment result anytime in the middle for the running or at the end of the execution, etc.
![](./img/QuickStart1.png)
Top 10 trials will be listed in the Overview page, you can browse all the trials in "Trials Detail" page.
![](./img/QuickStart2.png)
#### View trials detail page
Click the tab "Default Metric" to see the point graph of all trials. Hover to see its specific default metric and search space message.
![](./img/QuickStart3.png)
Click the tab "Hyper Parameter" to see the parallel graph.
* You can select the percentage to see top trials.
* Choose two axis to swap its positions
![](./img/QuickStart4.png)
Click the tab "Trial Duration" to see the bar graph.
![](./img/QuickStart5.png)
Below is the status of the all trials. Specifically:
* Trial detail: trial's id, trial's duration, start time, end time, status, accuracy and search space file.
* If you run on the OpenPAI platform, you can also see the hdfsLogPath.
* Kill: you can kill a job that status is running.
* Support to search for a specific trial.
![](./img/QuickStart6.png)
* Intermediate Result Grap
![](./img/QuickStart7.png)
## Related Topic
* [Try different Tuners](Builtin_Tuner.md)
* [Try different Assessors](Builtin_Assessors.md)
* [How to use command line tool nnictl](NNICTLDOC.md)
* [How to write a trial](Trials.md)
* [How to run an experiment on local (with multiple GPUs)?](tutorial_1_CR_exp_local_api.md)
* [How to run an experiment on multiple machines?](RemoteMachineMode.md)
* [How to run an experiment on OpenPAI?](PAIMode.md)
* [How to run an experiment on Kubernetes through Kubeflow?](KubeflowMode.md)
* [How to run an experiment on Kubernetes through FrameworkController?](FrameworkControllerMode.md)
\ No newline at end of file
docs/RELEASE.md
View file @ 035d58bc
# Release 0.5.0 - 01/14/2019 # ChangeLog
## Major Features
### New tuner and assessor supports
* Support [Metis tuner](./HowToChooseTuner.md#MetisTuner) as a new NNI tuner. Metis algorithm has been proofed to be well performed for **online** hyper-parameter tuning.
* Support [ENAS customized tuner](https://github.com/countif/enas_nni), a tuner contributed by github community user, is an algorithm for neural network search, it could learn neural network architecture via reinforcement learning and serve a better performance than NAS.
* Support [Curve fitting assessor](./HowToChooseTuner.md#Curvefitting) for early stop policy using learning curve extrapolation.
* Advanced Support of [Weight Sharing](./AdvancedNAS.md): Enable weight sharing for NAS tuners, currently through NFS.
## Release 0.5.1 - 1/31/2018
### Improvements
* Making [log directory](https://github.com/Microsoft/nni/blob/v0.5.1/docs/ExperimentConfig.md) configurable
* Support [different levels of logs](https://github.com/Microsoft/nni/blob/v0.5.1/docs/ExperimentConfig.md), making it easier for debugging
### Documentation
* Reorganized documentation & New Homepage Released: https://nni.readthedocs.io/en/latest/
### Bug Fixes and Other Changes
* Fix the bug of installation in python virtualenv, and refactor the installation logic
* Fix the bug of HDFS access failure on PAI mode after PAI is upgraded.
* Fix the bug that sometimes in-place flushed stdout makes experiment crash
## Release 0.5.0 - 01/14/2019
### Major Features
#### New tuner and assessor supports
* Support [Metis tuner](./Builtin_Tuner.md#MetisTuner) as a new NNI tuner. Metis algorithm has been proofed to be well performed for **online** hyper-parameter tuning.
* Support [ENAS customized tuner](https://github.com/countif/enas_nni), a tuner contributed by github community user, is an algorithm for neural network search, it could learn neural network architecture via reinforcement learning and serve a better performance than NAS.
* Support [Curve fitting assessor](./Builtin_Tuner.md#Curvefitting) for early stop policy using learning curve extrapolation.
* Advanced Support of [Weight Sharing](./AdvancedNAS.md): Enable weight sharing for NAS tuners, currently through NFS.
#### Training Service Enhancement
### Training Service Enhancement
* [FrameworkController Training service](./FrameworkControllerMode.md): Support run experiments using frameworkcontroller on kubernetes * [FrameworkController Training service](./FrameworkControllerMode.md): Support run experiments using frameworkcontroller on kubernetes
* FrameworkController is a Controller on kubernetes that is general enough to run (distributed) jobs with various machine learning frameworks, such as tensorflow, pytorch, MXNet. * FrameworkController is a Controller on kubernetes that is general enough to run (distributed) jobs with various machine learning frameworks, such as tensorflow, pytorch, MXNet.
* NNI provides unified and simple specification for job definition. * NNI provides unified and simple specification for job definition.
* MNIST example for how to use FrameworkController. * MNIST example for how to use FrameworkController.
### User Experience improvements #### User Experience improvements
* A better trial logging support for NNI experiments in PAI, Kubeflow and FrameworkController mode:
* A better trial logging support for NNI experiments in OpenPAI, Kubeflow and FrameworkController mode:
* An improved logging architecture to send stdout/stderr of trials to NNI manager via Http post. NNI manager will store trial's stdout/stderr messages in local log file. * An improved logging architecture to send stdout/stderr of trials to NNI manager via Http post. NNI manager will store trial's stdout/stderr messages in local log file.
* Show the link for trial log file on WebUI. * Show the link for trial log file on WebUI.
* Support to show final result's all key-value pairs. * Support to show final result's all key-value pairs.
# Release 0.4.1 - 12/14/2018 ## Release 0.4.1 - 12/14/2018
## Major Features
### New tuner supports
* Support [network morphism](./HowToChooseTuner.md#NetworkMorphism) as a new tuner
### Training Service improvements ### Major Features
* Migrate [Kubeflow training service](https://github.com/Microsoft/nni/blob/master/docs/KubeflowMode.md)'s dependency from kubectl CLI to [Kubernetes API](https://kubernetes.io/docs/concepts/overview/kubernetes-api/) client
* [Pytorch-operator](https://github.com/kubeflow/pytorch-operator) support for Kubeflow training service
* Improvement on local code files uploading to OpenPAI HDFS
* Fixed OpenPAI integration WebUI bug: WebUI doesn't show latest trial job status, which is caused by OpenPAI token expiration
### NNICTL improvements #### New tuner supports
* Show version information both in nnictl and WebUI. You can run **nnictl -v** to show your current installed NNI version
### WebUI improvements * Support [network morphism](./Builtin_Tuner.md#NetworkMorphism) as a new tuner
* Enable modify concurrency number during experiment
* Add feedback link to NNI github 'create issue' page
* Enable customize top 10 trials regarding to metric numbers (largest or smallest)
* Enable download logs for dispatcher & nnimanager
* Enable automatic scaling of axes for metric number
* Update annotation to support displaying real choice in searchspace
## New examples #### Training Service improvements
* [FashionMnist](https://github.com/Microsoft/nni/tree/master/examples/trials/network_morphism), work together with network morphism tuner
* [Distributed MNIST example](https://github.com/Microsoft/nni/tree/master/examples/trials/mnist-distributed-pytorch) written in PyTorch
* Migrate [Kubeflow training service](https://github.com/Microsoft/nni/blob/master/docs/KubeflowMode.md)'s dependency from kubectl CLI to [Kubernetes API](https://kubernetes.io/docs/concepts/overview/kubernetes-api/) client
* [Pytorch-operator](https://github.com/kubeflow/pytorch-operator) support for Kubeflow training service
* Improvement on local code files uploading to OpenPAI HDFS
* Fixed OpenPAI integration WebUI bug: WebUI doesn't show latest trial job status, which is caused by OpenPAI token expiration
# Release 0.4 - 12/6/2018 #### NNICTL improvements
## Major Features * Show version information both in nnictl and WebUI. You can run **nnictl -v** to show your current installed NNI version
* [Kubeflow Training service](./KubeflowMode.md)
#### WebUI improvements
* Enable modify concurrency number during experiment
* Add feedback link to NNI github 'create issue' page
* Enable customize top 10 trials regarding to metric numbers (largest or smallest)
* Enable download logs for dispatcher & nnimanager
* Enable automatic scaling of axes for metric number
* Update annotation to support displaying real choice in searchspace
### New examples
* [FashionMnist](https://github.com/Microsoft/nni/tree/master/examples/trials/network_morphism), work together with network morphism tuner
* [Distributed MNIST example](https://github.com/Microsoft/nni/tree/master/examples/trials/mnist-distributed-pytorch) written in PyTorch
## Release 0.4 - 12/6/2018
### Major Features
* [Kubeflow Training service](./KubeflowMode.md)
* Support tf-operator * Support tf-operator
* [Distributed trial example](../examples/trials/mnist-distributed/dist_mnist.py) on Kubeflow * [Distributed trial example](https://github.com/Microsoft/nni/tree/master/examples/trials/mnist-distributed/dist_mnist.py) on Kubeflow
* [Grid search tuner](../src/sdk/pynni/nni/README.md#Grid) * [Grid search tuner](Builtin_Tuner.md#GridSearch)
* [Hyperband tuner](../src/sdk/pynni/nni/README.md#Hyperband) * [Hyperband tuner](Builtin_Tuner.md#Hyperband)
* Support launch NNI experiment on MAC * Support launch NNI experiment on MAC
* WebUI * WebUI
* UI support for hyperband tuner * UI support for hyperband tuner
* Remove tensorboard button * Remove tensorboard button
* Show experiment error message * Show experiment error message
...@@ -63,89 +90,112 @@ ...@@ -63,89 +90,112 @@
* Support search a specific trial by trial number * Support search a specific trial by trial number
* Show trial's hdfsLogPath * Show trial's hdfsLogPath
* Download experiment parameters * Download experiment parameters
## Others
* Asynchronous dispatcher ### Others
* Docker file update, add pytorch library
* Refactor 'nnictl stop' process, send SIGTERM to nni manager process, rather than calling stop Rest API. * Asynchronous dispatcher
* OpenPAI training service bug fix * Docker file update, add pytorch library
* Support NNI Manager IP configuration(nniManagerIp) in PAI cluster config file, to fix the issue that user’s machine has no eth0 device * Refactor 'nnictl stop' process, send SIGTERM to nni manager process, rather than calling stop Rest API.
* OpenPAI training service bug fix
* Support NNI Manager IP configuration(nniManagerIp) in OpenPAI cluster config file, to fix the issue that user’s machine has no eth0 device
* File number in codeDir is capped to 1000 now, to avoid user mistakenly fill root dir for codeDir * File number in codeDir is capped to 1000 now, to avoid user mistakenly fill root dir for codeDir
* Don’t print useless ‘metrics is empty’ log int PAI job’s stdout. Only print useful message once new metrics are recorded, to reduce confusion when user checks PAI trial’s output for debugging purpose * Don’t print useless ‘metrics is empty’ log in OpenPAI job’s stdout. Only print useful message once new metrics are recorded, to reduce confusion when user checks OpenPAI trial’s output for debugging purpose
* Add timestamp at the beginning of each log entry in trial keeper. * Add timestamp at the beginning of each log entry in trial keeper.
# Release 0.3.0 - 11/2/2018 ## Release 0.3.0 - 11/2/2018
## NNICTL new features and updates
### NNICTL new features and updates
* Support running multiple experiments simultaneously. * Support running multiple experiments simultaneously.
Before v0.3, NNI only supports running single experiment once a time. After this realse, users are able to run multiple experiments simultaneously. Each experiment will require a unique port, the 1st experiment will be set to the default port as previous versions. You can specify a unique port for the rest experiments as below: Before v0.3, NNI only supports running single experiment once a time. After this realse, users are able to run multiple experiments simultaneously. Each experiment will require a unique port, the 1st experiment will be set to the default port as previous versions. You can specify a unique port for the rest experiments as below:
```nnictl create --port 8081 --config <config file path>``` ```bash
nnictl create --port 8081 --config <config file path>
```
* Support updating max trial number. * Support updating max trial number.
use ```nnictl update --help``` to learn more. Or refer to [NNICTL Spec](https://github.com/Microsoft/nni/blob/master/docs/NNICTLDOC.md) for the fully usage of NNICTL. use `nnictl update --help` to learn more. Or refer to [NNICTL Spec](https://github.com/Microsoft/nni/blob/master/docs/NNICTLDOC.md) for the fully usage of NNICTL.
### API new features and updates
## API new features and updates
* <span style="color:red">**breaking change**</span>: nn.get_parameters() is refactored to nni.get_next_parameter. All examples of prior releases can not run on v0.3, please clone nni repo to get new examples. If you had applied NNI to your own codes, please update the API accordingly. * <span style="color:red">**breaking change**</span>: nn.get_parameters() is refactored to nni.get_next_parameter. All examples of prior releases can not run on v0.3, please clone nni repo to get new examples. If you had applied NNI to your own codes, please update the API accordingly.
* New API **nni.get_sequence_id()**. * New API **nni.get_sequence_id()**.
Each trial job is allocated a unique sequence number, which can be retrieved by nni.get_sequence_id() API. Each trial job is allocated a unique sequence number, which can be retrieved by nni.get_sequence_id() API.
```git clone -b v0.3 https://github.com/Microsoft/nni.git``` ```bash
git clone -b v0.3 https://github.com/Microsoft/nni.git
```
* **nni.report_final_result(result)** API supports more data types for result parameter. * **nni.report_final_result(result)** API supports more data types for result parameter.
It can be of following types: It can be of following types:
* int * int
* float * float
* A python dict containing 'default' key, the value of 'default' key should be of type int or float. The dict can contain any other key value pairs. * A python dict containing 'default' key, the value of 'default' key should be of type int or float. The dict can contain any other key value pairs.
## New tuner support ### New tuner support
* **Batch Tuner** which iterates all parameter combination, can be used to submit batch trial jobs. * **Batch Tuner** which iterates all parameter combination, can be used to submit batch trial jobs.
## New examples ### New examples
* A NNI Docker image for public usage: * A NNI Docker image for public usage:
```docker pull msranni/nni:latest```
```bash
docker pull msranni/nni:latest
```
* New trial example: [NNI Sklearn Example](https://github.com/Microsoft/nni/tree/master/examples/trials/sklearn) * New trial example: [NNI Sklearn Example](https://github.com/Microsoft/nni/tree/master/examples/trials/sklearn)
* New competition example: [Kaggle Competition TGS Salt Example](https://github.com/Microsoft/nni/tree/master/examples/trials/kaggle-tgs-salt) * New competition example: [Kaggle Competition TGS Salt Example](https://github.com/Microsoft/nni/tree/master/examples/trials/kaggle-tgs-salt)
## Others ### Others
* UI refactoring, refer to [WebUI doc](WebUI.md) for how to work with the new UI. * UI refactoring, refer to [WebUI doc](WebUI.md) for how to work with the new UI.
* Continuous Integration: NNI had switched to Azure pipelines * Continuous Integration: NNI had switched to Azure pipelines
* [Known Issues in release 0.3.0](https://github.com/Microsoft/nni/labels/nni030knownissues). * [Known Issues in release 0.3.0](https://github.com/Microsoft/nni/labels/nni030knownissues).
## Release 0.2.0 - 9/29/2018
# Release 0.2.0 - 9/29/2018 ### Major Features
## Major Features
* Support [OpenPAI](https://github.com/Microsoft/pai) (aka pai) Training Service (See [here](./PAIMode.md) for instructions about how to submit NNI job in pai mode) * Support [OpenPAI](https://github.com/Microsoft/pai) Training Platform (See [here](./PAIMode.md) for instructions about how to submit NNI job in pai mode)
* Support training services on pai mode. NNI trials will be scheduled to run on OpenPAI cluster * Support training services on pai mode. NNI trials will be scheduled to run on OpenPAI cluster
* NNI trial's output (including logs and model file) will be copied to OpenPAI HDFS for further debugging and checking * NNI trial's output (including logs and model file) will be copied to OpenPAI HDFS for further debugging and checking
* Support [SMAC](https://www.cs.ubc.ca/~hutter/papers/10-TR-SMAC.pdf) tuner (See [here](HowToChooseTuner.md) for instructions about how to use SMAC tuner) * Support [SMAC](https://www.cs.ubc.ca/~hutter/papers/10-TR-SMAC.pdf) tuner (See [here](Builtin_Tuner.md) for instructions about how to use SMAC tuner)
* [SMAC](https://www.cs.ubc.ca/~hutter/papers/10-TR-SMAC.pdf) is based on Sequential Model-Based Optimization (SMBO). It adapts the most prominent previously used model class (Gaussian stochastic process models) and introduces the model class of random forests to SMBO to handle categorical parameters. The SMAC supported by NNI is a wrapper on [SMAC3](https://github.com/automl/SMAC3) * [SMAC](https://www.cs.ubc.ca/~hutter/papers/10-TR-SMAC.pdf) is based on Sequential Model-Based Optimization (SMBO). It adapts the most prominent previously used model class (Gaussian stochastic process models) and introduces the model class of random forests to SMBO to handle categorical parameters. The SMAC supported by NNI is a wrapper on [SMAC3](https://github.com/automl/SMAC3)
* Support NNI installation on [conda](https://conda.io/docs/index.html) and python virtual environment * Support NNI installation on [conda](https://conda.io/docs/index.html) and python virtual environment
* Others * Others
* Update ga squad example and related documentation * Update ga squad example and related documentation
* WebUI UX small enhancement and bug fix * WebUI UX small enhancement and bug fix
## Known Issues ### Known Issues
[Known Issues in release 0.2.0](https://github.com/Microsoft/nni/labels/nni020knownissues). [Known Issues in release 0.2.0](https://github.com/Microsoft/nni/labels/nni020knownissues).
# Release 0.1.0 - 9/10/2018 (initial release) ## Release 0.1.0 - 9/10/2018 (initial release)
Initial release of Neural Network Intelligence (NNI). Initial release of Neural Network Intelligence (NNI).
## Major Features ### Major Features
* Installation and Deployment
* Installation and Deployment
* Support pip install and source codes install * Support pip install and source codes install
* Support training services on local mode(including Multi-GPU mode) as well as multi-machines mode * Support training services on local mode(including Multi-GPU mode) as well as multi-machines mode
* Tuners, Assessors and Trial * Tuners, Assessors and Trial
* Support AutoML algorithms including: hyperopt_tpe, hyperopt_annealing, hyperopt_random, and evolution_tuner * Support AutoML algorithms including: hyperopt_tpe, hyperopt_annealing, hyperopt_random, and evolution_tuner
* Support assessor(early stop) algorithms including: medianstop algorithm * Support assessor(early stop) algorithms including: medianstop algorithm
* Provide Python API for user defined tuners and assessors * Provide Python API for user defined tuners and assessors
* Provide Python API for user to wrap trial code as NNI deployable codes * Provide Python API for user to wrap trial code as NNI deployable codes
* Experiments * Experiments
* Provide a command line toolkit 'nnictl' for experiments management * Provide a command line toolkit 'nnictl' for experiments management
* Provide a WebUI for viewing experiments details and managing experiments * Provide a WebUI for viewing experiments details and managing experiments
* Continuous Integration * Continuous Integration
* Support CI by providing out-of-box integration with [travis-ci](https://github.com/travis-ci) on ubuntu * Support CI by providing out-of-box integration with [travis-ci](https://github.com/travis-ci) on ubuntu
* Others * Others
* Support simple GPU job scheduling * Support simple GPU job scheduling
## Known Issues ### Known Issues
[Known Issues in release 0.1.0](https://github.com/Microsoft/nni/labels/nni010knownissues). [Known Issues in release 0.1.0](https://github.com/Microsoft/nni/labels/nni010knownissues).
docs/Reference.rst 0 → 100644
View file @ 035d58bc
References
==================
.. toctree::
:maxdepth: 3
Command Line <NNICTLDOC>
Annotation <AnnotationSpec>
Configuration<ExperimentConfig>
Search Space <SearchSpaceSpec>
\ No newline at end of file
docs/RemoteMachineMode.md
View file @ 035d58bc
**Run an Experiment on Multiple Machines** # Run an Experiment on Multiple Machines
===
NNI supports running an experiment on multiple machines through SSH channel, called `remote` mode. NNI assumes that you have access to those machines, and already setup the environment for running deep learning training code. NNI supports running an experiment on multiple machines through SSH channel, called `remote` mode. NNI assumes that you have access to those machines, and already setup the environment for running deep learning training code.
...@@ -14,15 +12,7 @@ e.g. Three machines and you login in with account `bob` (Note: the account is no ...@@ -14,15 +12,7 @@ e.g. Three machines and you login in with account `bob` (Note: the account is no
## Setup NNI environment ## Setup NNI environment
Install NNI on each of your machines following the install guide [here](GetStarted.md). Install NNI on each of your machines following the install guide [here](QuickStart.md).
For remote machines that are used only to run trials but not the nnictl, you can just install python SDK:
* __Install python SDK through pip__
```bash
python3 -m pip install --user --upgrade nni-sdk
```
## Run an experiment ## Run an experiment
......
docs/SQuAD_evolution_examples.md 0 → 100644
View file @ 035d58bc
# Automatic Model Architecture Search for Reading Comprehension
This example shows us how to use Genetic Algorithm to find good model architectures for Reading Comprehension.
## 1. Search Space
Since attention and recurrent neural network (RNN) have been proven effective in Reading Comprehension.
We conclude the search space as follow:
1. IDENTITY (Effectively means keep training).
2. INSERT-RNN-LAYER (Inserts a LSTM. Comparing the performance of GRU and LSTM in our experiment, we decided to use LSTM here.)
3. REMOVE-RNN-LAYER
4. INSERT-ATTENTION-LAYER(Inserts an attention layer.)
5. REMOVE-ATTENTION-LAYER
6. ADD-SKIP (Identity between random layers).
7. REMOVE-SKIP (Removes random skip).
![](https://github.com/Microsoft/nni/tree/master/examples/trials/ga_squad/ga_squad.png)
### New version
Also we have another version which time cost is less and performance is better. We will release soon.
## 2. How to run this example in local?
### 2.1 Use downloading script to download data
Execute the following command to download needed files
using the downloading script:
```bash
chmod +x ./download.sh
./download.sh
```
Or Download manually
1. download "dev-v1.1.json" and "train-v1.1.json" in https://rajpurkar.github.io/SQuAD-explorer/
```bash
wget https://rajpurkar.github.io/SQuAD-explorer/dataset/train-v1.1.json
wget https://rajpurkar.github.io/SQuAD-explorer/dataset/dev-v1.1.json
```
2. download "glove.840B.300d.txt" in https://nlp.stanford.edu/projects/glove/
```bash
wget http://nlp.stanford.edu/data/glove.840B.300d.zip
unzip glove.840B.300d.zip
```
### 2.2 Update configuration
Modify `nni/examples/trials/ga_squad/config.yml`, here is the default configuration:
```yaml
authorName: default
experimentName: example_ga_squad
trialConcurrency: 1
maxExecDuration: 1h
maxTrialNum: 1
#choice: local, remote
trainingServicePlatform: local
#choice: true, false
useAnnotation: false
tuner:
codeDir: ~/nni/examples/tuners/ga_customer_tuner
classFileName: customer_tuner.py
className: CustomerTuner
classArgs:
optimize_mode: maximize
trial:
command: python3 trial.py
codeDir: ~/nni/examples/trials/ga_squad
gpuNum: 0
```
In the "trial" part, if you want to use GPU to perform the architecture search, change `gpuNum` from `0` to `1`. You need to increase the `maxTrialNum` and `maxExecDuration`, according to how long you want to wait for the search result.
### 2.3 submit this job
```bash
nnictl create --config ~/nni/examples/trials/ga_squad/config.yml
```
## 3 Run this example on OpenPAI
Due to the memory limitation of upload, we only upload the source code and complete the data download and training on OpenPAI. This experiment requires sufficient memory that `memoryMB >= 32G`, and the training may last for several hours.
### 3.1 Update configuration
Modify `nni/examples/trials/ga_squad/config_pai.yml`, here is the default configuration:
```yaml
authorName: default
experimentName: example_ga_squad
trialConcurrency: 1
maxExecDuration: 1h
maxTrialNum: 10
#choice: local, remote, pai
trainingServicePlatform: pai
#choice: true, false
useAnnotation: false
#Your nni_manager ip
nniManagerIp: 10.10.10.10
tuner:
codeDir: https://github.com/Microsoft/nni/tree/master/examples/tuners/ga_customer_tuner
classFileName: customer_tuner.py
className: CustomerTuner
classArgs:
optimize_mode: maximize
trial:
command: chmod +x ./download.sh && ./download.sh && python3 trial.py
codeDir: .
gpuNum: 0
cpuNum: 1
memoryMB: 32869
#The docker image to run nni job on OpenPAI
image: msranni/nni:latest
#The hdfs directory to store data on OpenPAI, format 'hdfs://host:port/directory'
dataDir: hdfs://10.10.10.10:9000/username/nni
#The hdfs directory to store output data generated by nni, format 'hdfs://host:port/directory'
outputDir: hdfs://10.10.10.10:9000/username/nni
paiConfig:
#The username to login OpenPAI
userName: username
#The password to login OpenPAI
passWord: password
#The host of restful server of OpenPAI
host: 10.10.10.10
```
Please change the default value to your personal account and machine information. Including `nniManagerIp`, `dataDir`, `outputDir`, `userName`, `passWord` and `host`.
In the "trial" part, if you want to use GPU to perform the architecture search, change `gpuNum` from `0` to `1`. You need to increase the `maxTrialNum` and `maxExecDuration`, according to how long you want to wait for the search result.
`trialConcurrency` is the number of trials running concurrently, which is the number of GPUs you want to use, if you are setting `gpuNum` to 1.
### 3.2 submit this job
```bash
nnictl create --config ~/nni/examples/trials/ga_squad/config_pai.yml
```
## 4. Technical details about the trial
### 4.1 How does it works
The evolution-algorithm based architecture for question answering has two different parts just like any other examples: the trial and the tuner.
### 4.2 The trial
The trial has a lot of different files, functions and classes. Here we will only give most of those files a brief introduction:
* `attention.py` contains an implementation for attention mechanism in Tensorflow.
* `data.py` contains functions for data preprocessing.
* `evaluate.py` contains the evaluation script.
* `graph.py` contains the definition of the computation graph.
* `rnn.py` contains an implementation for GRU in Tensorflow.
* `train_model.py` is a wrapper for the whole question answering model.
Among those files, `trial.py` and `graph_to_tf.py` are special.
`graph_to_tf.py` has a function named as `graph_to_network`, here is its skeleton code:
```python
def graph_to_network(input1,
input2,
input1_lengths,
input2_lengths,
graph,
dropout_rate,
is_training,
num_heads=1,
rnn_units=256):
topology = graph.is_topology()
layers = dict()
layers_sequence_lengths = dict()
num_units = input1.get_shape().as_list()[-1]
layers[0] = input1*tf.sqrt(tf.cast(num_units, tf.float32)) + \
positional_encoding(input1, scale=False, zero_pad=False)
layers[1] = input2*tf.sqrt(tf.cast(num_units, tf.float32))
layers[0] = dropout(layers[0], dropout_rate, is_training)
layers[1] = dropout(layers[1], dropout_rate, is_training)
layers_sequence_lengths[0] = input1_lengths
layers_sequence_lengths[1] = input2_lengths
for _, topo_i in enumerate(topology):
if topo_i == '|':
continue
if graph.layers[topo_i].graph_type == LayerType.input.value:
# ......
elif graph.layers[topo_i].graph_type == LayerType.attention.value:
# ......
# More layers to handle
```
As we can see, this function is actually a compiler, that converts the internal model DAG configuration (which will be introduced in the `Model configuration format` section) `graph`, to a Tensorflow computation graph.
```python
topology = graph.is_topology()
```
performs topological sorting on the internal graph representation, and the code inside the loop:
```python
for _, topo_i in enumerate(topology):
```
performs actually conversion that maps each layer to a part in Tensorflow computation graph.
### 4.3 The tuner
The tuner is much more simple than the trial. They actually share the same `graph.py`. Besides, the tuner has a `customer_tuner.py`, the most important class in which is `CustomerTuner`:
```python
class CustomerTuner(Tuner):
# ......
def generate_parameters(self, parameter_id):
"""Returns a set of trial graph config, as a serializable object.
parameter_id : int
"""
if len(self.population) <= 0:
logger.debug("the len of poplution lower than zero.")
raise Exception('The population is empty')
pos = -1
for i in range(len(self.population)):
if self.population[i].result == None:
pos = i
break
if pos != -1:
indiv = copy.deepcopy(self.population[pos])
self.population.pop(pos)
temp = json.loads(graph_dumps(indiv.config))
else:
random.shuffle(self.population)
if self.population[0].result > self.population[1].result:
self.population[0] = self.population[1]
indiv = copy.deepcopy(self.population[0])
self.population.pop(1)
indiv.mutation()
graph = indiv.config
temp = json.loads(graph_dumps(graph))
# ......
```
As we can see, the overloaded method `generate_parameters` implements a pretty naive mutation algorithm. The code lines:
```python
if self.population[0].result > self.population[1].result:
self.population[0] = self.population[1]
indiv = copy.deepcopy(self.population[0])
```
controls the mutation process. It will always take two random individuals in the population, only keeping and mutating the one with better result.
### 4.4 Model configuration format
Here is an example of the model configuration, which is passed from the tuner to the trial in the architecture search procedure.
```json
{
"max_layer_num": 50,
"layers": [
{
"input_size": 0,
"type": 3,
"output_size": 1,
"input": [],
"size": "x",
"output": [4, 5],
"is_delete": false
},
{
"input_size": 0,
"type": 3,
"output_size": 1,
"input": [],
"size": "y",
"output": [4, 5],
"is_delete": false
},
{
"input_size": 1,
"type": 4,
"output_size": 0,
"input": [6],
"size": "x",
"output": [],
"is_delete": false
},
{
"input_size": 1,
"type": 4,
"output_size": 0,
"input": [5],
"size": "y",
"output": [],
"is_delete": false
},
{"Comment": "More layers will be here for actual graphs."}
]
}
```
Every model configuration will have a "layers" section, which is a JSON list of layer definitions. The definition of each layer is also a JSON object, where:
* `type` is the type of the layer. 0, 1, 2, 3, 4 corresponds to attention, self-attention, RNN, input and output layer respectively.
* `size` is the length of the output. "x", "y" correspond to document length / question length, respectively.
* `input_size` is the number of inputs the layer has.
* `input` is the indices of layers taken as input of this layer.
* `output` is the indices of layers use this layer's output as their input.
* `is_delete` means whether the layer is still available.
docs/SearchSpaceSpec.md
View file @ 035d58bc
## How to define search space? # Search Space
### Hyper-parameter Search Space ## Overview
* A search space configure example as follow: In NNI, tuner will sample parameters/architecture according to the search space, which is defined as a json file.
```python To define a search space, users should define the name of variable, the type of sampling strategy and its parameters.
* A example of search space definition as follow:
```yaml
{ {
"dropout_rate":{"_type":"uniform","_value":[0.1,0.5]}, "dropout_rate":{"_type":"uniform","_value":[0.1,0.5]},
"conv_size":{"_type":"choice","_value":[2,3,5,7]}, "conv_size":{"_type":"choice","_value":[2,3,5,7]},
...@@ -15,60 +19,65 @@ ...@@ -15,60 +19,65 @@
``` ```
The example define `dropout_rate` as variable which priori distribution is uniform distribution, and its value from `0.1` and `0.5`. Take the first line as an example. `dropout_rate` is defined as a variable whose priori distribution is a uniform distribution of a range from `0.1` and `0.5`.
The tuner will sample parameters/architecture by understanding the search space first.
## Types
User should define the name of variable, type and candidate value of variable. All types of sampling strategies and their parameter are listed here:
The candidate type and value for variable is here:
* {"_type":"choice","_value":options} * {"_type":"choice","_value":options}
* Which means the variable value is one of the options, which should be a list The elements of options can themselves be [nested] stochastic expressions. In this case, the stochastic choices that only appear in some of the options become conditional parameters. * Which means the variable value is one of the options, which should be a list. The elements of options can themselves be [nested] stochastic expressions. In this case, the stochastic choices that only appear in some of the options become conditional parameters.
<br/>
* {"_type":"randint","_value":[upper]} * {"_type":"randint","_value":[upper]}
* Which means the variable value is a random integer in the range [0, upper). The semantics of this distribution is that there is no more correlation in the loss function between nearby integer values, as compared with more distant integer values. This is an appropriate distribution for describing random seeds for example. If the loss function is probably more correlated for nearby integer values, then you should probably use one of the "quantized" continuous distributions, such as either quniform, qloguniform, qnormal or qlognormal. Note that if you want to change lower bound, you can use `quniform` for now. * Which means the variable value is a random integer in the range [0, upper). The semantics of this distribution is that there is no more correlation in the loss function between nearby integer values, as compared with more distant integer values. This is an appropriate distribution for describing random seeds for example. If the loss function is probably more correlated for nearby integer values, then you should probably use one of the "quantized" continuous distributions, such as either quniform, qloguniform, qnormal or qlognormal. Note that if you want to change lower bound, you can use `quniform` for now.
<br/>
* {"_type":"uniform","_value":[low, high]} * {"_type":"uniform","_value":[low, high]}
* Which means the variable value is a value uniformly between low and high. * Which means the variable value is a value uniformly between low and high.
* When optimizing, this variable is constrained to a two-sided interval. * When optimizing, this variable is constrained to a two-sided interval.
<br/>
* {"_type":"quniform","_value":[low, high, q]} * {"_type":"quniform","_value":[low, high, q]}
* Which means the variable value is a value like round(uniform(low, high) / q) * q * Which means the variable value is a value like round(uniform(low, high) / q) * q
* Suitable for a discrete value with respect to which the objective is still somewhat "smooth", but which should be bounded both above and below. If you want to uniformly choose integer from a range [low, high], you can write `_value` like this: `[low, high, 1]`. * Suitable for a discrete value with respect to which the objective is still somewhat "smooth", but which should be bounded both above and below. If you want to uniformly choose integer from a range [low, high], you can write `_value` like this: `[low, high, 1]`.
<br/>
* {"_type":"loguniform","_value":[low, high]} * {"_type":"loguniform","_value":[low, high]}
* Which means the variable value is a value drawn from a range [low, high] according to a loguniform distribution like exp(uniform(log(low), log(high))), so that the logarithm of the return value is uniformly distributed. * Which means the variable value is a value drawn from a range [low, high] according to a loguniform distribution like exp(uniform(log(low), log(high))), so that the logarithm of the return value is uniformly distributed.
* When optimizing, this variable is constrained to be positive. * When optimizing, this variable is constrained to be positive.
<br/>
* {"_type":"qloguniform","_value":[low, high, q]} * {"_type":"qloguniform","_value":[low, high, q]}
* Which means the variable value is a value like round(loguniform(low, high)) / q) * q * Which means the variable value is a value like round(loguniform(low, high)) / q) * q
* Suitable for a discrete variable with respect to which the objective is "smooth" and gets smoother with the size of the value, but which should be bounded both above and below. * Suitable for a discrete variable with respect to which the objective is "smooth" and gets smoother with the size of the value, but which should be bounded both above and below.
<br/>
* {"_type":"normal","_value":[label, mu, sigma]} * {"_type":"normal","_value":[label, mu, sigma]}
* Which means the variable value is a real value that's normally-distributed with mean mu and standard deviation sigma. When optimizing, this is an unconstrained variable. * Which means the variable value is a real value that's normally-distributed with mean mu and standard deviation sigma. When optimizing, this is an unconstrained variable.
<br/>
* {"_type":"qnormal","_value":[label, mu, sigma, q]} * {"_type":"qnormal","_value":[label, mu, sigma, q]}
* Which means the variable value is a value like round(normal(mu, sigma) / q) * q * Which means the variable value is a value like round(normal(mu, sigma) / q) * q
* Suitable for a discrete variable that probably takes a value around mu, but is fundamentally unbounded. * Suitable for a discrete variable that probably takes a value around mu, but is fundamentally unbounded.
<br/>
* {"_type":"lognormal","_value":[label, mu, sigma]} * {"_type":"lognormal","_value":[label, mu, sigma]}
* Which means the variable value is a value drawn according to exp(normal(mu, sigma)) so that the logarithm of the return value is normally distributed. When optimizing, this variable is constrained to be positive. * Which means the variable value is a value drawn according to exp(normal(mu, sigma)) so that the logarithm of the return value is normally distributed. When optimizing, this variable is constrained to be positive.
<br/>
* {"_type":"qlognormal","_value":[label, mu, sigma, q]} * {"_type":"qlognormal","_value":[label, mu, sigma, q]}
* Which means the variable value is a value like round(exp(normal(mu, sigma)) / q) * q * Which means the variable value is a value like round(exp(normal(mu, sigma)) / q) * q
* Suitable for a discrete variable with respect to which the objective is smooth and gets smoother with the size of the variable, which is bounded from one side. * Suitable for a discrete variable with respect to which the objective is smooth and gets smoother with the size of the variable, which is bounded from one side.
<br/>
Note that SMAC only supports a subset of the types above, including `choice`, `randint`, `uniform`, `loguniform`, `quniform(q=1)`. In the current version, SMAC does not support cascaded search space (i.e., conditional variable in SMAC). ## Search Space Types Supported by Each Tuner
| | choice | randint | uniform | quniform | loguniform | qloguniform | normal | qnormal | lognormal | qlognormal |
|:------:|:------:|:------:|:------:|:------:|:------:|:------:|:------:|:------:|:------:|:------:|
| TPE Tuner | &#10003; | &#10003; | &#10003; | &#10003; | &#10003; | &#10003; | &#10003; | &#10003; | &#10003; | &#10003; |
| Random Search Tuner| &#10003; | &#10003; | &#10003; | &#10003; | &#10003; | &#10003; | &#10003; | &#10003; | &#10003; | &#10003; |
| Anneal Tuner | &#10003; | &#10003; | &#10003; | &#10003; | &#10003; | &#10003; | &#10003; | &#10003; | &#10003; | &#10003; |
| Evolution Tuner | &#10003; | &#10003; | &#10003; | &#10003; | &#10003; | &#10003; | &#10003; | &#10003; | &#10003; | &#10003; |
| SMAC Tuner | &#10003; | &#10003; | &#10003; | &#10003; | &#10003; | | | | | |
| Batch Tuner | &#10003; | | | | | | | | | |
| Grid Search Tuner | &#10003; | | | &#10003; | | &#10003; | | | | |
| Hyperband Advisor | &#10003; | &#10003; | &#10003; | &#10003; | &#10003; | &#10003; | &#10003; | &#10003; | &#10003; | &#10003; |
| Metis Tuner | &#10003; | &#10003; | &#10003; | &#10003; | | | | | | |
Note that In Grid Search Tuner, for users' convenience, the definition of `quniform` and `qloguniform` change, where q here specifies the number of values that will be sampled. Details about them are listed as follows
Note that GridSearch Tuner only supports a subset of the types above, including `choice`, `quniform` and `qloguniform`, where q here specifies the number of values that will be sampled. Details about the last two type as follows
* Type 'quniform' will receive three values [low, high, q], where [low, high] specifies a range and 'q' specifies the number of values that will be sampled evenly. Note that q should be at least 2. It will be sampled in a way that the first sampled value is 'low', and each of the following values is (high-low)/q larger that the value in front of it. * Type 'quniform' will receive three values [low, high, q], where [low, high] specifies a range and 'q' specifies the number of values that will be sampled evenly. Note that q should be at least 2. It will be sampled in a way that the first sampled value is 'low', and each of the following values is (high-low)/q larger that the value in front of it.
* Type 'qloguniform' behaves like 'quniform' except that it will first change the range to [log(low), log(high)] and sample and then change the sampled value back. * Type 'qloguniform' behaves like 'quniform' except that it will first change the range to [log(low), log(high)] and sample and then change the sampled value back.
Note that Metis Tuner only support numerical `choice` now
docs/StartExperiment.md deleted 100644 → 0
View file @ b633c265
How to start an experiment
===
## 1.Introduce
There are few steps to start an new experiment of nni, here are the process.
<img src="./img/experiment_process.jpg" width="50%" height="50%" />
## 2.Details
### 2.1 Check environment
1. Check if there is an old experiment running
2. Check if the port of restfurl server is free.
3. Validate the content of config yaml file.
4. Prepare a config file to to record the information of this experiment.
### 2.2 Start restful server
Start an restful server process to manage nni experiment, the default port is 8080.
### 2.3 Check restful server
Check whether restful server process is successfully started and could get a response when send message to restful server.
### 2.4 Set experiment config
Call restful server to set experiment config before starting an experiment, experiment config includes the config values in config yaml file.
### 2.5 Check experiment cofig
Check the response content of restful server, if the status code of response is 200, the config is successfully set.
### 2.6 Start Experiment
Call restful server process to setup an experiment.
### 2.7 Check experiment
1. Check the response of restful server.
2. Handle error information.
3. Print success or error information to screen.
4. Save configuration information to config file of nnictl.
docs/howto_1_WriteTrial.md → docs/Trials.md
View file @ 035d58bc
**Write a Trial Run on NNI** # Write a Trial Run on NNI
===
A **Trial** in NNI is an individual attempt at applying a set of parameters on a model. A **Trial** in NNI is an individual attempt at applying a configuration (e.g., a set of hyper-parameters) on a model.
To define a NNI trial, you need to firstly define the set of parameters and then update the model. NNI provide two approaches for you to define a trial: `NNI API` and `NNI Python annotation`. To define an NNI trial, you need to firstly define the set of parameters (i.e., search space) and then update the model. NNI provide two approaches for you to define a trial: [NNI API](#nni-api) and [NNI Python annotation](#nni-annotation). You could also refer to [here](#more-examples) for more trial examples.
<a name="nni-api"></a>
## NNI API ## NNI API
>Step 1 - Prepare a SearchSpace parameters file. ### Step 1 - Prepare a SearchSpace parameters file.
An example is shown below: An example is shown below:
...@@ -20,61 +20,55 @@ An example is shown below: ...@@ -20,61 +20,55 @@ An example is shown below:
} }
``` ```
Refer to [SearchSpaceSpec.md](./SearchSpaceSpec.md) to learn more about search space. Refer to [SearchSpaceSpec.md](./SearchSpaceSpec.md) to learn more about search space. Tuner will generate configurations from this search space, that is, choosing a value for each hyperparameter from the range.
>Step 2 - Update model codes ### Step 2 - Update model codes
~~~~ - Import NNI
2.1 Declare NNI API
Include `import nni` in your trial code to use NNI APIs.
2.2 Get predefined parameters
Use the following code snippet:
RECEIVED_PARAMS = nni.get_next_parameter()
to get hyper-parameters' values assigned by tuner. `RECEIVED_PARAMS` is an object, for example:
{"conv_size": 2, "hidden_size": 124, "learning_rate": 0.0307, "dropout_rate": 0.2029}
2.3 Report NNI results Include `import nni` in your trial code to use NNI APIs.
Use the API:
`nni.report_intermediate_result(accuracy)`
to send `accuracy` to assessor. - Get configuration from Tuner
Use the API: ```python
RECEIVED_PARAMS = nni.get_next_parameter()
```
`RECEIVED_PARAMS` is an object, for example:
`{"conv_size": 2, "hidden_size": 124, "learning_rate": 0.0307, "dropout_rate": 0.2029}`.
`nni.report_final_result(accuracy)` - Report metric data periodically (optional)
to send `accuracy` to tuner. ```python
~~~~ nni.report_intermediate_result(metrics)
```
`metrics` could be any python object. If users use NNI built-in tuner/assessor, `metrics` can only have two formats: 1) a number e.g., float, int, 2) a dict object that has a key named `default` whose value is a number. This `metrics` is reported to [assessor](Builtin_Assessors.md). Usually, `metrics` could be periodically evaluated loss or accuracy.
**NOTE**: - Report performance of the configuration
~~~~ ```python
accuracy - The `accuracy` could be any python object, but if you use NNI built-in tuner/assessor, `accuracy` should be a numerical variable (e.g. float, int). nni.report_final_result(metrics)
assessor - The assessor will decide which trial should early stop based on the history performance of trial (intermediate result of one trial). ```
tuner - The tuner will generate next parameters/architecture based on the explore history (final result of all trials). `metrics` also could be any python object. If users use NNI built-in tuner/assessor, `metrics` follows the same format rule as that in `report_intermediate_result`, the number indicates the model's performance, for example, the model's accuracy, loss etc. This `metrics` is reported to [tuner](Builtin_Tuner.md).
~~~~
>Step 3 - Enable NNI API ### Step 3 - Enable NNI API
To enable NNI API mode, you need to set useAnnotation to *false* and provide the path of SearchSpace file (you just defined in step 1): To enable NNI API mode, you need to set useAnnotation to *false* and provide the path of SearchSpace file (you just defined in step 1):
``` ```yaml
useAnnotation: false useAnnotation: false
searchSpacePath: /path/to/your/search_space.json searchSpacePath: /path/to/your/search_space.json
``` ```
You can refer to [here](./ExperimentConfig.md) for more information about how to set up experiment configurations. You can refer to [here](ExperimentConfig.md) for more information about how to set up experiment configurations.
You can refer to [here](../examples/trials/README.md) for more information about how to write trial code using NNI APIs. *Please refer to [here](sdk_reference.md) for more APIs (e.g., `nni.get_sequence_id()`) provided by NNI.
<a name="nni-annotation"></a>
## NNI Python Annotation ## NNI Python Annotation
An alternative to write a trial is to use NNI's syntax for python. Simple as any annotation, NNI annotation is working like comments in your codes. You don't have to make structure or any other big changes to your existing codes. With a few lines of NNI annotation, you will be able to: An alternative to writing a trial is to use NNI's syntax for python. Simple as any annotation, NNI annotation is working like comments in your codes. You don't have to make structure or any other big changes to your existing codes. With a few lines of NNI annotation, you will be able to:
* annotate the variables you want to tune * annotate the variables you want to tune
* specify in which range you want to tune the variables * specify in which range you want to tune the variables
* annotate which variable you want to report as intermediate result to `assessor` * annotate which variable you want to report as intermediate result to `assessor`
...@@ -82,11 +76,14 @@ An alternative to write a trial is to use NNI's syntax for python. Simple as any ...@@ -82,11 +76,14 @@ An alternative to write a trial is to use NNI's syntax for python. Simple as any
Again, take MNIST as an example, it only requires 2 steps to write a trial with NNI Annotation. Again, take MNIST as an example, it only requires 2 steps to write a trial with NNI Annotation.
>Step 1 - Update codes with annotations ### Step 1 - Update codes with annotations
Please refer the following tensorflow code snippet for NNI Annotation, the highlighted 4 lines are annotations that help you to: (1) tune batch\_size and (2) dropout\_rate, (3) report test\_acc every 100 steps, and (4) at last report test\_acc as final result. The following is a tensorflow code snippet for NNI Annotation, where the highlighted four lines are annotations that help you to:
1. tune batch\_size and dropout\_rate
2. report test\_acc every 100 steps
3. at last report test\_acc as final result.
>What noteworthy is: as these new added codes are annotations, it does not actually change your previous codes logic, you can still run your code as usual in environments without NNI installed. What noteworthy is: as these newly added codes are annotations, it does not actually change your previous codes logic, you can still run your code as usual in environments without NNI installed.
```diff ```diff
with tf.Session() as sess: with tf.Session() as sess:
...@@ -114,21 +111,25 @@ with tf.Session() as sess: ...@@ -114,21 +111,25 @@ with tf.Session() as sess:
+ """@nni.report_final_result(test_acc)""" + """@nni.report_final_result(test_acc)"""
``` ```
>NOTE **NOTE**:
>>`@nni.variable` will take effect on its following line - `@nni.variable` will take effect on its following line, which is an assignment statement whose leftvalue must be specified by the keyword `name` in `@nni.variable`.
>> - `@nni.report_intermediate_result`/`@nni.report_final_result` will send the data to assessor/tuner at that line.
>>`@nni.report_intermediate_result`/`@nni.report_final_result` will send the data to assessor/tuner at that line.
>>
>>Please refer to [Annotation README](../tools/nni_annotation/README.md) for more information about annotation syntax and its usage.
For more information about annotation syntax and its usage, please refer to [Annotation](AnnotationSpec.md).
>Step 2 - Enable NNI Annotation
In the yaml configure file, you need to set *useAnnotation* to true to enable NNI annotation:
```yaml ### Step 2 - Enable NNI Annotation
In the YAML configure file, you need to set *useAnnotation* to true to enable NNI annotation:
```
useAnnotation: true useAnnotation: true
``` ```
## More Trial Example <a name="more-examples"></a>
## More Trial Examples
* [Automatic Model Architecture Search for Reading Comprehension.](../examples/trials/ga_squad/README.md) * [MNIST examples](mnist_examples.md)
* [Finding out best optimizer for Cifar10 classification](cifar10_examples.md)
* [How to tune Scikit-learn on NNI](sklearn_examples.md)
* [Automatic Model Architecture Search for Reading Comprehension.](SQuAD_evolution_examples.md)
* [Tuning GBDT on NNI](gbdt_example.md)
docs/Tutorials.rst 0 → 100644
View file @ 035d58bc
######################
Tutorials
######################
.. toctree::
Installation
Write Trial<Trials>
Tuners<tuners>
Assessors<assessors>
WebUI
Training Platform<training_services>
advanced
\ No newline at end of file
docs/WebUI.md
View file @ 035d58bc
...@@ -41,11 +41,10 @@ Click the tab "Trials Detail" to see the status of the all trials. Specifically: ...@@ -41,11 +41,10 @@ Click the tab "Trials Detail" to see the status of the all trials. Specifically:
![](./img/webui-img/detail-local.png) ![](./img/webui-img/detail-local.png)
* If you run a pai or kubeflow experiment, you can also see the hdfsLog. * If you run on OpenPAI or Kubeflow platform, you can also see the hdfsLog.
![](./img/webui-img/detail-pai.png) ![](./img/webui-img/detail-pai.png)
![](./img/webui-img/trialog.png)
* Kill: you can kill a job that status is running. * Kill: you can kill a job that status is running.
* Support to search for a specific trial. * Support to search for a specific trial.
......
docs/advanced.rst 0 → 100644
View file @ 035d58bc
Advanced Features
=====================
.. toctree::
MultiPhase<multiPhase>
AdvancedNAS
\ No newline at end of file
docs/assessors.rst 0 → 100644
View file @ 035d58bc
Assessors
==============
In order to save our computing resources, NNI supports an early stop policy and creates **Assessor** to finish this job.
Assessor receives the intermediate result from Trial and decides whether the Trial should be killed by specific algorithm. Once the Trial experiment meets the early stop conditions(which means assessor is pessimistic about the final results), the assessor will kill the trial and the status of trial will be `"EARLY_STOPPED"`.
Here is an experimental result of MNIST after using 'Curvefitting' Assessor in 'maximize' mode, you can see that assessor successfully **early stopped** many trials with bad hyperparameters in advance. If you use assessor, we may get better hyperparameters under the same computing resources.
*Implemented code directory: config_assessor.yml <https://github.com/Microsoft/nni/blob/master/examples/trials/mnist/config_assessor.yml>*
.. image:: ./img/Assessor.png
Like Tuners, users can either use built-in Assessors, or customize an Assessor on their own. Please refer to the following tutorials for detail:
.. toctree::
Builtin Assessors<Builtin_Assessors>
Customized Assessors<Customize_Assessor>
docs/cifar10_examples.md 0 → 100644
View file @ 035d58bc
# CIFAR-10 examples
## Overview
[CIFAR-10][3] classification is a common benchmark problem in machine learning. The CIFAR-10 dataset is the collection of images. It is one of the most widely used datasets for machine learning research which contains 60,000 32x32 color images in 10 different classes. Thus, we use CIFAR-10 classification as an example to introduce NNI usage.
### **Goals**
As we all know, the choice of model optimizer is directly affects the performance of the final metrics. The goal of this tutorial is to **tune a better performace optimizer** to train a relatively small convolutional neural network (CNN) for recognizing images.
In this example, we have selected the following common deep learning optimizer:
> "SGD", "Adadelta", "Adagrad", "Adam", "Adamax"
### **Experimental**
#### Preparations
This example requires pytorch. Pytorch install package should be chosen based on python version and cuda version.
Here is an example of the environment python==3.5 and cuda == 8.0, then using the following commands to install [pytorch][2]:
```bash
python3 -m pip install http://download.pytorch.org/whl/cu80/torch-0.4.1-cp35-cp35m-linux_x86_64.whl
python3 -m pip install torchvision
```
#### CIFAR-10 with NNI
**Search Space**
As we stated in the target, we target to find out the best `optimizer` for training CIFAR-10 classification. When using different optimizers, we also need to adjust `learning rates` and `network structure` accordingly. so we chose these three parameters as hyperparameters and write the following search space.
```json
{
"lr":{"_type":"choice", "_value":[0.1, 0.01, 0.001, 0.0001]},
"optimizer":{"_type":"choice", "_value":["SGD", "Adadelta", "Adagrad", "Adam", "Adamax"]},
"model":{"_type":"choice", "_value":["vgg", "resnet18", "googlenet", "densenet121", "mobilenet", "dpn92", "senet18"]}
}
```
*Implemented code directory: [search_space.json][8]*
**Trial**
The code for CNN training of each hyperparameters set, paying particular attention to the following points are specific for NNI:
* Use `nni.get_next_parameter()` to get next training hyperparameter set.
* Use `nni.report_intermediate_result(acc)` to report the intermedian result after finish each epoch.
* Use `nni.report_final_result(acc)` to report the final result before the trial end.
*Implemented code directory: [main.py][9]*
You can also use your previous code directly, refer to [How to define a trial][5] for modify.
**Config**
Here is the example of running this experiment on local(with multiple GPUs):
code directory: [examples/trials/cifar10_pytorch/config.yml][6]
Here is the example of running this experiment on OpenPAI:
code directory: [examples/trials/cifar10_pytorch/config_pai.yml][7]
*The complete examples we have implemented: [examples/trials/cifar10_pytorch/][1]*
#### Launch the experiment
We are ready for the experiment, let's now **run the config.yml file from your command line to start the experiment**.
```bash
nnictl create --config nni/examples/trials/cifar10_pytorch/config.yml
```
[1]: https://github.com/Microsoft/nni/tree/master/examples/trials/cifar10_pytorch
[2]: https://pytorch.org/
[3]: https://www.cs.toronto.edu/~kriz/cifar.html
[4]: https://github.com/Microsoft/nni/tree/master/examples/trials/cifar10_pytorch
[5]: https://github.com/Microsoft/nni/blob/master/docs/Trials.md
[6]: https://github.com/Microsoft/nni/blob/master/examples/trials/cifar10_pytorch/config.yml
[7]: https://github.com/Microsoft/nni/blob/master/examples/trials/cifar10_pytorch/config_pai.yml
[8]: https://github.com/Microsoft/nni/blob/master/examples/trials/cifar10_pytorch/search_space.json
[9]: https://github.com/Microsoft/nni/blob/master/examples/trials/cifar10_pytorch/main.py
\ No newline at end of file
docs/conf.py 0 → 100644
View file @ 035d58bc
# -*- coding: utf-8 -*-
#
# Configuration file for the Sphinx documentation builder.
#
# This file does only contain a selection of the most common options. For a
# full list see the documentation:
# http://www.sphinx-doc.org/en/master/config
# -- Path setup --------------------------------------------------------------
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
#
# import os
# import sys
# sys.path.insert(0, os.path.abspath('.'))
from recommonmark.parser import CommonMarkParser
from recommonmark.transform import AutoStructify
# -- Project information ---------------------------------------------------
project = 'Neural Network Intelligence'
copyright = '2019, Microsoft'
author = 'Microsoft'
# The short X.Y version
version = ''
# The full version, including alpha/beta/rc tags
release = 'v0.5.1'
# -- General configuration ---------------------------------------------------
# If your documentation needs a minimal Sphinx version, state it here.
#
# needs_sphinx = '1.0'
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = [
'sphinx.ext.autodoc',
'sphinx.ext.mathjax',
'sphinx_markdown_tables',
'sphinxarg.ext',
]
# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']
# The suffix(es) of source filenames.
# You can specify multiple suffix as a list of string:
#
source_parsers = {
'.md': CommonMarkParser
}
source_suffix = ['.rst', '.md']
# The master toctree document.
master_doc = 'index'
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
#
# This is also used if you do content translation via gettext catalogs.
# Usually you set "language" from the command line for these cases.
language = None
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
# This pattern also affects html_static_path and html_extra_path.
exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
# The name of the Pygments (syntax highlighting) style to use.
pygments_style = None
# -- Options for HTML output -------------------------------------------------
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
#
html_theme = 'sphinx_rtd_theme'
# Theme options are theme-specific and customize the look and feel of a theme
# further. For a list of options available for each theme, see the
# documentation.
#
html_theme_options = {
'logo_only': True,
}
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
# html_static_path = ['_static']
# Custom sidebar templates, must be a dictionary that maps document names
# to template names.
#
# The default sidebars (for documents that don't match any pattern) are
# defined by theme itself. Builtin themes are using these templates by
# default: ``['localtoc.html', 'relations.html', 'sourcelink.html',
# 'searchbox.html']``.
#
# html_sidebars = {}
html_logo = './img/nni_logo_dark.png'
# -- Options for HTMLHelp output ---------------------------------------------
# Output file base name for HTML help builder.
htmlhelp_basename = 'NeuralNetworkIntelligencedoc'
# -- Options for LaTeX output ------------------------------------------------
latex_elements = {
# The paper size ('letterpaper' or 'a4paper').
#
# 'papersize': 'letterpaper',
# The font size ('10pt', '11pt' or '12pt').
#
# 'pointsize': '10pt',
# Additional stuff for the LaTeX preamble.
#
# 'preamble': '',
# Latex figure (float) alignment
#
# 'figure_align': 'htbp',
}
# Grouping the document tree into LaTeX files. List of tuples
# (source start file, target name, title,
# author, documentclass [howto, manual, or own class]).
latex_documents = [
(master_doc, 'NeuralNetworkIntelligence.tex', 'Neural Network Intelligence Documentation',
'Microsoft', 'manual'),
]
# -- Options for manual page output ------------------------------------------
# One entry per manual page. List of tuples
# (source start file, name, description, authors, manual section).
man_pages = [
(master_doc, 'neuralnetworkintelligence', 'Neural Network Intelligence Documentation',
[author], 1)
]
# -- Options for Texinfo output ----------------------------------------------
# Grouping the document tree into Texinfo files. List of tuples
# (source start file, target name, title, author,
# dir menu entry, description, category)
texinfo_documents = [
(master_doc, 'NeuralNetworkIntelligence', 'Neural Network Intelligence Documentation',
author, 'NeuralNetworkIntelligence', 'One line description of project.',
'Miscellaneous'),
]
# -- Options for Epub output -------------------------------------------------
# Bibliographic Dublin Core info.
epub_title = project
# The unique identifier of the text. This can be a ISBN number
# or the project homepage.
#
# epub_identifier = ''
# A unique identification for the text.
#
# epub_uid = ''
# A list of files that should not be packed into the epub file.
epub_exclude_files = ['search.html']
# -- Extension configuration -------------------------------------------------
github_doc_root = 'https://github.com/Microsoft/nni/tree/master/doc/'
def setup(app):
app.add_config_value('recommonmark_config', {
'url_resolver': lambda url: github_doc_root + url if url.startswith('..') else url,
'enable_auto_toc_tree': False,
}, True)
app.add_transform(AutoStructify)
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment