Skip to content

GitLab

"test/ut/vscode:/vscode.git/clone" did not exist on "05c7d6e924ac405a9d15a5fe43cd37253c87f9b0"
Unverified Commit 611a45fc authored by chicm-ms's avatar chicm-ms Committed by GitHub
Browse files

Merge pull request #19 from microsoft/master 

pull code
parents 841d4677 e267a737
Show whitespace changes
Inline Side-by-side
Showing with 276 additions and 101 deletions
README.md
View file @ 611a45fc
......@@ -106,7 +106,7 @@ We encourage researchers and students leverage these projects to accelerate the
## **Install & Verify**
If you choose NNI Windows local mode and you use PowerShell to run script for the first time, you need to **run PowerShell as administrator** with this command first:
If you are using NNI on Windows and use PowerShell to run script for the first time, you need to **run PowerShell as administrator** with this command first:
```bash
Set-ExecutionPolicy -ExecutionPolicy Unrestricted
......@@ -114,7 +114,7 @@ If you choose NNI Windows local mode and you use PowerShell to run script for th
**Install through pip**
* We support Linux, MacOS and Windows(local mode) in current stage, Ubuntu 16.04 or higher, MacOS 10.14.1 along with Windows 10.1809 are tested and supported. Simply run the following `pip install` in an environment that has `python >= 3.5`.
* We support Linux, MacOS and Windows(local, remote and pai mode) in current stage, Ubuntu 16.04 or higher, MacOS 10.14.1 along with Windows 10.1809 are tested and supported. Simply run the following `pip install` in an environment that has `python >= 3.5`.
Linux and MacOS
......@@ -131,12 +131,12 @@ python -m pip install --upgrade nni
Note:
* `--user` can be added if you want to install NNI in your home directory, which does not require any special privileges.
* Currently NNI on Windows only support local mode. Anaconda or Miniconda is highly recommended to install NNI on Windows.
* Currently NNI on Windows support local, remote and pai mode. Anaconda or Miniconda is highly recommended to install NNI on Windows.
* If there is any error like `Segmentation fault`, please refer to [FAQ](docs/en_US/FAQ.md)
**Install through source code**
* We support Linux (Ubuntu 16.04 or higher), MacOS (10.14.1) and Windows local mode (10.1809) in our current stage.
* We support Linux (Ubuntu 16.04 or higher), MacOS (10.14.1) and Windows (10.1809) in our current stage.
Linux and MacOS
......@@ -155,12 +155,12 @@ Windows
```bash
git clone -b v0.7 https://github.com/Microsoft/nni.git
cd nni
powershell ./install.ps1
powershell .\install.ps1
```
For the system requirements of NNI, please refer to [Install NNI](docs/en_US/Installation.md)
For NNI Windows local mode, please refer to [NNI Windows local mode](docs/en_US/WindowsLocalMode.md)
For NNI on Windows, please refer to [NNI on Windows](docs/en_US/NniOnWindows.md)
**Verify install**
......@@ -185,7 +185,7 @@ Windows
* Run the MNIST example.
```bash
nnictl create --config nni/examples/trials/mnist/config_windows.yml
nnictl create --config nni\examples\trials\mnist\config_windows.yml
```
* Wait for the message `INFO: Successfully started experiment!` in the command line. This message indicates that your experiment has been successfully started. You can explore the experiment using the `Web UI url`.
......
README_zh_CN.md
View file @ 611a45fc
......@@ -47,32 +47,32 @@ NNI (Neural Network Intelligence) 是自动机器学习（AutoML）的工具包
</ul>
</td>
<td>
<a href="docs/zh_CN/Builtin_Tuner.md">Tuner（调参器）</a>
<a href="docs/zh_CN/BuiltinTuner.md">Tuner（调参器）</a>
<ul>
<li><a href="docs/zh_CN/Builtin_Tuner.md#TPE">TPE</a></li>
<li><a href="docs/zh_CN/Builtin_Tuner.md#Random">Random Search（随机搜索）</a></li>
<li><a href="docs/zh_CN/Builtin_Tuner.md#Anneal">Anneal（退火算法）</a></li>
<li><a href="docs/zh_CN/Builtin_Tuner.md#Evolution">Naive Evolution（进化算法）</a></li>
<li><a href="docs/zh_CN/Builtin_Tuner.md#SMAC">SMAC</a></li>
<li><a href="docs/zh_CN/Builtin_Tuner.md#Batch">Batch（批处理）</a></li>
<li><a href="docs/zh_CN/Builtin_Tuner.md#Grid">Grid Search（遍历搜索）</a></li>
<li><a href="docs/zh_CN/Builtin_Tuner.md#Hyperband">Hyperband</a></li>
<li><a href="docs/zh_CN/Builtin_Tuner.md#NetworkMorphism">Network Morphism</a></li>
<li><a href="docs/zh_CN/BuiltinTuner.md#TPE">TPE</a></li>
<li><a href="docs/zh_CN/BuiltinTuner.md#Random">Random Search（随机搜索）</a></li>
<li><a href="docs/zh_CN/BuiltinTuner.md#Anneal">Anneal（退火算法）</a></li>
<li><a href="docs/zh_CN/BuiltinTuner.md#Evolution">Naive Evolution（进化算法）</a></li>
<li><a href="docs/zh_CN/BuiltinTuner.md#SMAC">SMAC</a></li>
<li><a href="docs/zh_CN/BuiltinTuner.md#Batch">Batch（批处理）</a></li>
<li><a href="docs/zh_CN/BuiltinTuner.md#Grid">Grid Search（遍历搜索）</a></li>
<li><a href="docs/zh_CN/BuiltinTuner.md#Hyperband">Hyperband</a></li>
<li><a href="docs/zh_CN/BuiltinTuner.md#NetworkMorphism">Network Morphism</a></li>
<li><a href="examples/tuners/enas_nni/README_zh_CN.md">ENAS</a></li>
<li><a href="docs/zh_CN/Builtin_Tuner.md#NetworkMorphism#MetisTuner">Metis Tuner</a></li>
<li><a href="docs/zh_CN/Builtin_Tuner.md#BOHB">BOHB</a></li>
<li><a href="docs/zh_CN/BuiltinTuner.md#NetworkMorphism#MetisTuner">Metis Tuner</a></li>
<li><a href="docs/zh_CN/BuiltinTuner.md#BOHB">BOHB</a></li>
</ul>
<a href="docs/zh_CN/Builtin_Assessors.md#assessor">Assessor（评估器）</a>
<a href="docs/zh_CN/BuiltinAssessors.md#assessor">Assessor（评估器）</a>
<ul>
<li><a href="docs/zh_CN/Builtin_Assessors.md#Medianstop">Median Stop</a></li>
<li><a href="docs/zh_CN/Builtin_Assessors.md#Curvefitting">Curve Fitting</a></li>
<li><a href="docs/zh_CN/BuiltinAssessors.md#Medianstop">Median Stop</a></li>
<li><a href="docs/zh_CN/BuiltinAssessors.md#Curvefitting">Curve Fitting</a></li>
</ul>
</td>
<td>
<ul>
<li><a href="docs/zh_CN/LocalMode.md">本地计算机</a></li>
<li><a href="docs/zh_CN/RemoteMachineMode.md">远程计算机</a></li>
<li><a href="docs/zh_CN/PAIMode.md">OpenPAI</a></li>
<li><a href="docs/zh_CN/PaiMode.md">OpenPAI</a></li>
<li><a href="docs/zh_CN/KubeflowMode.md">Kubeflow</a></li>
<li><a href="docs/zh_CN/FrameworkControllerMode.md">基于 Kubernetes（AKS 等等）的 FrameworkController</a></li>
</ul>
......@@ -94,7 +94,10 @@ NNI (Neural Network Intelligence) 是自动机器学习（AutoML）的工具包
* [OpenPAI](https://github.com/Microsoft/pai)：作为开源平台，提供了完整的 AI 模型训练和资源管理能力，能轻松扩展，并支持各种规模的私有部署、云和混合环境。
* [FrameworkController](https://github.com/Microsoft/frameworkcontroller)：开源的通用 Kubernetes Pod 控制器，通过单个控制器来编排 Kubernetes 上所有类型的应用。
* [MMdnn](https://github.com/Microsoft/MMdnn)：一个完整、跨框架的解决方案，能够转换、可视化、诊断深度神经网络模型。 MMdnn 中的 "MM" 表示model management（模型管理），而 "dnn" 是 deep neural network（深度神经网络）的缩写。 我们鼓励研究人员和学生利用这些项目来加速 AI 开发和研究。
* [MMdnn](https://github.com/Microsoft/MMdnn)：一个完整、跨框架的解决方案，能够转换、可视化、诊断深度神经网络模型。 MMdnn 中的 "MM" 表示model management（模型管理），而 "dnn" 是 deep neural network（深度神经网络）的缩写。
* [SPTAG](https://github.com/Microsoft/SPTAG) : Space Partition Tree And Graph (SPTAG) 是用于大规模向量的最近邻搜索场景的开源库。
我们鼓励研究人员和学生利用这些项目来加速 AI 开发和研究。
## **安装和验证**
......@@ -106,9 +109,9 @@ NNI (Neural Network Intelligence) 是自动机器学习（AutoML）的工具包
**通过 pip 命令安装**
* 当前支持 Linux，MacOS 和 Windows（本机模式），在 Ubuntu 16.04 或更高版本，MacOS 10.14.1 以及 Windows 10.1809 上进行了测试。 在 `python >= 3.5` 的环境中，只需要运行 `pip install` 即可完成安装。
* 当前支持 Linux，MacOS 和 Windows（本机，远程，OpenPAI 模式），在 Ubuntu 16.04 或更高版本，MacOS 10.14.1 以及 Windows 10.1809 上进行了测试。 在 `python >= 3.5` 的环境中，只需要运行 `pip install` 即可完成安装。
Linux 和 MacOS
Linux 和 macOS
```bash
python3 -m pip install --upgrade nni
......@@ -123,14 +126,14 @@ python -m pip install --upgrade nni
注意：
* 如果需要将 NNI 安装到自己的 home 目录中，可使用 `--user`，这样也不需要任何特殊权限。
* 当前 NNI 在 Windows 上仅支持本机模式。 强烈推荐使用 Anaconda 在 Windows 上安装 NNI。
* 目前，Windows 上的 NNI 支持本机，远程和 OpenPAI 模式。 强烈推荐使用 Anaconda 或 Miniconda 在 Windows 上安装 NNI。
* 如果遇到如`Segmentation fault` 这样的任何错误请参考[常见问题](docs/zh_CN/FAQ.md)。
**通过源代码安装**
* 当前支持 Linux（Ubuntu 16.04 或更高版本），MacOS（10.14.1）以及 Windows 10（1809 版）下的本机模式。
* 当前支持 Linux（Ubuntu 16.04 或更高版本），MacOS（10.14.1）以及 Windows 10（1809 版）。
Linux 和 MacOS
Linux 和 macOS
* 在 `python >= 3.5` 的环境中运行命令： `git` 和 `wget`，确保安装了这两个组件。
......@@ -152,7 +155,7 @@ Windows
参考[安装 NNI](docs/zh_CN/Installation.md) 了解系统需求。
参考 [NNI Windows 本机模式](docs/zh_CN/WindowsLocalMode.md)，了解更多信息。
Windows 上参考 [Windows 上使用 NNI](docs/zh_CN/NniOnWindows.md)。
**验证安装**
......@@ -224,11 +227,11 @@ You can use these commands to get more information about the experiment
## **入门**
* [安装 NNI](docs/zh_CN/Installation.md)
* [使用命令行工具 nnictl](docs/zh_CN/NNICTLDOC.md)
* [使用命令行工具 nnictl](docs/zh_CN/Nnictl.md)
* [使用 NNIBoard](docs/zh_CN/WebUI.md)
* [如何定义搜索空间](docs/zh_CN/SearchSpaceSpec.md)
* [如何编写 Trial 代码](docs/zh_CN/Trials.md)
* [如何选择 Tuner、搜索算法](docs/zh_CN/Builtin_Tuner.md)
* [如何选择 Tuner、搜索算法](docs/zh_CN/BuiltinTuner.md)
* [配置 Experiment](docs/zh_CN/ExperimentConfig.md)
* [如何使用 Annotation](docs/zh_CN/Trials.md#nni-python-annotation)
......@@ -236,12 +239,12 @@ You can use these commands to get more information about the experiment
* [在本机运行 Experiment (支持多 GPU 卡)](docs/zh_CN/LocalMode.md)
* [在多机上运行 Experiment](docs/zh_CN/RemoteMachineMode.md)
* [在 OpenPAI 上运行 Experiment](docs/zh_CN/PAIMode.md)
* [在 OpenPAI 上运行 Experiment](docs/zh_CN/PaiMode.md)
* [在 Kubeflow 上运行 Experiment。](docs/zh_CN/KubeflowMode.md)
* [尝试不同的 Tuner](docs/zh_CN/tuners.rst)
* [尝试不同的 Assessor](docs/zh_CN/assessors.rst)
* [实现自定义 Tuner](docs/zh_CN/Customize_Tuner.md)
* [实现自定义 Assessor](docs/zh_CN/Customize_Assessor.md)
* [实现自定义 Tuner](docs/zh_CN/CustomizeTuner.md)
* [实现自定义 Assessor](docs/zh_CN/CustomizeAssessor.md)
* [使用进化算法为阅读理解任务找到好模型](examples/trials/ga_squad/README_zh_CN.md)
## **贡献**
......@@ -250,9 +253,9 @@ You can use these commands to get more information about the experiment
推荐新贡献者从标有 **good first issue** 的简单需求开始。
如要安装 NNI 开发环境，参考： [配置 NNI 开发环境](docs/zh_CN/SetupNNIDeveloperEnvironment.md)。
如要安装 NNI 开发环境，参考：[配置 NNI 开发环境](docs/zh_CN/SetupNniDeveloperEnvironment.md)。
在写代码之前，请查看并熟悉 NNI 代码贡献指南：[贡献](docs/zh_CN/CONTRIBUTING.md)。
在写代码之前，请查看并熟悉 NNI 代码贡献指南：[贡献](docs/zh_CN/Contributing.md)。
我们正在编写[如何调试](docs/zh_CN/HowToDebug.md) 的页面，欢迎提交建议和问题。
......
deployment/pypi/Makefile
View file @ 611a45fc
......@@ -20,22 +20,28 @@ ifeq ($(version_ts), true)
NNI_VERSION_VALUE := $(NNI_VERSION_VALUE).$(TIME_STAMP)
endif
NNI_VERSION_TEMPLATE = 999.0.0-developing
NNI_YARN_TARBALL ?= $(CWD)nni-yarn.tar.gz
NNI_YARN_FOLDER ?= $(CWD)nni-yarn
NNI_YARN := PATH=$(CWD)node-$(OS_SPEC)-x64/bin:$${PATH} $(NNI_YARN_FOLDER)/bin/yarn
.PHONY: build
build:
python3 -m pip install --user --upgrade setuptools wheel
wget https://aka.ms/nni/nodejs-download/$(OS_SPEC) -O $(CWD)node-$(OS_SPEC)-x64.tar.xz
wget -q https://aka.ms/nni/nodejs-download/$(OS_SPEC) -O $(CWD)node-$(OS_SPEC)-x64.tar.xz
rm -rf $(CWD)node-$(OS_SPEC)-x64
mkdir $(CWD)node-$(OS_SPEC)-x64
tar xf $(CWD)node-$(OS_SPEC)-x64.tar.xz -C node-$(OS_SPEC)-x64 --strip-components 1
cd $(CWD)../../src/nni_manager && yarn && yarn build
cd $(CWD)../../src/webui && yarn && yarn build
wget -q https://aka.ms/yarn-download -O $(NNI_YARN_TARBALL)
rm -rf $(NNI_YARN_FOLDER)
mkdir $(NNI_YARN_FOLDER)
tar -xf $(NNI_YARN_TARBALL) -C $(NNI_YARN_FOLDER) --strip-components 1
cd $(CWD)../../src/nni_manager && $(NNI_YARN) && $(NNI_YARN) build
cd $(CWD)../../src/webui && $(NNI_YARN) && $(NNI_YARN) build
rm -rf $(CWD)nni
cp -r $(CWD)../../src/nni_manager/dist $(CWD)nni
cp -r $(CWD)../../src/webui/build $(CWD)nni/static
cp $(CWD)../../src/nni_manager/package.json $(CWD)nni
sed -ie 's/$(NNI_VERSION_TEMPLATE)/$(NNI_VERSION_VALUE)/' $(CWD)nni/package.json
cd $(CWD)nni && yarn --prod
cd $(CWD)nni && $(NNI_YARN) --prod
cd $(CWD) && sed -ie 's/$(NNI_VERSION_TEMPLATE)/$(NNI_VERSION_VALUE)/' setup.py && python3 setup.py bdist_wheel -p $(WHEEL_SPEC)
cd $(CWD)
......
docs/en_US/AnnotationSpec.md
View file @ 611a45fc
......@@ -36,8 +36,8 @@ There are 10 types to express your search space as follows:
* `@nni.variable(nni.choice(option1,option2,...,optionN),name=variable)`
Which means the variable value is one of the options, which should be a list The elements of options can themselves be stochastic expressions
* `@nni.variable(nni.randint(upper),name=variable)`
Which means the variable value is a random integer in the range [0, upper).
* `@nni.variable(nni.randint(lower, upper),name=variable)`
Which means the variable value is a value like round(uniform(low, high)). For now, the type of chosen value is float. If you want to use integer value, please convert it explicitly.
* `@nni.variable(nni.uniform(low, high),name=variable)`
Which means the variable value is a value uniformly between low and high.
* `@nni.variable(nni.quniform(low, high, q),name=variable)`
......
docs/en_US/BuiltinTuner.md
View file @ 611a45fc
......@@ -2,7 +2,7 @@
NNI provides state-of-the-art tuning algorithm as our builtin-tuners and makes them easy to use. Below is the brief summary of NNI currently built-in Tuners:
Note: Click the **Tuner's name** to get a detailed description of the algorithm, click the corresponding **Usage** to get the Tuner's installation requirements, suggested scenario and using example. Here is an [article](./Blog/HPOComparison.md) about the comparison of different Tuners on several problems.
Note: Click the **Tuner's name** to get a detailed description of the algorithm, click the corresponding **Usage** to get the Tuner's installation requirements, suggested scenario and using example. Here is an [article](./CommunitySharings/HPOComparison.md) about the comparison of different Tuners on several problems.
Currently we support the following algorithms:
......
docs/en_US/ExperimentConfig.md
View file @ 611a45fc
......@@ -399,6 +399,15 @@ machineList:
__gpuIndices__ is used to specify designated GPU devices for NNI, if it is set, only the specified GPU devices are used for NNI trial jobs. Single or multiple GPU indices can be specified, multiple GPU indices are seperated by comma(,), such as `1` or `0,1,3`.
* __maxTrialNumPerGpu__
__maxTrialNumPerGpu__ is used to specify the max concurrency trial number on a GPU device.
* __useActiveGpu__
__useActiveGpu__ is used to specify whether to use a GPU if there is another process. By default, NNI will use the GPU only if there is no another active process in the GPU, if __useActiveGpu__ is set to true, NNI will use the GPU regardless of another processes. This field is not applicable for NNI on Windows.
* __machineList__
__machineList__ should be set if __trainingServicePlatform__ is set to remote, or it should be empty.
......@@ -433,6 +442,14 @@ machineList:
__gpuIndices__ is used to specify designated GPU devices for NNI on this remote machine, if it is set, only the specified GPU devices are used for NNI trial jobs. Single or multiple GPU indices can be specified, multiple GPU indices are seperated by comma(,), such as `1` or `0,1,3`.
* __maxTrialNumPerGpu__
__maxTrialNumPerGpu__ is used to specify the max concurrency trial number on a GPU device.
* __useActiveGpu__
__useActiveGpu__ is used to specify whether to use a GPU if there is another process. By default, NNI will use the GPU only if there is no another active process in the GPU, if __useActiveGpu__ is set to true, NNI will use the GPU regardless of another processes. This field is not applicable for NNI on Windows.
* __kubeflowConfig__:
* __operator__
......
docs/en_US/FAQ.md
View file @ 611a45fc
......@@ -36,8 +36,8 @@ Unable to open the WebUI may have the following reasons:
* If you still can't see the WebUI after you use the server IP, you can check the proxy and the firewall of your machine. Or use the browser on the machine where you start your NNI experiment.
* Another reason may be your experiment is failed and NNI may fail to get the experiment infomation. You can check the log of NNImanager in the following directory: ~/nni/experiment/[your_experiment_id] /log/nnimanager.log
### Windows local mode problems
Please refer to [NNI Windows local mode](WindowsLocalMode.md)
### NNI on Windows problems
Please refer to [NNI on Windows](NniOnWindows.md)
### Help us improve
Please inquiry the problem in https://github.com/Microsoft/nni/issues to see whether there are other people already reported the problem, create a new one if there are no existing issues been created.
docs/en_US/GeneralNasInterfaces.md 0 → 100644
View file @ 611a45fc
# General Programming Interface for Neural Architecture Search
Automatic neural architecture search is taking an increasingly important role on finding better models. Recent research works have proved the feasibility of automatic NAS, and also found some models that could beat manually designed and tuned models. Some of representative works are [NASNet][2], [ENAS][1], [DARTS][3], [Network Morphism][4], and [Evolution][5]. There are new innovations keeping emerging. However, it takes great efforts to implement those algorithms, and it is hard to reuse code base of one algorithm for implementing another.
To facilitate NAS innovations (e.g., design/implement new NAS models, compare different NAS models side-by-side), an easy-to-use and flexibile programming interface is crucial.
## Programming interface
A new programming interface for designing and searching for a model is often demanded in two scenarios. 1) When designing a neural network, the designer may have multiple choices for a layer, sub-model, or connection, and not sure which one or a combination performs the best. It would be appealing to have an easy way to express the candidate layers/sub-models they want to try. 2) For the researchers who are working on automatic NAS, they want to have an unified way to express the search space of neural architectures. And making unchanged trial code adapted to different searching algorithms.
We designed a simple and flexible programming interface based on [NNI annotation](./AnnotationSpec.md). It is elaborated through examples below.
### Example: choose an operator for a layer
When designing the following model there might be several choices in the fourth layer that may make this model perform good. In the script of this model, we can use annotation for the fourth layer as shown in the figure. In this annotation, there are five fields in total:
![](../img/example_layerchoice.png)
* __layer_choice__: It is a list of function calls, each function should have defined in user's script or imported libraries. The input arguments of the function should follow the format: `def XXX(inputs, arg2, arg3, ...)`, where `inputs` is a list with two elements. One is the list of `fixed_inputs`, and the other is a list of the chosen inputs from `optional_inputs`. `conv` and `pool` in the figure are examples of function definition. For the function calls in this list, no need to write the first argument (i.e., `input`). Note that only one of the function calls are chosen for this layer.
* __fixed_inputs__: It is a list of variables, the variable could be an output tensor from a previous layer. The variable could be `layer_output` of another nni.mutable_layer before this layer, or other python variables before this layer. All the variables in this list will be fed into the chosen function in `layer_choice` (as the first element of the `input` list).
* __optional_inputs__: It is a list of variables, the variable could be an output tensor from a previous layer. The variable could be `layer_output` of another nni.mutable_layer before this layer, or other python variables before this layer. Only `input_num` variables will be fed into the chosen function in `layer_choice` (as the second element of the `input` list).
* __optional_input_size__: It indicates how many inputs are chosen from `input_candidates`. It could be a number or a range. A range [1,3] means it chooses 1, 2, or 3 inputs.
* __layer_output__: The name of the output(s) of this layer, in this case it represents the return of the function call in `layer_choice`. This will be a variable name that can be used in the following python code or nni.mutable_layer(s).
There are two ways to write annotation for this example. For the upper one, `input` of the function calls is `[[],[out3]]`. For the bottom one, `input` is `[[out3],[]]`.
### Example: choose input connections for a layer
Designing connections of layers is critical for making a high performance model. With our provided interface, users could annotate which connections a layer takes (as inputs). They could choose several ones from a set of connections. Below is an example which chooses two inputs from three candidate inputs for `concat`. Here `concat` always takes the output of its previous layer using `fixed_inputs`.
![](../img/example_connectchoice.png)
### Example: choose both operators and connections
In this example, we choose one from the three operators and choose two connections for it. As there are multiple variables in `inputs`, we call `concat` at the beginning of the functions.
![](../img/example_combined.png)
### Example: [ENAS][1] macro search space
To illustrate the convenience of the programming interface, we use the interface to implement the trial code of "ENAS + macro search space". The left figure is the macro search space in ENAS paper.
![](../img/example_enas.png)
## Unified NAS search space specification
After finishing the trial code through the annotation above, users have implicitly specified the search space of neural architectures in the code. Based on the code, NNI will automatcailly generate a search space file which could be fed into tuning algorithms. This search space file follows the following `json` format.
```json
{
"mutable_1": {
"layer_1": {
"layer_choice": ["conv(ch=128)", "pool", "identity"],
"optional_inputs": ["out1", "out2", "out3"],
"optional_input_size": 2
},
"layer_2": {
...
}
}
}
```
Accordingly, a specified neural architecture (generated by tuning algorithm) is expressed as follows:
```json
{
"mutable_1": {
"layer_1": {
"chosen_layer": "pool",
"chosen_inputs": ["out1", "out3"]
},
"layer_2": {
...
}
}
}
```
With the specification of the format of search space and architecture (choice) expression, users are free to implement various (general) tuning algorithms for neural architecture search on NNI. One future work is to provide a general NAS algorihtm.
=============================================================
## Neural architecture search on NNI
### Basic flow of experiment execution
NNI's annotation compiler transforms the annotated trial code to the code that could receive architecture choice and build the corresponding model (i.e., graph). The NAS search space can be seen as a full graph (here, full graph means enabling all the provided operators and connections to build a graph), the architecture chosen by the tuning algorithm is a subgraph in it. By default, the compiled trial code only builds and executes the subgraph.
![](../img/nas_on_nni.png)
The above figure shows how the trial code runs on NNI. `nnictl` processes user trial code to generate a search space file and compiled trial code. The former is fed to tuner, and the latter is used to run trilas.
[__TODO__] Simple example of NAS on NNI.
### Weight sharing
Sharing weights among chosen architectures (i.e., trials) could speedup model search. For example, properly inheriting weights of completed trials could speedup the converge of new trials. One-Shot NAS (e.g., ENAS, Darts) is more aggressive, the training of different architectures (i.e., subgraphs) shares the same copy of the weights in full graph.
![](../img/nas_weight_share.png)
We believe weight sharing (transferring) plays a key role on speeding up NAS, while finding efficient ways of sharing weights is still a hot research topic. We provide a key-value store for users to store and load weights. Tuners and Trials use a provided KV client lib to access the storage.
[__TODO__] Example of weight sharing on NNI.
### Support of One-Shot NAS
One-Shot NAS is a popular approach to find good neural architecture within a limited time and resource budget. Basically, it builds a full graph based on the search space, and uses gradient descent to at last find the best subgraph. There are different training approaches, such as [training subgraphs (per mini-batch)][1], [training full graph through dropout][6], [training with architecture weights (regularization)][3]. Here we focus on the first approach, i.e., training subgraphs (ENAS).
With the same annotated trial code, users could choose One-Shot NAS as execution mode on NNI. Specifically, the compiled trial code builds the full graph (rather than subgraph demonstrated above), it receives a chosen architecture and training this architecture on the full graph for a mini-batch, then request another chosen architecture. It is supported by [NNI multi-phase](./multiPhase.md). We support this training approach because training a subgraph is very fast, building the graph every time training a subgraph induces too much overhead.
![](../img/one-shot_training.png)
The design of One-Shot NAS on NNI is shown in the above figure. One-Shot NAS usually only has one trial job with full graph. NNI supports running multiple such trial jobs each of which runs independently. As One-Shot NAS is not stable, running multiple instances helps find better model. Moreover, trial jobs are also able to synchronize weights during running (i.e., there is only one copy of weights, like asynchroneous parameter-server mode). This may speedup converge.
[__TODO__] Example of One-Shot NAS on NNI.
## General tuning algorithms for NAS
Like hyperparameter tuning, a relatively general algorithm for NAS is required. The general programming interface makes this task easier to some extent. We have a RL-based tuner algorithm for NAS from our contributors. We expect efforts from community to design and implement better NAS algorithms.
[__TODO__] More tuning algorithms for NAS.
## Export best neural architecture and code
[__TODO__] After the NNI experiment is done, users could run `nnictl experiment export --code` to export the trial code with the best neural architecture.
## Conclusion and Future work
There could be different NAS algorithms and execution modes, but they could be supported with the same programming interface as demonstrated above.
There are many interesting research topics in this area, both system and machine learning.
[1]: https://arxiv.org/abs/1802.03268
[2]: https://arxiv.org/abs/1707.07012
[3]: https://arxiv.org/abs/1806.09055
[4]: https://arxiv.org/abs/1806.10282
[5]: https://arxiv.org/abs/1703.01041
[6]: http://proceedings.mlr.press/v80/bender18a/bender18a.pdf
docs/en_US/Installation.md
View file @ 611a45fc
# Installation of NNI
Currently we support installation on Linux, Mac and Windows(local mode).
Currently we support installation on Linux, Mac and Windows(local, remote and pai mode).
## **Installation on Linux & Mac**
......@@ -15,7 +15,7 @@ Currently we support installation on Linux, Mac and Windows(local mode).
Prerequisite: `python >=3.5`, `git`, `wget`
```bash
git clone -b v0.7 https://github.com/Microsoft/nni.git
git clone -b v0.8 https://github.com/Microsoft/nni.git
cd nni
./install.sh
```
......@@ -48,9 +48,9 @@ Currently we support installation on Linux, Mac and Windows(local mode).
you can install NNI as administrator or current user as follows:
```bash
git clone -b v0.7 https://github.com/Microsoft/nni.git
git clone -b v0.8 https://github.com/Microsoft/nni.git
cd nni
powershell ./install.ps1
powershell .\install.ps1
```
## **System requirements**
......
docs/en_US/MnistExamples.md
View file @ 611a45fc
......@@ -51,7 +51,7 @@ This example is to show how to use hyperband to tune the model. There is one mor
This example is to show that NNI also support nested search space. The search space file is an example of how to define nested search space.
`code directory: examples/trials/mnist-cascading-search-space/`
`code directory: examples/trials/mnist-nested-search-space/`
<a name="mnist-kubeflow-tf"></a>
**distributed MNIST (tensorflow) using kubeflow**
......
docs/en_US/WindowsLocalMode.md → docs/en_US/NniOnWindows.md
View file @ 611a45fc
# Windows Local Mode (experimental feature)
# NNI on Windows (experimental feature)
Currently we only support local mode on Windows. Windows 10.1809 is well tested and recommended.
Currently we support local, remote and pai mode on Windows. Windows 10.1809 is well tested and recommended.
## **Installation on Windows**
**Anaconda or Miniconda python(64-bit) is highly recommended.**
When you use PowerShell to run script for the first time, you need **run PowerShell as administrator** with this command:
```bash
Set-ExecutionPolicy -ExecutionPolicy Unrestricted
```
* __Install NNI through pip__
Prerequisite: `python(64-bit) >= 3.5`
```bash
python -m pip install --upgrade nni
```
* __Install NNI through source code__
Prerequisite: `python >=3.5`, `git`, `PowerShell`
```bash
git clone -b v0.7 https://github.com/Microsoft/nni.git
cd nni
powershell ./install.ps1
```
please refer to [Installation](Installation.md#installation-on-windows) for more details.
When these things are done, use the **config_windows.yml** configuration to start an experiment for validation.
```bash
nnictl create --config nni/examples/trials/mnist/config_windows.yml
nnictl create --config nni\examples\trials\mnist\config_windows.yml
```
For other examples you need to change trial command `python3` into `python` in each example YAML.
......
docs/en_US/Nnictl.md
View file @ 611a45fc
......@@ -124,21 +124,21 @@ Debug mode will disable version check function in Trialkeeper.
nnictl stop
```
1. If there is an id specified, and the id matches the running experiment, nnictl will stop the corresponding experiment, or will print error message.
2. If there is an id specified, and the id matches the running experiment, nnictl will stop the corresponding experiment, or will print error message.
```bash
nnictl stop [experiment_id]
```
1. Users could use 'nnictl stop all' to stop all experiments.
3. Users could use 'nnictl stop all' to stop all experiments.
```bash
nnictl stop all
```
1. If the id ends with *, nnictl will stop all experiments whose ids matchs the regular.
1. If the id does not exist but match the prefix of an experiment id, nnictl will stop the matched experiment.
1. If the id does not exist but match multiple prefix of the experiment ids, nnictl will give id information.
4. If the id ends with *, nnictl will stop all experiments whose ids matchs the regular.
5. If the id does not exist but match the prefix of an experiment id, nnictl will stop the matched experiment.
6. If the id does not exist but match multiple prefix of the experiment ids, nnictl will give id information.
<a name="update"></a>
......@@ -650,3 +650,4 @@ Debug mode will disable version check function in Trialkeeper.
```bash
nnictl --version
```
\ No newline at end of file
docs/en_US/QuickStart.md
View file @ 611a45fc
......@@ -2,7 +2,7 @@
## Installation
We support Linux MacOS and Windows(local mode) in current stage, Ubuntu 16.04 or higher, MacOS 10.14.1 and Windows 10.1809 are tested and supported. Simply run the following `pip install` in an environment that has `python >= 3.5`.
We support Linux MacOS and Windows in current stage, Ubuntu 16.04 or higher, MacOS 10.14.1 and Windows 10.1809 are tested and supported. Simply run the following `pip install` in an environment that has `python >= 3.5`.
#### Linux and MacOS
```bash
......@@ -10,7 +10,7 @@ We support Linux MacOS and Windows(local mode) in current stage, Ubuntu 16.04 or
```
#### Windows
If you choose Windows local mode and use PowerShell to run script, you need run below PowerShell command as administrator at first time.
If you are using NNI on Windows, you need run below PowerShell command as administrator at first time.
```bash
Set-ExecutionPolicy -ExecutionPolicy Unrestricted
```
......@@ -151,10 +151,10 @@ Run the **config.yml** file from your command line to start MNIST experiment.
#### Windows
Run the **config_windows.yml** file from your command line to start MNIST experiment.
**Note**, if you're using windows local mode, it needs to change `python3` to `python` in the config.yml file, or use the config_windows.yml file to start the experiment.
**Note**, if you're using NNI on Windows, it needs to change `python3` to `python` in the config.yml file, or use the config_windows.yml file to start the experiment.
```bash
nnictl create --config nni/examples/trials/mnist/config_windows.yml
nnictl create --config nni\examples\trials\mnist\config_windows.yml
```
Note, **nnictl** is a command line tool, which can be used to control experiments, such as start/stop/resume an experiment, start/stop NNIBoard, etc. Click [here](Nnictl.md) for more usage of `nnictl`
......
docs/en_US/RemoteMachineMode.md
View file @ 611a45fc
......@@ -55,7 +55,8 @@ machineList:
username: bob
passwd: bob123
```
You can use different systems to run experiments on the remote machine.
#### Linux and MacOS
Simply filling the `machineList` section and then run:
```bash
......@@ -64,5 +65,14 @@ nnictl create --config ~/nni/examples/trials/mnist-annotation/config_remote.yml
to start the experiment.
#### Windows
Simply filling the `machineList` section and then run:
```bash
nnictl create --config %userprofile%\nni\examples\trials\mnist-annotation\config_remote.yml
```
to start the experiment.
## version check
NNI support version check feature in since version 0.6, [refer](PaiMode.md)
\ No newline at end of file
docs/en_US/SearchSpaceSpec.md
View file @ 611a45fc
......@@ -29,16 +29,16 @@ All types of sampling strategies and their parameter are listed here:
* Which means the variable's value is one of the options. Here 'options' should be a list. Each element of options is a number of string. It could also be a nested sub-search-space, this sub-search-space takes effect only when the corresponding element is chosen. The variables in this sub-search-space could be seen as conditional variables.
* An simple [example](../../examples/trials/mnist-cascading-search-space/search_space.json) of [nested] search space definition. If an element in the options list is a dict, it is a sub-search-space, and for our built-in tuners you have to add a key '_name' in this dict, which helps you to identify which element is chosen. Accordingly, here is a [sample](../../examples/trials/mnist-cascading-search-space/sample.json) which users can get from nni with nested search space definition. Tuners which support nested search space is as follows:
* An simple [example](https://github.com/microsoft/nni/tree/master/examples/trials/mnist-nested-search-space/search_space.json) of [nested] search space definition. If an element in the options list is a dict, it is a sub-search-space, and for our built-in tuners you have to add a key '_name' in this dict, which helps you to identify which element is chosen. Accordingly, here is a [sample](https://github.com/microsoft/nni/tree/master/examples/trials/mnist-nested-search-space/sample.json) which users can get from nni with nested search space definition. Tuners which support nested search space is as follows:
- Random Search
- TPE
- Anneal
- Evolution
* {"_type":"randint","_value":[upper]}
* {"_type":"randint","_value":[lower, upper]}
* Which means the variable value is a random integer in the range [0, upper). The semantics of this distribution is that there is no more correlation in the loss function between nearby integer values, as compared with more distant integer values. This is an appropriate distribution for describing random seeds for example. If the loss function is probably more correlated for nearby integer values, then you should probably use one of the "quantized" continuous distributions, such as either quniform, qloguniform, qnormal or qlognormal. Note that if you want to change lower bound, you can use `quniform` for now.
* For now, we implment the "randint" distribution with "quniform", which means the variable value is a value like round(uniform(lower, upper)). The type of chosen value is float. If you want to use integer value, please convert it explicitly.
* {"_type":"uniform","_value":[low, high]}
* Which means the variable value is a value uniformly between low and high.
......@@ -86,9 +86,19 @@ All types of sampling strategies and their parameter are listed here:
| Hyperband Advisor | &#10003; | &#10003; | &#10003; | &#10003; | &#10003; | &#10003; | &#10003; | &#10003; | &#10003; | &#10003; |
| Metis Tuner | &#10003; | &#10003; | &#10003; | &#10003; | | | | | | |
Note that In Grid Search Tuner, for users' convenience, the definition of `quniform` and `qloguniform` change, where q here specifies the number of values that will be sampled. Details about them are listed as follows
* Type 'quniform' will receive three values [low, high, q], where [low, high] specifies a range and 'q' specifies the number of values that will be sampled evenly. Note that q should be at least 2. It will be sampled in a way that the first sampled value is 'low', and each of the following values is (high-low)/q larger that the value in front of it.
* Type 'qloguniform' behaves like 'quniform' except that it will first change the range to [log(low), log(high)] and sample and then change the sampled value back.
Known Limitations:
Note that Metis Tuner only support numerical `choice` now
* Note that In Grid Search Tuner, for users' convenience, the definition of `quniform` and `qloguniform` change, where q here specifies the number of values that will be sampled. Details about them are listed as follows
* Type 'quniform' will receive three values [low, high, q], where [low, high] specifies a range and 'q' specifies the number of values that will be sampled evenly. Note that q should be at least 2. It will be sampled in a way that the first sampled value is 'low', and each of the following values is (high-low)/q larger that the value in front of it.
* Type 'qloguniform' behaves like 'quniform' except that it will first change the range to [log(low), log(high)] and sample and then change the sampled value back.
* Note that Metis Tuner only supports numerical `choice` now
* Note that for nested search space:
* Only Random Search/TPE/Anneal/Evolution tuner supports nested search space
* We do not support nested search space "Hyper Parameter" parallel graph now, the enhancement is being considered in #1110(https://github.com/microsoft/nni/issues/1110), any suggestions or discussions or contributions are warmly welcomed
docs/en_US/WebUI.md
View file @ 611a45fc
......@@ -6,6 +6,8 @@ Click the tab "Overview".
* See the experiment trial profile and search space message.
* Support to download the experiment result.
* Support to export nni-manager and dispatcher log file.
* If you have any question, you can click "Feedback" to report it.
![](../img/webui-img/over1.png)
* See good performance trials.
......@@ -52,6 +54,14 @@ Click the tab "Trials Detail" to see the status of the all trials. Specifically:
![](../img/webui-img/detail-local.png)
* The button named "Add column" can select which column to show in the table. If you run an experiment that final result is dict, you can see other keys in the table.
![](../img/webui-img/addColumn.png)
* You can use the button named "Copy as python" to copy trial's parameters.
![](../img/webui-img/copyParameter.png)
* If you run on OpenPAI or Kubeflow platform, you can also see the hdfsLog.
![](../img/webui-img/detail-pai.png)
......
docs/en_US/community_sharings.rst
View file @ 611a45fc
......@@ -8,4 +8,5 @@ In addtion to the official tutorilas and examples, we encourage community contri
:maxdepth: 2
NNI Practice Sharing<nni_practice_sharing>
AutoML Practice Sharing<automl_practice_sharing>
Neural Architecture Search Comparison<CommunitySharings/NasComparison>
Hyper-parameter Tuning Algorithm Comparsion<CommunitySharings/HpoComparison>
docs/en_US/nni_practice_sharing.rst
View file @ 611a45fc
......@@ -8,4 +8,3 @@ Sharing the practice of leveraging NNI to tune models and systems.
:maxdepth: 2
Tuning SVD of Recommenders on NNI<CommunitySharings/NniPracticeSharing/RecommendersSvd>
\ No newline at end of file
Auto-tuning AutoGBDT and RocksDB on NNI<CommunitySharings/NniPracticeSharing/HpoComparison>
\ No newline at end of file
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment