Merge pull request #181 from microsoft/master

merge master

README.md

@@ -17,7 +17,7 @@
 NNI (Neural Network Intelligence) is a toolkit to help users run automated machine learning (AutoML) experiments. 
 The tool dispatches and runs trial jobs generated by tuning algorithms to search the best neural architecture and/or hyper-parameters in different environments like local machine, remote servers and cloud.
 
-### **NNI [v0.7](https://github.com/Microsoft/nni/releases) has been released!**
+### **NNI [v0.8](https://github.com/Microsoft/nni/releases) has been released!**
 <p align="center">
   <a href="#nni-v05-has-been-released"><img src="docs/img/overview.svg" /></a>
 </p>
@@ -57,17 +57,17 @@ The tool dispatches and runs trial jobs generated by tuning algorithms to search
           <li><a href="docs/en_US/BuiltinTuner.md#TPE">TPE</a></li>
           <li><a href="docs/en_US/BuiltinTuner.md#Random">Random Search</a></li>
           <li><a href="docs/en_US/BuiltinTuner.md#Anneal">Anneal</a></li>
-          <li><a href="docs/en_US/BuiltinTuner.md#Evolution">Naive Evolution</a></li>
+          <li><a href="docs/en_US/BuiltinTuner.md#Evolution">Naïve Evolution</a></li>
           <li><a href="docs/en_US/BuiltinTuner.md#SMAC">SMAC</a></li>
           <li><a href="docs/en_US/BuiltinTuner.md#Batch">Batch</a></li>
-          <li><a href="docs/en_US/BuiltinTuner.md#Grid">Grid Search</a></li>
+          <li><a href="docs/en_US/BuiltinTuner.md#GridSearch">Grid Search</a></li>
           <li><a href="docs/en_US/BuiltinTuner.md#Hyperband">Hyperband</a></li>
           <li><a href="docs/en_US/BuiltinTuner.md#NetworkMorphism">Network Morphism</a></li>
           <li><a href="examples/tuners/enas_nni/README.md">ENAS</a></li>
-          <li><a href="docs/en_US/BuiltinTuner.md#NetworkMorphism#MetisTuner">Metis Tuner</a></li>
+          <li><a href="docs/en_US/BuiltinTuner.md#MetisTuner">Metis Tuner</a></li>
           <li><a href="docs/en_US/BuiltinTuner.md#BOHB">BOHB</a></li>
         </ul>
-          <a href="docs/en_US/BuiltinAssessors.md#assessor">Assessor</a> 
+          <a href="docs/en_US/BuiltinAssessors.md">Assessor</a> 
         <ul>
           <li><a href="docs/en_US/BuiltinAssessors.md#Medianstop">Median Stop</a></li>
           <li><a href="docs/en_US/BuiltinAssessors.md#Curvefitting">Curve Fitting</a></li>

README_zh_CN.md

@@ -55,14 +55,14 @@ NNI (Neural Network Intelligence) 是自动机器学习（AutoML）的工具包
           <li><a href="docs/zh_CN/BuiltinTuner.md#Evolution">Naive Evolution（进化算法）</a></li>
           <li><a href="docs/zh_CN/BuiltinTuner.md#SMAC">SMAC</a></li>
           <li><a href="docs/zh_CN/BuiltinTuner.md#Batch">Batch（批处理）</a></li>
-          <li><a href="docs/zh_CN/BuiltinTuner.md#Grid">Grid Search（遍历搜索）</a></li>
+          <li><a href="docs/zh_CN/BuiltinTuner.md#GridSearch">Grid Search（遍历搜索）</a></li>
           <li><a href="docs/zh_CN/BuiltinTuner.md#Hyperband">Hyperband</a></li>
           <li><a href="docs/zh_CN/BuiltinTuner.md#NetworkMorphism">Network Morphism</a></li>
           <li><a href="examples/tuners/enas_nni/README_zh_CN.md">ENAS</a></li>
-          <li><a href="docs/zh_CN/BuiltinTuner.md#NetworkMorphism#MetisTuner">Metis Tuner</a></li>
+          <li><a href="docs/zh_CN/BuiltinTuner.md#MetisTuner">Metis Tuner</a></li>
           <li><a href="docs/zh_CN/BuiltinTuner.md#BOHB">BOHB</a></li>
         </ul>
-          <a href="docs/zh_CN/BuiltinAssessors.md#assessor">Assessor（评估器）</a> 
+          <a href="docs/zh_CN/BuiltinAssessors.md">Assessor（评估器）</a> 
         <ul>
           <li><a href="docs/zh_CN/BuiltinAssessors.md#Medianstop">Median Stop</a></li>
           <li><a href="docs/zh_CN/BuiltinAssessors.md#Curvefitting">Curve Fitting</a></li>
@@ -150,7 +150,7 @@ Windows
 ```bash
   git clone -b v0.7 https://github.com/Microsoft/nni.git
   cd nni
-  powershell ./install.ps1
+  powershell .\install.ps1
 ```
 
 参考[安装 NNI](docs/zh_CN/Installation.md) 了解系统需求。
@@ -180,7 +180,7 @@ Windows
 * 运行 MNIST 示例。
 
 ```bash
-    nnictl create --config nni/examples/trials/mnist/config_windows.yml
+    nnictl create --config nni\examples\trials\mnist\config_windows.yml
 ```
 
 * 在命令行中等待输出 `INFO: Successfully started experiment!`。 此消息表明 Experiment 已成功启动。 通过命令行输出的 `Web UI url` 来访问 Experiment 的界面。

docs/en_US/BatchTuner.md

@@ -5,4 +5,4 @@ Batch Tuner on NNI
 
 Batch tuner allows users to simply provide several configurations (i.e., choices of hyper-parameters) for their trial code. After finishing all the configurations, the experiment is done. Batch tuner only supports the type choice in [search space spec](SearchSpaceSpec.md).
 
-Suggested sceanrio: If the configurations you want to try have been decided, you can list them in searchspace file (using choice) and run them using batch tuner.
+Suggested scenario: If the configurations you want to try have been decided, you can list them in SearchSpace file (using choice) and run them using batch tuner.
\ No newline at end of file

docs/en_US/BuiltinAssessors.md → docs/en_US/BuiltinAssessor.md

@@ -2,7 +2,7 @@
 
 NNI provides state-of-the-art tuning algorithm in our builtin-assessors and makes them easy to use. Below is the brief overview of NNI current builtin Assessors:
 
-Note: Click the **Assessor's name** to get a detailed description of the algorithm, click the corresponding **Usage** to get the Assessor's installation requirements, suggested scenario and using example.
+Note: Click the **Assessor's name** to get the Assessor's installation requirements, suggested scenario and using example. The link for a detailed description of the algorithm is at the end of the suggested scenario of each Assessor.
 
 Currently we support the following Assessors:
 
@@ -25,7 +25,7 @@ Note: Please follow the format when you write your `config.yml` file.
 
 **Suggested scenario**
 
-It is applicable in a wide range of performance curves, thus, can be used in various scenarios to speed up the tuning progress.
+It is applicable in a wide range of performance curves, thus, can be used in various scenarios to speed up the tuning progress. [Detailed Description](./MedianstopAssessor.md)
 
 **Requirement of classArg**
 
@@ -53,7 +53,7 @@ assessor:
 
 **Suggested scenario**
 
-It is applicable in a wide range of performance curves, thus, can be used in various scenarios to speed up the tuning progress. Even better, it's able to handle and assess curves with similar performance.
+It is applicable in a wide range of performance curves, thus, can be used in various scenarios to speed up the tuning progress. Even better, it's able to handle and assess curves with similar performance. [Detailed Description](./CurvefittingAssessor.md)
 
 **Requirement of classArg**
 

docs/en_US/BuiltinTuner.md

@@ -2,7 +2,7 @@
 
 NNI provides state-of-the-art tuning algorithm as our builtin-tuners and makes them easy to use. Below is the brief summary of NNI currently built-in Tuners:
 
-Note: Click the **Tuner's name** to get a detailed description of the algorithm, click the corresponding **Usage** to get the Tuner's installation requirements, suggested scenario and using example. Here is an [article](./CommunitySharings/HPOComparison.md) about the comparison of different Tuners on several problems.
+Note: Click the **Tuner's name** to get the Tuner's installation requirements, suggested scenario and using example. The link for a detailed description of the algorithm is at the end of the suggested scenario of each tuner. Here is an [article](./CommunitySharings/HpoComparision.md) about the comparison of different Tuners on several problems.
 
 Currently we support the following algorithms:
 
@@ -36,7 +36,8 @@ Note: Please follow the format when you write your `config.yml` file. Some built
 
 **Suggested scenario**
 
-TPE, as a black-box optimization, can be used in various scenarios and shows good performance in general. Especially when you have limited computation resource and can only try a small number of trials. From a large amount of experiments, we could found that TPE is far better than Random Search.
+TPE, as a black-box optimization, can be used in various scenarios and shows good performance in general. Especially when you have limited computation resource and can only try a small number of trials. From a large amount of experiments, we could found that TPE is far better than Random Search. [Detailed Description](./HyperoptTuner.md)
+
 
 **Requirement of classArg**
 
@@ -62,7 +63,7 @@ tuner:
 
 **Suggested scenario**
 
-Random search is suggested when each trial does not take too long (e.g., each trial can be completed very soon, or early stopped by assessor quickly), and you have enough computation resource. Or you want to uniformly explore the search space. Random Search could be considered as baseline of search algorithm.
+Random search is suggested when each trial does not take too long (e.g., each trial can be completed very soon, or early stopped by assessor quickly), and you have enough computation resource. Or you want to uniformly explore the search space. Random Search could be considered as baseline of search algorithm. [Detailed Description](./HyperoptTuner.md)
 
 **Requirement of classArg:**
 
@@ -86,7 +87,8 @@ tuner:
 
 **Suggested scenario**
 
-Anneal is suggested when each trial does not take too long, and you have enough computation resource(almost same with Random Search). Or the variables in search space could be sample from some prior distribution.
+Anneal is suggested when each trial does not take too long, and you have enough computation resource(almost same with Random Search). Or the variables in search space could be sample from some prior distribution. [Detailed Description](./HyperoptTuner.md)
+
 
 **Requirement of classArg**
 
@@ -112,7 +114,8 @@ tuner:
 
 **Suggested scenario**
 
-Its requirement of computation resource is relatively high. Specifically, it requires large initial population to avoid falling into local optimum. If your trial is short or leverages assessor, this tuner is a good choice. And, it is more suggested when your trial code supports weight transfer, that is, the trial could inherit the converged weights from its parent(s). This can greatly speed up the training progress.
+Its requirement of computation resource is relatively high. Specifically, it requires large initial population to avoid falling into local optimum. If your trial is short or leverages assessor, this tuner is a good choice. And, it is more suggested when your trial code supports weight transfer, that is, the trial could inherit the converged weights from its parent(s). This can greatly speed up the training progress. [Detailed Description](./EvolutionTuner.md)
+
 
 **Usage example**
 
@@ -144,7 +147,7 @@ nnictl package install --name=SMAC
 
 **Suggested scenario**
 
-Similar to TPE, SMAC is also a black-box tuner which can be tried in various scenarios, and is suggested when computation resource is limited. It is optimized for discrete hyperparameters, thus, suggested when most of your hyperparameters are discrete.
+Similar to TPE, SMAC is also a black-box tuner which can be tried in various scenarios, and is suggested when computation resource is limited. It is optimized for discrete hyperparameters, thus, suggested when most of your hyperparameters are discrete. [Detailed Description](./SmacTuner.md)
 
 **Requirement of classArg**
 
@@ -170,7 +173,7 @@ tuner:
 
 **Suggested scenario**
 
-If the configurations you want to try have been decided, you can list them in searchspace file (using `choice`) and run them using batch tuner.
+If the configurations you want to try have been decided, you can list them in searchspace file (using `choice`) and run them using batch tuner. [Detailed Description](./BatchTuner.md)
 
 **Usage example**
 
@@ -211,7 +214,7 @@ The search space file including the high-level key `combine_params`. The type of
 
 Note that the only acceptable types of search space are `choice`, `quniform`, `qloguniform`. **The number `q` in `quniform` and `qloguniform` has special meaning (different from the spec in [search space spec](./SearchSpaceSpec.md)). It means the number of values that will be sampled evenly from the range `low` and `high`.**
 
-It is suggested when search space is small, it is feasible to exhaustively sweeping the whole search space.
+It is suggested when search space is small, it is feasible to exhaustively sweeping the whole search space. [Detailed Description](./GridsearchTuner.md)
 
 **Usage example**
 
@@ -231,7 +234,7 @@ tuner:
 
 **Suggested scenario**
 
-It is suggested when you have limited computation resource but have relatively large search space. It performs well in the scenario that intermediate result (e.g., accuracy) can reflect good or bad of final result (e.g., accuracy) to some extent.
+It is suggested when you have limited computation resource but have relatively large search space. It performs well in the scenario that intermediate result (e.g., accuracy) can reflect good or bad of final result (e.g., accuracy) to some extent. [Detailed Description](./HyperbandAdvisor.md)
 
 **Requirement of classArg**
 
@@ -265,7 +268,7 @@ NetworkMorphism requires [pyTorch](https://pytorch.org/get-started/locally), so
 
 **Suggested scenario**
 
-It is suggested that you want to apply deep learning methods to your task (your own dataset) but you have no idea of how to choose or design a network. You modify the [example](https://github.com/Microsoft/nni/tree/master/examples/trials/network_morphism/cifar10/cifar10_keras.py) to fit your own dataset and your own data augmentation method. Also you can change the batch size, learning rate or optimizer. It is feasible for different tasks to find a good network architecture. Now this tuner only supports the computer vision domain.
+It is suggested that you want to apply deep learning methods to your task (your own dataset) but you have no idea of how to choose or design a network. You modify the [example](https://github.com/Microsoft/nni/tree/master/examples/trials/network_morphism/cifar10/cifar10_keras.py) to fit your own dataset and your own data augmentation method. Also you can change the batch size, learning rate or optimizer. It is feasible for different tasks to find a good network architecture. Now this tuner only supports the computer vision domain. [Detailed Description](./NetworkmorphismTuner.md)
 
 **Requirement of classArg**
 
@@ -305,7 +308,7 @@ Metis Tuner requires [sklearn](https://scikit-learn.org/), so users should insta
 
 **Suggested scenario**
 
-Similar to TPE and SMAC, Metis is a black-box tuner. If your system takes a long time to finish each trial, Metis is more favorable than other approaches such as random search. Furthermore, Metis provides guidance on the subsequent trial. Here is an [example](https://github.com/Microsoft/nni/tree/master/examples/trials/auto-gbdt/search_space_metis.json) about the use of Metis. User only need to send the final result like `accuracy` to tuner, by calling the nni SDK.
+Similar to TPE and SMAC, Metis is a black-box tuner. If your system takes a long time to finish each trial, Metis is more favorable than other approaches such as random search. Furthermore, Metis provides guidance on the subsequent trial. Here is an [example](https://github.com/Microsoft/nni/tree/master/examples/trials/auto-gbdt/search_space_metis.json) about the use of Metis. User only need to send the final result like `accuracy` to tuner, by calling the nni SDK. [Detailed Description](./MetisTuner.md)
 
 **Requirement of classArg**
 
@@ -339,7 +342,7 @@ nnictl package install --name=BOHB
 
 **Suggested scenario**
 
-Similar to Hyperband, it is suggested when you have limited computation resource but have relatively large search space. It performs well in the scenario that intermediate result (e.g., accuracy) can reflect good or bad of final result (e.g., accuracy) to some extent. In this case, it may converges to a better configuration due to bayesian optimization usage.
+Similar to Hyperband, it is suggested when you have limited computation resource but have relatively large search space. It performs well in the scenario that intermediate result (e.g., accuracy) can reflect good or bad of final result (e.g., accuracy) to some extent. In this case, it may converges to a better configuration due to bayesian optimization usage. [Detailed Description](./BohbAdvisor.md)
 
 **Requirement of classArg**
 

docs/en_US/CommunitySharings/HpoComparision.md

@@ -5,17 +5,17 @@ Comparison of Hyperparameter Optimization algorithms on several problems.
 
 Hyperparameter Optimization algorithms are list below:
 
-- [Random Search](../Builtin_Tuner.md#Random)
-- [Grid Search](../Builtin_Tuner.md#Random)
-- [Evolution](../Builtin_Tuner.md#Evolution)
-- [Anneal](../Builtin_Tuner.md#Anneal)
-- [Metis](../Builtin_Tuner.md#MetisTuner)
-- [TPE](../Builtin_Tuner.md#TPE)
-- [SMAC](../Builtin_Tuner.md#SMAC)
-- [HyperBand](../Builtin_Tuner.md#Hyperband)
-- [BOHB](../Builtin_Tuner.md#BOHB)
-
-All algorithms run in NNI local environment。
+- [Random Search](../BuiltinTuner.md)
+- [Grid Search](../BuiltinTuner.md)
+- [Evolution](../BuiltinTuner.md)
+- [Anneal](../BuiltinTuner.md)
+- [Metis](../BuiltinTuner.md)
+- [TPE](../BuiltinTuner.md)
+- [SMAC](../BuiltinTuner.md)
+- [HyperBand](../BuiltinTuner.md)
+- [BOHB](../BuiltinTuner.md)
+
+All algorithms run in NNI local environment.
 
 Machine Environment：
 

docs/en_US/CommunitySharings/NniPracticeSharing/RecommendersSvd.md

@@ -2,12 +2,12 @@
 
 In this tutorial, we first introduce a github repo [Recommenders](https://github.com/Microsoft/Recommenders). It is a repository that provides examples and best practices for building recommendation systems, provided as Jupyter notebooks. It has various models that are popular and widely deployed in recommendation systems. To provide a complete end-to-end experience, they present each example in five key tasks, as shown below:
 
- - [Prepare Data](https://github.com/Microsoft/Recommenders/blob/master/notebooks/01_prepare_data/README.md): Preparing and loading data for each recommender algorithm
+- [Prepare Data](https://github.com/Microsoft/Recommenders/blob/master/notebooks/01_prepare_data/README.md): Preparing and loading data for each recommender algorithm.
 - [Model](https://github.com/Microsoft/Recommenders/blob/master/notebooks/02_model/README.md): Building models using various classical and deep learning recommender algorithms such as Alternating Least Squares ([ALS](https://spark.apache.org/docs/latest/api/python/_modules/pyspark/ml/recommendation.html#ALS)) or eXtreme Deep Factorization Machines ([xDeepFM](https://arxiv.org/abs/1803.05170)).
-- [Evaluate](https://github.com/Microsoft/Recommenders/blob/master/notebooks/03_evaluate/README.md): Evaluating algorithms with offline metrics
-- [Model Select and Optimize](https://github.com/Microsoft/Recommenders/blob/master/notebooks/04_model_select_and_optimize/README.md): Tuning and optimizing hyperparameters for recommender models
-- [Operationalize](https://github.com/Microsoft/Recommenders/blob/master/notebooks/05_operationalize/README.md): Operationalizing models in a production environment on Azure
+- [Evaluate](https://github.com/Microsoft/Recommenders/blob/master/notebooks/03_evaluate/README.md): Evaluating algorithms with offline metrics.
+- [Model Select and Optimize](https://github.com/Microsoft/Recommenders/blob/master/notebooks/04_model_select_and_optimize/README.md): Tuning and optimizing hyperparameters for recommender models.
+- [Operationalize](https://github.com/Microsoft/Recommenders/blob/master/notebooks/05_operationalize/README.md): Operationalizing models in a production environment on Azure.
 
-The fourth task is tuning and optimizing the model's hyperparametrs, this is where NNI could help. To give a concrete example that NNI tunes the models in Recommenders, let's demonstrate with the model [SVD](https://github.com/Microsoft/Recommenders/blob/master/notebooks/02_model/surprise_svd_deep_dive.ipynb), and data Movielens100k. There are more than 10 hyperparameters to be tuned in this model. 
+The fourth task is tuning and optimizing the model's hyperparameters, this is where NNI could help. To give a concrete example that NNI tunes the models in Recommenders, let's demonstrate with the model [SVD](https://github.com/Microsoft/Recommenders/blob/master/notebooks/02_model/surprise_svd_deep_dive.ipynb), and data Movielens100k. There are more than 10 hyperparameters to be tuned in this model.
 
 [This Jupyter notebook](https://github.com/Microsoft/Recommenders/blob/master/notebooks/04_model_select_and_optimize/nni_surprise_svd.ipynb) provided by Recommenders is a very detailed step-by-step tutorial for this example. It uses different built-in tuning algorithms in NNI, including `Annealing`, `SMAC`, `Random Search`, `TPE`, `Hyperband`, `Metis` and `Evolution`. Finally, the results of different tuning algorithms are compared. Please go through this notebook to learn how to use NNI to tune SVD model, then you could further use NNI to tune other models in Recommenders.

docs/en_US/FAQ.md

@@ -24,7 +24,7 @@ Your machine don't have eth0 device, please set [nniManagerIp](ExperimentConfig.
 When the duration of experiment reaches the maximum duration, nniManager will not create new trials, but the existing trials will continue unless user manually stop the experiment. 
 
 ### Could not stop an experiment using `nnictl stop`
-If you upgrade your NNI or you delete some config files of NNI when there is an experiment running, this kind of issue may happen because the loss of config file. You could use `ps -ef | grep node` to find the pid of your experiment, and use `kill -9 {pid}` to kill it manually.
+If you upgrade your NNI or you delete some config files of NNI when there is an experiment running, this kind of issue may happen because the loss of config file. You could use `ps -ef | grep node` to find the PID of your experiment, and use `kill -9 {pid}` to kill it manually.
 
 ### Could not get `default metric` in webUI of virtual machines
 Config the network mode to bridge mode or other mode that could make virtual machine's host accessible from external machine, and make sure the port of virtual machine is not forbidden by firewall.
@@ -34,7 +34,7 @@ Unable to open the WebUI may have the following reasons:
 
 * http://127.0.0.1, http://172.17.0.1 and http://10.0.0.15 are referred to localhost, if you start your experiment on the server or remote machine. You can replace the IP to your server IP to view the WebUI, like http://[your_server_ip]:8080
 * If you still can't see the WebUI after you use the server IP, you can check the proxy and the firewall of your machine. Or use the browser on the machine where you start your NNI experiment.
-* Another reason may be your experiment is failed and NNI may fail to get the experiment infomation. You can check the log of NNImanager in the following directory: ~/nni/experiment/[your_experiment_id] /log/nnimanager.log
+* Another reason may be your experiment is failed and NNI may fail to get the experiment information. You can check the log of NNIManager in the following directory: ~/nni/experiment/[your_experiment_id] /log/nnimanager.log
 
 ### NNI on Windows problems
 Please refer to [NNI on Windows](NniOnWindows.md) 

docs/en_US/GeneralNasInterfaces.md

-# General Programming Interface for Neural Architecture Search
+# General Programming Interface for Neural Architecture Search (experimental feature)
+
+_*This is an experimental feature, currently, we only implemented the general NAS programming interface. Weight sharing and one-shot NAS based on this programming interface will be supported in the following releases._
 
 Automatic neural architecture search is taking an increasingly important role on finding better models. Recent research works have proved the feasibility of automatic NAS, and also found some models that could beat manually designed and tuned models. Some of representative works are [NASNet][2], [ENAS][1], [DARTS][3], [Network Morphism][4], and [Evolution][5]. There are new innovations keeping emerging. However, it takes great efforts to implement those algorithms, and it is hard to reuse code base of one algorithm for implementing another.
 
-To facilitate NAS innovations (e.g., design/implement new NAS models, compare different NAS models side-by-side), an easy-to-use and flexibile programming interface is crucial.
+To facilitate NAS innovations (e.g., design/implement new NAS models, compare different NAS models side-by-side), an easy-to-use and flexible programming interface is crucial.
 
 ## Programming interface
 
@@ -10,19 +12,21 @@ To facilitate NAS innovations (e.g., design/implement new NAS models, compare di
 
  We designed a simple and flexible programming interface based on [NNI annotation](./AnnotationSpec.md). It is elaborated through examples below.
 
- ### Example: choose an operator for a layer
+### Example: choose an operator for a layer
 
 When designing the following model there might be several choices in the fourth layer that may make this model perform good. In the script of this model, we can use annotation for the fourth layer as shown in the figure. In this annotation, there are five fields in total:
 
 ![](../img/example_layerchoice.png)
 
-* __layer_choice__: It is a list of function calls, each function should have defined in user's script or imported libraries. The input arguments of the function should follow the format: `def XXX(inputs, arg2, arg3, ...)`, where `inputs` is a list with two elements. One is the list of `fixed_inputs`, and the other is a list of the chosen inputs from `optional_inputs`. `conv` and `pool` in the figure are examples of function definition. For the function calls in this list, no need to write the first argument (i.e., `input`). Note that only one of the function calls are chosen for this layer.
-* __fixed_inputs__: It is a list of variables, the variable could be an output tensor from a previous layer. The variable could be `layer_output` of another nni.mutable_layer before this layer, or other python variables before this layer. All the variables in this list will be fed into the chosen function in `layer_choice` (as the first element of the `input` list).
-* __optional_inputs__: It is a list of variables, the variable could be an output tensor from a previous layer. The variable could be `layer_output` of another nni.mutable_layer before this layer, or other python variables before this layer. Only `input_num` variables will be fed into the chosen function in `layer_choice` (as the second element of the `input` list).
+* __layer_choice__: It is a list of function calls, each function should have defined in user's script or imported libraries. The input arguments of the function should follow the format: `def XXX(inputs, arg2, arg3, ...)`, where inputs is a list with two elements. One is the list of `fixed_inputs`, and the other is a list of the chosen inputs from `optional_inputs`. `conv` and `pool` in the figure are examples of function definition. For the function calls in this list, no need to write the first argument (i.e., input). Note that only one of the function calls are chosen for this layer.
+* __fixed_inputs__: It is a list of variables, the variable could be an output tensor from a previous layer. The variable could be `layer_output` of another `nni.mutable_layer` before this layer, or other python variables before this layer. All the variables in this list will be fed into the chosen function in `layer_choice` (as the first element of the input list).
+* __optional_inputs__: It is a list of variables, the variable could be an output tensor from a previous layer. The variable could be `layer_output` of another `nni.mutable_layer` before this layer, or other python variables before this layer. Only `optional_input_size` variables will be fed into the chosen function in `layer_choice` (as the second element of the input list).
 * __optional_input_size__: It indicates how many inputs are chosen from `input_candidates`. It could be a number or a range. A range [1,3] means it chooses 1, 2, or 3 inputs.
-* __layer_output__: The name of the output(s) of this layer, in this case it represents the return of the function call in `layer_choice`. This will be a variable name that can be used in the following python code or nni.mutable_layer(s).
+* __layer_output__: The name of the output(s) of this layer, in this case it represents the return of the function call in `layer_choice`. This will be a variable name that can be used in the following python code or `nni.mutable_layer`.
+
+There are two ways to write annotation for this example. For the upper one, input of the function calls is `[[],[out3]]`. For the bottom one, input is `[[out3],[]]`.
 
-There are two ways to write annotation for this example. For the upper one, `input` of the function calls is `[[],[out3]]`. For the bottom one, `input` is `[[out3],[]]`.
+__Debugging__: We provided an `nnictl trial codegen` command to help debugging your code of NAS programming on NNI. If your trial with trial_id `XXX` in your experiment `YYY` is failed, you could run `nnictl trial codegen YYY --trial_id XXX` to generate an executable code for this trial under your current directory. With this code, you can directly run the trial command without NNI to check why this trial is failed. Basically, this command is to compile your trial code and replace the NNI NAS code with the real chosen layers and inputs.
 
 ### Example: choose input connections for a layer
 
@@ -32,7 +36,7 @@ Designing connections of layers is critical for making a high performance model.
 
 ### Example: choose both operators and connections
 
-In this example, we choose one from the three operators and choose two connections for it. As there are multiple variables in `inputs`, we call `concat` at the beginning of the functions.
+In this example, we choose one from the three operators and choose two connections for it. As there are multiple variables in inputs, we call `concat` at the beginning of the functions.
 
 ![](../img/example_combined.png)
 
@@ -42,10 +46,9 @@ To illustrate the convenience of the programming interface, we use the interface
 
 ![](../img/example_enas.png)
 
-
 ## Unified NAS search space specification
 
-After finishing the trial code through the annotation above, users have implicitly specified the search space of neural architectures in the code. Based on the code, NNI will automatcailly generate a search space file which could be fed into tuning algorithms. This search space file follows the following `json` format.
+After finishing the trial code through the annotation above, users have implicitly specified the search space of neural architectures in the code. Based on the code, NNI will automatically generate a search space file which could be fed into tuning algorithms. This search space file follows the following JSON format.
 
 ```json
 {
@@ -78,7 +81,7 @@ Accordingly, a specified neural architecture (generated by tuning algorithm) is
 }
 ```
 
-With the specification of the format of search space and architecture (choice) expression, users are free to implement various (general) tuning algorithms for neural architecture search on NNI. One future work is to provide a general NAS algorihtm.
+With the specification of the format of search space and architecture (choice) expression, users are free to implement various (general) tuning algorithms for neural architecture search on NNI. One future work is to provide a general NAS algorithm.
 
 =============================================================
 
@@ -92,9 +95,9 @@ NNI's annotation compiler transforms the annotated trial code to the code that c
 
 The above figure shows how the trial code runs on NNI. `nnictl` processes user trial code to generate a search space file and compiled trial code. The former is fed to tuner, and the latter is used to run trials. 
 
-[__TODO__] Simple example of NAS on NNI.
+[Simple example of NAS on NNI](https://github.com/microsoft/nni/tree/v0.8/examples/trials/mnist-nas).
 
-### Weight sharing
+### [__TODO__] Weight sharing
 
 Sharing weights among chosen architectures (i.e., trials) could speedup model search. For example, properly inheriting weights of completed trials could speedup the converge of new trials. One-Shot NAS (e.g., ENAS, Darts) is more aggressive, the training of different architectures (i.e., subgraphs) shares the same copy of the weights in full graph.
 
@@ -102,9 +105,9 @@ Sharing weights among chosen architectures (i.e., trials) could speedup model se
 
 We believe weight sharing (transferring) plays a key role on speeding up NAS, while finding efficient ways of sharing weights is still a hot research topic. We provide a key-value store for users to store and load weights. Tuners and Trials use a provided KV client lib to access the storage.
 
-[__TODO__] Example of weight sharing on NNI.
+Example of weight sharing on NNI.
 
-### Support of One-Shot NAS
+### [__TODO__] Support of One-Shot NAS
 
 One-Shot NAS is a popular approach to find good neural architecture within a limited time and resource budget. Basically, it builds a full graph based on the search space, and uses gradient descent to at last find the best subgraph. There are different training approaches, such as [training subgraphs (per mini-batch)][1], [training full graph through dropout][6], [training with architecture weights (regularization)][3]. Here we focus on the first approach, i.e., training subgraphs (ENAS).
 
@@ -112,20 +115,20 @@ With the same annotated trial code, users could choose One-Shot NAS as execution
 
 ![](../img/one-shot_training.png)
 
-The design of One-Shot NAS on NNI is shown in the above figure. One-Shot NAS usually only has one trial job with full graph. NNI supports running multiple such trial jobs each of which runs independently. As One-Shot NAS is not stable, running multiple instances helps find better model. Moreover, trial jobs are also able to synchronize weights during running (i.e., there is only one copy of weights, like asynchroneous parameter-server mode). This may speedup converge.
+The design of One-Shot NAS on NNI is shown in the above figure. One-Shot NAS usually only has one trial job with full graph. NNI supports running multiple such trial jobs each of which runs independently. As One-Shot NAS is not stable, running multiple instances helps find better model. Moreover, trial jobs are also able to synchronize weights during running (i.e., there is only one copy of weights, like asynchronous parameter-server mode). This may speedup converge.
 
-[__TODO__] Example of One-Shot NAS on NNI.
+Example of One-Shot NAS on NNI.
 
 
-## General tuning algorithms for NAS
+## [__TODO__] General tuning algorithms for NAS
 
 Like hyperparameter tuning, a relatively general algorithm for NAS is required. The general programming interface makes this task easier to some extent. We have a RL-based tuner algorithm for NAS from our contributors. We expect efforts from community to design and implement better NAS algorithms.
 
-[__TODO__] More tuning algorithms for NAS.
+More tuning algorithms for NAS.
 
-## Export best neural architecture and code
+## [__TODO__] Export best neural architecture and code
 
-[__TODO__] After the NNI experiment is done, users could run `nnictl experiment export --code` to export the trial code with the best neural architecture.
+After the NNI experiment is done, users could run `nnictl experiment export --code` to export the trial code with the best neural architecture.
 
 ## Conclusion and Future work
 
@@ -133,7 +136,6 @@ There could be different NAS algorithms and execution modes, but they could be s
 
 There are many interesting research topics in this area, both system and machine learning.
 
-
 [1]: https://arxiv.org/abs/1802.03268
 [2]: https://arxiv.org/abs/1707.07012
 [3]: https://arxiv.org/abs/1806.09055

docs/en_US/Installation.md

@@ -7,6 +7,7 @@ Currently we support installation on Linux, Mac and Windows(local, remote and pa
 * __Install NNI through pip__
 
   Prerequisite: `python >= 3.5`
+
   ```bash
   python3 -m pip install --upgrade nni
   ```
@@ -14,6 +15,7 @@ Currently we support installation on Linux, Mac and Windows(local, remote and pa
 * __Install NNI through source code__
 
   Prerequisite: `python >=3.5`, `git`, `wget`
+
   ```bash
   git clone -b v0.8 https://github.com/Microsoft/nni.git
   cd nni
@@ -26,13 +28,14 @@ Currently we support installation on Linux, Mac and Windows(local, remote and pa
 
 ## **Installation on Windows**
 
-  When you use PowerShell to run script for the first time, you need **run PowerShell as administrator** with this command:
+When you use PowerShell to run script for the first time, you need **run PowerShell as administrator** with this command:
 
-  ```bash
-  Set-ExecutionPolicy -ExecutionPolicy Unrestricted
-  ```
+```powershell
+Set-ExecutionPolicy -ExecutionPolicy Unrestricted
+```
 
   Anaconda or Miniconda is highly recommended.
+
 * __Install NNI through pip__
 
   Prerequisite: `python(64-bit) >= 3.5`

docs/en_US/NniOnWindows.md

@@ -4,7 +4,7 @@ Currently we support local, remote and pai mode on Windows. Windows 10.1809 is w
 
 ## **Installation on Windows**
 
-  please refer to [Installation](Installation.md#installation-on-windows) for more details.
+  please refer to [Installation](Installation.md) for more details.
 
 When these things are done, use the **config_windows.yml** configuration to start an experiment for validation.
 

docs/en_US/PaiMode.md

@@ -31,7 +31,7 @@ trial:
   gpuNum: 0
   cpuNum: 1
   memoryMB: 8196
-  image: openpai/pai.example.tensorflow
+  image: msranni/nni:latest
   dataDir: hdfs://10.1.1.1:9000/nni
   outputDir: hdfs://10.1.1.1:9000/nni
 # Configuration to access OpenPAI Cluster

docs/en_US/Release.md

 # ChangeLog
 
+# Release 0.8 - 6/4/2019
+## Major Features
+* [Support NNI on Windows for PAI/Remote mode]
+    * NNI running on windows for remote mode
+    * NNI running on windows for PAI mode
+* [Advanced features for using GPU]
+   * Run multiple trial jobs on the same GPU for local and remote mode
+   * Run trial jobs on the GPU running non-NNI jobs
+* [Kubeflow v1beta2 operator]
+   * Support Kubeflow TFJob/PyTorchJob v1beta2
+* [General NAS programming interface](./GeneralNasInterfaces.md)
+   * Provide NAS programming interface for users to easily express their neural architecture search space through NNI annotation
+   * Provide a new command `nnictl trial codegen` for debugging the NAS code
+   * Tutorial of NAS programming interface, example of NAS on mnist, customized random tuner for NAS
+* [Support resume tuner/advisor's state for experiment resume]
+   * For experiment resume, tuner/advisor will be resumed by replaying finished trial data
+* [Web Portal]
+   * Improve the design of copying trial's parameters
+   * Support 'randint' type in hyper-parameter graph
+   * Use should ComponentUpdate to avoid unnecessary render
+## Bug fix and other changes
+* [Bug fix that `nnictl update` has inconsistent command styles]
+* [Support import data for SMAC tuner]
+* [Bug fix that experiment state transition from ERROR back to RUNNING]
+* [Fix bug of table entries]
+* [Nested search space refinement]
+* [Refine 'randint' type and support lower bound]
+* [Comparison of different hyper-parameter tuning algorithm](./CommunitySharings/HpoComparision.md)
+* [Comparison of NAS algorithm](./CommunitySharings/NasComparision.md)
+* [NNI practice on Recommenders](./CommunitySharings/NniPracticeSharing/RecommendersSvd.md)
+
 ## Release 0.7 - 4/29/2018
 
 ### Major Features

docs/en_US/SearchSpaceSpec.md

@@ -38,7 +38,7 @@ All types of sampling strategies and their parameter are listed here:
 
 * {"_type":"randint","_value":[lower, upper]}
 
-  * For now, we implment the "randint" distribution with "quniform", which means the variable value is a value like round(uniform(lower, upper)). The type of chosen value is float. If you want to use integer value, please convert it explicitly.
+  * For now, we implement the "randint" distribution with "quniform", which means the variable value is a value like round(uniform(lower, upper)). The type of chosen value is float. If you want to use integer value, please convert it explicitly.
 
 * {"_type":"uniform","_value":[low, high]}
   * Which means the variable value is a value uniformly between low and high.

docs/en_US/advanced.rst

@@ -4,3 +4,4 @@ Advanced Features
 ..  toctree::
     MultiPhase<MultiPhase>
     AdvancedNas<AdvancedNas>
+    NAS Programming Interface<GeneralNasInterfaces>
\ No newline at end of file

docs/en_US/builtin_assessor.rst

@@ -4,6 +4,6 @@ Builtin-Assessors
 ..  toctree::
     :maxdepth: 1
 
-    Overview<BuiltinAssessors>
+    Overview<BuiltinAssessor>
     Medianstop<MedianstopAssessor>
     Curvefitting<CurvefittingAssessor>
\ No newline at end of file

docs/zh_CN/AdvancedNas.md

@@ -101,4 +101,4 @@ sudo mount -t nfs 10.10.10.10:/tmp/nni/shared /mnt/nfs/nni
 
 ## 样例
 
-详细内容参考：[简单的参数共享样例](https://github.com/Microsoft/nni/tree/master/test/async_sharing_test)。 基于上一个 [ga_squad](https://github.com/Microsoft/nni/tree/master/examples/trials/ga_squad) 样例，还提供了新的 [样例](https://github.com/Microsoft/nni/tree/master/examples/trials/weight_sharing/ga_squad)。
+详细内容参考：[简单的参数共享样例](https://github.com/Microsoft/nni/tree/master/test/async_sharing_test)。 基于已有的 [ga_squad](https://github.com/Microsoft/nni/tree/master/examples/trials/ga_squad) 样例，还提供了新的 [样例](https://github.com/Microsoft/nni/tree/master/examples/trials/weight_sharing/ga_squad)。
\ No newline at end of file

docs/zh_CN/AnnotationSpec.md

@@ -34,7 +34,7 @@ NNI 中，有 4 种类型的 Annotation；
 NNI 支持如下 10 种类型来表示搜索空间：
 
 - `@nni.variable(nni.choice(option1,option2,...,optionN),name=variable)` 变量值是选项中的一种，这些变量可以是任意的表达式。
-- `@nni.variable(nni.randint(upper),name=variable)` 变量可以是范围 [0, upper) 中的任意整数。
+- `@nni.variable(nni.randint(lower, upper),name=variable)` 变量值的公式为：round(uniform(low, high))。 目前，值的类型为 float。 如果要使用整数，需要显式转换。
 - `@nni.variable(nni.uniform(low, high),name=variable)` 变量值会是 low 和 high 之间均匀分布的某个值。
 - `@nni.variable(nni.quniform(low, high, q),name=variable)` 变量值会是 low 和 high 之间均匀分布的某个值，公式为：round(uniform(low, high) / q) * q
 - `@nni.variable(nni.loguniform(low, high),name=variable)` 变量值是 exp(uniform(low, high)) 的点，数值以对数均匀分布。

docs/zh_CN/CommunitySharings/NniPracticeSharing/RecommendersSvd.md

@@ -2,11 +2,11 @@
 
 本教程中，会首先介绍 GitHub 存储库：[Recommenders](https://github.com/Microsoft/Recommenders)。 它使用 Jupyter Notebook 提供了构建推荐系统的一些示例和实践技巧。 其中大量的模型被广泛的应用于推荐系统中。 为了提供完整的体验，每个示例都通过以下五个关键任务中展示：
 
-- [准备数据](https://github.com/Microsoft/Recommenders/blob/master/notebooks/01_prepare_data/README.md)：为每个推荐算法准备并读取数据。 
-    - [模型](https://github.com/Microsoft/Recommenders/blob/master/notebooks/02_model/README.md)：使用各种经典的以及深度学习推荐算法，如交替最小二乘法（[ALS](https://spark.apache.org/docs/latest/api/python/_modules/pyspark/ml/recommendation.html#ALS)）或极限深度分解机（[xDeepFM](https://arxiv.org/abs/1803.05170)）。
-    - [评估](https://github.com/Microsoft/Recommenders/blob/master/notebooks/03_evaluate/README.md)：使用离线指标来评估算法。
-    - [模型选择和优化](https://github.com/Microsoft/Recommenders/blob/master/notebooks/04_model_select_and_optimize/README.md)：为推荐算法模型调优超参。
-    - [运营](https://github.com/Microsoft/Recommenders/blob/master/notebooks/05_operationalize/README.md)：在 Azure 的生产环境上运行模型。
+- [准备数据](https://github.com/Microsoft/Recommenders/blob/master/notebooks/01_prepare_data/README.md)：为每个 Recommender 算法准备并读取数据。
+- [模型](https://github.com/Microsoft/Recommenders/blob/master/notebooks/02_model/README.md)：使用各种经典的以及深度学习推荐算法，如交替最小二乘法（[ALS](https://spark.apache.org/docs/latest/api/python/_modules/pyspark/ml/recommendation.html#ALS)）或极限深度分解机（[xDeepFM](https://arxiv.org/abs/1803.05170)）。
+- [评估](https://github.com/Microsoft/Recommenders/blob/master/notebooks/03_evaluate/README.md)：使用离线指标来评估算法。
+- [模型选择和优化](https://github.com/Microsoft/Recommenders/blob/master/notebooks/04_model_select_and_optimize/README.md)：为推荐算法模型调优超参。
+- [运营](https://github.com/Microsoft/Recommenders/blob/master/notebooks/05_operationalize/README.md)：在 Azure 的生产环境上运行模型。
 
 在第四项调优模型超参的任务上，NNI 可以发挥作用。 在 NNI 上调优推荐模型的具体示例，采用了 [SVD](https://github.com/Microsoft/Recommenders/blob/master/notebooks/02_model/surprise_svd_deep_dive.ipynb) 算法，以及数据集 Movielens100k。 此模型有超过 10 个超参需要调优。
 

docs/zh_CN/ExperimentConfig.md

@@ -425,6 +425,14 @@ machineList:
     
     **gpuIndices** 用于指定 GPU。设置此值后，只有指定的 GPU 会被用来运行 Trial 任务。 可指定单个或多个 GPU 的索引，多个 GPU 之间用逗号（,）隔开，例如 `1` 或 `0,1,3`。
   
+  - **maxTrialNumPerGpu**
+    
+    **maxTrialNumPerGpu** 用于指定每个 GPU 设备上最大并发的 Trial 数量。
+  
+  - **useActiveGpu**
+    
+    **useActiveGpu** 用于指定 NNI 是否使用还有其它进程的 GPU。 默认情况下，NNI 只会使用没有其它进程的空闲 GPU，如果 **useActiveGpu** 设置为 true，NNI 会使用所有 GPU。 此字段不适用于 Windows 版的 NNI。
+
 - **machineList**
   
   如果 **trainingServicePlatform** 为 remote，则需要设置 **machineList**。否则应将其置为空。
@@ -461,6 +469,14 @@ machineList:
     
     **gpuIndices** 用于指定 GPU。设置此值后，远程计算机上只有指定的 GPU 会被用来运行 Trial 任务。 可指定单个或多个 GPU 的索引，多个 GPU 之间用逗号（,）隔开，例如 `1` 或 `0,1,3`。
   
+  - **maxTrialNumPerGpu**
+    
+    **maxTrialNumPerGpu** 用于指定每个 GPU 设备上最大并发的 Trial 数量。
+  
+  - **useActiveGpu**
+    
+    **useActiveGpu** 用于指定 NNI 是否使用还有其它进程的 GPU。 默认情况下，NNI 只会使用没有其它进程的空闲 GPU，如果 **useActiveGpu** 设置为 true，NNI 会使用所有 GPU。 此字段不适用于 Windows 版的 NNI。
+
 - **kubeflowConfig**:
   
   - **operator**