Merge pull request #135 from Microsoft/master

merge master

README.md

@@ -10,6 +10,7 @@
 [![Bugs](https://img.shields.io/github/issues/Microsoft/nni/bug.svg)](https://github.com/Microsoft/nni/issues?q=is%3Aissue+is%3Aopen+label%3Abug)
 [![Pull Requests](https://img.shields.io/github/issues-pr-raw/Microsoft/nni.svg)](https://github.com/Microsoft/nni/pulls?q=is%3Apr+is%3Aopen)
 [![Version](https://img.shields.io/github/release/Microsoft/nni.svg)](https://github.com/Microsoft/nni/releases) [![Join the chat at https://gitter.im/Microsoft/nni](https://badges.gitter.im/Microsoft/nni.svg)](https://gitter.im/Microsoft/nni?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
+[![Documentation Status](https://readthedocs.org/projects/nni/badge/?version=latest)](https://nni.readthedocs.io/en/latest/?badge=latest)
 
 [简体中文](README_zh_CN.md)
 

README_zh_CN.md

@@ -186,7 +186,7 @@ NNI (Neural Network Intelligence) 是自动机器学习（AutoML）的工具包
 * [使用命令行工具 nnictl](docs/zh_CN/NNICTLDOC.md)
 * [使用 NNIBoard](docs/zh_CN/WebUI.md)
 * [如何定义搜索空间](docs/zh_CN/SearchSpaceSpec.md)
-* [如何定义一次 Trial](docs/zh_CN/Trials.md)
+* [如何编写 Trial 代码](docs/zh_CN/Trials.md)
 * [如何选择 Tuner、搜索算法](docs/zh_CN/Builtin_Tuner.md)
 * [配置 Experiment](docs/zh_CN/ExperimentConfig.md)
 * [如何使用 Annotation](docs/zh_CN/Trials.md#nni-python-annotation)

deployment/deployment-pipelines.yml

@@ -26,7 +26,7 @@ jobs:
         echo 'building release package...'
         make build
       fi
-
+    condition: eq( variables['upload_package'], 'true' )
     displayName: 'build nni bdsit_wheel'
   - script: |
       cd deployment/pypi
@@ -38,21 +38,36 @@ jobs:
         echo 'uploading release package to pypi...'
         python3 -m twine upload -u $(pypi_user) -p $(pypi_pwd) dist/*
       fi
+    condition: eq( variables['upload_package'], 'true' )
     displayName: 'upload nni package to pypi/testpypi'
 
   - script: |
-      export IMG_TAG=`git describe --tags --abbrev=0`
       cd deployment/docker
-      docker login -u $(docker_hub_user) -p $(docker_hub_pwd)
 
       if [ $(build_type) = 'prerelease' ]
       then
+        docker login -u $(docker_hub_dev_user) -p $(docker_hub_dev_pwd)
+        export IMG_NAME=$(dev_docker_img)
+        export IMG_TAG=`git describe --tags --abbrev=0`.`date +%y%m%d%H%M`
         echo 'updating docker file for testpyi...'
-        sed -ie 's/RUN python3 -m pip --no-cache-dir install nni/RUN python3 -m pip install --user --no-cache-dir --index-url https:\/\/test.pypi.org\/simple\/ --extra-index-url https:\/\/pypi.org\/simple nni/' Dockerfile
+        sed -ie 's/RUN python3 -m pip --no-cache-dir install nni/RUN python3 -m pip install --user --no-cache-dir --index-url https:\/\/test.pypi.org\/simple --extra-index-url https:\/\/pypi.org\/simple nni/' Dockerfile
+      else
+        docker login -u $(docker_hub_user) -p $(docker_hub_pwd)
+        export IMG_NAME=msranni/nni
+        export IMG_TAG=`git describe --tags --abbrev=0`
       fi
-      echo $IMG_TAG
-      docker build -f Dockerfile -t msranni/nni:$IMG_TAG .
-      docker push msranni/nni:$IMG_TAG
+      echo $IMG_NAME:$IMG_TAG
+      cat Dockerfile
+      docker build -f Dockerfile -t $IMG_NAME:$IMG_TAG .
+      docker push $IMG_NAME:$IMG_TAG
+
+      if [ $(update_latest_tag) = 'true' ]
+      then
+        docker tag $IMG_NAME:$IMG_TAG $IMG_NAME:latest
+        docker push $IMG_NAME:latest
+      fi
+
+    condition: eq( variables['build_docker_img'], 'true' )
     displayName: 'build and upload nni docker image'
 
 - job: 'Build_upload_nni_macos'
@@ -83,7 +98,7 @@ jobs:
         echo 'building release package...'
         PATH=$HOME/Library/Python/3.7/bin:$PATH make build
       fi
-
+    condition: eq( variables['upload_package'], 'true' )
     displayName: 'build nni bdsit_wheel'
   - script: |
       cd deployment/pypi
@@ -95,4 +110,5 @@ jobs:
         echo 'uploading release package to pypi...'
         python3 -m twine upload -u $(pypi_user) -p $(pypi_pwd) dist/*
       fi
+    condition: eq( variables['upload_package'], 'true' )
     displayName: 'upload nni package to pypi/testpypi'

docs/en_US/RELEASE.md

@@ -2,8 +2,8 @@
 
 ## Release 0.5.1 - 1/31/2018
 ### Improvements
-* Making [log directory](https://github.com/Microsoft/nni/blob/v0.5.1/docs/ExperimentConfig.md) configurable
-* Support [different levels of logs](https://github.com/Microsoft/nni/blob/v0.5.1/docs/ExperimentConfig.md), making it easier for debugging 
+* Making [log directory](https://github.com/Microsoft/nni/blob/v0.5.1/docs/en_US/ExperimentConfig.md) configurable
+* Support [different levels of logs](https://github.com/Microsoft/nni/blob/v0.5.1/docs/en_US/ExperimentConfig.md), making it easier for debugging 
 
 ### Documentation
 * Reorganized documentation & New Homepage Released: https://nni.readthedocs.io/en/latest/
@@ -20,9 +20,9 @@
 
 #### New tuner and assessor supports
 
-* Support [Metis tuner](./Builtin_Tuner.md#MetisTuner) as a new NNI tuner. Metis algorithm has been proofed to be well performed for **online** hyper-parameter tuning.
+* Support [Metis tuner](metisTuner.md) as a new NNI tuner. Metis algorithm has been proofed to be well performed for **online** hyper-parameter tuning.
 * Support [ENAS customized tuner](https://github.com/countif/enas_nni), a tuner contributed by github community user, is an algorithm for neural network search, it could learn neural network architecture via reinforcement learning and serve a better performance than NAS.
-* Support [Curve fitting assessor](./Builtin_Tuner.md#Curvefitting) for early stop policy using learning curve extrapolation.
+* Support [Curve fitting assessor](curvefittingAssessor.md) for early stop policy using learning curve extrapolation.
 * Advanced Support of [Weight Sharing](./AdvancedNAS.md): Enable weight sharing for NAS tuners, currently through NFS.
 
 #### Training Service Enhancement
@@ -45,7 +45,7 @@
 
 #### New tuner supports
 
-* Support [network morphism](./Builtin_Tuner.md#NetworkMorphism) as a new tuner
+* Support [network morphism](networkmorphismTuner.md) as a new tuner
 
 #### Training Service improvements
 
@@ -79,8 +79,8 @@
 * [Kubeflow Training service](./KubeflowMode.md)
   * Support tf-operator
   * [Distributed trial example](https://github.com/Microsoft/nni/tree/master/examples/trials/mnist-distributed/dist_mnist.py) on Kubeflow
-* [Grid search tuner](Builtin_Tuner.md#GridSearch) 
-* [Hyperband tuner](Builtin_Tuner.md#Hyperband)
+* [Grid search tuner](gridsearchTuner.md) 
+* [Hyperband tuner](hyperbandAdvisor.md)
 * Support launch NNI experiment on MAC
 * WebUI
   * UI support for hyperband tuner
@@ -163,7 +163,7 @@
 * Support [OpenPAI](https://github.com/Microsoft/pai) Training Platform (See [here](./PAIMode.md) for instructions about how to submit NNI job in pai mode)
   * Support training services on pai mode. NNI trials will be scheduled to run on OpenPAI cluster
   * NNI trial's output (including logs and model file) will be copied to OpenPAI HDFS for further debugging and checking
-* Support [SMAC](https://www.cs.ubc.ca/~hutter/papers/10-TR-SMAC.pdf) tuner (See [here](Builtin_Tuner.md) for instructions about how to use SMAC tuner)
+* Support [SMAC](https://www.cs.ubc.ca/~hutter/papers/10-TR-SMAC.pdf) tuner (See [here](smacTuner.md) for instructions about how to use SMAC tuner)
   * [SMAC](https://www.cs.ubc.ca/~hutter/papers/10-TR-SMAC.pdf) is based on Sequential Model-Based Optimization (SMBO). It adapts the most prominent previously used model class (Gaussian stochastic process models) and introduces the model class of random forests to SMBO to handle categorical parameters. The SMAC supported by NNI is a wrapper on [SMAC3](https://github.com/automl/SMAC3)
 * Support NNI installation on [conda](https://conda.io/docs/index.html) and python virtual environment
 * Others

docs/en_US/Tutorials.rst

@@ -3,6 +3,8 @@ Tutorials
 ######################
 
 ..  toctree::
+    :maxdepth: 2
+
     Installation
     Write Trial<Trials>
     Tuners<tuners>

docs/en_US/curvefittingAssessor.md

@@ -6,18 +6,16 @@ Curve Fitting Assessor is a LPA(learning, predicting, assessing) algorithm. It s
 
 In this algorithm, we use 12 curves to fit the learning curve, the large set of parametric curve models are chosen from [reference paper][1]. The learning curves' shape coincides with our prior knowlwdge about the form of learning curves: They are typically increasing, saturating functions.
 
-<p align="center">
-<img src="./learning_curve.PNG" alt="drawing"/>
-</p>
+![](../img/curvefitting_learning_curve.PNG)
 
 We combine all learning curve models into a single, more powerful model. This combined model is given by a weighted linear combination:
-<p align="center">
-<img src="./f_comb.gif" alt="drawing"/>
-</p>
+
+![](../img/curvefitting_f_comb.gif)
+
 where the new combined parameter vector
-<p align="center">
-<img src="./expression_xi.gif" alt="drawing"/>
-</p>
+
+![](../img/curvefitting_expression_xi.gif)
+
 Assuming additive a Gaussian noise and the noise parameter is initialized to its maximum likelihood estimate.
 
 We determine the maximum probability value of the new combined parameter vector by learing the historical data. Use such value to predict the future trial performance, and stop the inadequate experiments to save computing resource.
@@ -32,9 +30,7 @@ Concretely,this algorithm goes through three stages of learning, predicting and
 
 The figure below is the result of our algorithm on MNIST trial history data, where the green point represents the data obtained by Assessor, the blue point represents the future but unknown data, and the red line is the Curve predicted by the Curve fitting assessor.
 
-<p align="center">
-<img src="./example_of_curve_fitting.PNG" alt="drawing"/>
-</p>
+![](../img/curvefitting_example.PNG)
 
 ## 2. Usage
 To use Curve Fitting Assessor, you should add the following spec in your experiment's YAML config file:

docs/requirements.txt

@@ -8,4 +8,5 @@ hyperopt
 json_tricks
 numpy
 scipy
-coverage
+coverage
\ No newline at end of file
+sklearn
\ No newline at end of file

docs/zh_CN/Builtin_Assessors.md

@@ -64,6 +64,7 @@ assessor:
 * **optimize_mode** (*maximize 或 minimize, 可选, 默认值为 maximize*) - 如果为 'maximize', Assessor 会在结果小于期望值时**终止** Trial。 如果为 'minimize'，Assessor 会在结果大于期望值时**终止** Trial。
 * **start_step** (*int, 可选, 默认值为 6*) - 只有收到 start_step 个中间结果后，才开始判断是否一个 Trial 应该被终止。
 * **threshold** (*float, 可选, 默认值为 0.95*) - 用来确定提前终止较差结果的阈值。 例如，如果 threshold = 0.95, optimize_mode = maximize，最好的历史结果是 0.9，那么会在 Trial 的预测值低于 0.95 * 0.9 = 0.855 时停止。
+* **gap** (*int, 可选, 默认值为 1*) - Assessor 两次评估之间的间隔次数。 例如：如果 gap = 2, start_step = 6，就会评估第 6, 8, 10, 12... 个中间结果。
 
 **使用样例：**
 
@@ -76,4 +77,5 @@ assessor:
       optimize_mode: maximize
       start_step: 6
       threshold: 0.95
+      gap: 1
 ```
\ No newline at end of file

docs/zh_CN/curvefittingAssessor.md

@@ -6,21 +6,15 @@ Curve Fitting Assessor 是一个 LPA (learning, predicting, assessing，即学
 
 此算法中，使用了 12 条曲线来拟合学习曲线，从[参考论文](http://aad.informatik.uni-freiburg.de/papers/15-IJCAI-Extrapolation_of_Learning_Curves.pdf)中选择了大量的参数曲线模型。 学习曲线的形状与先验知识是一致的：都是典型的递增的、饱和的函数。
 
-<p align="center">
-<img src="./learning_curve.PNG" alt="drawing"/>
-</p>
+![](../img/curvefitting_learning_curve.PNG)
 
 所有学习曲线模型被合并到了单个，更强大的模型中。 合并的模型通过加权线性混合：
 
-<p align="center">
-<img src="./f_comb.gif" alt="drawing"/>
-</p>
+![](../img/curvefitting_f_comb.gif)
 
 合并后的参数向量
 
-<p align="center">
-<img src="./expression_xi.gif" alt="drawing"/>
-</p>
+![](../img/curvefitting_expression_xi.gif)
 
 假设增加一个高斯噪声，且噪声参数初始化为最大似然估计。
 
@@ -36,9 +30,7 @@ Curve Fitting Assessor 是一个 LPA (learning, predicting, assessing，即学
 
 下图显示了此算法在 MNIST Trial 历史数据上结果。其中绿点表示 Assessor 获得的数据，蓝点表示将来，但未知的数据，红色线条是 Curve fitting Assessor 的预测曲线。
 
-<p align="center">
-<img src="./example_of_curve_fitting.PNG" alt="drawing"/>
-</p>
+![](../img/curvefitting_example.PNG)
 
 ## 2. 用法
 
@@ -61,6 +53,10 @@ Curve Fitting Assessor 是一个 LPA (learning, predicting, assessing，即学
           * 阈值的默认值是 0.95。
           # 注意：如果选择了 minimize 模式，要让 threshold >= 1.0 (如 threshold=1.1)
           threshold: 0.95
+          # (可选) gap 是两次评估之间的间隔次数。
+          # 例如：如果 gap = 2, start_step = 6，就会评估第 6, 8, 10, 12... 个中间结果。
+          * gap 的默认值是 1。
+          gap: 1
     
 
 ## 3. 文件结构