Port markdown docs to rst and introduce "githublink" (#3107)

docs/en_US/TrialExample/Cifar10Examples.rst

+CIFAR-10 examples
+=================
+
+Overview
+--------
+
+`CIFAR-10 <https://www.cs.toronto.edu/~kriz/cifar.html>`__ classification is a common benchmark problem in machine learning. The CIFAR-10 dataset is the collection of images. It is one of the most widely used datasets for machine learning research which contains 60,000 32x32 color images in 10 different classes. Thus, we use CIFAR-10 classification as an example to introduce NNI usage.
+
+**Goals**
+^^^^^^^^^^^^^
+
+As we all know, the choice of model optimizer is directly affects the performance of the final metrics. The goal of this tutorial is to **tune a better performace optimizer** to train a relatively small convolutional neural network (CNN) for recognizing images.
+
+In this example, we have selected the following common deep learning optimizer:
+
+..
+
+   "SGD", "Adadelta", "Adagrad", "Adam", "Adamax"
+
+
+**Experimental**
+^^^^^^^^^^^^^^^^^^^^
+
+Preparations
+^^^^^^^^^^^^
+
+This example requires PyTorch. PyTorch install package should be chosen based on python version and cuda version.
+
+Here is an example of the environment python==3.5 and cuda == 8.0, then using the following commands to install `PyTorch <https://pytorch.org/>`__\ :
+
+.. code-block:: bash
+
+   python3 -m pip install http://download.pytorch.org/whl/cu80/torch-0.4.1-cp35-cp35m-linux_x86_64.whl
+   python3 -m pip install torchvision
+
+CIFAR-10 with NNI
+^^^^^^^^^^^^^^^^^
+
+**Search Space**
+
+As we stated in the target, we target to find out the best ``optimizer`` for training CIFAR-10 classification. When using different optimizers, we also need to adjust ``learning rates`` and ``network structure`` accordingly. so we chose these three parameters as hyperparameters and write the following search space.
+
+.. code-block:: json
+
+   {
+       "lr":{"_type":"choice", "_value":[0.1, 0.01, 0.001, 0.0001]},
+       "optimizer":{"_type":"choice", "_value":["SGD", "Adadelta", "Adagrad", "Adam", "Adamax"]},
+       "model":{"_type":"choice", "_value":["vgg", "resnet18", "googlenet", "densenet121", "mobilenet", "dpn92", "senet18"]}
+   }
+
+*Implemented code directory: :githublink:`search_space.json <examples/trials/cifar10_pytorch/search_space.json>`*
+
+**Trial**
+
+The code for CNN training of each hyperparameters set, paying particular attention to the following points are specific for NNI:
+
+
+* Use ``nni.get_next_parameter()`` to get next training hyperparameter set.
+* Use ``nni.report_intermediate_result(acc)`` to report the intermedian result after finish each epoch.
+* Use ``nni.report_final_result(acc)`` to report the final result before the trial end.
+
+*Implemented code directory: :githublink:`main.py <examples/trials/cifar10_pytorch/main.py>`*
+
+You can also use your previous code directly, refer to `How to define a trial <Trials.rst>`__ for modify.
+
+**Config**
+
+Here is the example of running this experiment on local(with multiple GPUs):
+
+code directory: :githublink:`examples/trials/cifar10_pytorch/config.yml <examples/trials/cifar10_pytorch/config.yml>`
+
+Here is the example of running this experiment on OpenPAI:
+
+code directory: :githublink:`examples/trials/cifar10_pytorch/config_pai.yml <examples/trials/cifar10_pytorch/config_pai.yml>`
+
+*The complete examples we have implemented: :githublink:`examples/trials/cifar10_pytorch/ <examples/trials/cifar10_pytorch>`*
+
+Launch the experiment
+^^^^^^^^^^^^^^^^^^^^^
+
+We are ready for the experiment, let's now **run the config.yml file from your command line to start the experiment**.
+
+.. code-block:: bash
+
+   nnictl create --config nni/examples/trials/cifar10_pytorch/config.yml

docs/en_US/TrialExample/EfficientNet.rst

+EfficientNet
+============
+
+`EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks <https://arxiv.org/abs/1905.11946>`__
+
+Use Grid search to find the best combination of alpha, beta and gamma for EfficientNet-B1, as discussed in Section 3.3 in paper. Search space, tuner, configuration examples are provided here.
+
+Instructions
+------------
+
+:githublink:`Example code <examples/trials/efficientnet>`
+
+
+#. Set your working directory here in the example code directory.
+#. Run ``git clone https://github.com/ultmaster/EfficientNet-PyTorch`` to clone the `ultmaster modified version <https://github.com/ultmaster/EfficientNet-PyTorch>`__ of the original `EfficientNet-PyTorch <https://github.com/lukemelas/EfficientNet-PyTorch>`__. The modifications were done to adhere to the original `Tensorflow version <https://github.com/tensorflow/tpu/tree/master/models/official/efficientnet>`__ as close as possible (including EMA, label smoothing and etc.); also added are the part which gets parameters from tuner and reports intermediate/final results. Clone it into ``EfficientNet-PyTorch``\ ; the files like ``main.py``\ , ``train_imagenet.sh`` will appear inside, as specified in the configuration files.
+#. Run ``nnictl create --config config_local.yml`` (use ``config_pai.yml`` for OpenPAI) to find the best EfficientNet-B1. Adjust the training service (PAI/local/remote), batch size in the config files according to the environment.
+
+For training on ImageNet, read ``EfficientNet-PyTorch/train_imagenet.sh``. Download ImageNet beforehand and extract it adhering to `PyTorch format <https://pytorch.org/docs/stable/torchvision/datasets.html#imagenet>`__ and then replace ``/mnt/data/imagenet`` in with the location of the ImageNet storage. This file should also be a good example to follow for mounting ImageNet into the container on OpenPAI.
+
+Results
+-------
+
+The follow image is a screenshot, demonstrating the relationship between acc@1 and alpha, beta, gamma.
+
+
+.. image:: ../../img/efficientnet_search_result.png
+   :target: ../../img/efficientnet_search_result.png
+   :alt: 
+

docs/en_US/TrialExample/GbdtExample.rst

+GBDT in nni
+===========
+
+Gradient boosting is a machine learning technique for regression and classification problems, which produces a prediction model in the form of an ensemble of weak prediction models, typically decision trees. It builds the model in a stage-wise fashion as other boosting methods do, and it generalizes them by allowing optimization of an arbitrary differentiable loss function.
+
+Gradient boosting decision tree has many popular implementations, such as `lightgbm <https://github.com/Microsoft/LightGBM>`__\ , `xgboost <https://github.com/dmlc/xgboost>`__\ , and `catboost <https://github.com/catboost/catboost>`__\ , etc. GBDT is a great tool for solving the problem of traditional machine learning problem. Since GBDT is a robust algorithm, it could use in many domains. The better hyper-parameters for GBDT, the better performance you could achieve.
+
+NNI is a great platform for tuning hyper-parameters, you could try various builtin search algorithm in nni and run multiple trials concurrently.
+
+1. Search Space in GBDT
+-----------------------
+
+There are many hyper-parameters in GBDT, but what kind of parameters will affect the performance or speed? Based on some practical experience, some suggestion here(Take lightgbm as example):
+
+..
+
+   * For better accuracy
+   * ``learning_rate``. The range of ``learning rate`` could be [0.001, 0.9].
+
+
+
+* 
+  ``num_leaves``. ``num_leaves`` is related to ``max_depth``\ , you don't have to tune both of them.
+
+* 
+  ``bagging_freq``. ``bagging_freq`` could be [1, 2, 4, 8, 10]
+
+* 
+  ``num_iterations``. May larger if underfitting.
+
+..
+
+   * For speed up
+   * ``bagging_fraction``. The range of ``bagging_fraction`` could be [0.7, 1.0].
+
+
+
+* 
+  ``feature_fraction``. The range of ``feature_fraction`` could be [0.6, 1.0].
+
+* 
+  ``max_bin``.
+
+..
+
+   * To avoid overfitting
+   * ``min_data_in_leaf``. This depends on your dataset.
+
+
+
+* 
+  ``min_sum_hessian_in_leaf``. This depend on your dataset.
+
+* 
+  ``lambda_l1`` and ``lambda_l2``.
+
+* 
+  ``min_gain_to_split``.
+
+* 
+  ``num_leaves``.
+
+Reference link:
+`lightgbm <https://lightgbm.readthedocs.io/en/latest/Parameters-Tuning.html>`__ and `autoxgoboost <https://github.com/ja-thomas/autoxgboost/blob/master/poster_2018.pdf>`__
+
+2. Task description
+-------------------
+
+Now we come back to our example "auto-gbdt" which run in lightgbm and nni. The data including :githublink:`train data <examples/trials/auto-gbdt/data/regression.train>` and :githublink:`test data <examples/trials/auto-gbdt/data/regression.train>`.
+Given the features and label in train data, we train a GBDT regression model and use it to predict.
+
+3. How to run in nni
+--------------------
+
+3.1 Install all the requirments
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. code-block:: bash
+
+   pip install lightgbm
+   pip install pandas
+
+3.2 Prepare your trial code
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+You need to prepare a basic code as following:
+
+.. code-block:: python
+
+   ...
+
+   def get_default_parameters():
+       ...
+       return params
+
+
+   def load_data(train_path='./data/regression.train', test_path='./data/regression.test'):
+       '''
+       Load or create dataset
+       '''
+       ...
+
+       return lgb_train, lgb_eval, X_test, y_test
+
+   def run(lgb_train, lgb_eval, params, X_test, y_test):
+       # train
+       gbm = lgb.train(params,
+                       lgb_train,
+                       num_boost_round=20,
+                       valid_sets=lgb_eval,
+                       early_stopping_rounds=5)
+       # predict
+       y_pred = gbm.predict(X_test, num_iteration=gbm.best_iteration)
+
+       # eval
+       rmse = mean_squared_error(y_test, y_pred) ** 0.5
+       print('The rmse of prediction is:', rmse)
+
+   if __name__ == '__main__':
+       lgb_train, lgb_eval, X_test, y_test = load_data()
+
+       PARAMS = get_default_parameters()
+       # train
+       run(lgb_train, lgb_eval, PARAMS, X_test, y_test)
+
+3.3 Prepare your search space.
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+If you like to tune ``num_leaves``\ , ``learning_rate``\ , ``bagging_fraction`` and ``bagging_freq``\ , you could write a :githublink:`search_space.json <examples/trials/auto-gbdt/search_space.json>` as follow:
+
+.. code-block:: json
+
+   {
+       "num_leaves":{"_type":"choice","_value":[31, 28, 24, 20]},
+       "learning_rate":{"_type":"choice","_value":[0.01, 0.05, 0.1, 0.2]},
+       "bagging_fraction":{"_type":"uniform","_value":[0.7, 1.0]},
+       "bagging_freq":{"_type":"choice","_value":[1, 2, 4, 8, 10]}
+   }
+
+More support variable type you could reference `here <../Tutorial/SearchSpaceSpec.rst>`__.
+
+3.4 Add SDK of nni into your code.
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. code-block:: diff
+
+   +import nni
+   ...
+
+   def get_default_parameters():
+       ...
+       return params
+
+
+   def load_data(train_path='./data/regression.train', test_path='./data/regression.test'):
+       '''
+       Load or create dataset
+       '''
+       ...
+
+       return lgb_train, lgb_eval, X_test, y_test
+
+   def run(lgb_train, lgb_eval, params, X_test, y_test):
+       # train
+       gbm = lgb.train(params,
+                       lgb_train,
+                       num_boost_round=20,
+                       valid_sets=lgb_eval,
+                       early_stopping_rounds=5)
+       # predict
+       y_pred = gbm.predict(X_test, num_iteration=gbm.best_iteration)
+
+       # eval
+       rmse = mean_squared_error(y_test, y_pred) ** 0.5
+       print('The rmse of prediction is:', rmse)
+   +   nni.report_final_result(rmse)
+
+   if __name__ == '__main__':
+       lgb_train, lgb_eval, X_test, y_test = load_data()
+   +   RECEIVED_PARAMS = nni.get_next_parameter()
+       PARAMS = get_default_parameters()
+   +   PARAMS.update(RECEIVED_PARAMS)
+
+       # train
+       run(lgb_train, lgb_eval, PARAMS, X_test, y_test)
+
+3.5 Write a config file and run it.
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+In the config file, you could set some settings including:
+
+
+* Experiment setting: ``trialConcurrency``\ , ``maxExecDuration``\ , ``maxTrialNum``\ , ``trial gpuNum``\ , etc.
+* Platform setting: ``trainingServicePlatform``\ , etc.
+* Path seeting: ``searchSpacePath``\ , ``trial codeDir``\ , etc.
+* Algorithm setting: select ``tuner`` algorithm, ``tuner optimize_mode``\ , etc.
+
+An config.yml as follow:
+
+.. code-block:: yaml
+
+   authorName: default
+   experimentName: example_auto-gbdt
+   trialConcurrency: 1
+   maxExecDuration: 10h
+   maxTrialNum: 10
+   #choice: local, remote, pai
+   trainingServicePlatform: local
+   searchSpacePath: search_space.json
+   #choice: true, false
+   useAnnotation: false
+   tuner:
+     #choice: TPE, Random, Anneal, Evolution, BatchTuner
+     #SMAC (SMAC should be installed through nnictl)
+     builtinTunerName: TPE
+     classArgs:
+       #choice: maximize, minimize
+       optimize_mode: minimize
+   trial:
+     command: python3 main.py
+     codeDir: .
+     gpuNum: 0
+
+Run this experiment with command as follow:
+
+.. code-block:: bash
+
+   nnictl create --config ./config.yml

docs/en_US/TrialExample/KDExample.rst

+Knowledge Distillation on NNI
+=============================
+
+KnowledgeDistill
+----------------
+
+Knowledge distillation support, in `Distilling the Knowledge in a Neural Network <https://arxiv.org/abs/1503.02531>`__\ ,  the compressed model is trained to mimic a pre-trained, larger model.  This training setting is also referred to as "teacher-student",  where the large model is the teacher and the small model is the student.
+
+
+.. image:: ../../img/distill.png
+   :target: ../../img/distill.png
+   :alt: 
+
+
+Usage
+^^^^^
+
+PyTorch code
+
+.. code-block:: python
+
+   from knowledge_distill.knowledge_distill import KnowledgeDistill
+   kd = KnowledgeDistill(kd_teacher_model, kd_T=5)
+   alpha = 1
+   beta = 0.8
+   for batch_idx, (data, target) in enumerate(train_loader):
+       data, target = data.to(device), target.to(device)
+       optimizer.zero_grad()
+       output = model(data)
+       loss = F.cross_entropy(output, target)
+       # you only to add the following line to fine-tune with knowledge distillation
+       loss = alpha * loss + beta * kd.loss(data=data, student_out=output)
+       loss.backward()
+
+User configuration for KnowledgeDistill
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+
+* **kd_teacher_model:** The pre-trained teacher model 
+* **kd_T:** Temperature for smoothing teacher model's output
+
+The complete code can be found `here <https://github.com/microsoft/nni/tree/v1.3/examples/model_compress/knowledge_distill/>`__

docs/en_US/TrialExample/MnistExamples.rst

+.. role:: raw-html(raw)
+   :format: html
+
+
+MNIST examples
+==============
+
+CNN MNIST classifier for deep learning is similar to ``hello world`` for programming languages. Thus, we use MNIST as example to introduce different features of NNI. The examples are listed below:
+
+
+* `MNIST with NNI API (TensorFlow v1.x) <#mnist-tfv1>`__
+* `MNIST with NNI API (TensorFlow v2.x) <#mnist-tfv2>`__
+* `MNIST with NNI annotation <#mnist-annotation>`__
+* `MNIST in keras <#mnist-keras>`__
+* `MNIST -- tuning with batch tuner <#mnist-batch>`__
+* `MNIST -- tuning with hyperband <#mnist-hyperband>`__
+* `MNIST -- tuning within a nested search space <#mnist-nested>`__
+* `distributed MNIST (tensorflow) using kubeflow <#mnist-kubeflow-tf>`__
+* `distributed MNIST (pytorch) using kubeflow <#mnist-kubeflow-pytorch>`__
+
+:raw-html:`<a name="mnist-tfv1"></a>`
+**MNIST with NNI API (TensorFlow v1.x)**
+
+This is a simple network which has two convolutional layers, two pooling layers and a fully connected layer. We tune hyperparameters, such as dropout rate, convolution size, hidden size, etc. It can be tuned with most NNI built-in tuners, such as TPE, SMAC, Random. We also provide an exmaple YAML file which enables assessor.
+
+``code directory: examples/trials/mnist-tfv1/``
+
+:raw-html:`<a name="mnist-tfv2"></a>`
+**MNIST with NNI API (TensorFlow v2.x)**
+
+Same network to the example above, but written in TensorFlow v2.x Keras API.
+
+``code directory: examples/trials/mnist-tfv2/``
+
+:raw-html:`<a name="mnist-annotation"></a>`
+**MNIST with NNI annotation**
+
+This example is similar to the example above, the only difference is that this example uses NNI annotation to specify search space and report results, while the example above uses NNI apis to receive configuration and report results.
+
+``code directory: examples/trials/mnist-annotation/``
+
+:raw-html:`<a name="mnist-keras"></a>`
+**MNIST in keras**
+
+This example is implemented in keras. It is also a network for MNIST dataset, with two convolution layers, one pooling layer, and two fully connected layers.
+
+``code directory: examples/trials/mnist-keras/``
+
+:raw-html:`<a name="mnist-batch"></a>`
+**MNIST -- tuning with batch tuner**
+
+This example is to show how to use batch tuner. Users simply list all the configurations they want to try in the search space file. NNI will try all of them.
+
+``code directory: examples/trials/mnist-batch-tune-keras/``
+
+:raw-html:`<a name="mnist-hyperband"></a>`
+**MNIST -- tuning with hyperband**
+
+This example is to show how to use hyperband to tune the model. There is one more key ``STEPS`` in the received configuration for trials to control how long it can run (e.g., number of iterations).
+
+``code directory: examples/trials/mnist-hyperband/``
+
+:raw-html:`<a name="mnist-nested"></a>`
+**MNIST -- tuning within a nested search space**
+
+This example is to show that NNI also support nested search space. The search space file is an example of how to define nested search space.
+
+``code directory: examples/trials/mnist-nested-search-space/``
+
+:raw-html:`<a name="mnist-kubeflow-tf"></a>`
+**distributed MNIST (tensorflow) using kubeflow**
+
+This example is to show how to run distributed training on kubeflow through NNI. Users can simply provide distributed training code and a configure file which specifies the kubeflow mode. For example, what is the command to run ps and what is the command to run worker, and how many resources they consume. This example is implemented in tensorflow, thus, uses kubeflow tensorflow operator.
+
+``code directory: examples/trials/mnist-distributed/``
+
+:raw-html:`<a name="mnist-kubeflow-pytorch"></a>`
+**distributed MNIST (pytorch) using kubeflow**
+
+Similar to the previous example, the difference is that this example is implemented in pytorch, thus, it uses kubeflow pytorch operator.
+
+``code directory: examples/trials/mnist-distributed-pytorch/``

docs/en_US/TrialExample/OpEvoExamples.rst

+.. role:: raw-html(raw)
+   :format: html
+
+
+Tuning Tensor Operators on NNI
+==============================
+
+Overview
+--------
+
+Abundant applications raise the demands of training and inference deep neural networks (DNNs) efficiently on diverse hardware platforms ranging from cloud servers to embedded devices. Moreover, computational graph-level optimization of deep neural network, like tensor operator fusion, may introduce new tensor operators. Thus, manually optimized tensor operators provided by hardware-specific libraries have limitations in terms of supporting new hardware platforms or supporting new operators, so automatically optimizing tensor operators on diverse hardware platforms is essential for large-scale deployment and application of deep learning technologies in the real-world problems.
+
+Tensor operator optimization is substantially a combinatorial optimization problem. The objective function is the performance of a tensor operator on specific hardware platform, which should be maximized with respect to the hyper-parameters of corresponding device code, such as how to tile a matrix or whether to unroll a loop. Unlike many typical problems of this type, such as travelling salesman problem, the objective function of tensor operator optimization is a black box and expensive to sample. One has to compile a device code with a specific configuration and run it on real hardware to get the corresponding performance metric. Therefore, a desired method for optimizing tensor operators should find the best configuration with as few samples as possible.
+
+The expensive objective function makes solving tensor operator optimization problem with traditional combinatorial optimization methods, for example, simulated annealing and evolutionary algorithms, almost impossible. Although these algorithms inherently support combinatorial search spaces, they do not take sample-efficiency into account,
+thus thousands of or even more samples are usually needed, which is unacceptable when tuning tensor operators in product environments. On the other hand, sequential model based optimization (SMBO) methods are proved sample-efficient for optimizing black-box functions with continuous search spaces. However, when optimizing ones with combinatorial search spaces, SMBO methods are not as sample-efficient as their continuous counterparts, because there is lack of prior assumptions about the objective functions, such as continuity and differentiability in the case of continuous search spaces. For example, if one could assume an objective function with a continuous search space is infinitely differentiable, a Gaussian process with a radial basis function (RBF) kernel could be used to model the objective function. In this way, a sample provides not only a single value at a point but also the local properties of the objective function in its neighborhood or even global properties,
+which results in a high sample-efficiency. In contrast, SMBO methods for combinatorial optimization suffer poor sample-efficiency due to the lack of proper prior assumptions and surrogate models which can leverage them.
+
+OpEvo is recently proposed for solving this challenging problem. It efficiently explores the search spaces of tensor operators by introducing a topology-aware mutation operation based on q-random walk distribution to leverage the topological structures over the search spaces. Following this example, you can use OpEvo to tune three representative types of tensor operators selected from two popular neural networks, BERT and AlexNet. Three comparison baselines, AutoTVM, G-BFS and N-A2C, are also provided. Please refer to `OpEvo: An Evolutionary Method for Tensor Operator Optimization <https://arxiv.org/abs/2006.05664>`__ for detailed explanation about these algorithms.
+
+Environment Setup
+-----------------
+
+We prepared a dockerfile for setting up experiment environments. Before starting, please make sure the Docker daemon is running and the driver of your GPU accelerator is properly installed. Enter into the example folder ``examples/trials/systems/opevo`` and run below command to build and instantiate a Docker image from the dockerfile.
+
+.. code-block:: bash
+
+   # if you are using Nvidia GPU
+   make cuda-env
+   # if you are using AMD GPU
+   make rocm-env
+
+Run Experiments:
+----------------
+
+Three representative kinds of tensor operators, **matrix multiplication**\ ,** batched matrix multiplication** and **2D convolution**\ , are chosen from BERT and AlexNet, and tuned with NNI. The ``Trial`` code for all tensor operators is ``/root/compiler_auto_tune_stable.py``\ , and ``Search Space`` files and ``config`` files for each tuning algorithm locate in ``/root/experiments/``\ , which are categorized by tensor operators. Here ``/root`` refers to the root of the container.
+
+For tuning the operators of matrix multiplication, please run below commands from ``/root``\ :
+
+.. code-block:: bash
+
+   # (N, K) x (K, M) represents a matrix of shape (N, K) multiplies a matrix of shape (K, M)
+
+   # (512, 1024) x (1024, 1024)
+   # tuning with OpEvo
+   nnictl create --config experiments/mm/N512K1024M1024/config_opevo.yml
+   # tuning with G-BFS
+   nnictl create --config experiments/mm/N512K1024M1024/config_gbfs.yml
+   # tuning with N-A2C
+   nnictl create --config experiments/mm/N512K1024M1024/config_na2c.yml
+   # tuning with AutoTVM
+   OP=matmul STEP=512 N=512 M=1024 K=1024 P=NN ./run.s
+
+   # (512, 1024) x (1024, 4096)
+   # tuning with OpEvo
+   nnictl create --config experiments/mm/N512K1024M4096/config_opevo.yml
+   # tuning with G-BFS
+   nnictl create --config experiments/mm/N512K1024M4096/config_gbfs.yml
+   # tuning with N-A2C
+   nnictl create --config experiments/mm/N512K1024M4096/config_na2c.yml
+   # tuning with AutoTVM
+   OP=matmul STEP=512 N=512 M=1024 K=4096 P=NN ./run.sh
+
+   # (512, 4096) x (4096, 1024)
+   # tuning with OpEvo
+   nnictl create --config experiments/mm/N512K4096M1024/config_opevo.yml
+   # tuning with G-BFS
+   nnictl create --config experiments/mm/N512K4096M1024/config_gbfs.yml
+   # tuning with N-A2C
+   nnictl create --config experiments/mm/N512K4096M1024/config_na2c.yml
+   # tuning with AutoTVM
+   OP=matmul STEP=512 N=512 M=4096 K=1024 P=NN ./run.sh
+
+For tuning the operators of batched matrix multiplication, please run below commands from ``/root``\ :
+
+.. code-block:: bash
+
+   # batched matrix with batch size 960 and shape of matrix (128, 128) multiplies batched matrix with batch size 960 and shape of matrix (128, 64)
+   # tuning with OpEvo
+   nnictl create --config experiments/bmm/B960N128K128M64PNN/config_opevo.yml
+   # tuning with AutoTVM
+   OP=batch_matmul STEP=512 B=960 N=128 K=128 M=64 P=NN ./run.sh
+
+   # batched matrix with batch size 960 and shape of matrix (128, 128) is transposed first and then multiplies batched matrix with batch size 960 and shape of matrix (128, 64)
+   # tuning with OpEvo
+   nnictl create --config experiments/bmm/B960N128K128M64PTN/config_opevo.yml
+   # tuning with AutoTVM
+   OP=batch_matmul STEP=512 B=960 N=128 K=128 M=64 P=TN ./run.sh
+
+   # batched matrix with batch size 960 and shape of matrix (128, 64) is transposed first and then right multiplies batched matrix with batch size 960 and shape of matrix (128, 64).
+   # tuning with OpEvo
+   nnictl create --config experiments/bmm/B960N128K64M128PNT/config_opevo.yml
+   # tuning with AutoTVM
+   OP=batch_matmul STEP=512 B=960 N=128 K=64 M=128 P=NT ./run.sh
+
+For tuning the operators of 2D convolution, please run below commands from ``/root``\ :
+
+.. code-block:: bash
+
+   # image tensor of shape (512, 3, 227, 227) convolves with kernel tensor of shape (64, 3, 11, 11) with stride 4 and padding 0
+   # tuning with OpEvo
+   nnictl create --config experiments/conv/N512C3HW227F64K11ST4PD0/config_opevo.yml
+   # tuning with AutoTVM
+   OP=convfwd_direct STEP=512 N=512 C=3 H=227 W=227 F=64 K=11 ST=4 PD=0 ./run.sh
+
+   # image tensor of shape (512, 64, 27, 27) convolves with kernel tensor of shape (192, 64, 5, 5) with stride 1 and padding 2
+   # tuning with OpEvo
+   nnictl create --config experiments/conv/N512C64HW27F192K5ST1PD2/config_opevo.yml
+   # tuning with AutoTVM
+   OP=convfwd_direct STEP=512 N=512 C=64 H=27 W=27 F=192 K=5 ST=1 PD=2 ./run.sh
+
+Please note that G-BFS and N-A2C are only designed for tuning tiling schemes of multiplication of matrices with only power of 2 rows and columns, so they are not compatible with other types of configuration spaces, thus not eligible to tune the operators of batched matrix multiplication and 2D convolution. Here, AutoTVM is implemented by its authors in the TVM project, so the tuning results are printed on the screen rather than reported to NNI manager. The port 8080 of the container is bind to the host on the same port, so one can access the NNI Web UI through ``host_ip_addr:8080`` and monitor tuning process as below screenshot.
+
+:raw-html:`<img src="../../../examples/trials/systems/opevo/screenshot.png" />`
+
+Citing OpEvo
+------------
+
+If you feel OpEvo is helpful, please consider citing the paper as follows:
+
+.. code-block:: bash
+
+   @misc{gao2020opevo,
+       title={OpEvo: An Evolutionary Method for Tensor Operator Optimization},
+       author={Xiaotian Gao and Cui Wei and Lintao Zhang and Mao Yang},
+       year={2020},
+       eprint={2006.05664},
+       archivePrefix={arXiv},
+       primaryClass={cs.LG}
+   }

docs/en_US/TrialExample/RocksdbExamples.rst

+Tuning RocksDB on NNI
+=====================
+
+Overview
+--------
+
+`RocksDB <https://github.com/facebook/rocksdb>`__ is a popular high performance embedded key-value database used in production systems at various web-scale enterprises including Facebook, Yahoo!, and LinkedIn.. It is a fork of `LevelDB <https://github.com/google/leveldb>`__ by Facebook optimized to exploit many central processing unit (CPU) cores, and make efficient use of fast storage, such as solid-state drives (SSD), for input/output (I/O) bound workloads.
+
+The performance of RocksDB is highly contingent on its tuning. However, because of the complexity of its underlying technology and a large number of configurable parameters, a good configuration is sometimes hard to obtain. NNI can help to address this issue. NNI supports many kinds of tuning algorithms to search the best configuration of RocksDB, and support many kinds of environments like local machine, remote servers and cloud. 
+
+This example illustrates how to use NNI to search the best configuration of RocksDB for a ``fillrandom`` benchmark supported by a benchmark tool ``db_bench``\ , which is an official benchmark tool provided by RocksDB itself. Therefore, before running this example, please make sure NNI is installed and `\ ``db_bench`` <https://github.com/facebook/rocksdb/wiki/Benchmarking-tools>`__ is in your ``PATH``. Please refer to `here <../Tutorial/QuickStart.md>`__ for detailed information about installation and preparing of NNI environment, and `here <https://github.com/facebook/rocksdb/blob/master/INSTALL.rst>`__ for compiling RocksDB as well as ``db_bench``.
+
+We also provide a simple script :githublink:`db_bench_installation.sh <examples/trials/systems/rocksdb-fillrandom/db_bench_installation.sh>` helping to compile and install ``db_bench`` as well as its dependencies on Ubuntu. Installing RocksDB on other systems can follow the same procedure.
+
+*code directory: :githublink:`example/trials/systems/rocksdb-fillrandom <examples/trials/systems/rocksdb-fillrandom>`*
+
+Experiment setup
+----------------
+
+There are mainly three steps to setup an experiment of tuning systems on NNI. Define search space with a ``json`` file, write a benchmark code, and start NNI experiment by passing a config file to NNI manager.
+
+Search Space
+^^^^^^^^^^^^
+
+For simplicity, this example tunes three parameters, ``write_buffer_size``\ , ``min_write_buffer_num`` and ``level0_file_num_compaction_trigger``\ , for writing 16M keys with 20 Bytes of key size and 100 Bytes of value size randomly, based on writing operations per second (OPS). ``write_buffer_size`` sets the size of a single memtable. Once memtable exceeds this size, it is marked immutable and a new one is created. ``min_write_buffer_num`` is the minimum number of memtables to be merged before flushing to storage. Once the number of files in level 0 reaches ``level0_file_num_compaction_trigger``\ , level 0 to level 1 compaction is triggered.
+
+In this example, the search space is specified by a ``search_space.json`` file as shown below. Detailed explanation of search space could be found `here <../Tutorial/SearchSpaceSpec.rst>`__.
+
+.. code-block:: json
+
+   {
+       "write_buffer_size": {
+           "_type": "quniform",
+           "_value": [2097152, 16777216, 1048576]
+       },
+       "min_write_buffer_number_to_merge": {
+           "_type": "quniform",
+           "_value": [2, 16, 1]
+       },
+       "level0_file_num_compaction_trigger": {
+           "_type": "quniform",
+           "_value": [2, 16, 1]
+       }
+   }
+
+*code directory: :githublink:`example/trials/systems/rocksdb-fillrandom/search_space.json <examples/trials/systems/rocksdb-fillrandom/search_space.json>`*
+
+Benchmark code
+^^^^^^^^^^^^^^
+
+Benchmark code should receive a configuration from NNI manager, and report the corresponding benchmark result back. Following NNI APIs are designed for this purpose. In this example, writing operations per second (OPS) is used as a performance metric. Please refer to `here <Trials.rst>`__ for detailed information.
+
+
+* Use ``nni.get_next_parameter()`` to get next system configuration.
+* Use ``nni.report_final_result(metric)`` to report the benchmark result.
+
+*code directory: :githublink:`example/trials/systems/rocksdb-fillrandom/main.py <examples/trials/systems/rocksdb-fillrandom/main.py>`*
+
+Config file
+^^^^^^^^^^^
+
+One could start a NNI experiment with a config file. A config file for NNI is a ``yaml`` file usually including experiment settings (\ ``trialConcurrency``\ , ``maxExecDuration``\ , ``maxTrialNum``\ , ``trial gpuNum``\ , etc.), platform settings (\ ``trainingServicePlatform``\ , etc.), path settings (\ ``searchSpacePath``\ , ``trial codeDir``\ , etc.) and tuner settings (\ ``tuner``\ , ``tuner optimize_mode``\ , etc.). Please refer to `here <../Tutorial/QuickStart.rst>`__ for more information.
+
+Here is an example of tuning RocksDB with SMAC algorithm:
+
+*code directory: :githublink:`example/trials/systems/rocksdb-fillrandom/config_smac.yml <examples/trials/systems/rocksdb-fillrandom/config_smac.yml>`*
+
+Here is an example of tuning RocksDB with TPE algorithm:
+
+*code directory: :githublink:`example/trials/systems/rocksdb-fillrandom/config_tpe.yml <examples/trials/systems/rocksdb-fillrandom/config_tpe.yml>`*
+
+Other tuners can be easily adopted in the same way. Please refer to `here <../Tuner/BuiltinTuner.rst>`__ for more information.
+
+Finally, we could enter the example folder and start the experiment using following commands:
+
+.. code-block:: bash
+
+   # tuning RocksDB with SMAC tuner
+   nnictl create --config ./config_smac.yml
+   # tuning RocksDB with TPE tuner
+   nnictl create --config ./config_tpe.yml
+
+Experiment results
+------------------
+
+We ran these two examples on the same machine with following details:
+
+
+* 16 * Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz
+* 465 GB of rotational hard drive with ext4 file system
+* 128 GB of RAM
+* Kernel version: 4.15.0-58-generic
+* NNI version: v1.0-37-g1bd24577
+* RocksDB version: 6.4
+* RocksDB DEBUG_LEVEL: 0
+
+The detailed experiment results are shown in the below figure. Horizontal axis is sequential order of trials. Vertical axis is the metric, write OPS in this example. Blue dots represent trials for tuning RocksDB with SMAC tuner, and orange dots stand for trials for tuning RocksDB with TPE tuner. 
+
+
+.. image:: https://github.com/microsoft/nni/tree/v1.9/examples/trials/systems/rocksdb-fillrandom/plot.png
+   :target: https://github.com/microsoft/nni/tree/v1.9/examples/trials/systems/rocksdb-fillrandom/plot.png
+   :alt: image
+
+
+Following table lists the best trials and corresponding parameters and metric obtained by the two tuners. Unsurprisingly, both of them found the same optimal configuration for ``fillrandom`` benchmark.
+
+.. list-table::
+   :header-rows: 1
+   :widths: auto
+
+   * - Tuner
+     - Best trial
+     - Best OPS
+     - write_buffer_size
+     - min_write_buffer_number_to_merge
+     - level0_file_num_compaction_trigger
+   * - SMAC
+     - 255
+     - 779289
+     - 2097152
+     - 7.0
+     - 7.0
+   * - TPE
+     - 169
+     - 761456
+     - 2097152
+     - 7.0
+     - 7.0
+

docs/en_US/TrialExample/SklearnExamples.rst

+Scikit-learn in NNI
+===================
+
+`Scikit-learn <https://github.com/scikit-learn/scikit-learn>`__ is a popular machine learning tool for data mining and data analysis. It supports many kinds of machine learning models like LinearRegression, LogisticRegression, DecisionTree, SVM etc. How to make the use of scikit-learn more efficiency is a valuable topic.
+
+NNI supports many kinds of tuning algorithms to search the best models and/or hyper-parameters for scikit-learn, and support many kinds of environments like local machine, remote servers and cloud.
+
+1. How to run the example
+-------------------------
+
+To start using NNI, you should install the NNI package, and use the command line tool ``nnictl`` to start an experiment. For more information about installation and preparing for the environment,  please refer `here <../Tutorial/QuickStart.rst>`__.
+
+After you installed NNI, you could enter the corresponding folder and start the experiment using following commands:
+
+.. code-block:: bash
+
+   nnictl create --config ./config.yml
+
+2. Description of the example
+-----------------------------
+
+2.1 classification
+^^^^^^^^^^^^^^^^^^
+
+This example uses the dataset of digits, which is made up of 1797 8x8 images, and each image is a hand-written digit, the goal is to classify these images into 10 classes.
+
+In this example, we use SVC as the model, and choose some parameters of this model, including ``"C", "kernel", "degree", "gamma" and "coef0"``. For more information of these parameters, please `refer <https://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html>`__.
+
+2.2 regression
+^^^^^^^^^^^^^^
+
+This example uses the Boston Housing Dataset, this dataset consists of price of houses in various places in Boston and the information such as Crime (CRIM), areas of non-retail business in the town (INDUS), the age of people who own the house (AGE) etc., to predict the house price of Boston.
+
+In this example, we tune different kinds of regression models including ``"LinearRegression", "SVR", "KNeighborsRegressor", "DecisionTreeRegressor"`` and some parameters like ``"svr_kernel", "knr_weights"``. You could get more details about these models from `here <https://scikit-learn.org/stable/supervised_learning.html#supervised-learning>`__.
+
+3. How to write scikit-learn code using NNI
+-------------------------------------------
+
+It is easy to use NNI in your scikit-learn code, there are only a few steps.
+
+
+* 
+  **step 1**
+
+  Prepare a search_space.json to storage your choose spaces.
+  For example, if you want to choose different models, you may try:
+
+  .. code-block:: json
+
+     {
+       "model_name":{"_type":"choice","_value":["LinearRegression", "SVR", "KNeighborsRegressor", "DecisionTreeRegressor"]}
+     }
+
+  If you want to choose different models and parameters, you could put them together in a search_space.json file.
+
+  .. code-block:: json
+
+     {
+       "model_name":{"_type":"choice","_value":["LinearRegression", "SVR", "KNeighborsRegressor", "DecisionTreeRegressor"]},
+       "svr_kernel": {"_type":"choice","_value":["linear", "poly", "rbf"]},
+       "knr_weights": {"_type":"choice","_value":["uniform", "distance"]}
+     }
+
+  Then you could read these values as a dict from your python code, please get into the step 2.
+
+* 
+  **step 2**
+
+  At the beginning of your python code, you should ``import nni`` to insure the packages works normally.
+
+  First, you should use ``nni.get_next_parameter()`` function to get your parameters given by NNI. Then you could use these parameters to update your code.
+  For example, if you define your search_space.json like following format:
+
+  .. code-block:: json
+
+     {
+       "C": {"_type":"uniform","_value":[0.1, 1]},
+       "kernel": {"_type":"choice","_value":["linear", "rbf", "poly", "sigmoid"]},
+       "degree": {"_type":"choice","_value":[1, 2, 3, 4]},
+       "gamma": {"_type":"uniform","_value":[0.01, 0.1]},
+       "coef0": {"_type":"uniform","_value":[0.01, 0.1]}
+     }
+
+  You may get a parameter dict like this:
+
+  .. code-block:: python
+
+     params = {
+           'C': 1.0,
+           'kernel': 'linear',
+           'degree': 3,
+           'gamma': 0.01,
+           'coef0': 0.01
+     }
+
+  Then you could use these variables to write your scikit-learn code.
+
+* 
+  **step 3**
+
+  After you finished your training, you could get your own score of the model, like your precision, recall or MSE etc. NNI needs your score to tuner algorithms and generate next group of parameters, please report the score back to NNI and start next trial job.
+
+  You just need to use ``nni.report_final_result(score)`` to communicate with NNI after you process your scikit-learn code. Or if you have multiple scores in the steps of training, you could also report them back to NNI using ``nni.report_intemediate_result(score)``. Note, you may not report intermediate result of your job, but you must report back your final result.

docs/en_US/TrialExample/SquadEvolutionExamples.rst

+Automatic Model Architecture Search for Reading Comprehension
+=============================================================
+
+This example shows us how to use Genetic Algorithm to find good model architectures for Reading Comprehension.
+
+1. Search Space
+---------------
+
+Since attention and RNN have been proven effective in Reading Comprehension, we conclude the search space as follow:
+
+
+#. IDENTITY (Effectively means keep training).
+#. INSERT-RNN-LAYER (Inserts a LSTM. Comparing the performance of GRU and LSTM in our experiment, we decided to use LSTM here.)
+#. REMOVE-RNN-LAYER
+#. INSERT-ATTENTION-LAYER(Inserts an attention layer.)
+#. REMOVE-ATTENTION-LAYER
+#. ADD-SKIP (Identity between random layers).
+#. REMOVE-SKIP (Removes random skip).
+
+
+.. image:: ../../../examples/trials/ga_squad/ga_squad.png
+   :target: ../../../examples/trials/ga_squad/ga_squad.png
+   :alt: 
+
+
+New version
+^^^^^^^^^^^
+
+Also we have another version which time cost is less and performance is better. We will release soon.
+
+2. How to run this example in local?
+------------------------------------
+
+2.1 Use downloading script to download data
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Execute the following command to download needed files
+using the downloading script:
+
+.. code-block:: bash
+
+   chmod +x ./download.sh
+   ./download.sh
+
+Or Download manually
+
+
+#. download "dev-v1.1.json" and "train-v1.1.json" in https://rajpurkar.github.io/SQuAD-explorer/
+
+.. code-block:: bash
+
+   wget https://rajpurkar.github.io/SQuAD-explorer/dataset/train-v1.1.json
+   wget https://rajpurkar.github.io/SQuAD-explorer/dataset/dev-v1.1.json
+
+
+#. download "glove.840B.300d.txt" in https://nlp.stanford.edu/projects/glove/
+
+.. code-block:: bash
+
+   wget http://nlp.stanford.edu/data/glove.840B.300d.zip
+   unzip glove.840B.300d.zip
+
+2.2 Update configuration
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+Modify ``nni/examples/trials/ga_squad/config.yml``\ , here is the default configuration:
+
+.. code-block:: yaml
+
+   authorName: default
+   experimentName: example_ga_squad
+   trialConcurrency: 1
+   maxExecDuration: 1h
+   maxTrialNum: 1
+   #choice: local, remote
+   trainingServicePlatform: local
+   #choice: true, false
+   useAnnotation: false
+   tuner:
+     codeDir: ~/nni/examples/tuners/ga_customer_tuner
+     classFileName: customer_tuner.py
+     className: CustomerTuner
+     classArgs:
+       optimize_mode: maximize
+   trial:
+     command: python3 trial.py
+     codeDir: ~/nni/examples/trials/ga_squad
+     gpuNum: 0
+
+In the "trial" part, if you want to use GPU to perform the architecture search, change ``gpuNum`` from ``0`` to ``1``. You need to increase the ``maxTrialNum`` and ``maxExecDuration``\ , according to how long you want to wait for the search result.
+
+2.3 submit this job
+^^^^^^^^^^^^^^^^^^^
+
+.. code-block:: bash
+
+   nnictl create --config ~/nni/examples/trials/ga_squad/config.yml
+
+3 Run this example on OpenPAI
+-----------------------------
+
+Due to the memory limitation of upload, we only upload the source code and complete the data download and training on OpenPAI. This experiment requires sufficient memory that ``memoryMB >= 32G``\ , and the training may last for several hours.
+
+3.1 Update configuration
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+Modify ``nni/examples/trials/ga_squad/config_pai.yml``\ , here is the default configuration:
+
+.. code-block:: yaml
+
+   authorName: default
+   experimentName: example_ga_squad
+   trialConcurrency: 1
+   maxExecDuration: 1h
+   maxTrialNum: 10
+   #choice: local, remote, pai
+   trainingServicePlatform: pai
+   #choice: true, false
+   useAnnotation: false
+   #Your nni_manager ip
+   nniManagerIp: 10.10.10.10
+   tuner:
+     codeDir: https://github.com/Microsoft/nni/tree/v1.9/examples/tuners/ga_customer_tuner
+     classFileName: customer_tuner.py
+     className: CustomerTuner
+     classArgs:
+       optimize_mode: maximize
+   trial:
+     command: chmod +x ./download.sh && ./download.sh && python3 trial.py
+     codeDir: .
+     gpuNum: 0
+     cpuNum: 1
+     memoryMB: 32869
+     #The docker image to run nni job on OpenPAI
+     image: msranni/nni:latest
+   paiConfig:
+     #The username to login OpenPAI
+     userName: username
+     #The password to login OpenPAI
+     passWord: password
+     #The host of restful server of OpenPAI
+     host: 10.10.10.10
+
+Please change the default value to your personal account and machine information. Including ``nniManagerIp``\ , ``userName``\ , ``passWord`` and ``host``.
+
+In the "trial" part, if you want to use GPU to perform the architecture search, change ``gpuNum`` from ``0`` to ``1``. You need to increase the ``maxTrialNum`` and ``maxExecDuration``\ , according to how long you want to wait for the search result.
+
+``trialConcurrency`` is the number of trials running concurrently, which is the number of GPUs you want to use, if you are setting ``gpuNum`` to 1.
+
+3.2 submit this job
+^^^^^^^^^^^^^^^^^^^
+
+.. code-block:: bash
+
+   nnictl create --config ~/nni/examples/trials/ga_squad/config_pai.yml
+
+4. Technical details about the trial
+------------------------------------
+
+4.1 How does it works
+^^^^^^^^^^^^^^^^^^^^^
+
+The evolution-algorithm based architecture for question answering has two different parts just like any other examples: the trial and the tuner.
+
+4.2 The trial
+^^^^^^^^^^^^^
+
+The trial has a lot of different files, functions and classes. Here we will only give most of those files a brief introduction:
+
+
+* ``attention.py`` contains an implementation for attention mechanism in Tensorflow.
+* ``data.py`` contains functions for data preprocessing.
+* ``evaluate.py`` contains the evaluation script.
+* ``graph.py`` contains the definition of the computation graph.
+* ``rnn.py`` contains an implementation for GRU in Tensorflow.
+* ``train_model.py`` is a wrapper for the whole question answering model.
+
+Among those files, ``trial.py`` and ``graph_to_tf.py`` are special.
+
+``graph_to_tf.py`` has a function named as ``graph_to_network``\ , here is its skeleton code:
+
+.. code-block:: python
+
+   def graph_to_network(input1,
+                        input2,
+                        input1_lengths,
+                        input2_lengths,
+                        graph,
+                        dropout_rate,
+                        is_training,
+                        num_heads=1,
+                        rnn_units=256):
+       topology = graph.is_topology()
+       layers = dict()
+       layers_sequence_lengths = dict()
+       num_units = input1.get_shape().as_list()[-1]
+       layers[0] = input1*tf.sqrt(tf.cast(num_units, tf.float32)) + \
+           positional_encoding(input1, scale=False, zero_pad=False)
+       layers[1] = input2*tf.sqrt(tf.cast(num_units, tf.float32))
+       layers[0] = dropout(layers[0], dropout_rate, is_training)
+       layers[1] = dropout(layers[1], dropout_rate, is_training)
+       layers_sequence_lengths[0] = input1_lengths
+       layers_sequence_lengths[1] = input2_lengths
+       for _, topo_i in enumerate(topology):
+           if topo_i == '|':
+               continue
+           if graph.layers[topo_i].graph_type == LayerType.input.value:
+               # ......
+           elif graph.layers[topo_i].graph_type == LayerType.attention.value:
+               # ......
+           # More layers to handle
+
+As we can see, this function is actually a compiler, that converts the internal model DAG configuration (which will be introduced in the ``Model configuration format`` section) ``graph``\ , to a Tensorflow computation graph.
+
+.. code-block:: python
+
+   topology = graph.is_topology()
+
+performs topological sorting on the internal graph representation, and the code inside the loop:
+
+.. code-block:: python
+
+   for _, topo_i in enumerate(topology):
+
+performs actually conversion that maps each layer to a part in Tensorflow computation graph.
+
+4.3 The tuner
+^^^^^^^^^^^^^
+
+The tuner is much more simple than the trial. They actually share the same ``graph.py``. Besides, the tuner has a ``customer_tuner.py``\ , the most important class in which is ``CustomerTuner``\ :
+
+.. code-block:: python
+
+   class CustomerTuner(Tuner):
+       # ......
+
+       def generate_parameters(self, parameter_id):
+           """Returns a set of trial graph config, as a serializable object.
+           parameter_id : int
+           """
+           if len(self.population) <= 0:
+               logger.debug("the len of poplution lower than zero.")
+               raise Exception('The population is empty')
+           pos = -1
+           for i in range(len(self.population)):
+               if self.population[i].result == None:
+                   pos = i
+                   break
+           if pos != -1:
+               indiv = copy.deepcopy(self.population[pos])
+               self.population.pop(pos)
+               temp = json.loads(graph_dumps(indiv.config))
+           else:
+               random.shuffle(self.population)
+               if self.population[0].result > self.population[1].result:
+                   self.population[0] = self.population[1]
+               indiv = copy.deepcopy(self.population[0])
+               self.population.pop(1)
+               indiv.mutation()
+               graph = indiv.config
+               temp =  json.loads(graph_dumps(graph))
+
+       # ......
+
+As we can see, the overloaded method ``generate_parameters`` implements a pretty naive mutation algorithm. The code lines:
+
+.. code-block:: python
+
+               if self.population[0].result > self.population[1].result:
+                   self.population[0] = self.population[1]
+               indiv = copy.deepcopy(self.population[0])
+
+controls the mutation process. It will always take two random individuals in the population, only keeping and mutating the one with better result.
+
+4.4 Model configuration format
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Here is an example of the model configuration, which is passed from the tuner to the trial in the architecture search procedure.
+
+.. code-block:: json
+
+   {
+       "max_layer_num": 50,
+       "layers": [
+           {
+               "input_size": 0,
+               "type": 3,
+               "output_size": 1,
+               "input": [],
+               "size": "x",
+               "output": [4, 5],
+               "is_delete": false
+           },
+           {
+               "input_size": 0,
+               "type": 3,
+               "output_size": 1,
+               "input": [],
+               "size": "y",
+               "output": [4, 5],
+               "is_delete": false
+           },
+           {
+               "input_size": 1,
+               "type": 4,
+               "output_size": 0,
+               "input": [6],
+               "size": "x",
+               "output": [],
+               "is_delete": false
+           },
+           {
+               "input_size": 1,
+               "type": 4,
+               "output_size": 0,
+               "input": [5],
+               "size": "y",
+               "output": [],
+               "is_delete": false
+           },
+           {"Comment": "More layers will be here for actual graphs."}
+       ]
+   }
+
+Every model configuration will have a "layers" section, which is a JSON list of layer definitions. The definition of each layer is also a JSON object, where:
+
+
+* ``type`` is the type of the layer. 0, 1, 2, 3, 4 corresponds to attention, self-attention, RNN, input and output layer respectively.
+* ``size`` is the length of the output. "x", "y" correspond to document length / question length, respectively.
+* ``input_size`` is the number of inputs the layer has.
+* ``input`` is the indices of layers taken as input of this layer.
+* ``output`` is the indices of layers use this layer's output as their input.
+* ``is_delete`` means whether the layer is still available.

docs/en_US/TrialExample/Trials.rst

+.. role:: raw-html(raw)
+   :format: html
+
+
+Write a Trial Run on NNI
+========================
+
+A **Trial** in NNI is an individual attempt at applying a configuration (e.g., a set of hyper-parameters) to a model.
+
+To define an NNI trial, you need to first define the set of parameters (i.e., search space) and then update the model. NNI provides two approaches for you to define a trial: `NNI API <#nni-api>`__ and `NNI Python annotation <#nni-annotation>`__. You could also refer to `here <#more-examples>`__ for more trial examples.
+
+:raw-html:`<a name="nni-api"></a>`
+
+NNI API
+-------
+
+Step 1 - Prepare a SearchSpace parameters file.
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+An example is shown below:
+
+.. code-block:: json
+
+   {
+       "dropout_rate":{"_type":"uniform","_value":[0.1,0.5]},
+       "conv_size":{"_type":"choice","_value":[2,3,5,7]},
+       "hidden_size":{"_type":"choice","_value":[124, 512, 1024]},
+       "learning_rate":{"_type":"uniform","_value":[0.0001, 0.1]}
+   }
+
+Refer to `SearchSpaceSpec.md <../Tutorial/SearchSpaceSpec.rst>`__ to learn more about search spaces. Tuner will generate configurations from this search space, that is, choosing a value for each hyperparameter from the range.
+
+Step 2 - Update model code
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+
+* 
+  Import NNI
+
+    Include ``import nni`` in your trial code to use NNI APIs.
+
+* 
+  Get configuration from Tuner
+
+.. code-block:: python
+
+   RECEIVED_PARAMS = nni.get_next_parameter()
+
+``RECEIVED_PARAMS`` is an object, for example:
+
+``{"conv_size": 2, "hidden_size": 124, "learning_rate": 0.0307, "dropout_rate": 0.2029}``.
+
+
+* Report metric data periodically (optional)
+
+.. code-block:: python
+
+   nni.report_intermediate_result(metrics)
+
+``metrics`` can be any python object. If users use the NNI built-in tuner/assessor, ``metrics`` can only have two formats: 1) a number e.g., float, int, or 2) a dict object that has a key named ``default`` whose value is a number. These ``metrics`` are reported to `assessor <../Assessor/BuiltinAssessor.rst>`__. Often, ``metrics`` includes the periodically evaluated loss or accuracy.
+
+
+* Report performance of the configuration
+
+.. code-block:: python
+
+   nni.report_final_result(metrics)
+
+``metrics`` can also be any python object. If users use the NNI built-in tuner/assessor, ``metrics`` follows the same format rule as that in ``report_intermediate_result``\ , the number indicates the model's performance, for example, the model's accuracy, loss etc. These ``metrics`` are reported to `tuner <../Tuner/BuiltinTuner.rst>`__.
+
+Step 3 - Enable NNI API
+^^^^^^^^^^^^^^^^^^^^^^^
+
+To enable NNI API mode, you need to set useAnnotation to *false* and provide the path of the SearchSpace file was defined in step 1:
+
+.. code-block:: yaml
+
+   useAnnotation: false
+   searchSpacePath: /path/to/your/search_space.json
+
+You can refer to `here <../Tutorial/ExperimentConfig.rst>`__ for more information about how to set up experiment configurations.
+
+Please refer to `here </sdk_reference.html>`__ for more APIs (e.g., ``nni.get_sequence_id()``\ ) provided by NNI.
+
+:raw-html:`<a name="nni-annotation"></a>`
+
+NNI Python Annotation
+---------------------
+
+An alternative to writing a trial is to use NNI's syntax for python. NNI annotations are simple, similar to comments. You don't have to make structural changes to your existing code. With a few lines of NNI annotation, you will be able to:
+
+
+* annotate the variables you want to tune
+* specify the range  in which you want to tune the variables
+* annotate which variable you want to report as an intermediate result to ``assessor``
+* annotate which variable you want to report as the final result (e.g. model accuracy) to ``tuner``.
+
+Again, take MNIST as an example, it only requires 2 steps to write a trial with NNI Annotation.
+
+Step 1 - Update codes with annotations
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The following is a TensorFlow code snippet for NNI Annotation where the highlighted four lines are annotations that:
+
+
+#. tune batch_size and dropout_rate
+#. report test_acc every 100 steps
+#. lastly report test_acc as the final result.
+
+It's worth noting that, as these newly added codes are merely annotations, you can still run your code as usual in environments without NNI installed.
+
+.. code-block:: diff
+
+   with tf.Session() as sess:
+       sess.run(tf.global_variables_initializer())
+   +   """@nni.variable(nni.choice(50, 250, 500), name=batch_size)"""
+       batch_size = 128
+       for i in range(10000):
+           batch = mnist.train.next_batch(batch_size)
+   +       """@nni.variable(nni.choice(0.1, 0.5), name=dropout_rate)"""
+           dropout_rate = 0.5
+           mnist_network.train_step.run(feed_dict={mnist_network.images: batch[0],
+                                                   mnist_network.labels: batch[1],
+                                                   mnist_network.keep_prob: dropout_rate})
+           if i % 100 == 0:
+               test_acc = mnist_network.accuracy.eval(
+                   feed_dict={mnist_network.images: mnist.test.images,
+                               mnist_network.labels: mnist.test.labels,
+                               mnist_network.keep_prob: 1.0})
+   +           """@nni.report_intermediate_result(test_acc)"""
+
+       test_acc = mnist_network.accuracy.eval(
+           feed_dict={mnist_network.images: mnist.test.images,
+                       mnist_network.labels: mnist.test.labels,
+                       mnist_network.keep_prob: 1.0})
+   +   """@nni.report_final_result(test_acc)"""
+
+**NOTE**\ :
+
+
+* ``@nni.variable`` will affect its following line which should be an assignment statement whose left-hand side must be the same as the keyword ``name`` in the ``@nni.variable`` statement.
+* ``@nni.report_intermediate_result``\ /\ ``@nni.report_final_result`` will send the data to assessor/tuner at that line.
+
+For more information about annotation syntax and its usage, please refer to `Annotation <../Tutorial/AnnotationSpec.rst>`__.
+
+Step 2 - Enable NNI Annotation
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+In the YAML configure file, you need to set *useAnnotation* to true to enable NNI annotation:
+
+.. code-block:: bash
+
+   useAnnotation: true
+
+Standalone mode for debugging
+-----------------------------
+
+NNI supports a standalone mode for trial code to run without starting an NNI experiment. This is for finding out bugs in trial code more conveniently. NNI annotation natively supports standalone mode, as the added NNI related lines are comments. For NNI trial APIs, the APIs have changed behaviors in standalone mode, some APIs return dummy values, and some APIs do not really report values. Please refer to the following table for the full list of these APIs.
+
+.. code-block:: python
+
+   # NOTE: please assign default values to the hyperparameters in your trial code
+   nni.get_next_parameter # return {}
+   nni.report_final_result # have log printed on stdout, but does not report
+   nni.report_intermediate_result # have log printed on stdout, but does not report
+   nni.get_experiment_id # return "STANDALONE"
+   nni.get_trial_id # return "STANDALONE"
+   nni.get_sequence_id # return 0
+
+You can try standalone mode with the :githublink:`mnist example <examples/trials/mnist-tfv1>`. Simply run ``python3 mnist.py`` under the code directory. The trial code should successfully run with the default hyperparameter values.
+
+For more information on debugging, please refer to `How to Debug <../Tutorial/HowToDebug.rst>`__
+
+Where are my trials?
+--------------------
+
+Local Mode
+^^^^^^^^^^
+
+In NNI, every trial has a dedicated directory for them to output their own data. In each trial, an environment variable called ``NNI_OUTPUT_DIR`` is exported. Under this directory, you can find each trial's code, data, and other logs. In addition, each trial's log (including stdout) will be re-directed to a file named ``trial.log`` under that directory.
+
+If NNI Annotation is used, the trial's converted code is in another temporary directory. You can check that in a file named ``run.sh`` under the directory indicated by ``NNI_OUTPUT_DIR``. The second line (i.e., the ``cd`` command) of this file will change directory to the actual directory where code is located. Below is an example of ``run.sh``\ :
+
+.. code-block:: bash
+
+   #!/bin/bash
+   cd /tmp/user_name/nni/annotation/tmpzj0h72x6 #This is the actual directory
+   export NNI_PLATFORM=local
+   export NNI_SYS_DIR=/home/user_name/nni-experiments/$experiment_id$/trials/$trial_id$
+   export NNI_TRIAL_JOB_ID=nrbb2
+   export NNI_OUTPUT_DIR=/home/user_name/nni-experiments/$eperiment_id$/trials/$trial_id$
+   export NNI_TRIAL_SEQ_ID=1
+   export MULTI_PHASE=false
+   export CUDA_VISIBLE_DEVICES=
+   eval python3 mnist.py 2>/home/user_name/nni-experiments/$experiment_id$/trials/$trial_id$/stderr
+   echo $? `date +%s%3N` >/home/user_name/nni-experiments/$experiment_id$/trials/$trial_id$/.nni/state
+
+Other Modes
+^^^^^^^^^^^
+
+When running trials on other platforms like remote machine or PAI, the environment variable ``NNI_OUTPUT_DIR`` only refers to the output directory of the trial, while the trial code and ``run.sh`` might not be there. However, the ``trial.log`` will be transmitted back to the local machine in the trial's directory, which defaults to ``~/nni-experiments/$experiment_id$/trials/$trial_id$/``
+
+For more information, please refer to `HowToDebug <../Tutorial/HowToDebug.rst>`__.
+
+:raw-html:`<a name="more-examples"></a>`
+
+More Trial Examples
+-------------------
+
+
+* `MNIST examples <MnistExamples.rst>`__
+* `Finding out best optimizer for Cifar10 classification <Cifar10Examples.rst>`__
+* `How to tune Scikit-learn on NNI <SklearnExamples.rst>`__
+* `Automatic Model Architecture Search for Reading Comprehension. <SquadEvolutionExamples.rst>`__
+* `Tuning GBDT on NNI <GbdtExample.rst>`__
+* `Tuning RocksDB on NNI <RocksdbExamples.rst>`__

docs/en_US/Tuner/BatchTuner.rst

+Batch Tuner on NNI
+==================
+
+Batch Tuner
+-----------
+
+Batch tuner allows users to simply provide several configurations (i.e., choices of hyper-parameters) for their trial code. After finishing all the configurations, the experiment is done. Batch tuner only supports the type ``choice`` in the `search space spec <../Tutorial/SearchSpaceSpec.rst>`__.
+
+Suggested scenario: If the configurations you want to try have been decided, you can list them in the SearchSpace file (using ``choice``\ ) and run them using the batch tuner.

docs/en_US/Tuner/BohbAdvisor.rst

+BOHB Advisor on NNI
+===================
+
+1. Introduction
+---------------
+
+BOHB is a robust and efficient hyperparameter tuning algorithm mentioned in `this reference paper <https://arxiv.org/abs/1807.01774>`__. BO is an abbreviation for "Bayesian Optimization" and HB is an abbreviation for "Hyperband".
+
+BOHB relies on HB (Hyperband) to determine how many configurations to evaluate with which budget, but it **replaces the random selection of configurations at the beginning of each HB iteration by a model-based search (Bayesian Optimization)**. Once the desired number of configurations for the iteration is reached, the standard successive halving procedure is carried out using these configurations. We keep track of the performance of all function evaluations g(x, b) of configurations x on all budgets b to use as a basis for our models in later iterations.
+
+Below we divide the introduction of the BOHB process into two parts:
+
+HB (Hyperband)
+^^^^^^^^^^^^^^
+
+We follow Hyperband’s way of choosing the budgets and continue to use SuccessiveHalving. For more details, you can refer to the `Hyperband in NNI <HyperbandAdvisor.rst>`__ and the `reference paper for Hyperband <https://arxiv.org/abs/1603.06560>`__. This procedure is summarized by the pseudocode below.
+
+
+.. image:: ../../img/bohb_1.png
+   :target: ../../img/bohb_1.png
+   :alt: 
+
+
+BO (Bayesian Optimization)
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The BO part of BOHB closely resembles TPE with one major difference: we opted for a single multidimensional KDE compared to the hierarchy of one-dimensional KDEs used in TPE in order to better handle interaction effects in the input space.
+
+Tree Parzen Estimator(TPE): uses a KDE (kernel density estimator) to model the densities.
+
+
+.. image:: ../../img/bohb_2.png
+   :target: ../../img/bohb_2.png
+   :alt: 
+
+
+To fit useful KDEs, we require a minimum number of data points Nmin; this is set to d + 1 for our experiments, where d is the number of hyperparameters. To build a model as early as possible, we do not wait until Nb = \|Db\|, where the number of observations for budget b is large enough to satisfy q · Nb ≥ Nmin. Instead, after initializing with Nmin + 2 random configurations, we choose the
+
+
+.. image:: ../../img/bohb_3.png
+   :target: ../../img/bohb_3.png
+   :alt: 
+
+
+best and worst configurations, respectively, to model the two densities.
+
+Note that we also sample a constant fraction named **random fraction** of the configurations uniformly at random.
+
+2. Workflow
+-----------
+
+
+.. image:: ../../img/bohb_6.jpg
+   :target: ../../img/bohb_6.jpg
+   :alt: 
+
+
+This image shows the workflow of BOHB. Here we set max_budget = 9, min_budget = 1, eta = 3, others as default. In this case, s_max = 2, so we will continuously run the {s=2, s=1, s=0, s=2, s=1, s=0, ...} cycle. In each stage of SuccessiveHalving (the orange box), we will pick the top 1/eta configurations and run them again with more budget, repeating the SuccessiveHalving stage until the end of this iteration. At the same time, we collect the configurations, budgets and final metrics of each trial and use these to build a multidimensional KDEmodel with the key "budget".
+ Multidimensional KDE is used to guide the selection of configurations for the next iteration.
+
+The sampling procedure (using Multidimensional KDE to guide selection) is summarized by the pseudocode below.
+
+
+.. image:: ../../img/bohb_4.png
+   :target: ../../img/bohb_4.png
+   :alt: 
+
+
+3. Usage
+--------
+
+BOHB advisor requires the `ConfigSpace <https://github.com/automl/ConfigSpace>`__ package. ConfigSpace can be installed using the following command.
+
+.. code-block:: bash
+
+   nnictl package install --name=BOHB
+
+To use BOHB, you should add the following spec in your experiment's YAML config file:
+
+.. code-block:: yaml
+
+   advisor:
+     builtinAdvisorName: BOHB
+     classArgs:
+       optimize_mode: maximize
+       min_budget: 1
+       max_budget: 27
+       eta: 3
+       min_points_in_model: 7
+       top_n_percent: 15
+       num_samples: 64
+       random_fraction: 0.33
+       bandwidth_factor: 3.0
+       min_bandwidth: 0.001
+
+**classArgs Requirements:**
+
+
+* **optimize_mode** (*maximize or minimize, optional, default = maximize*\ ) - If 'maximize', tuners will try to maximize metrics. If 'minimize', tuner will try to minimize metrics.
+* **min_budget** (*int, optional, default = 1*\ ) - The smallest budget to assign to a trial job, (budget can be the number of mini-batches or epochs). Needs to be positive.
+* **max_budget** (*int, optional, default = 3*\ ) - The largest budget to assign to a trial job, (budget can be the number of mini-batches or epochs). Needs to be larger than min_budget.
+* **eta** (*int, optional, default = 3*\ ) - In each iteration, a complete run of sequential halving is executed. In it, after evaluating each configuration on the same subset size, only a fraction of 1/eta of them 'advances' to the next round. Must be greater or equal to 2.
+* **min_points_in_model**\ (*int, optional, default = None*\ ): number of observations to start building a KDE. Default 'None' means dim+1; when the number of completed trials in this budget is equal to or larger than ``max{dim+1, min_points_in_model}``\ , BOHB will start to build a KDE model of this budget then use said KDE model to guide configuration selection. Needs to be positive. (dim means the number of hyperparameters in search space)
+* **top_n_percent**\ (*int, optional, default = 15*\ ): percentage (between 1 and 99) of the observations which are considered good. Good points and bad points are used for building KDE models. For example, if you have 100 observed trials and top_n_percent is 15, then the top 15% of points will be used for building the good points models "l(x)". The remaining 85% of points will be used for building the bad point models "g(x)".
+* **num_samples**\ (*int, optional, default = 64*\ ): number of samples to optimize EI (default 64). In this case, we will sample "num_samples" points and compare the result of l(x)/g(x). Then we will return the one with the maximum l(x)/g(x) value as the next configuration if the optimize_mode is ``maximize``. Otherwise, we return the smallest one.
+* **random_fraction**\ (*float, optional, default = 0.33*\ ): fraction of purely random configurations that are sampled from the prior without the model.
+* **bandwidth_factor**\ (*float, optional, default = 3.0*\ ): to encourage diversity, the points proposed to optimize EI are sampled from a 'widened' KDE where the bandwidth is multiplied by this factor. We suggest using the default value if you are not familiar with KDE.
+* **min_bandwidth**\ (*float, optional, default = 0.001*\ ): to keep diversity, even when all (good) samples have the same value for one of the parameters, a minimum bandwidth (default: 1e-3) is used instead of zero. We suggest using the default value if you are not familiar with KDE.
+
+*Please note that the float type currently only supports decimal representations. You have to use 0.333 instead of 1/3 and 0.001 instead of 1e-3.*
+
+4. File Structure
+-----------------
+
+The advisor has a lot of different files, functions, and classes. Here, we will only give most of those files a brief introduction:
+
+
+* ``bohb_advisor.py`` Definition of BOHB, handles interaction with the dispatcher, including generating new trials and processing results. Also includes the implementation of the HB (Hyperband) part.
+* ``config_generator.py`` Includes the implementation of the BO (Bayesian Optimization) part. The function *get_config* can generate new configurations based on BO; the function *new_result* will update the model with the new result.
+
+5. Experiment
+-------------
+
+MNIST with BOHB
+^^^^^^^^^^^^^^^
+
+code implementation: :githublink:`examples/trials/mnist-advisor <examples/trials/>`
+
+We chose BOHB to build a CNN on the MNIST dataset. The following is our experimental final results:
+
+
+.. image:: ../../img/bohb_5.png
+   :target: ../../img/bohb_5.png
+   :alt: 
+
+
+More experimental results can be found in the `reference paper <https://arxiv.org/abs/1807.01774>`__. We can see that BOHB makes good use of previous results and has a balanced trade-off in exploration and exploitation.

docs/en_US/Tuner/BuiltinTuner.rst

+.. role:: raw-html(raw)
+   :format: html
+
+
+HyperParameter Tuning with NNI Built-in Tuners
+==============================================
+
+To fit a machine/deep learning model into different tasks/problems, hyperparameters always need to be tuned. Automating the process of hyperparaeter tuning always requires a good tuning algorithm. NNI has provided state-of-the-art tuning algorithms as part of our built-in tuners and makes them easy to use. Below is the brief summary of NNI's current built-in tuners:
+
+Note: Click the **Tuner's name** to get the Tuner's installation requirements, suggested scenario, and an example configuration. A link for a detailed description of each algorithm is located at the end of the suggested scenario for each tuner. Here is an `article <../CommunitySharings/HpoComparison.rst>`__ comparing different Tuners on several problems.
+
+Currently, we support the following algorithms:
+
+.. list-table::
+   :header-rows: 1
+   :widths: auto
+
+   * - Tuner
+     - Brief Introduction of Algorithm
+   * - `TPE <#TPE>`__
+     - The Tree-structured Parzen Estimator (TPE) is a sequential model-based optimization (SMBO) approach. SMBO methods sequentially construct models to approximate the performance of hyperparameters based on historical measurements, and then subsequently choose new hyperparameters to test based on this model. `Reference Paper <https://papers.nips.cc/paper/4443-algorithms-for-hyper-parameter-optimization.pdf>`__
+   * - `Random Search <#Random>`__
+     - In Random Search for Hyper-Parameter Optimization show that Random Search might be surprisingly simple and effective. We suggest that we could use Random Search as the baseline when we have no knowledge about the prior distribution of hyper-parameters. `Reference Paper <http://www.jmlr.org/papers/volume13/bergstra12a/bergstra12a.pdf>`__
+   * - `Anneal <#Anneal>`__
+     - This simple annealing algorithm begins by sampling from the prior, but tends over time to sample from points closer and closer to the best ones observed. This algorithm is a simple variation on the random search that leverages smoothness in the response surface. The annealing rate is not adaptive.
+   * - `Naïve Evolution <#Evolution>`__
+     - Naïve Evolution comes from Large-Scale Evolution of Image Classifiers. It randomly initializes a population-based on search space. For each generation, it chooses better ones and does some mutation (e.g., change a hyperparameter, add/remove one layer) on them to get the next generation. Naïve Evolution requires many trials to work, but it's very simple and easy to expand new features. `Reference paper <https://arxiv.org/pdf/1703.01041.pdf>`__
+   * - `SMAC <#SMAC>`__
+     - SMAC is based on Sequential Model-Based Optimization (SMBO). It adapts the most prominent previously used model class (Gaussian stochastic process models) and introduces the model class of random forests to SMBO, in order to handle categorical parameters. The SMAC supported by NNI is a wrapper on the SMAC3 GitHub repo. Notice, SMAC needs to be installed by ``nnictl package`` command. `Reference Paper, <https://www.cs.ubc.ca/~hutter/papers/10-TR-SMAC.pdf>`__ `GitHub Repo <https://github.com/automl/SMAC3>`__
+   * - `Batch tuner <#Batch>`__
+     - Batch tuner allows users to simply provide several configurations (i.e., choices of hyper-parameters) for their trial code. After finishing all the configurations, the experiment is done. Batch tuner only supports the type choice in search space spec.
+   * - `Grid Search <#GridSearch>`__
+     - Grid Search performs an exhaustive searching through a manually specified subset of the hyperparameter space defined in the searchspace file. Note that the only acceptable types of search space are choice, quniform, randint.
+   * - `Hyperband <#Hyperband>`__
+     - Hyperband tries to use limited resources to explore as many configurations as possible and returns the most promising ones as a final result. The basic idea is to generate many configurations and run them for a small number of trials. The half least-promising configurations are thrown out, the remaining are further trained along with a selection of new configurations. The size of these populations is sensitive to resource constraints (e.g. allotted search time). `Reference Paper <https://arxiv.org/pdf/1603.06560.pdf>`__
+   * - `Network Morphism <#NetworkMorphism>`__
+     - Network Morphism provides functions to automatically search for deep learning architectures. It generates child networks that inherit the knowledge from their parent network which it is a morph from. This includes changes in depth, width, and skip-connections. Next, it estimates the value of a child network using historic architecture and metric pairs. Then it selects the most promising one to train. `Reference Paper <https://arxiv.org/abs/1806.10282>`__
+   * - `Metis Tuner <#MetisTuner>`__
+     - Metis offers the following benefits when it comes to tuning parameters: While most tools only predict the optimal configuration, Metis gives you two outputs: (a) current prediction of optimal configuration, and (b) suggestion for the next trial. No more guesswork. While most tools assume training datasets do not have noisy data, Metis actually tells you if you need to re-sample a particular hyper-parameter. `Reference Paper <https://www.microsoft.com/en-us/research/publication/metis-robustly-tuning-tail-latencies-cloud-systems/>`__
+   * - `BOHB <#BOHB>`__
+     - BOHB is a follow-up work to Hyperband. It targets the weakness of Hyperband that new configurations are generated randomly without leveraging finished trials. For the name BOHB, HB means Hyperband, BO means Bayesian Optimization. BOHB leverages finished trials by building multiple TPE models, a proportion of new configurations are generated through these models. `Reference Paper <https://arxiv.org/abs/1807.01774>`__
+   * - `GP Tuner <#GPTuner>`__
+     - Gaussian Process Tuner is a sequential model-based optimization (SMBO) approach with Gaussian Process as the surrogate. `Reference Paper <https://papers.nips.cc/paper/4443-algorithms-for-hyper-parameter-optimization.pdf>`__\ , `Github Repo <https://github.com/fmfn/BayesianOptimization>`__
+   * - `PPO Tuner <#PPOTuner>`__
+     - PPO Tuner is a Reinforcement Learning tuner based on PPO algorithm. `Reference Paper <https://arxiv.org/abs/1707.06347>`__
+   * - `PBT Tuner <#PBTTuner>`__
+     - PBT Tuner is a simple asynchronous optimization algorithm which effectively utilizes a fixed computational budget to jointly optimize a population of models and their hyperparameters to maximize performance. `Reference Paper <https://arxiv.org/abs/1711.09846v1>`__
+
+
+Usage of Built-in Tuners
+------------------------
+
+Using a built-in tuner provided by the NNI SDK requires one to declare the  **builtinTunerName** and **classArgs** in the ``config.yml`` file. In this part, we will introduce each tuner along with information about usage and suggested scenarios, classArg requirements, and an example configuration.
+
+Note: Please follow the format when you write your ``config.yml`` file. Some built-in tuners need to be installed using ``nnictl package``\ , like SMAC.
+
+:raw-html:`<a name="TPE"></a>`
+
+TPE
+^^^
+
+..
+
+   Built-in Tuner Name: **TPE**
+
+
+**Suggested scenario**
+
+TPE, as a black-box optimization, can be used in various scenarios and shows good performance in general. Especially when you have limited computation resources and can only try a small number of trials. From a large amount of experiments, we found that TPE is far better than Random Search. `Detailed Description <./HyperoptTuner.rst>`__
+
+**classArgs Requirements:**
+
+
+* **optimize_mode** (*maximize or minimize, optional, default = maximize*\ ) - If 'maximize', the tuner will try to maximize metrics. If 'minimize', the tuner will try to minimize metrics.
+
+Note: We have optimized the parallelism of TPE for large-scale trial concurrency. For the principle of optimization or turn-on optimization, please refer to `TPE document <./HyperoptTuner.rst>`__.
+
+**Example Configuration:**
+
+.. code-block:: yaml
+
+   # config.yml
+   tuner:
+     builtinTunerName: TPE
+     classArgs:
+       optimize_mode: maximize
+
+:raw-html:`<br>`
+
+:raw-html:`<a name="Random"></a>`
+
+Random Search
+^^^^^^^^^^^^^
+
+..
+
+   Built-in Tuner Name: **Random**
+
+
+**Suggested scenario**
+
+Random search is suggested when each trial does not take very long (e.g., each trial can be completed very quickly, or early stopped by the assessor), and you have enough computational resources. It's also useful if you want to uniformly explore the search space. Random Search can be considered a baseline search algorithm. `Detailed Description <./HyperoptTuner.rst>`__
+
+**Example Configuration:**
+
+.. code-block:: yaml
+
+   # config.yml
+   tuner:
+     builtinTunerName: Random
+
+:raw-html:`<br>`
+
+:raw-html:`<a name="Anneal"></a>`
+
+Anneal
+^^^^^^
+
+..
+
+   Built-in Tuner Name: **Anneal**
+
+
+**Suggested scenario**
+
+Anneal is suggested when each trial does not take very long and you have enough computation resources (very similar to Random Search). It's also useful when the variables in the search space can be sample from some prior distribution. `Detailed Description <./HyperoptTuner.rst>`__
+
+**classArgs Requirements:**
+
+
+* **optimize_mode** (*maximize or minimize, optional, default = maximize*\ ) - If 'maximize', the tuner will try to maximize metrics. If 'minimize', the tuner will try to minimize metrics.
+
+**Example Configuration:**
+
+.. code-block:: yaml
+
+   # config.yml
+   tuner:
+     builtinTunerName: Anneal
+     classArgs:
+       optimize_mode: maximize
+
+:raw-html:`<br>`
+
+:raw-html:`<a name="Evolution"></a>`
+
+Naïve Evolution
+^^^^^^^^^^^^^^^
+
+..
+
+   Built-in Tuner Name: **Evolution**
+
+
+**Suggested scenario**
+
+Its computational resource requirements are relatively high. Specifically, it requires a large initial population to avoid falling into a local optimum. If your trial is short or leverages assessor, this tuner is a good choice. It is also suggested when your trial code supports weight transfer; that is, the trial could inherit the converged weights from its parent(s). This can greatly speed up the training process. `Detailed Description <./EvolutionTuner.rst>`__
+
+**classArgs Requirements:**
+
+
+* 
+  **optimize_mode** (*maximize or minimize, optional, default = maximize*\ ) - If 'maximize', the tuner will try to maximize metrics. If 'minimize', the tuner will try to minimize metrics.
+
+* 
+  **population_size** (*int value (should > 0), optional, default = 20*\ ) - the initial size of the population (trial num) in the evolution tuner. It's suggested that ``population_size`` be much larger than ``concurrency`` so users can get the most out of the algorithm (and at least ``concurrency``\ , or the tuner will fail on its first generation of parameters).
+
+**Example Configuration:**
+
+.. code-block:: yaml
+
+   # config.yml
+   tuner:
+     builtinTunerName: Evolution
+     classArgs:
+       optimize_mode: maximize
+       population_size: 100
+
+:raw-html:`<br>`
+
+:raw-html:`<a name="SMAC"></a>`
+
+SMAC
+^^^^
+
+..
+
+   Built-in Tuner Name: **SMAC**
+
+
+**Please note that SMAC doesn't support running on Windows currently. For the specific reason, please refer to this `GitHub issue <https://github.com/automl/SMAC3/issues/483>`__.**
+
+**Installation**
+
+SMAC needs to be installed by following command before the first usage. As a reminder, ``swig`` is required for SMAC: for Ubuntu ``swig`` can be installed with ``apt``.
+
+.. code-block:: bash
+
+   nnictl package install --name=SMAC
+
+**Suggested scenario**
+
+Similar to TPE, SMAC is also a black-box tuner that can be tried in various scenarios and is suggested when computational resources are limited. It is optimized for discrete hyperparameters, thus, it's suggested when most of your hyperparameters are discrete. `Detailed Description <./SmacTuner.rst>`__
+
+**classArgs Requirements:**
+
+
+* **optimize_mode** (*maximize or minimize, optional, default = maximize*\ ) - If 'maximize', the tuner will try to maximize metrics. If 'minimize', the tuner will try to minimize metrics.
+* **config_dedup** (*True or False, optional, default = False*\ ) - If True, the tuner will not generate a configuration that has been already generated. If False, a configuration may be generated twice, but it is rare for a relatively large search space.
+
+**Example Configuration:**
+
+.. code-block:: yaml
+
+   # config.yml
+   tuner:
+     builtinTunerName: SMAC
+     classArgs:
+       optimize_mode: maximize
+
+:raw-html:`<br>`
+
+:raw-html:`<a name="Batch"></a>`
+
+Batch Tuner
+^^^^^^^^^^^
+
+..
+
+   Built-in Tuner Name: BatchTuner
+
+
+**Suggested scenario**
+
+If the configurations you want to try have been decided beforehand, you can list them in search space file (using ``choice``\ ) and run them using batch tuner. `Detailed Description <./BatchTuner.rst>`__
+
+**Example Configuration:**
+
+.. code-block:: yaml
+
+   # config.yml
+   tuner:
+     builtinTunerName: BatchTuner
+
+:raw-html:`<br>`
+
+Note that the search space for BatchTuner should look like:
+
+.. code-block:: json
+
+   {
+       "combine_params":
+       {
+           "_type" : "choice",
+           "_value" : [{"optimizer": "Adam", "learning_rate": 0.00001},
+                       {"optimizer": "Adam", "learning_rate": 0.0001},
+                       {"optimizer": "Adam", "learning_rate": 0.001},
+                       {"optimizer": "SGD", "learning_rate": 0.01},
+                       {"optimizer": "SGD", "learning_rate": 0.005},
+                       {"optimizer": "SGD", "learning_rate": 0.0002}]
+       }
+   }
+
+The search space file should include the high-level key ``combine_params``. The type of params in the search space must be ``choice`` and the ``values`` must include all the combined params values.
+
+:raw-html:`<a name="GridSearch"></a>`
+
+Grid Search
+^^^^^^^^^^^
+
+..
+
+   Built-in Tuner Name: **Grid Search**
+
+
+**Suggested scenario**
+
+Note that the only acceptable types within the search space are ``choice``\ , ``quniform``\ , and ``randint``.
+
+This is suggested when the search space is small. It's suggested when it is feasible to exhaustively sweep the whole search space. `Detailed Description <./GridsearchTuner.rst>`__
+
+**Example Configuration:**
+
+.. code-block:: yaml
+
+   # config.yml
+   tuner:
+     builtinTunerName: GridSearch
+
+:raw-html:`<br>`
+
+:raw-html:`<a name="Hyperband"></a>`
+
+Hyperband
+^^^^^^^^^
+
+..
+
+   Built-in Advisor Name: **Hyperband**
+
+
+**Suggested scenario**
+
+This is suggested when you have limited computational resources but have a relatively large search space. It performs well in scenarios where intermediate results can indicate good or bad final results to some extent. For example, when models that are more accurate early on in training are also more accurate later on. `Detailed Description <./HyperbandAdvisor.rst>`__
+
+**classArgs Requirements:**
+
+
+* **optimize_mode** (*maximize or minimize, optional, default = maximize*\ ) - If 'maximize', the tuner will try to maximize metrics. If 'minimize', the tuner will try to minimize metrics.
+* **R** (*int, optional, default = 60*\ ) - the maximum budget given to a trial (could be the number of mini-batches or epochs). Each trial should use TRIAL_BUDGET to control how long they run.
+* **eta** (*int, optional, default = 3*\ ) - ``(eta-1)/eta`` is the proportion of discarded trials.
+* **exec_mode** (*serial or parallelism, optional, default = parallelism*\ ) - If 'parallelism', the tuner will try to use available resources to start new bucket immediately. If 'serial', the tuner will only start new bucket after the current bucket is done.
+
+**Example Configuration:**
+
+.. code-block:: yaml
+
+   # config.yml
+   advisor:
+     builtinAdvisorName: Hyperband
+     classArgs:
+       optimize_mode: maximize
+       R: 60
+       eta: 3
+
+:raw-html:`<br>`
+
+:raw-html:`<a name="NetworkMorphism"></a>`
+
+Network Morphism
+^^^^^^^^^^^^^^^^
+
+..
+
+   Built-in Tuner Name: **NetworkMorphism**
+
+
+**Installation**
+
+NetworkMorphism requires :githublink:`PyTorch <examples/trials/network_morphism/requirements.txt>`.
+
+**Suggested scenario**
+
+This is suggested when you want to apply deep learning methods to your task but you have no idea how to choose or design a network. You may modify this :githublink:`example <examples/trials/network_morphism/cifar10/cifar10_keras.py>` to fit your own dataset and your own data augmentation method. Also you can change the batch size, learning rate, or optimizer. Currently, this tuner only supports the computer vision domain. `Detailed Description <./NetworkmorphismTuner.rst>`__
+
+**classArgs Requirements:**
+
+
+* **optimize_mode** (*maximize or minimize, optional, default = maximize*\ ) - If 'maximize', the tuner will try to maximize metrics. If 'minimize', the tuner will try to minimize metrics.
+* **task** (*('cv'), optional, default = 'cv'*\ ) - The domain of the experiment. For now, this tuner only supports the computer vision (CV) domain.
+* **input_width** (*int, optional, default = 32*\ ) - input image width
+* **input_channel** (*int, optional, default = 3*\ ) - input image channel
+* **n_output_node** (*int, optional, default = 10*\ ) - number of classes
+
+**Example Configuration:**
+
+.. code-block:: yaml
+
+   # config.yml
+   tuner:
+     builtinTunerName: NetworkMorphism
+       classArgs:
+         optimize_mode: maximize
+         task: cv
+         input_width: 32
+         input_channel: 3
+         n_output_node: 10
+
+:raw-html:`<br>`
+
+:raw-html:`<a name="MetisTuner"></a>`
+
+Metis Tuner
+^^^^^^^^^^^
+
+..
+
+   Built-in Tuner Name: **MetisTuner**
+
+
+Note that the only acceptable types of search space types are ``quniform``\ , ``uniform``\ , ``randint``\ , and numerical ``choice``. Only numerical values are supported since the values will be used to evaluate the 'distance' between different points.
+
+**Suggested scenario**
+
+Similar to TPE and SMAC, Metis is a black-box tuner. If your system takes a long time to finish each trial, Metis is more favorable than other approaches such as random search. Furthermore, Metis provides guidance on subsequent trials. Here is an :githublink:`example <examples/trials/auto-gbdt/search_space_metis.json>` on the use of Metis. Users only need to send the final result, such as ``accuracy``\ , to the tuner by calling the NNI SDK. `Detailed Description <./MetisTuner.rst>`__
+
+**classArgs Requirements:**
+
+
+* **optimize_mode** (*'maximize' or 'minimize', optional, default = 'maximize'*\ ) - If 'maximize', the tuner will try to maximize metrics. If 'minimize', the tuner will try to minimize metrics.
+
+**Example Configuration:**
+
+.. code-block:: yaml
+
+   # config.yml
+   tuner:
+     builtinTunerName: MetisTuner
+     classArgs:
+       optimize_mode: maximize
+
+:raw-html:`<br>`
+
+:raw-html:`<a name="BOHB"></a>`
+
+BOHB Advisor
+^^^^^^^^^^^^
+
+..
+
+   Built-in Tuner Name: **BOHB**
+
+
+**Installation**
+
+BOHB advisor requires `ConfigSpace <https://github.com/automl/ConfigSpace>`__ package. ConfigSpace can be installed using the following command.
+
+.. code-block:: bash
+
+   nnictl package install --name=BOHB
+
+**Suggested scenario**
+
+Similar to Hyperband, BOHB is suggested when you have limited computational resources but have a relatively large search space. It performs well in scenarios where intermediate results can indicate good or bad final results to some extent. In this case, it may converge to a better configuration than Hyperband due to its usage of Bayesian optimization. `Detailed Description <./BohbAdvisor.rst>`__
+
+**classArgs Requirements:**
+
+
+* **optimize_mode** (*maximize or minimize, optional, default = maximize*\ ) - If 'maximize', tuners will try to maximize metrics. If 'minimize', tuner will try to minimize metrics.
+* **min_budget** (*int, optional, default = 1*\ ) - The smallest budget to assign to a trial job, (budget can be the number of mini-batches or epochs). Needs to be positive.
+* **max_budget** (*int, optional, default = 3*\ ) - The largest budget to assign to a trial job, (budget can be the number of mini-batches or epochs). Needs to be larger than min_budget.
+* **eta** (*int, optional, default = 3*\ ) - In each iteration, a complete run of sequential halving is executed. In it, after evaluating each configuration on the same subset size, only a fraction of 1/eta of them 'advances' to the next round. Must be greater or equal to 2.
+* **min_points_in_model**\ (*int, optional, default = None*\ ): number of observations to start building a KDE. Default 'None' means dim+1; when the number of completed trials in this budget is equal to or larger than ``max{dim+1, min_points_in_model}``\ , BOHB will start to build a KDE model of this budget then use said KDE model to guide configuration selection. Needs to be positive. (dim means the number of hyperparameters in search space)
+* **top_n_percent**\ (*int, optional, default = 15*\ ): percentage (between 1 and 99) of the observations which are considered good. Good points and bad points are used for building KDE models. For example, if you have 100 observed trials and top_n_percent is 15, then the top 15% of points will be used for building the good points models "l(x)". The remaining 85% of points will be used for building the bad point models "g(x)".
+* **num_samples**\ (*int, optional, default = 64*\ ): number of samples to optimize EI (default 64). In this case, we will sample "num_samples" points and compare the result of l(x)/g(x). Then we will return the one with the maximum l(x)/g(x) value as the next configuration if the optimize_mode is ``maximize``. Otherwise, we return the smallest one.
+* **random_fraction**\ (*float, optional, default = 0.33*\ ): fraction of purely random configurations that are sampled from the prior without the model.
+* **bandwidth_factor**\ (*float, optional, default = 3.0*\ ): to encourage diversity, the points proposed to optimize EI are sampled from a 'widened' KDE where the bandwidth is multiplied by this factor. We suggest using the default value if you are not familiar with KDE.
+* **min_bandwidth**\ (*float, optional, default = 0.001*\ ): to keep diversity, even when all (good) samples have the same value for one of the parameters, a minimum bandwidth (default: 1e-3) is used instead of zero. We suggest using the default value if you are not familiar with KDE.
+
+*Please note that the float type currently only supports decimal representations. You have to use 0.333 instead of 1/3 and 0.001 instead of 1e-3.*
+
+**Example Configuration:**
+
+.. code-block:: yaml
+
+   advisor:
+     builtinAdvisorName: BOHB
+     classArgs:
+       optimize_mode: maximize
+       min_budget: 1
+       max_budget: 27
+       eta: 3
+
+:raw-html:`<a name="GPTuner"></a>`
+
+GP Tuner
+^^^^^^^^
+
+..
+
+   Built-in Tuner Name: **GPTuner**
+
+
+Note that the only acceptable types within the search space are ``randint``\ , ``uniform``\ , ``quniform``\ ,  ``loguniform``\ , ``qloguniform``\ , and numerical ``choice``. Only numerical values are supported since the values will be used to evaluate the 'distance' between different points.
+
+**Suggested scenario**
+
+As a strategy in a Sequential Model-based Global Optimization (SMBO) algorithm, GP Tuner uses a proxy optimization problem (finding the maximum of the acquisition function) that, albeit still a hard problem, is cheaper (in the computational sense) to solve and common tools can be employed to solve it. Therefore, GP Tuner is most adequate for situations where the function to be optimized is very expensive to evaluate. GP can be used when computational resources are limited. However, GP Tuner has a computational cost that grows at *O(N^3)* due to the requirement of inverting the Gram matrix, so it's not suitable when lots of trials are needed. `Detailed Description <./GPTuner.rst>`__
+
+**classArgs Requirements:**
+
+
+* **optimize_mode** (*'maximize' or 'minimize', optional, default = 'maximize'*\ ) - If 'maximize', the tuner will try to maximize metrics. If 'minimize', the tuner will try to minimize metrics.
+* **utility** (*'ei', 'ucb' or 'poi', optional, default = 'ei'*\ ) - The utility function (acquisition function). 'ei', 'ucb', and 'poi' correspond to 'Expected Improvement', 'Upper Confidence Bound', and 'Probability of Improvement', respectively.
+* **kappa** (*float, optional, default = 5*\ ) - Used by the 'ucb' utility function. The bigger ``kappa`` is, the more exploratory the tuner will be.
+* **xi** (*float, optional, default = 0*\ ) - Used by the 'ei' and 'poi' utility functions. The bigger ``xi`` is, the more exploratory the tuner will be.
+* **nu** (*float, optional, default = 2.5*\ ) - Used to specify the Matern kernel. The smaller nu, the less smooth the approximated function is.
+* **alpha** (*float, optional, default = 1e-6*\ ) - Used to specify the Gaussian Process Regressor. Larger values correspond to an increased noise level in the observations.
+* **cold_start_num** (*int, optional, default = 10*\ ) - Number of random explorations to perform before the Gaussian Process. Random exploration can help by diversifying the exploration space.
+* **selection_num_warm_up** (*int, optional, default = 1e5*\ ) - Number of random points to evaluate when getting the point which maximizes the acquisition function.
+* **selection_num_starting_points** (*int, optional, default = 250*\ ) - Number of times to run L-BFGS-B from a random starting point after the warmup.
+
+**Example Configuration:**
+
+.. code-block:: yaml
+
+   # config.yml
+   tuner:
+     builtinTunerName: GPTuner
+     classArgs:
+       optimize_mode: maximize
+       utility: 'ei'
+       kappa: 5.0
+       xi: 0.0
+       nu: 2.5
+       alpha: 1e-6
+       cold_start_num: 10
+       selection_num_warm_up: 100000
+       selection_num_starting_points: 250
+
+:raw-html:`<a name="PPOTuner"></a>`
+
+PPO Tuner
+^^^^^^^^^
+
+..
+
+   Built-in Tuner Name: **PPOTuner**
+
+
+Note that the only acceptable types within the search space are ``layer_choice`` and ``input_choice``. For ``input_choice``\ , ``n_chosen`` can only be 0, 1, or [0, 1]. Note, the search space file for NAS is usually automatically generated through the command `nnictl ss_gen <../Tutorial/Nnictl.rst>`__.
+
+**Suggested scenario**
+
+PPOTuner is a Reinforcement Learning tuner based on the PPO algorithm. PPOTuner can be used when using the NNI NAS interface to do neural architecture search. In general, the Reinforcement Learning algorithm needs more computing resources, though the PPO algorithm is relatively more efficient than others. It's recommended to use this tuner when you have a large amount of computional resources available. You could try it on a very simple task, such as the :githublink:`mnist-nas <examples/trials/mnist-nas>` example. `See details <./PPOTuner.rst>`__
+
+**classArgs Requirements:**
+
+
+* **optimize_mode** (*'maximize' or 'minimize'*\ ) - If 'maximize', the tuner will try to maximize metrics. If 'minimize', the tuner will try to minimize metrics.
+* **trials_per_update** (*int, optional, default = 20*\ ) - The number of trials to be used for one update. It must be divisible by minibatch_size. ``trials_per_update`` is recommended to be an exact multiple of ``trialConcurrency`` for better concurrency of trials.
+* **epochs_per_update** (*int, optional, default = 4*\ ) - The number of epochs for one update.
+* **minibatch_size** (*int, optional, default = 4*\ ) - Mini-batch size (i.e., number of trials for a mini-batch) for the update. Note that trials_per_update must be divisible by minibatch_size.
+* **ent_coef** (*float, optional, default = 0.0*\ ) - Policy entropy coefficient in the optimization objective.
+* **lr** (*float, optional, default = 3e-4*\ ) - Learning rate of the model (lstm network); constant.
+* **vf_coef** (*float, optional, default = 0.5*\ ) - Value function loss coefficient in the optimization objective.
+* **max_grad_norm** (*float, optional, default = 0.5*\ ) - Gradient norm clipping coefficient.
+* **gamma** (*float, optional, default = 0.99*\ ) - Discounting factor.
+* **lam** (*float, optional, default = 0.95*\ ) - Advantage estimation discounting factor (lambda in the paper).
+* **cliprange** (*float, optional, default = 0.2*\ ) - Cliprange in the PPO algorithm, constant.
+
+**Example Configuration:**
+
+.. code-block:: yaml
+
+   # config.yml
+   tuner:
+     builtinTunerName: PPOTuner
+     classArgs:
+       optimize_mode: maximize
+
+:raw-html:`<a name="PBTTuner"></a>`
+
+PBT Tuner
+^^^^^^^^^
+
+..
+
+   Built-in Tuner Name: **PBTTuner**
+
+
+**Suggested scenario**
+
+Population Based Training (PBT) bridges and extends parallel search methods and sequential optimization methods. It requires relatively small computation resource, by inheriting weights from currently good-performing ones to explore better ones periodically. With PBTTuner, users finally get a trained model, rather than a configuration that could reproduce the trained model by training the model from scratch. This is because model weights are inherited periodically through the whole search process. PBT can also be seen as a training approach. If you don't need to get a specific configuration, but just expect a good model, PBTTuner is a good choice. `See details <./PBTTuner.rst>`__
+
+**classArgs requirements:**
+
+
+* **optimize_mode** (*'maximize' or 'minimize'*\ ) - If 'maximize', the tuner will target to maximize metrics. If 'minimize', the tuner will target to minimize metrics.
+* **all_checkpoint_dir** (*str, optional, default = None*\ ) - Directory for trials to load and save checkpoint, if not specified, the directory would be "~/nni/checkpoint/\ :raw-html:`<exp-id>`\ ". Note that if the experiment is not local mode, users should provide a path in a shared storage which can be accessed by all the trials.
+* **population_size** (*int, optional, default = 10*\ ) - Number of trials in a population. Each step has this number of trials. In our implementation, one step is running each trial by specific training epochs set by users.
+* **factors** (*tuple, optional, default = (1.2, 0.8)*\ ) - Factors for perturbation of hyperparameters.
+* **fraction** (*float, optional, default = 0.2*\ ) - Fraction for selecting bottom and top trials.
+
+**Usage example**
+
+.. code-block:: yaml
+
+   # config.yml
+   tuner:
+     builtinTunerName: PBTTuner
+     classArgs:
+       optimize_mode: maximize
+
+Note that, to use this tuner, your trial code should be modified accordingly, please refer to `the document of PBTTuner <./PBTTuner.rst>`__ for details.
+
+**Reference and Feedback**
+------------------------------
+
+
+* To `report a bug <https://github.com/microsoft/nni/issues/new?template=bug-report.rst>`__ for this feature in GitHub;
+* To `file a feature or improvement request <https://github.com/microsoft/nni/issues/new?template=enhancement.rst>`__ for this feature in GitHub;
+* To know more about :githublink:`Feature Engineering with NNI <docs/en_US/FeatureEngineering/Overview.rst>`\ ;
+* To know more about :githublink:`NAS with NNI <docs/en_US/NAS/Overview.rst>`\ ;
+* To know more about :githublink:`Model Compression with NNI <docs/en_US/Compression/Overview.rst>`\ ;

docs/en_US/Tuner/CustomizeAdvisor.rst

+**How To** - Customize Your Own Advisor
+===========================================
+
+*Warning: API is subject to change in future releases.*
+
+Advisor targets the scenario that the automl algorithm wants the methods of both tuner and assessor. Advisor is similar to tuner on that it receives trial parameters request, final results, and generate trial parameters. Also, it is similar to assessor on that it receives intermediate results, trial's end state, and could send trial kill command. Note that, if you use Advisor, tuner and assessor are not allowed to be used at the same time.
+
+If a user want to implement a customized Advisor, she/he only needs to:
+
+**1. Define an Advisor inheriting from the MsgDispatcherBase class.** For example:
+
+.. code-block:: python
+
+   from nni.runtime.msg_dispatcher_base import MsgDispatcherBase
+
+   class CustomizedAdvisor(MsgDispatcherBase):
+       def __init__(self, ...):
+           ...
+
+**2. Implement the methods with prefix ``handle_`` except ``handle_request``**.. You might find `docs </sdk_reference.html#nni.runtime.msg_dispatcher_base.MsgDispatcherBase>`__ for ``MsgDispatcherBase`` helpful.
+
+**3. Configure your customized Advisor in experiment YAML config file.**
+
+Similar to tuner and assessor. NNI needs to locate your customized Advisor class and instantiate the class, so you need to specify the location of the customized Advisor class and pass literal values as parameters to the ``__init__`` constructor.
+
+.. code-block:: yaml
+
+   advisor:
+     codeDir: /home/abc/myadvisor
+     classFileName: my_customized_advisor.py
+     className: CustomizedAdvisor
+     # Any parameter need to pass to your advisor class __init__ constructor
+     # can be specified in this optional classArgs field, for example
+     classArgs:
+       arg1: value1
+
+**Note that** The working directory of your advisor is ``<home>/nni-experiments/<experiment_id>/log``\ , which can be retrieved with environment variable ``NNI_LOG_DIRECTORY``.
+
+Example
+-------
+
+Here we provide an :githublink:`example <examples/tuners/mnist_keras_customized_advisor>`.

docs/en_US/Tuner/CustomizeTuner.rst

+Customize-Tuner
+===============
+
+Customize Tuner
+---------------
+
+NNI provides state-of-the-art tuning algorithm in builtin-tuners. NNI supports to build a tuner by yourself for tuning demand.
+
+If you want to implement your own tuning algorithm, you can implement a customized Tuner, there are three things to do:
+
+
+#. Inherit the base Tuner class
+#. Implement receive_trial_result, generate_parameter and update_search_space function
+#. Configure your customized tuner in experiment YAML config file
+
+Here is an example:
+
+**1. Inherit the base Tuner class**
+
+.. code-block:: python
+
+   from nni.tuner import Tuner
+
+   class CustomizedTuner(Tuner):
+       def __init__(self, ...):
+           ...
+
+**2. Implement receive_trial_result, generate_parameter and update_search_space function**
+
+.. code-block:: python
+
+   from nni.tuner import Tuner
+
+   class CustomizedTuner(Tuner):
+       def __init__(self, ...):
+           ...
+
+       def receive_trial_result(self, parameter_id, parameters, value, **kwargs):
+           '''
+           Receive trial's final result.
+           parameter_id: int
+           parameters: object created by 'generate_parameters()'
+           value: final metrics of the trial, including default metric
+           '''
+           # your code implements here.
+       ...
+
+       def generate_parameters(self, parameter_id, **kwargs):
+           '''
+           Returns a set of trial (hyper-)parameters, as a serializable object
+           parameter_id: int
+           '''
+           # your code implements here.
+           return your_parameters
+       ...
+
+       def update_search_space(self, search_space):
+           '''
+           Tuners are advised to support updating search space at run-time.
+           If a tuner can only set search space once before generating first hyper-parameters,
+           it should explicitly document this behaviour.
+           search_space: JSON object created by experiment owner
+           '''
+           # your code implements here.
+       ...
+
+``receive_trial_result`` will receive the ``parameter_id, parameters, value`` as parameters input. Also, Tuner will receive the ``value`` object are exactly same value that Trial send.
+
+The ``your_parameters`` return from ``generate_parameters`` function, will be package as json object by NNI SDK. NNI SDK will unpack json object so the Trial will receive the exact same ``your_parameters`` from Tuner.
+
+For example:
+If the you implement the ``generate_parameters`` like this:
+
+.. code-block:: python
+
+   def generate_parameters(self, parameter_id, **kwargs):
+       '''
+       Returns a set of trial (hyper-)parameters, as a serializable object
+       parameter_id: int
+       '''
+       # your code implements here.
+       return {"dropout": 0.3, "learning_rate": 0.4}
+
+It means your Tuner will always generate parameters ``{"dropout": 0.3, "learning_rate": 0.4}``. Then Trial will receive ``{"dropout": 0.3, "learning_rate": 0.4}`` by calling API ``nni.get_next_parameter()``. Once the trial ends with a result (normally some kind of metrics), it can send the result to Tuner by calling API ``nni.report_final_result()``\ , for example ``nni.report_final_result(0.93)``. Then your Tuner's ``receive_trial_result`` function will receied the result like：
+
+.. code-block:: python
+
+   parameter_id = 82347
+   parameters = {"dropout": 0.3, "learning_rate": 0.4}
+   value = 0.93
+
+**Note that** The working directory of your tuner is ``<home>/nni-experiments/<experiment_id>/log``\ , which can be retrieved with environment variable ``NNI_LOG_DIRECTORY``\ , therefore, if you want to access a file (e.g., ``data.txt``\ ) in the directory of your own tuner, you cannot use ``open('data.txt', 'r')``. Instead, you should use the following:
+
+.. code-block:: python
+
+   _pwd = os.path.dirname(__file__)
+   _fd = open(os.path.join(_pwd, 'data.txt'), 'r')
+
+This is because your tuner is not executed in the directory of your tuner (i.e., ``pwd`` is not the directory of your own tuner).
+
+**3. Configure your customized tuner in experiment YAML config file**
+
+NNI needs to locate your customized tuner class and instantiate the class, so you need to specify the location of the customized tuner class and pass literal values as parameters to the __init__ constructor.
+
+.. code-block:: yaml
+
+   tuner:
+     codeDir: /home/abc/mytuner
+     classFileName: my_customized_tuner.py
+     className: CustomizedTuner
+     # Any parameter need to pass to your tuner class __init__ constructor
+     # can be specified in this optional classArgs field, for example
+     classArgs:
+       arg1: value1
+
+More detail example you could see:
+
+..
+
+   * :githublink:`evolution-tuner <src/sdk/pynni/nni/evolution_tuner>`
+   * :githublink:`hyperopt-tuner <src/sdk/pynni/nni/hyperopt_tuner>`
+   * :githublink:`evolution-based-customized-tuner <examples/tuners/ga_customer_tuner>`
+
+
+Write a more advanced automl algorithm
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The methods above are usually enough to write a general tuner. However, users may also want more methods, for example, intermediate results, trials' state (e.g., the methods in assessor), in order to have a more powerful automl algorithm. Therefore, we have another concept called ``advisor`` which directly inherits from ``MsgDispatcherBase`` in :githublink:`src/sdk/pynni/nni/msg_dispatcher_base.py <src/sdk/pynni/nni/msg_dispatcher_base.py>`. Please refer to `here <CustomizeAdvisor.rst>`__ for how to write a customized advisor.

docs/en_US/Tuner/EvolutionTuner.rst

+Naive Evolution Tuners on NNI
+=============================
+
+Naive Evolution
+---------------
+
+Naive Evolution comes from `Large-Scale Evolution of Image Classifiers <https://arxiv.org/pdf/1703.01041.pdf>`__. It randomly initializes a population based on the search space. For each generation, it chooses better ones and does some mutation (e.g., changes a hyperparameter, adds/removes one layer, etc.) on them to get the next generation. Naive Evolution requires many trials to works but it's very simple and it's easily expanded with new features.

docs/en_US/Tuner/GPTuner.rst

+GP Tuner on NNI
+===============
+
+GP Tuner
+--------
+
+Bayesian optimization works by constructing a posterior distribution of functions (a Gaussian Process) that best describes the function you want to optimize. As the number of observations grows, the posterior distribution improves, and the algorithm becomes more certain of which regions in parameter space are worth exploring and which are not.
+
+GP Tuner is designed to minimize/maximize the number of steps required to find a combination of parameters that are close to the optimal combination. To do so, this method uses a proxy optimization problem (finding the maximum of the acquisition function) that, albeit still a hard problem, is cheaper (in the computational sense) to solve, and it's amenable to common tools. Therefore, Bayesian Optimization is suggested for situations where sampling the function to be optimized is very expensive.
+
+Note that the only acceptable types within the search space are ``randint``\ , ``uniform``\ , ``quniform``\ ,  ``loguniform``\ , ``qloguniform``\ , and numerical ``choice``.
+
+This optimization approach is described in Section 3 of `Algorithms for Hyper-Parameter Optimization <https://papers.nips.cc/paper/4443-algorithms-for-hyper-parameter-optimization.pdf>`__.

docs/en_US/Tuner/GridsearchTuner.rst

+Grid Search on NNI
+==================
+
+Grid Search
+-----------
+
+Grid Search performs an exhaustive search through a manually specified subset of the hyperparameter space defined in the searchspace file. 
+
+Note that the only acceptable types within the search space are ``choice``\ , ``quniform``\ , and ``randint``.

docs/en_US/Tuner/HyperbandAdvisor.rst

+Hyperband on NNI
+================
+
+1. Introduction
+---------------
+
+`Hyperband <https://arxiv.org/pdf/1603.06560.pdf>`__ is a popular autoML algorithm. The basic idea of Hyperband is to create several buckets, each having ``n`` randomly generated hyperparameter configurations, each configuration using ``r`` resources (e.g., epoch number, batch number). After the ``n`` configurations are finished, it chooses the top ``n/eta`` configurations and runs them using increased ``r*eta`` resources. At last, it chooses the best configuration it has found so far.
+
+2. Implementation with full parallelism
+---------------------------------------
+
+First, this is an example of how to write an autoML algorithm based on MsgDispatcherBase, rather than Tuner and Assessor. Hyperband is implemented in this way because it integrates the functions of both Tuner and Assessor, thus, we call it Advisor.
+
+Second, this implementation fully leverages Hyperband's internal parallelism. Specifically, the next bucket is not started strictly after the current bucket. Instead, it starts when there are available resources. If you want to use full parallelism mode, set ``exec_mode`` with ``parallelism``. 
+
+Or if you want to set ``exec_mode`` with ``serial`` according to the original algorithm. In this mode, the next bucket will start strictly after the current bucket.
+
+``parallelism`` mode may lead to multiple unfinished buckets, and there is at most one unfinished bucket under ``serial`` mode. The advantage of ``parallelism`` mode is to make full use of resources, which may reduce the experiment duration multiple times. The following two pictures are the results of quick verification using `nas-bench-201 <../NAS/Benchmarks.rst>`__\ , picture above is in ``parallelism`` mode, picture below is in ``serial`` mode.
+
+
+.. image:: ../../img/hyperband_parallelism.png
+   :target: ../../img/hyperband_parallelism.png
+   :alt: parallelism mode
+
+
+
+.. image:: ../../img/hyperband_serial.png
+   :target: ../../img/hyperband_serial.png
+   :alt: serial mode
+
+
+If you want to reproduce these results, refer to the example under ``examples/trials/benchmarking/`` for details.
+
+3. Usage
+--------
+
+To use Hyperband, you should add the following spec in your experiment's YAML config file:
+
+.. code-block:: bash
+
+   advisor:
+     #choice: Hyperband
+     builtinAdvisorName: Hyperband
+     classArgs:
+       #R: the maximum trial budget
+       R: 100
+       #eta: proportion of discarded trials
+       eta: 3
+       #choice: maximize, minimize
+       optimize_mode: maximize
+       #choice: serial, parallelism
+       exec_mode: parallelism
+
+Note that once you use Advisor, you are not allowed to add a Tuner and Assessor spec in the config file. If you use Hyperband, among the hyperparameters (i.e., key-value pairs) received by a trial, there will be one more key called ``TRIAL_BUDGET`` defined by user. **By using this ``TRIAL_BUDGET``\ , the trial can control how long it runs**.
+
+For ``report_intermediate_result(metric)`` and ``report_final_result(metric)`` in your trial code, **\ ``metric`` should be either a number or a dict which has a key ``default`` with a number as its value**. This number is the one you want to maximize or minimize, for example, accuracy or loss.
+
+``R`` and ``eta`` are the parameters of Hyperband that you can change. ``R`` means the maximum trial budget that can be allocated to a configuration. Here, trial budget could mean the number of epochs or mini-batches. This ``TRIAL_BUDGET`` should be used by the trial to control how long it runs. Refer to the example under ``examples/trials/mnist-advisor/`` for details.
+
+``eta`` means ``n/eta`` configurations from ``n`` configurations will survive and rerun using more budgets.
+
+Here is a concrete example of ``R=81`` and ``eta=3``\ :
+
+.. list-table::
+   :header-rows: 1
+   :widths: auto
+
+   * -
+     - s=4
+     - s=3
+     - s=2
+     - s=1
+     - s=0
+   * - i
+     - n r
+     - n r
+     - n r
+     - n r
+     - n r
+   * - 0
+     - 81 1
+     - 27 3
+     - 9 9
+     - 6 27
+     - 5 81
+   * - 1
+     - 27 3
+     - 9 9
+     - 3 27
+     - 2 81
+     -
+   * - 2
+     - 9 9
+     - 3 27
+     - 1 81
+     -
+     -
+   * - 3
+     - 3 27
+     - 1 81
+     -
+     -
+     -
+   * - 4
+     - 1 81
+     -
+     -
+     -
+     -
+
+
+``s`` means bucket, ``n`` means the number of configurations that are generated, the corresponding ``r`` means how many budgets these configurations run. ``i`` means round, for example, bucket 4 has 5 rounds, bucket 3 has 4 rounds.
+
+For information about writing trial code, please refer to the instructions under ``examples/trials/mnist-hyperband/``.
+
+4. Future improvements
+----------------------
+
+The current implementation of Hyperband can be further improved by supporting a simple early stop algorithm since it's possible that not all the configurations in the top ``n/eta`` perform well. Any unpromising configurations should be stopped early.
+
+In the current implementation, configurations are generated randomly which follows the design in the `paper <https://arxiv.org/pdf/1603.06560.pdf>`__. As an improvement, configurations could be generated more wisely by leveraging advanced algorithms.

docs/en_US/Tuner/HyperoptTuner.rst

+TPE, Random Search, Anneal Tuners on NNI
+========================================
+
+TPE
+---
+
+The Tree-structured Parzen Estimator (TPE) is a sequential model-based optimization (SMBO) approach. SMBO methods sequentially construct models to approximate the performance of hyperparameters based on historical measurements, and then subsequently choose new hyperparameters to test based on this model. The TPE approach models P(x|y) and P(y) where x represents hyperparameters and y the associated evaluation matric. P(x|y) is modeled by transforming the generative process of hyperparameters, replacing the distributions of the configuration prior with non-parametric densities. This optimization approach is described in detail in `Algorithms for Hyper-Parameter Optimization <https://papers.nips.cc/paper/4443-algorithms-for-hyper-parameter-optimization.pdf>`__. ​
+
+Parallel TPE optimization
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+TPE approaches were actually run asynchronously in order to make use of multiple compute nodes and to avoid wasting time waiting for trial evaluations to complete. The original algorithm design was optimized for sequential computation. If we were to use TPE with much concurrency, its performance will be bad. We have optimized this case using the Constant Liar algorithm. For these principles of optimization, please refer to our `research blog <../CommunitySharings/ParallelizingTpeSearch.rst>`__.
+
+Usage
+^^^^^
+
+ To use TPE, you should add the following spec in your experiment's YAML config file:
+
+.. code-block:: yaml
+
+   tuner:
+     builtinTunerName: TPE
+     classArgs:
+       optimize_mode: maximize
+       parallel_optimize: True
+       constant_liar_type: min
+
+**classArgs requirements:**
+
+
+* **optimize_mode** (*maximize or minimize, optional, default = maximize*\ ) - If 'maximize', tuners will try to maximize metrics. If 'minimize', tuner will try to minimize metrics.
+* **parallel_optimize** (*bool, optional, default = False*\ ) - If True, TPE will use the Constant Liar algorithm to optimize parallel hyperparameter tuning. Otherwise, TPE will not discriminate between sequential or parallel situations.
+* **constant_liar_type** (*min or max or mean, optional, default = min*\ ) - The type of constant liar to use, will logically be determined on the basis of the values taken by y at X. There are three possible values, min{Y}, max{Y}, and mean{Y}.
+
+Random Search
+-------------
+
+In `Random Search for Hyper-Parameter Optimization <http://www.jmlr.org/papers/volume13/bergstra12a/bergstra12a.pdf>`__ we show that Random Search might be surprisingly effective despite its simplicity. We suggest using Random Search as a baseline when no knowledge about the prior distribution of hyper-parameters is available.
+
+Anneal
+------
+
+This simple annealing algorithm begins by sampling from the prior but tends over time to sample from points closer and closer to the best ones observed. This algorithm is a simple variation on random search that leverages smoothness in the response surface. The annealing rate is not adaptive.