the source code of NNI for DCU

docs/en_US/Tuner/BatchTuner.rst

+Batch Tuner
+===========
+
+Batch tuner allows users to simply provide several configurations (i.e., choices of hyper-parameters) for their trial code. After finishing all the configurations, the experiment is done. Batch tuner only supports the type ``choice`` in the `search space spec <../Tutorial/SearchSpaceSpec.rst>`__.
+
+Suggested scenario: If the configurations you want to try have been decided, you can list them in the SearchSpace file (using ``choice``) and run them using the batch tuner.
+
+Usage
+-----
+
+Example Configuration
+^^^^^^^^^^^^^^^^^^^^^
+
+.. code-block:: yaml
+
+   # config.yml
+   tuner:
+     name: BatchTuner
+
+Note that the search space for BatchTuner should look like:
+
+.. code-block:: json
+
+   {
+       "combine_params":
+       {
+           "_type" : "choice",
+           "_value" : [{"optimizer": "Adam", "learning_rate": 0.00001},
+                       {"optimizer": "Adam", "learning_rate": 0.0001},
+                       {"optimizer": "Adam", "learning_rate": 0.001},
+                       {"optimizer": "SGD", "learning_rate": 0.01},
+                       {"optimizer": "SGD", "learning_rate": 0.005},
+                       {"optimizer": "SGD", "learning_rate": 0.0002}]
+       }
+   }
+
+The search space file should include the high-level key ``combine_params``. The type of params in the search space must be ``choice`` and the ``values`` must include all the combined params values.

docs/en_US/Tuner/BohbAdvisor.rst

+BOHB Advisor
+============
+
+BOHB is a robust and efficient hyperparameter tuning algorithm mentioned in `this reference paper <https://arxiv.org/abs/1807.01774>`__. BO is an abbreviation for "Bayesian Optimization" and HB is an abbreviation for "Hyperband".
+
+BOHB relies on HB (Hyperband) to determine how many configurations to evaluate with which budget, but it **replaces the random selection of configurations at the beginning of each HB iteration by a model-based search (Bayesian Optimization)**. Once the desired number of configurations for the iteration is reached, the standard successive halving procedure is carried out using these configurations. We keep track of the performance of all function evaluations g(x, b) of configurations x on all budgets b to use as a basis for our models in later iterations.
+
+Below we divide the introduction of the BOHB process into two parts:
+
+HB (Hyperband)
+^^^^^^^^^^^^^^
+
+We follow Hyperband’s way of choosing the budgets and continue to use SuccessiveHalving. For more details, you can refer to the `Hyperband in NNI <HyperbandAdvisor.rst>`__ and the `reference paper for Hyperband <https://arxiv.org/abs/1603.06560>`__. This procedure is summarized by the pseudocode below.
+
+
+.. image:: ../../img/bohb_1.png
+   :target: ../../img/bohb_1.png
+   :alt: 
+
+
+BO (Bayesian Optimization)
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The BO part of BOHB closely resembles TPE with one major difference: we opted for a single multidimensional KDE compared to the hierarchy of one-dimensional KDEs used in TPE in order to better handle interaction effects in the input space.
+
+Tree Parzen Estimator(TPE): uses a KDE (kernel density estimator) to model the densities.
+
+
+.. image:: ../../img/bohb_2.png
+   :target: ../../img/bohb_2.png
+   :alt: 
+
+
+To fit useful KDEs, we require a minimum number of data points Nmin; this is set to d + 1 for our experiments, where d is the number of hyperparameters. To build a model as early as possible, we do not wait until Nb = \|Db\|, where the number of observations for budget b is large enough to satisfy q · Nb ≥ Nmin. Instead, after initializing with Nmin + 2 random configurations, we choose the
+
+
+.. image:: ../../img/bohb_3.png
+   :target: ../../img/bohb_3.png
+   :alt: 
+
+
+best and worst configurations, respectively, to model the two densities.
+
+Note that we also sample a constant fraction named **random fraction** of the configurations uniformly at random.
+
+Workflow
+--------
+
+
+.. image:: ../../img/bohb_6.jpg
+   :target: ../../img/bohb_6.jpg
+   :alt: 
+
+
+This image shows the workflow of BOHB. Here we set max_budget = 9, min_budget = 1, eta = 3, others as default. In this case, s_max = 2, so we will continuously run the {s=2, s=1, s=0, s=2, s=1, s=0, ...} cycle. In each stage of SuccessiveHalving (the orange box), we will pick the top 1/eta configurations and run them again with more budget, repeating the SuccessiveHalving stage until the end of this iteration. At the same time, we collect the configurations, budgets and final metrics of each trial and use these to build a multidimensional KDEmodel with the key "budget".
+ Multidimensional KDE is used to guide the selection of configurations for the next iteration.
+
+The sampling procedure (using Multidimensional KDE to guide selection) is summarized by the pseudocode below.
+
+
+.. image:: ../../img/bohb_4.png
+   :target: ../../img/bohb_4.png
+   :alt: 
+
+
+Usage
+-----
+
+Installation
+^^^^^^^^^^^^
+
+BOHB advisor requires the `ConfigSpace <https://github.com/automl/ConfigSpace>`__ package. ConfigSpace can be installed using the following command.
+
+.. code-block:: bash
+
+   pip install nni[BOHB]
+
+classArgs Requirements
+^^^^^^^^^^^^^^^^^^^^^^
+
+* **optimize_mode** (*maximize or minimize, optional, default = maximize*) - If 'maximize', tuners will try to maximize metrics. If 'minimize', tuner will try to minimize metrics.
+* **min_budget** (*int, optional, default = 1*) - The smallest budget to assign to a trial job, (budget can be the number of mini-batches or epochs). Needs to be positive.
+* **max_budget** (*int, optional, default = 3*) - The largest budget to assign to a trial job, (budget can be the number of mini-batches or epochs). Needs to be larger than min_budget.
+* **eta** (*int, optional, default = 3*) - In each iteration, a complete run of sequential halving is executed. In it, after evaluating each configuration on the same subset size, only a fraction of 1/eta of them 'advances' to the next round. Must be greater or equal to 2.
+* **min_points_in_model** (*int, optional, default = None*): number of observations to start building a KDE. Default 'None' means dim+1; when the number of completed trials in this budget is equal to or larger than ``max{dim+1, min_points_in_model}``, BOHB will start to build a KDE model of this budget then use said KDE model to guide configuration selection. Needs to be positive. (dim means the number of hyperparameters in search space)
+* **top_n_percent** (*int, optional, default = 15*): percentage (between 1 and 99) of the observations which are considered good. Good points and bad points are used for building KDE models. For example, if you have 100 observed trials and top_n_percent is 15, then the top 15% of points will be used for building the good points models "l(x)". The remaining 85% of points will be used for building the bad point models "g(x)".
+* **num_samples** (*int, optional, default = 64*): number of samples to optimize EI (default 64). In this case, we will sample "num_samples" points and compare the result of l(x)/g(x). Then we will return the one with the maximum l(x)/g(x) value as the next configuration if the optimize_mode is ``maximize``. Otherwise, we return the smallest one.
+* **random_fraction** (*float, optional, default = 0.33*): fraction of purely random configurations that are sampled from the prior without the model.
+* **bandwidth_factor** (*float, optional, default = 3.0*): to encourage diversity, the points proposed to optimize EI are sampled from a 'widened' KDE where the bandwidth is multiplied by this factor. We suggest using the default value if you are not familiar with KDE.
+* **min_bandwidth** (*float, optional, default = 0.001*): to keep diversity, even when all (good) samples have the same value for one of the parameters, a minimum bandwidth (default: 1e-3) is used instead of zero. We suggest using the default value if you are not familiar with KDE.
+
+*Please note that the float type currently only supports decimal representations. You have to use 0.333 instead of 1/3 and 0.001 instead of 1e-3.*
+
+
+Config File
+^^^^^^^^^^^
+
+To use BOHB, you should add the following spec in your experiment's YAML config file:
+
+.. code-block:: yaml
+
+   advisor:
+     builtinAdvisorName: BOHB
+     classArgs:
+       optimize_mode: maximize
+       min_budget: 1
+       max_budget: 27
+       eta: 3
+       min_points_in_model: 7
+       top_n_percent: 15
+       num_samples: 64
+       random_fraction: 0.33
+       bandwidth_factor: 3.0
+       min_bandwidth: 0.001
+
+**classArgs Requirements:**
+
+
+* **optimize_mode** (*maximize or minimize, optional, default = maximize*) - If 'maximize', tuners will try to maximize metrics. If 'minimize', tuner will try to minimize metrics.
+* **min_budget** (*int, optional, default = 1*) - The smallest budget to assign to a trial job, (budget can be the number of mini-batches or epochs). Needs to be positive.
+* **max_budget** (*int, optional, default = 3*) - The largest budget to assign to a trial job, (budget can be the number of mini-batches or epochs). Needs to be larger than min_budget.
+* **eta** (*int, optional, default = 3*) - In each iteration, a complete run of sequential halving is executed. In it, after evaluating each configuration on the same subset size, only a fraction of 1/eta of them 'advances' to the next round. Must be greater or equal to 2.
+* **min_points_in_model** (*int, optional, default = None*): number of observations to start building a KDE. Default 'None' means dim+1; when the number of completed trials in this budget is equal to or larger than ``max{dim+1, min_points_in_model}``, BOHB will start to build a KDE model of this budget then use said KDE model to guide configuration selection. Needs to be positive. (dim means the number of hyperparameters in search space)
+* **top_n_percent** (*int, optional, default = 15*): percentage (between 1 and 99) of the observations which are considered good. Good points and bad points are used for building KDE models. For example, if you have 100 observed trials and top_n_percent is 15, then the top 15% of points will be used for building the good points models "l(x)". The remaining 85% of points will be used for building the bad point models "g(x)".
+* **num_samples** (*int, optional, default = 64*): number of samples to optimize EI (default 64). In this case, we will sample "num_samples" points and compare the result of l(x)/g(x). Then we will return the one with the maximum l(x)/g(x) value as the next configuration if the optimize_mode is ``maximize``. Otherwise, we return the smallest one.
+* **random_fraction** (*float, optional, default = 0.33*): fraction of purely random configurations that are sampled from the prior without the model.
+* **bandwidth_factor** (*float, optional, default = 3.0*): to encourage diversity, the points proposed to optimize EI are sampled from a 'widened' KDE where the bandwidth is multiplied by this factor. We suggest using the default value if you are not familiar with KDE.
+* **min_bandwidth** (*float, optional, default = 0.001*): to keep diversity, even when all (good) samples have the same value for one of the parameters, a minimum bandwidth (default: 1e-3) is used instead of zero. We suggest using the default value if you are not familiar with KDE.
+* **config_space** (*str, optional*): directly use a .pcs file serialized by `ConfigSpace <https://automl.github.io/ConfigSpace/>` in "pcs new" format. In this case, search space file (if provided in config) will be ignored. Note that this path needs to be an absolute path. Relative path is currently not supported.
+
+*Please note that the float type currently only supports decimal representations. You have to use 0.333 instead of 1/3 and 0.001 instead of 1e-3.*
+
+File Structure
+--------------
+
+The advisor has a lot of different files, functions, and classes. Here, we will only give most of those files a brief introduction:
+
+
+* ``bohb_advisor.py`` Definition of BOHB, handles interaction with the dispatcher, including generating new trials and processing results. Also includes the implementation of the HB (Hyperband) part.
+* ``config_generator.py`` Includes the implementation of the BO (Bayesian Optimization) part. The function *get_config* can generate new configurations based on BO; the function *new_result* will update the model with the new result.
+
+Experiment
+----------
+
+MNIST with BOHB
+^^^^^^^^^^^^^^^
+
+code implementation: :githublink:`examples/trials/mnist-advisor <examples/trials/>`
+
+We chose BOHB to build a CNN on the MNIST dataset. The following is our experimental final results:
+
+
+.. image:: ../../img/bohb_5.png
+   :target: ../../img/bohb_5.png
+   :alt: 
+
+
+More experimental results can be found in the `reference paper <https://arxiv.org/abs/1807.01774>`__. We can see that BOHB makes good use of previous results and has a balanced trade-off in exploration and exploitation.

docs/en_US/Tuner/BuiltinTuner.rst

+HyperParameter Tuning with NNI Built-in Tuners
+==============================================
+
+To fit a machine/deep learning model into different tasks/problems, hyperparameters always need to be tuned. Automating the process of hyperparaeter tuning always requires a good tuning algorithm. NNI has provided state-of-the-art tuning algorithms as part of our built-in tuners and makes them easy to use. Below is the brief summary of NNI's current built-in tuners:
+
+Note: Click the **Tuner's name** to get the Tuner's installation requirements, suggested scenario, and an example configuration. A link for a detailed description of each algorithm is located at the end of the suggested scenario for each tuner. Here is an `article <../CommunitySharings/HpoComparison.rst>`__ comparing different Tuners on several problems.
+
+Currently, we support the following algorithms:
+
+.. list-table::
+   :header-rows: 1
+   :widths: auto
+
+   * - Tuner
+     - Brief Introduction of Algorithm
+
+   * - `TPE <./TpeTuner.rst>`__
+     - The Tree-structured Parzen Estimator (TPE) is a sequential model-based optimization (SMBO) approach. SMBO methods sequentially construct models to approximate the performance of hyperparameters based on historical measurements, and then subsequently choose new hyperparameters to test based on this model. `Reference Paper <https://papers.nips.cc/paper/4443-algorithms-for-hyper-parameter-optimization.pdf>`__
+
+       TPE, as a black-box optimization, can be used in various scenarios and shows good performance in general. Especially when you have limited computation resources and can only try a small number of trials. From a large amount of experiments, we found that TPE is far better than Random Search.
+
+   * - `Random Search <./RandomTuner.rst>`__
+     - In Random Search for Hyper-Parameter Optimization show that Random Search might be surprisingly simple and effective. We suggest that we could use Random Search as the baseline when we have no knowledge about the prior distribution of hyper-parameters. `Reference Paper <http://www.jmlr.org/papers/volume13/bergstra12a/bergstra12a.pdf>`__
+
+       Random search is suggested when each trial does not take very long (e.g., each trial can be completed very quickly, or early stopped by the assessor), and you have enough computational resources. It's also useful if you want to uniformly explore the search space. Random Search can be considered a baseline search algorithm.
+
+   * - `Anneal <./AnnealTuner.rst>`__
+     - This simple annealing algorithm begins by sampling from the prior, but tends over time to sample from points closer and closer to the best ones observed. This algorithm is a simple variation on the random search that leverages smoothness in the response surface. The annealing rate is not adaptive.
+
+       Anneal is suggested when each trial does not take very long and you have enough computation resources (very similar to Random Search). It's also useful when the variables in the search space can be sample from some prior distribution.
+
+   * - `Naïve Evolution <./EvolutionTuner.rst>`__
+     - Naïve Evolution comes from Large-Scale Evolution of Image Classifiers. It randomly initializes a population-based on search space. For each generation, it chooses better ones and does some mutation (e.g., change a hyperparameter, add/remove one layer) on them to get the next generation. Naïve Evolution requires many trials to work, but it's very simple and easy to expand new features. `Reference paper <https://arxiv.org/pdf/1703.01041.pdf>`__
+
+       Its computational resource requirements are relatively high. Specifically, it requires a large initial population to avoid falling into a local optimum. If your trial is short or leverages assessor, this tuner is a good choice. It is also suggested when your trial code supports weight transfer; that is, the trial could inherit the converged weights from its parent(s). This can greatly speed up the training process.
+
+   * - `SMAC <./SmacTuner.rst>`__
+     - SMAC is based on Sequential Model-Based Optimization (SMBO). It adapts the most prominent previously used model class (Gaussian stochastic process models) and introduces the model class of random forests to SMBO, in order to handle categorical parameters. The SMAC supported by NNI is a wrapper on the SMAC3 GitHub repo. Notice, SMAC needs to be installed by ``pip install nni[SMAC]`` command. `Reference Paper, <https://www.cs.ubc.ca/~hutter/papers/10-TR-SMAC.pdf>`__ `GitHub Repo <https://github.com/automl/SMAC3>`__
+
+       **Please note that SMAC doesn't support running on Windows currently**. For the specific reason, please refer to this `GitHub issue <https://github.com/automl/SMAC3/issues/483>`__.
+
+       Similar to TPE, SMAC is also a black-box tuner that can be tried in various scenarios and is suggested when computational resources are limited. It is optimized for discrete hyperparameters, thus, it's suggested when most of your hyperparameters are discrete.
+
+   * - `Batch tuner <./BatchTuner.rst>`__
+     - Batch tuner allows users to simply provide several configurations (i.e., choices of hyper-parameters) for their trial code. After finishing all the configurations, the experiment is done. Batch tuner only supports the type choice in search space spec.
+
+       If the configurations you want to try have been decided beforehand, you can list them in search space file (using ``choice``) and run them using batch tuner.
+       `Detailed Description <./BatchTuner.rst>`__
+
+   * - `Grid Search <./GridsearchTuner.rst>`__
+     - Grid Search performs an exhaustive searching through the search space.
+
+       This is suggested when the search space is small. It's suggested when it is feasible to exhaustively sweep the whole search space.
+
+   * - `Hyperband <./HyperbandAdvisor.rst>`__
+     - Hyperband tries to use limited resources to explore as many configurations as possible and returns the most promising ones as a final result. The basic idea is to generate many configurations and run them for a small number of trials. The half least-promising configurations are thrown out, the remaining are further trained along with a selection of new configurations. The size of these populations is sensitive to resource constraints (e.g. allotted search time). `Reference Paper <https://arxiv.org/pdf/1603.06560.pdf>`__
+
+       This is suggested when you have limited computational resources but have a relatively large search space. It performs well in scenarios where intermediate results can indicate good or bad final results to some extent. For example, when models that are more accurate early on in training are also more accurate later on.
+
+   * - `Network Morphism <./NetworkmorphismTuner.rst>`__
+     - Network Morphism provides functions to automatically search for deep learning architectures. It generates child networks that inherit the knowledge from their parent network which it is a morph from. This includes changes in depth, width, and skip-connections. Next, it estimates the value of a child network using historic architecture and metric pairs. Then it selects the most promising one to train. `Reference Paper <https://arxiv.org/abs/1806.10282>`__
+
+       This is suggested when you want to apply deep learning methods to your task but you have no idea how to choose or design a network. You may modify this :githublink:`example <examples/trials/network_morphism/cifar10/cifar10_keras.py>` to fit your own dataset and your own data augmentation method. Also you can change the batch size, learning rate, or optimizer. Currently, this tuner only supports the computer vision domain.
+
+   * - `Metis Tuner <./MetisTuner.rst>`__
+     - Metis offers the following benefits when it comes to tuning parameters: While most tools only predict the optimal configuration, Metis gives you two outputs: (a) current prediction of optimal configuration, and (b) suggestion for the next trial. No more guesswork. While most tools assume training datasets do not have noisy data, Metis actually tells you if you need to re-sample a particular hyper-parameter. `Reference Paper <https://www.microsoft.com/en-us/research/publication/metis-robustly-tuning-tail-latencies-cloud-systems/>`__
+
+       Similar to TPE and SMAC, Metis is a black-box tuner. If your system takes a long time to finish each trial, Metis is more favorable than other approaches such as random search. Furthermore, Metis provides guidance on subsequent trials. Here is an :githublink:`example <examples/trials/auto-gbdt/search_space_metis.json>` on the use of Metis. Users only need to send the final result, such as ``accuracy``, to the tuner by calling the NNI SDK.
+
+       Note that the only acceptable types of search space types are ``quniform``, ``uniform``, ``randint``, and numerical ``choice``. Only numerical values are supported since the values will be used to evaluate the 'distance' between different points.
+
+   * - `BOHB <./BohbAdvisor.rst>`__
+     - BOHB is a follow-up work to Hyperband. It targets the weakness of Hyperband that new configurations are generated randomly without leveraging finished trials. For the name BOHB, HB means Hyperband, BO means Bayesian Optimization. BOHB leverages finished trials by building multiple TPE models, a proportion of new configurations are generated through these models. `Reference Paper <https://arxiv.org/abs/1807.01774>`__
+
+       Similar to Hyperband, BOHB is suggested when you have limited computational resources but have a relatively large search space. It performs well in scenarios where intermediate results can indicate good or bad final results to some extent. In this case, it may converge to a better configuration than Hyperband due to its usage of Bayesian optimization.
+
+   * - `GP Tuner <./GPTuner.rst>`__
+     - Gaussian Process Tuner is a sequential model-based optimization (SMBO) approach with Gaussian Process as the surrogate. `Reference Paper <https://papers.nips.cc/paper/4443-algorithms-for-hyper-parameter-optimization.pdf>`__, `Github Repo <https://github.com/fmfn/BayesianOptimization>`__
+
+       Note that the only acceptable types within the search space are ``randint``, ``uniform``, ``quniform``, ``loguniform``, ``qloguniform``, and numerical ``choice``. Only numerical values are supported since the values will be used to evaluate the 'distance' between different points.
+
+       As a strategy in a Sequential Model-based Global Optimization (SMBO) algorithm, GP Tuner uses a proxy optimization problem (finding the maximum of the acquisition function) that, albeit still a hard problem, is cheaper (in the computational sense) to solve and common tools can be employed to solve it. Therefore, GP Tuner is most adequate for situations where the function to be optimized is very expensive to evaluate. GP can be used when computational resources are limited. However, GP Tuner has a computational cost that grows at *O(N^3)* due to the requirement of inverting the Gram matrix, so it's not suitable when lots of trials are needed.
+
+   * - `PBT Tuner <./PBTTuner.rst>`__
+     - PBT Tuner is a simple asynchronous optimization algorithm which effectively utilizes a fixed computational budget to jointly optimize a population of models and their hyperparameters to maximize performance. `Reference Paper <https://arxiv.org/abs/1711.09846v1>`__
+
+       Population Based Training (PBT) bridges and extends parallel search methods and sequential optimization methods. It requires relatively small computation resource, by inheriting weights from currently good-performing ones to explore better ones periodically. With PBTTuner, users finally get a trained model, rather than a configuration that could reproduce the trained model by training the model from scratch. This is because model weights are inherited periodically through the whole search process. PBT can also be seen as a training approach. If you don't need to get a specific configuration, but just expect a good model, PBTTuner is a good choice.
+
+   * - `DNGO Tuner <./DngoTuner.rst>`__
+     - Use of neural networks as an alternative to GPs to model distributions over functions in bayesian optimization.
+
+       Applicable to large scale hyperparameter optimization. Bayesian optimization that rapidly finds competitive models on benchmark object recognition tasks using convolutional networks, and image caption generation using neural language models.
+
+Usage of Built-in Tuners
+------------------------
+
+Using a built-in tuner provided by the NNI SDK requires one to declare the  **name** and **classArgs** in the ``config.yml`` file.
+Click tuners' name in above table to see their specification.
+
+Note: Some built-in tuners have dependencies that need to be installed using ``pip install nni[<tuner>]``, like SMAC's dependencies can be installed using ``pip install nni[SMAC]``.

docs/en_US/Tuner/CustomizeAdvisor.rst

+**How To** - Customize Your Own Advisor
+=======================================
+
+*Warning: API is subject to change in future releases.*
+
+Advisor targets the scenario that the automl algorithm wants the methods of both tuner and assessor. Advisor is similar to tuner on that it receives trial parameters request, final results, and generate trial parameters. Also, it is similar to assessor on that it receives intermediate results, trial's end state, and could send trial kill command. Note that, if you use Advisor, tuner and assessor are not allowed to be used at the same time.
+
+If a user want to implement a customized Advisor, she/he only needs to:
+
+**1. Define an Advisor inheriting from the MsgDispatcherBase class.** For example:
+
+.. code-block:: python
+
+   from nni.runtime.msg_dispatcher_base import MsgDispatcherBase
+
+   class CustomizedAdvisor(MsgDispatcherBase):
+       def __init__(self, ...):
+           ...
+
+**2. Implement the methods with prefix "handle_" except "handle_request""**
+
+You might find `docs <../autotune_ref.rst#Advisor>`__ for ``MsgDispatcherBase`` helpful.
+
+**3. Configure your customized Advisor in experiment YAML config file.**
+
+Similar to tuner and assessor. NNI needs to locate your customized Advisor class and instantiate the class, so you need to specify the location of the customized Advisor class and pass literal values as parameters to the ``__init__`` constructor.
+
+.. code-block:: yaml
+
+   advisor:
+     codeDir: /home/abc/myadvisor
+     classFileName: my_customized_advisor.py
+     className: CustomizedAdvisor
+     # Any parameter need to pass to your advisor class __init__ constructor
+     # can be specified in this optional classArgs field, for example
+     classArgs:
+       arg1: value1
+
+**Note that** The working directory of your advisor is ``<home>/nni-experiments/<experiment_id>/log``, which can be retrieved with environment variable ``NNI_LOG_DIRECTORY``.
+
+Example
+-------
+
+Here we provide an :githublink:`example <examples/tuners/mnist_keras_customized_advisor>`.

docs/en_US/Tuner/CustomizeTuner.rst

+Customize-Tuner
+===============
+
+NNI provides state-of-the-art tuning algorithm in builtin-tuners. NNI supports to build a tuner by yourself for tuning demand.
+
+If you want to implement your own tuning algorithm, you can implement a customized Tuner, there are three things to do:
+
+
+#. Inherit the base Tuner class
+#. Implement receive_trial_result, generate_parameter and update_search_space function
+#. Configure your customized tuner in experiment YAML config file
+
+Here is an example:
+
+**1. Inherit the base Tuner class**
+
+.. code-block:: python
+
+   from nni.tuner import Tuner
+
+   class CustomizedTuner(Tuner):
+       def __init__(self, ...):
+           ...
+
+**2. Implement receive_trial_result, generate_parameter and update_search_space function**
+
+.. code-block:: python
+
+   from nni.tuner import Tuner
+
+   class CustomizedTuner(Tuner):
+       def __init__(self, ...):
+           ...
+
+       def receive_trial_result(self, parameter_id, parameters, value, **kwargs):
+           '''
+           Receive trial's final result.
+           parameter_id: int
+           parameters: object created by 'generate_parameters()'
+           value: final metrics of the trial, including default metric
+           '''
+           # your code implements here.
+       ...
+
+       def generate_parameters(self, parameter_id, **kwargs):
+           '''
+           Returns a set of trial (hyper-)parameters, as a serializable object
+           parameter_id: int
+           '''
+           # your code implements here.
+           return your_parameters
+       ...
+
+       def update_search_space(self, search_space):
+           '''
+           Tuners are advised to support updating search space at run-time.
+           If a tuner can only set search space once before generating first hyper-parameters,
+           it should explicitly document this behaviour.
+           search_space: JSON object created by experiment owner
+           '''
+           # your code implements here.
+       ...
+
+``receive_trial_result`` will receive the ``parameter_id, parameters, value`` as parameters input. Also, Tuner will receive the ``value`` object are exactly same value that Trial send.
+
+The ``your_parameters`` return from ``generate_parameters`` function, will be package as json object by NNI SDK. NNI SDK will unpack json object so the Trial will receive the exact same ``your_parameters`` from Tuner.
+
+For example:
+If the you implement the ``generate_parameters`` like this:
+
+.. code-block:: python
+
+   def generate_parameters(self, parameter_id, **kwargs):
+       '''
+       Returns a set of trial (hyper-)parameters, as a serializable object
+       parameter_id: int
+       '''
+       # your code implements here.
+       return {"dropout": 0.3, "learning_rate": 0.4}
+
+It means your Tuner will always generate parameters ``{"dropout": 0.3, "learning_rate": 0.4}``. Then Trial will receive ``{"dropout": 0.3, "learning_rate": 0.4}`` by calling API ``nni.get_next_parameter()``. Once the trial ends with a result (normally some kind of metrics), it can send the result to Tuner by calling API ``nni.report_final_result()``, for example ``nni.report_final_result(0.93)``. Then your Tuner's ``receive_trial_result`` function will receied the result like：
+
+.. code-block:: python
+
+   parameter_id = 82347
+   parameters = {"dropout": 0.3, "learning_rate": 0.4}
+   value = 0.93
+
+**Note that** The working directory of your tuner is ``<home>/nni-experiments/<experiment_id>/log``, which can be retrieved with environment variable ``NNI_LOG_DIRECTORY``, therefore, if you want to access a file (e.g., ``data.txt``) in the directory of your own tuner, you cannot use ``open('data.txt', 'r')``. Instead, you should use the following:
+
+.. code-block:: python
+
+   _pwd = os.path.dirname(__file__)
+   _fd = open(os.path.join(_pwd, 'data.txt'), 'r')
+
+This is because your tuner is not executed in the directory of your tuner (i.e., ``pwd`` is not the directory of your own tuner).
+
+**3. Configure your customized tuner in experiment YAML config file**
+
+NNI needs to locate your customized tuner class and instantiate the class, so you need to specify the location of the customized tuner class and pass literal values as parameters to the __init__ constructor.
+
+.. code-block:: yaml
+
+   tuner:
+     codeDir: /home/abc/mytuner
+     classFileName: my_customized_tuner.py
+     className: CustomizedTuner
+     # Any parameter need to pass to your tuner class __init__ constructor
+     # can be specified in this optional classArgs field, for example
+     classArgs:
+       arg1: value1
+
+More detail example you could see:
+
+..
+
+   * :githublink:`evolution-tuner <nni/algorithms/hpo/evolution_tuner.py>`
+   * :githublink:`hyperopt-tuner <nni/algorithms/hpo/hyperopt_tuner.py>`
+   * :githublink:`evolution-based-customized-tuner <examples/tuners/ga_customer_tuner>`
+
+
+Write a more advanced automl algorithm
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The methods above are usually enough to write a general tuner. However, users may also want more methods, for example, intermediate results, trials' state (e.g., the methods in assessor), in order to have a more powerful automl algorithm. Therefore, we have another concept called ``advisor`` which directly inherits from ``MsgDispatcherBase`` in :githublink:`msg_dispatcher_base.py <nni/runtime/msg_dispatcher_base.py>`. Please refer to `here <CustomizeAdvisor.rst>`__ for how to write a customized advisor.

docs/en_US/Tuner/DngoTuner.rst

+DNGO Tuner
+==========
+
+Usage
+-----
+
+Installation
+^^^^^^^^^^^^
+
+classArgs requirements
+^^^^^^^^^^^^^^^^^^^^^^
+
+* **optimize_mode** (*'maximize' or 'minimize'*) - If 'maximize', the tuner will target to maximize metrics. If 'minimize', the tuner will target to minimize metrics.
+* **sample_size** (*int, default = 1000*) - Number of samples to select in each iteration. The best one will be picked from the samples as the next trial.
+* **trials_per_update** (*int, default = 20*) - Number of trials to collect before updating the model.
+* **num_epochs_per_training** (*int, default = 500*) - Number of epochs to train DNGO model.
+
+Example Configuration
+^^^^^^^^^^^^^^^^^^^^^
+
+.. code-block:: yaml
+
+   # config.yml
+   tuner:
+     name: DNGOTuner
+     classArgs:
+       optimize_mode: maximize

docs/en_US/Tuner/EvolutionTuner.rst

+Naive Evolution Tuner
+=====================
+
+Naive Evolution comes from `Large-Scale Evolution of Image Classifiers <https://arxiv.org/pdf/1703.01041.pdf>`__. It randomly initializes a population based on the search space. For each generation, it chooses better ones and does some mutation (e.g., changes a hyperparameter, adds/removes one layer, etc.) on them to get the next generation. Naive Evolution requires many trials to works but it's very simple and it's easily expanded with new features.
+
+Usage
+-----
+
+classArgs Requirements
+^^^^^^^^^^^^^^^^^^^^^^
+
+* 
+  **optimize_mode** (*maximize or minimize, optional, default = maximize*) - If 'maximize', the tuner will try to maximize metrics. If 'minimize', the tuner will try to minimize metrics.
+
+* 
+  **population_size** (*int value (should > 0), optional, default = 20*) - the initial size of the population (trial num) in the evolution tuner. It's suggested that ``population_size`` be much larger than ``concurrency`` so users can get the most out of the algorithm (and at least ``concurrency``, or the tuner will fail on its first generation of parameters).
+
+Example Configuration
+^^^^^^^^^^^^^^^^^^^^^
+
+.. code-block:: yaml
+
+   # config.yml
+   tuner:
+     name: Evolution
+     classArgs:
+       optimize_mode: maximize
+       population_size: 100
+

docs/en_US/Tuner/GPTuner.rst

+GP Tuner
+========
+
+Bayesian optimization works by constructing a posterior distribution of functions (a Gaussian Process) that best describes the function you want to optimize. As the number of observations grows, the posterior distribution improves, and the algorithm becomes more certain of which regions in parameter space are worth exploring and which are not.
+
+GP Tuner is designed to minimize/maximize the number of steps required to find a combination of parameters that are close to the optimal combination. To do so, this method uses a proxy optimization problem (finding the maximum of the acquisition function) that, albeit still a hard problem, is cheaper (in the computational sense) to solve, and it's amenable to common tools. Therefore, Bayesian Optimization is suggested for situations where sampling the function to be optimized is very expensive.
+
+Note that the only acceptable types within the search space are ``randint``, ``uniform``, ``quniform``, ``loguniform``, ``qloguniform``, and numerical ``choice``.
+
+This optimization approach is described in Section 3 of `Algorithms for Hyper-Parameter Optimization <https://papers.nips.cc/paper/4443-algorithms-for-hyper-parameter-optimization.pdf>`__.
+
+Usage
+-----
+
+classArgs requirements
+^^^^^^^^^^^^^^^^^^^^^^
+
+* **optimize_mode** (*'maximize' or 'minimize', optional, default = 'maximize'*) - If 'maximize', the tuner will try to maximize metrics. If 'minimize', the tuner will try to minimize metrics.
+* **utility** (*'ei', 'ucb' or 'poi', optional, default = 'ei'*) - The utility function (acquisition function). 'ei', 'ucb', and 'poi' correspond to 'Expected Improvement', 'Upper Confidence Bound', and 'Probability of Improvement', respectively.
+* **kappa** (*float, optional, default = 5*) - Used by the 'ucb' utility function. The bigger ``kappa`` is, the more exploratory the tuner will be.
+* **xi** (*float, optional, default = 0*) - Used by the 'ei' and 'poi' utility functions. The bigger ``xi`` is, the more exploratory the tuner will be.
+* **nu** (*float, optional, default = 2.5*) - Used to specify the Matern kernel. The smaller nu, the less smooth the approximated function is.
+* **alpha** (*float, optional, default = 1e-6*) - Used to specify the Gaussian Process Regressor. Larger values correspond to an increased noise level in the observations.
+* **cold_start_num** (*int, optional, default = 10*) - Number of random explorations to perform before the Gaussian Process. Random exploration can help by diversifying the exploration space.
+* **selection_num_warm_up** (*int, optional, default = 1e5*) - Number of random points to evaluate when getting the point which maximizes the acquisition function.
+* **selection_num_starting_points** (*int, optional, default = 250*) - Number of times to run L-BFGS-B from a random starting point after the warmup.
+
+Example Configuration
+^^^^^^^^^^^^^^^^^^^^^
+
+.. code-block:: yaml
+
+   # config.yml
+   tuner:
+     name: GPTuner
+     classArgs:
+       optimize_mode: maximize
+       utility: 'ei'
+       kappa: 5.0
+       xi: 0.0
+       nu: 2.5
+       alpha: 1e-6
+       cold_start_num: 10
+       selection_num_warm_up: 100000
+       selection_num_starting_points: 250

docs/en_US/Tuner/GridsearchTuner.rst

+Grid Search Tuner
+=================
+
+Grid Search performs an exhaustive search through a search space.
+
+For uniform and normal distributed parameters, grid search tuner samples them at progressively decreased intervals.
+
+Usage
+-----
+
+Grid search tuner has no argument.
+
+Example Configuration
+^^^^^^^^^^^^^^^^^^^^^
+
+.. code-block:: yaml
+
+   tuner:
+     name: GridSearch

docs/en_US/Tuner/HyperbandAdvisor.rst

+Hyperband Advisor
+=================
+
+`Hyperband <https://arxiv.org/pdf/1603.06560.pdf>`__ is a popular autoML algorithm. The basic idea of Hyperband is to create several buckets, each having ``n`` randomly generated hyperparameter configurations, each configuration using ``r`` resources (e.g., epoch number, batch number). After the ``n`` configurations are finished, it chooses the top ``n/eta`` configurations and runs them using increased ``r*eta`` resources. At last, it chooses the best configuration it has found so far.
+
+Implementation with full parallelism
+------------------------------------
+
+First, this is an example of how to write an autoML algorithm based on MsgDispatcherBase, rather than Tuner and Assessor. Hyperband is implemented in this way because it integrates the functions of both Tuner and Assessor, thus, we call it Advisor.
+
+Second, this implementation fully leverages Hyperband's internal parallelism. Specifically, the next bucket is not started strictly after the current bucket. Instead, it starts when there are available resources. If you want to use full parallelism mode, set ``exec_mode`` with ``parallelism``. 
+
+Or if you want to set ``exec_mode`` with ``serial`` according to the original algorithm. In this mode, the next bucket will start strictly after the current bucket.
+
+``parallelism`` mode may lead to multiple unfinished buckets, and there is at most one unfinished bucket under ``serial`` mode. The advantage of ``parallelism`` mode is to make full use of resources, which may reduce the experiment duration multiple times. The following two pictures are the results of quick verification using `nas-bench-201 <../NAS/Benchmarks.rst>`__, picture above is in ``parallelism`` mode, picture below is in ``serial`` mode.
+
+
+.. image:: ../../img/hyperband_parallelism.png
+   :target: ../../img/hyperband_parallelism.png
+   :alt: parallelism mode
+
+
+
+.. image:: ../../img/hyperband_serial.png
+   :target: ../../img/hyperband_serial.png
+   :alt: serial mode
+
+
+If you want to reproduce these results, refer to the example under ``examples/trials/benchmarking/`` for details.
+
+Usage
+-----
+
+Config file
+^^^^^^^^^^^
+
+To use Hyperband, you should add the following spec in your experiment's YAML config file:
+
+.. code-block:: bash
+
+   advisor:
+     #choice: Hyperband
+     builtinAdvisorName: Hyperband
+     classArgs:
+       #R: the maximum trial budget
+       R: 100
+       #eta: proportion of discarded trials
+       eta: 3
+       #choice: maximize, minimize
+       optimize_mode: maximize
+       #choice: serial, parallelism
+       exec_mode: parallelism
+
+Note that once you use Advisor, you are not allowed to add a Tuner and Assessor spec in the config file. If you use Hyperband, among the hyperparameters (i.e., key-value pairs) received by a trial, there will be one more key called ``TRIAL_BUDGET`` defined by user. **By using this ``TRIAL_BUDGET``, the trial can control how long it runs**.
+
+For ``report_intermediate_result(metric)`` and ``report_final_result(metric)`` in your trial code, **``metric`` should be either a number or a dict which has a key ``default`` with a number as its value**. This number is the one you want to maximize or minimize, for example, accuracy or loss.
+
+``R`` and ``eta`` are the parameters of Hyperband that you can change. ``R`` means the maximum trial budget that can be allocated to a configuration. Here, trial budget could mean the number of epochs or mini-batches. This ``TRIAL_BUDGET`` should be used by the trial to control how long it runs. Refer to the example under ``examples/trials/mnist-advisor/`` for details.
+
+``eta`` means ``n/eta`` configurations from ``n`` configurations will survive and rerun using more budgets.
+
+Here is a concrete example of ``R=81`` and ``eta=3``:
+
+.. list-table::
+   :header-rows: 1
+   :widths: auto
+
+   * -
+     - s=4
+     - s=3
+     - s=2
+     - s=1
+     - s=0
+   * - i
+     - n r
+     - n r
+     - n r
+     - n r
+     - n r
+   * - 0
+     - 81 1
+     - 27 3
+     - 9 9
+     - 6 27
+     - 5 81
+   * - 1
+     - 27 3
+     - 9 9
+     - 3 27
+     - 2 81
+     -
+   * - 2
+     - 9 9
+     - 3 27
+     - 1 81
+     -
+     -
+   * - 3
+     - 3 27
+     - 1 81
+     -
+     -
+     -
+   * - 4
+     - 1 81
+     -
+     -
+     -
+     -
+
+
+``s`` means bucket, ``n`` means the number of configurations that are generated, the corresponding ``r`` means how many budgets these configurations run. ``i`` means round, for example, bucket 4 has 5 rounds, bucket 3 has 4 rounds.
+
+For information about writing trial code, please refer to the instructions under ``examples/trials/mnist-hyperband/``.
+
+classArgs requirements
+^^^^^^^^^^^^^^^^^^^^^^
+
+
+* **optimize_mode** (*maximize or minimize, optional, default = maximize*) - If 'maximize', the tuner will try to maximize metrics. If 'minimize', the tuner will try to minimize metrics.
+* **R** (*int, optional, default = 60*) - the maximum budget given to a trial (could be the number of mini-batches or epochs). Each trial should use TRIAL_BUDGET to control how long they run.
+* **eta** (*int, optional, default = 3*) - ``(eta-1)/eta`` is the proportion of discarded trials.
+* **exec_mode** (*serial or parallelism, optional, default = parallelism*) - If 'parallelism', the tuner will try to use available resources to start new bucket immediately. If 'serial', the tuner will only start new bucket after the current bucket is done.
+
+Example Configuration
+^^^^^^^^^^^^^^^^^^^^^
+
+.. code-block:: yaml
+
+   # config.yml
+   advisor:
+     builtinAdvisorName: Hyperband
+     classArgs:
+       optimize_mode: maximize
+       R: 60
+       eta: 3
+
+Future improvements
+-------------------
+
+The current implementation of Hyperband can be further improved by supporting a simple early stop algorithm since it's possible that not all the configurations in the top ``n/eta`` perform well. Any unpromising configurations should be stopped early.
+
+In the current implementation, configurations are generated randomly which follows the design in the `paper <https://arxiv.org/pdf/1603.06560.pdf>`__. As an improvement, configurations could be generated more wisely by leveraging advanced algorithms.

docs/en_US/Tuner/MetisTuner.rst

+Metis Tuner
+===========
+
+`Metis <https://www.microsoft.com/en-us/research/publication/metis-robustly-tuning-tail-latencies-cloud-systems/>`__ offers several benefits over other tuning algorithms. While most tools only predict the optimal configuration, Metis gives you two outputs, a prediction for the optimal configuration and a suggestion for the next trial. No more guess work!
+
+While most tools assume training datasets do not have noisy data, Metis actually tells you if you need to resample a particular hyper-parameter.
+
+While most tools have problems of being exploitation-heavy, Metis' search strategy balances exploration, exploitation, and (optional) resampling.
+
+Metis belongs to the class of sequential model-based optimization (SMBO) algorithms and it is based on the Bayesian Optimization framework. To model the parameter-vs-performance space, Metis uses both a Gaussian Process and GMM. Since each trial can impose a high time cost, Metis heavily trades inference computations with naive trials. At each iteration, Metis does two tasks:
+
+
+* 
+  It finds the global optimal point in the Gaussian Process space. This point represents the optimal configuration.
+
+* 
+  It identifies the next hyper-parameter candidate. This is achieved by inferring the potential information gain of exploration, exploitation, and resampling.
+
+Note that the only acceptable types within the search space are ``quniform``, ``uniform``, ``randint``, and numerical ``choice``.
+
+More details can be found in our `paper <https://www.microsoft.com/en-us/research/publication/metis-robustly-tuning-tail-latencies-cloud-systems/>`__.
+
+Usage
+-----
+
+classArgs requirements
+^^^^^^^^^^^^^^^^^^^^^^
+
+* **optimize_mode** (*'maximize' or 'minimize', optional, default = 'maximize'*) - If 'maximize', the tuner will try to maximize metrics. If 'minimize', the tuner will try to minimize metrics.
+
+Example Configuration
+^^^^^^^^^^^^^^^^^^^^^
+
+.. code-block:: yaml
+
+   # config.yml
+   tuner:
+     name: MetisTuner
+     classArgs:
+       optimize_mode: maximize

docs/en_US/Tuner/NetworkmorphismTuner.rst

+Network Morphism Tuner
+======================
+
+`Autokeras <https://arxiv.org/abs/1806.10282>`__ is a popular autoML tool using Network Morphism. The basic idea of Autokeras is to use Bayesian Regression to estimate the metric of the Neural Network Architecture. Each time, it generates several child networks from father networks. Then it uses a naïve Bayesian regression to estimate its metric value from the history of trained results of network and metric value pairs. Next, it chooses the child which has the best, estimated performance and adds it to the training queue. Inspired by the work of Autokeras and referring to its `code <https://github.com/jhfjhfj1/autokeras>`__, we implemented our Network Morphism method on the NNI platform.
+
+If you want to know more about network morphism trial usage, please see the :githublink:`Readme.md <examples/trials/network_morphism/README.rst>`.
+
+Usage
+-----
+
+Installation
+^^^^^^^^^^^^
+
+NetworkMorphism requires :githublink:`PyTorch <examples/trials/network_morphism/requirements.txt>`.
+
+classArgs Requirements
+^^^^^^^^^^^^^^^^^^^^^^
+
+* **optimize_mode** (*maximize or minimize, optional, default = maximize*) - If 'maximize', the tuner will try to maximize metrics. If 'minimize', the tuner will try to minimize metrics.
+* **task** (*('cv'), optional, default = 'cv'*) - The domain of the experiment. For now, this tuner only supports the computer vision (CV) domain.
+* **input_width** (*int, optional, default = 32*) - input image width
+* **input_channel** (*int, optional, default = 3*) - input image channel
+* **n_output_node** (*int, optional, default = 10*) - number of classes
+
+
+
+Config File
+^^^^^^^^^^^
+
+To use Network Morphism, you should modify the following spec in your ``config.yml`` file:
+
+.. code-block:: yaml
+
+   tuner:
+     #choice: NetworkMorphism
+     name: NetworkMorphism
+     classArgs:
+       #choice: maximize, minimize
+       optimize_mode: maximize
+       #for now, this tuner only supports cv domain
+       task: cv
+       #modify to fit your input image width
+       input_width: 32
+       #modify to fit your input image channel
+       input_channel: 3
+       #modify to fit your number of classes
+       n_output_node: 10
+
+Example Configuration
+^^^^^^^^^^^^^^^^^^^^^
+
+.. code-block:: yaml
+
+   # config.yml
+   tuner:
+     name: NetworkMorphism
+       classArgs:
+         optimize_mode: maximize
+         task: cv
+         input_width: 32
+         input_channel: 3
+         n_output_node: 10
+
+In the training procedure, it generates a JSON file which represents a Network Graph. Users can call the "json_to_graph()" function to build a PyTorch or Keras model from this JSON file.
+
+.. code-block:: python
+
+   import nni
+   from nni.networkmorphism_tuner.graph import json_to_graph
+
+   def build_graph_from_json(ir_model_json):
+       """build a pytorch model from json representation
+       """
+       graph = json_to_graph(ir_model_json)
+       model = graph.produce_torch_model()
+       return model
+
+   # trial get next parameter from network morphism tuner
+   RCV_CONFIG = nni.get_next_parameter()
+   # call the function to build pytorch model or keras model
+   net = build_graph_from_json(RCV_CONFIG)
+
+   # training procedure
+   # ....
+
+   # report the final accuracy to NNI
+   nni.report_final_result(best_acc)
+
+If you want to save and load the **best model**, the following methods are recommended.
+
+.. code-block:: python
+
+   # 1. Use NNI API
+   ## You can get the best model ID from WebUI
+   ## or `nni-experiments/experiment_id/log/model_path/best_model.txt'
+
+   ## read the json string from model file and load it with NNI API
+   with open("best-model.json") as json_file:
+       json_of_model = json_file.read()
+   model = build_graph_from_json(json_of_model)
+
+   # 2. Use Framework API (Related to Framework)
+   ## 2.1 Keras API
+
+   ## Save the model with Keras API in the trial code
+   ## it's better to save model with id in nni local mode
+   model_id = nni.get_sequence_id()
+   ## serialize model to JSON
+   model_json = model.to_json()
+   with open("model-{}.json".format(model_id), "w") as json_file:
+       json_file.write(model_json)
+   ## serialize weights to HDF5
+   model.save_weights("model-{}.h5".format(model_id))
+
+   ## Load the model with Keras API if you want to reuse the model
+   ## load json and create model
+   model_id = "" # id of the model you want to reuse
+   with open('model-{}.json'.format(model_id), 'r') as json_file:
+       loaded_model_json = json_file.read()
+   loaded_model = model_from_json(loaded_model_json)
+   ## load weights into new model
+   loaded_model.load_weights("model-{}.h5".format(model_id))
+
+   ## 2.2 PyTorch API
+
+   ## Save the model with PyTorch API in the trial code
+   model_id = nni.get_sequence_id()
+   torch.save(model, "model-{}.pt".format(model_id))
+
+   ## Load the model with PyTorch API if you want to reuse the model
+   model_id = "" # id of the model you want to reuse
+   loaded_model = torch.load("model-{}.pt".format(model_id))
+
+File Structure
+--------------
+
+The tuner has a lot of different files, functions, and classes. Here, we will give most of those files only a brief introduction:
+
+
+* 
+  ``networkmorphism_tuner.py`` is a tuner which uses network morphism techniques.
+
+* 
+  ``bayesian.py`` is a Bayesian method to estimate the metric of unseen model based on the models we have already searched.
+
+* ``graph.py``  is the meta graph data structure. The class Graph represents the neural architecture graph of a model.
+
+  * Graph extracts the neural architecture graph from a model.
+  * Each node in the graph is an intermediate tensor between layers.
+  * Each layer is an edge in the graph.
+  * Notably, multiple edges may refer to the same layer.
+
+* 
+  ``graph_transformer.py`` includes some graph transformers which widen, deepen, or add skip-connections to the graph.
+
+* 
+  ``layers.py``  includes all the layers we use in our model.
+
+* ``layer_transformer.py`` includes some layer transformers which widen, deepen, or add skip-connections to the layer.
+* ``nn.py`` includes the class which generates the initial network.
+* ``metric.py`` some metric classes including Accuracy and MSE.
+* ``utils.py`` is the example search network architectures for the ``cifar10`` dataset, using Keras.
+
+The Network Representation Json Example
+---------------------------------------
+
+Here is an example of the intermediate representation JSON file we defined, which is passed from the tuner to the trial in the architecture search procedure. Users can call the "json_to_graph()" function in the trial code to build a PyTorch or Keras model from this JSON file.
+
+.. code-block:: json
+
+   {
+        "input_shape": [32, 32, 3],
+        "weighted": false,
+        "operation_history": [],
+        "layer_id_to_input_node_ids": {"0": [0],"1": [1],"2": [2],"3": [3],"4": [4],"5": [5],"6": [6],"7": [7],"8": [8],"9": [9],"10": [10],"11": [11],"12": [12],"13": [13],"14": [14],"15": [15],"16": [16]
+        },
+        "layer_id_to_output_node_ids": {"0": [1],"1": [2],"2": [3],"3": [4],"4": [5],"5": [6],"6": [7],"7": [8],"8": [9],"9": [10],"10": [11],"11": [12],"12": [13],"13": [14],"14": [15],"15": [16],"16": [17]
+        },
+        "adj_list": {
+            "0": [[1, 0]],
+            "1": [[2, 1]],
+            "2": [[3, 2]],
+            "3": [[4, 3]],
+            "4": [[5, 4]],
+            "5": [[6, 5]],
+            "6": [[7, 6]],
+            "7": [[8, 7]],
+            "8": [[9, 8]],
+            "9": [[10, 9]],
+            "10": [[11, 10]],
+            "11": [[12, 11]],
+            "12": [[13, 12]],
+            "13": [[14, 13]],
+            "14": [[15, 14]],
+            "15": [[16, 15]],
+            "16": [[17, 16]],
+            "17": []
+        },
+        "reverse_adj_list": {
+            "0": [],
+            "1": [[0, 0]],
+            "2": [[1, 1]],
+            "3": [[2, 2]],
+            "4": [[3, 3]],
+            "5": [[4, 4]],
+            "6": [[5, 5]],
+            "7": [[6, 6]],
+            "8": [[7, 7]],
+            "9": [[8, 8]],
+            "10": [[9, 9]],
+            "11": [[10, 10]],
+            "12": [[11, 11]],
+            "13": [[12, 12]],
+            "14": [[13, 13]],
+            "15": [[14, 14]],
+            "16": [[15, 15]],
+            "17": [[16, 16]]
+        },
+        "node_list": [
+            [0, [32, 32, 3]],
+            [1, [32, 32, 3]],
+            [2, [32, 32, 64]],
+            [3, [32, 32, 64]],
+            [4, [16, 16, 64]],
+            [5, [16, 16, 64]],
+            [6, [16, 16, 64]],
+            [7, [16, 16, 64]],
+            [8, [8, 8, 64]],
+            [9, [8, 8, 64]],
+            [10, [8, 8, 64]],
+            [11, [8, 8, 64]],
+            [12, [4, 4, 64]],
+            [13, [64]],
+            [14, [64]],
+            [15, [64]],
+            [16, [64]],
+            [17, [10]]
+        ],
+        "layer_list": [
+            [0, ["StubReLU", 0, 1]],
+            [1, ["StubConv2d", 1, 2, 3, 64, 3]],
+            [2, ["StubBatchNormalization2d", 2, 3, 64]],
+            [3, ["StubPooling2d", 3, 4, 2, 2, 0]],
+            [4, ["StubReLU", 4, 5]],
+            [5, ["StubConv2d", 5, 6, 64, 64, 3]],
+            [6, ["StubBatchNormalization2d", 6, 7, 64]],
+            [7, ["StubPooling2d", 7, 8, 2, 2, 0]],
+            [8, ["StubReLU", 8, 9]],
+            [9, ["StubConv2d", 9, 10, 64, 64, 3]],
+            [10, ["StubBatchNormalization2d", 10, 11, 64]],
+            [11, ["StubPooling2d", 11, 12, 2, 2, 0]],
+            [12, ["StubGlobalPooling2d", 12, 13]],
+            [13, ["StubDropout2d", 13, 14, 0.25]],
+            [14, ["StubDense", 14, 15, 64, 64]],
+            [15, ["StubReLU", 15, 16]],
+            [16, ["StubDense", 16, 17, 64, 10]]
+        ]
+    }
+
+You can consider the model to be a `directed acyclic graph <https://en.wikipedia.org/wiki/Directed_acyclic_graph>`__. The definition of each model is a JSON object where:
+
+
+* ``input_shape`` is a list of integers which do not include the batch axis.
+* ``weighted`` means whether the weights and biases in the neural network should be included in the graph.
+* ``operation_history`` is a list saving all the network morphism operations.
+* ``layer_id_to_input_node_ids`` is a dictionary mapping from layer identifiers to their input nodes identifiers.
+* ``layer_id_to_output_node_ids`` is a dictionary mapping from layer identifiers to their output nodes identifiers
+* ``adj_list`` is a two-dimensional list; the adjacency list of the graph. The first dimension is identified by tensor identifiers. In each edge list, the elements are two-element tuples of (tensor identifier, layer identifier).
+* ``reverse_adj_list`` is a reverse adjacent list in the same format as adj_list.
+* ``node_list`` is a list of integers. The indices of the list are the identifiers.
+* 
+  ``layer_list`` is a list of stub layers. The indices of the list are the identifiers.
+
+
+  * 
+    For ``StubConv (StubConv1d, StubConv2d, StubConv3d)``, the numbering follows the format: its node input id (or id list), node output id, input_channel, filters, kernel_size, stride, and padding.
+
+  * 
+    For ``StubDense``, the numbering follows the format: its node input id (or id list), node output id, input_units, and units.
+
+  * 
+    For ``StubBatchNormalization (StubBatchNormalization1d, StubBatchNormalization2d, StubBatchNormalization3d)``, the numbering follows the format: its node input id (or id list), node output id, and features numbers.
+
+  * 
+    For ``StubDropout(StubDropout1d, StubDropout2d, StubDropout3d)``, the numbering follows the format: its node input id (or id list), node output id, and dropout rate.
+
+  * 
+    For ``StubPooling (StubPooling1d, StubPooling2d, StubPooling3d)``, the numbering follows the format: its node input id (or id list), node output id, kernel_size, stride, and padding.
+
+  * 
+    For else layers, the numbering follows the format: its node input id (or id list) and node output id.
+
+TODO
+----
+
+Next step, we will change the API from s fixed network generator to a network generator with more available operators. We will use ONNX instead of JSON later as the intermediate representation spec in the future.

docs/en_US/Tuner/PBTTuner.rst

+PBT Tuner
+=========
+
+Population Based Training (PBT) comes from `Population Based Training of Neural Networks <https://arxiv.org/abs/1711.09846v1>`__. It's a simple asynchronous optimization algorithm which effectively utilizes a fixed computational budget to jointly optimize a population of models and their hyperparameters to maximize performance. Importantly, PBT discovers a schedule of hyperparameter settings rather than following the generally sub-optimal strategy of trying to find a single fixed set to use for the whole course of training. 
+
+
+.. image:: ../../img/pbt.jpg
+   :target: ../../img/pbt.jpg
+   :alt: 
+
+
+PBTTuner initializes a population with several trials (i.e., ``population_size``). There are four steps in the above figure, each trial only runs by one step. How long is one step is controlled by trial code, e.g., one epoch. When a trial starts, it loads a checkpoint specified by PBTTuner and continues to run one step, then saves checkpoint to a directory specified by PBTTuner and exits. The trials in a population run steps synchronously, that is, after all the trials finish the ``i``-th step, the ``(i+1)``-th step can be started. Exploitation and exploration of PBT are executed between two consecutive steps.
+
+Usage
+-----
+
+Provide checkpoint directory
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Since some trials need to load other trial's checkpoint, users should provide a directory (i.e., ``all_checkpoint_dir``) which is accessible by every trial. It is easy for local mode, users could directly use the default directory or specify any directory on the local machine. For other training services, users should follow `the document of those training services <../TrainingService/Overview.rst>`__ to provide a directory in a shared storage, such as NFS, Azure storage.
+
+Modify your trial code
+^^^^^^^^^^^^^^^^^^^^^^
+
+Before running a step, a trial needs to load a checkpoint, the checkpoint directory is specified in hyper-parameter configuration generated by PBTTuner, i.e., ``params['load_checkpoint_dir']``. Similarly, the directory for saving checkpoint is also included in the configuration, i.e., ``params['save_checkpoint_dir']``. Here, ``all_checkpoint_dir`` is base folder of ``load_checkpoint_dir`` and ``save_checkpoint_dir`` whose format is ``all_checkpoint_dir/<population-id>/<step>``.
+
+.. code-block:: python
+
+   params = nni.get_next_parameter()
+   # the path of the checkpoint to load
+   load_path = os.path.join(params['load_checkpoint_dir'], 'model.pth')
+   # load checkpoint from `load_path`
+   ...
+   # run one step
+   ...
+   # the path for saving a checkpoint
+   save_path = os.path.join(params['save_checkpoint_dir'], 'model.pth')
+   # save checkpoint to `save_path`
+   ...
+
+The complete example code can be found :githublink:`here <examples/trials/mnist-pbt-tuner-pytorch>`.
+
+classArgs requirements
+^^^^^^^^^^^^^^^^^^^^^^
+
+* **optimize_mode** (*'maximize' or 'minimize'*) - If 'maximize', the tuner will target to maximize metrics. If 'minimize', the tuner will target to minimize metrics.
+* **all_checkpoint_dir** (*str, optional, default = None*) - Directory for trials to load and save checkpoint, if not specified, the directory would be "~/nni/checkpoint/\ :raw-html:`<exp-id>`\ ". Note that if the experiment is not local mode, users should provide a path in a shared storage which can be accessed by all the trials.
+* **population_size** (*int, optional, default = 10*) - Number of trials in a population. Each step has this number of trials. In our implementation, one step is running each trial by specific training epochs set by users.
+* **factors** (*tuple, optional, default = (1.2, 0.8)*) - Factors for perturbation of hyperparameters.
+* **fraction** (*float, optional, default = 0.2*) - Fraction for selecting bottom and top trials.
+
+Experiment config
+^^^^^^^^^^^^^^^^^
+
+Below is an exmaple of PBTTuner configuration in experiment config file. **Note that Assessor is not allowed if PBTTuner is used.**
+
+.. code-block:: yaml
+
+   # config.yml
+   tuner:
+     name: PBTTuner
+     classArgs:
+       optimize_mode: maximize
+       all_checkpoint_dir: /the/path/to/store/checkpoints
+       population_size: 10
+
+Example Configuration
+^^^^^^^^^^^^^^^^^^^^^
+
+.. code-block:: yaml
+
+   # config.yml
+   tuner:
+     name: PBTTuner
+     classArgs:
+       optimize_mode: maximize

docs/en_US/Tuner/RandomTuner.rst

+Random Tuner
+============
+
+In `Random Search for Hyper-Parameter Optimization <http://www.jmlr.org/papers/volume13/bergstra12a/bergstra12a.pdf>`__ we show that Random Search might be surprisingly effective despite its simplicity.
+We suggest using Random Search as a baseline when no knowledge about the prior distribution of hyper-parameters is available.
+
+Usage
+-----
+
+Example Configuration
+
+.. code-block:: yaml
+
+   tuner:
+     name: Random
+     classArgs:
+       seed: 100  # optional

docs/en_US/Tuner/SmacTuner.rst

+SMAC Tuner
+==========
+
+`SMAC <https://www.cs.ubc.ca/~hutter/papers/10-TR-SMAC.pdf>`__ is based on Sequential Model-Based Optimization (SMBO). It adapts the most prominent previously used model class (Gaussian stochastic process models) and introduces the model class of random forests to SMBO in order to handle categorical parameters. The SMAC supported by nni is a wrapper on `the SMAC3 github repo <https://github.com/automl/SMAC3>`__.
+
+Note that SMAC on nni only supports a subset of the types in the `search space spec <../Tutorial/SearchSpaceSpec.rst>`__: ``choice``, ``randint``, ``uniform``, ``loguniform``, and ``quniform``.
+
+Usage
+-----
+
+Installation
+^^^^^^^^^^^^
+
+SMAC has dependencies that need to be installed by following command before the first usage. As a reminder, ``swig`` is required for SMAC: for Ubuntu ``swig`` can be installed with ``apt``.
+
+.. code-block:: bash
+
+   pip install nni[SMAC]
+
+classArgs requirements
+^^^^^^^^^^^^^^^^^^^^^^
+
+* **optimize_mode** (*maximize or minimize, optional, default = maximize*) - If 'maximize', the tuner will try to maximize metrics. If 'minimize', the tuner will try to minimize metrics.
+* **config_dedup** (*True or False, optional, default = False*) - If True, the tuner will not generate a configuration that has been already generated. If False, a configuration may be generated twice, but it is rare for a relatively large search space.
+
+Example Configuration
+^^^^^^^^^^^^^^^^^^^^^
+
+.. code-block:: yaml
+
+   # config.yml
+   tuner:
+     name: SMAC
+     classArgs:
+       optimize_mode: maximize

docs/en_US/Tuner/TpeTuner.rst

+TPE Tuner
+=========
+
+The Tree-structured Parzen Estimator (TPE) is a sequential model-based optimization (SMBO) approach.
+SMBO methods sequentially construct models to approximate the performance of hyperparameters based on historical measurements,
+and then subsequently choose new hyperparameters to test based on this model.
+
+The TPE approach models P(x|y) and P(y) where x represents hyperparameters and y the associated evaluation matric.
+P(x|y) is modeled by transforming the generative process of hyperparameters,
+replacing the distributions of the configuration prior with non-parametric densities.
+
+This optimization approach is described in detail in `Algorithms for Hyper-Parameter Optimization <https://papers.nips.cc/paper/4443-algorithms-for-hyper-parameter-optimization.pdf>`__.
+
+Parallel TPE optimization
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+TPE approaches were actually run asynchronously in order to make use of multiple compute nodes and to avoid wasting time waiting for trial evaluations to complete.
+The original algorithm design was optimized for sequential computation.
+If we were to use TPE with much concurrency, its performance will be bad.
+We have optimized this case using the Constant Liar algorithm.
+For these principles of optimization, please refer to our `research blog <../CommunitySharings/ParallelizingTpeSearch.rst>`__.
+
+Usage
+-----
+
+ To use TPE, you should add the following spec in your experiment's YAML config file:
+
+.. code-block:: yaml
+
+    ## minimal config ##
+    tuner:
+      name: TPE
+      classArgs:
+        optimize_mode: minimize
+
+.. code-block:: yaml
+
+    ## advanced config ##
+    tuner:
+      name: TPE
+      classArgs:
+        optimize_mode: maximize
+        seed: 12345
+        tpe_args:
+          constant_liar_type: 'mean'
+          n_startup_jobs: 10
+          n_ei_candidates: 20
+          linear_forgetting: 100
+          prior_weight: 0
+          gamma: 0.5
+        
+classArgs
+^^^^^^^^^
+
+.. list-table::
+    :widths: 10 20 10 60
+    :header-rows: 1
+
+    * - Field
+      - Type
+      - Default
+      - Description
+    
+    * - ``optimize_mode``
+      - ``'minimize' | 'maximize'``
+      - ``'minimize'``
+      - Whether to minimize or maximize trial metrics.
+
+    * - ``seed``
+      - ``int | null``
+      - ``null``
+      - The random seed.
+
+    * - ``tpe_args.constant_liar_type``
+      - ``'best' | 'worst' | 'mean' | null``
+      - ``'best'``
+      - TPE algorithm itself does not support parallel tuning. This parameter specifies how to optimize for trial_concurrency > 1. How each liar works is explained in paper's section 6.1.
+
+        In general ``best`` suit for small trial number and ``worst`` suit for large trial number.
+
+    * - ``tpe_args.n_startup_jobs``
+      - ``int``
+      - ``20``
+      - The first N hyper-parameters are generated fully randomly for warming up.
+
+        If the search space is large, you can increase this value. Or if max_trial_number is small, you may want to decrease it.
+
+    * - ``tpe_args.n_ei_candidates``
+      - ``int``
+      - ``24``
+      - For each iteration TPE samples EI for N sets of parameters and choose the best one. (loosely speaking)
+
+    * - ``tpe_args.linear_forgetting``
+      - ``int``
+      - ``25``
+      - TPE will lower the weights of old trials. This controls how many iterations it takes for a trial to start decay.
+
+    * - ``tpe_args.prior_weight``
+      - ``float``
+      - ``1.0``
+      - TPE treats user provided search space as prior.
+        When generating new trials, it also incorporates the prior in trial history by transforming the search space to
+        one trial configuration (i.e., each parameter of this configuration chooses the mean of its candidate range).
+        Here, prior_weight determines the weight of this trial configuration in the history trial configurations.
+
+        With prior weight 1.0, the search space is treated as one good trial.
+        For example, "normal(0, 1)" effectly equals to a trial with x = 0 which has yielded good result.
+
+    * - ``tpe_args.gamma``
+      - ``float``
+      - ``0.25``
+      - Controls how many trials are considered "good".
+
+        The number is calculated as "min(gamma * sqrt(N), linear_forgetting)".

docs/en_US/Tutorial/AnnotationSpec.rst

+NNI Annotation
+==============
+
+Overview
+--------
+
+To improve user experience and reduce user effort, we design an annotation grammar. Using NNI annotation, users can adapt their code to NNI just by adding some standalone annotating strings, which does not affect the execution of the original code.
+
+Below is an example:
+
+.. code-block:: python
+
+   '''@nni.variable(nni.choice(0.1, 0.01, 0.001), name=learning_rate)'''
+   learning_rate = 0.1
+
+The meaning of this example is that NNI will choose one of several values (0.1, 0.01, 0.001) to assign to the learning_rate variable. Specifically, this first line is an NNI annotation, which is a single string. Following is an assignment statement. What nni does here is to replace the right value of this assignment statement according to the information provided by the annotation line.
+
+In this way, users could either run the python code directly or launch NNI to tune hyper-parameter in this code, without changing any codes.
+
+Types of Annotation:
+--------------------
+
+In NNI, there are mainly four types of annotation:
+
+1. Annotate variables
+^^^^^^^^^^^^^^^^^^^^^
+
+   ``'''@nni.variable(sampling_algo, name)'''``
+
+``@nni.variable`` is used in NNI to annotate a variable.
+
+**Arguments**
+
+
+* **sampling_algo**\ : Sampling algorithm that specifies a search space. User should replace it with a built-in NNI sampling function whose name consists of an ``nni.`` identification and a search space type specified in `SearchSpaceSpec <SearchSpaceSpec.rst>`__ such as ``choice`` or ``uniform``.
+* **name**\ : The name of the variable that the selected value will be assigned to. Note that this argument should be the same as the left value of the following assignment statement.
+
+There are 10 types to express your search space as follows:
+
+
+* ``@nni.variable(nni.choice(option1,option2,...,optionN),name=variable)``
+  Which means the variable value is one of the options, which should be a list The elements of options can themselves be stochastic expressions
+* ``@nni.variable(nni.randint(lower, upper),name=variable)``
+  Which means the variable value is a value like round(uniform(low, high)). For now, the type of chosen value is float. If you want to use integer value, please convert it explicitly.
+* ``@nni.variable(nni.uniform(low, high),name=variable)``
+  Which means the variable value is a value uniformly between low and high.
+* ``@nni.variable(nni.quniform(low, high, q),name=variable)``
+  Which means the variable value is a value like clip(round(uniform(low, high) / q) * q, low, high), where the clip operation is used to constraint the generated value in the bound.
+* ``@nni.variable(nni.loguniform(low, high),name=variable)``
+  Which means the variable value is a value drawn according to exp(uniform(low, high)) so that the logarithm of the return value is uniformly distributed.
+* ``@nni.variable(nni.qloguniform(low, high, q),name=variable)``
+  Which means the variable value is a value like clip(round(loguniform(low, high) / q) * q, low, high), where the clip operation is used to constraint the generated value in the bound.
+* ``@nni.variable(nni.normal(mu, sigma),name=variable)``
+  Which means the variable value is a real value that's normally-distributed with mean mu and standard deviation sigma.
+* ``@nni.variable(nni.qnormal(mu, sigma, q),name=variable)``
+  Which means the variable value is a value like round(normal(mu, sigma) / q) * q
+* ``@nni.variable(nni.lognormal(mu, sigma),name=variable)``
+  Which means the variable value is a value drawn according to exp(normal(mu, sigma))
+* ``@nni.variable(nni.qlognormal(mu, sigma, q),name=variable)``
+  Which means the variable value is a value like round(exp(normal(mu, sigma)) / q) * q
+
+Below is an example:
+
+.. code-block:: python
+
+   '''@nni.variable(nni.choice(0.1, 0.01, 0.001), name=learning_rate)'''
+   learning_rate = 0.1
+
+2. Annotate functions
+^^^^^^^^^^^^^^^^^^^^^
+
+   ``'''@nni.function_choice(*functions, name)'''``
+
+``@nni.function_choice`` is used to choose one from several functions.
+
+**Arguments**
+
+
+* **functions**\ : Several functions that are waiting to be selected from. Note that it should be a complete function call with arguments. Such as ``max_pool(hidden_layer, pool_size)``.
+* **name**\ : The name of the function that will be replaced in the following assignment statement.
+
+An example here is:
+
+.. code-block:: python
+
+   """@nni.function_choice(max_pool(hidden_layer, pool_size), avg_pool(hidden_layer, pool_size), name=max_pool)"""
+   h_pooling = max_pool(hidden_layer, pool_size)
+
+3. Annotate intermediate result
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+   ``'''@nni.report_intermediate_result(metrics)'''``
+
+``@nni.report_intermediate_result`` is used to report intermediate result, whose usage is the same as ``nni.report_intermediate_result`` in the doc of `Write a trial run on NNI <../TrialExample/Trials.rst>`__
+
+4. Annotate final result
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+   ``'''@nni.report_final_result(metrics)'''``
+
+``@nni.report_final_result`` is used to report the final result of the current trial, whose usage is the same as ``nni.report_final_result`` in the doc of `Write a trial run on NNI <../TrialExample/Trials.rst>`__

docs/en_US/Tutorial/Contributing.rst

+Contributing to Neural Network Intelligence (NNI)
+=================================================
+
+Great!! We are always on the lookout for more contributors to our code base.
+
+Firstly, if you are unsure or afraid of anything, just ask or submit the issue or pull request anyways. You won't be yelled at for giving your best effort. The worst that can happen is that you'll be politely asked to change something. We appreciate any sort of contributions and don't want a wall of rules to get in the way of that.
+
+However, for those individuals who want a bit more guidance on the best way to contribute to the project, read on. This document will cover all the points we're looking for in your contributions, raising your chances of quickly merging or addressing your contributions.
+
+Looking for a quickstart, get acquainted with our `Get Started <QuickStart.rst>`__ guide.
+
+There are a few simple guidelines that you need to follow before providing your hacks.
+
+Raising Issues
+--------------
+
+When raising issues, please specify the following:
+
+
+* Setup details needs to be filled as specified in the issue template clearly for the reviewer to check.
+* A scenario where the issue occurred (with details on how to reproduce it).
+* Errors and log messages that are displayed by the software.
+* Any other details that might be useful.
+
+Submit Proposals for New Features
+---------------------------------
+
+
+* 
+  There is always something more that is required, to make it easier to suit your use-cases. Feel free to join the discussion on new features or raise a PR with your proposed change.
+
+* 
+  Fork the repository under your own github handle. After cloning the repository. Add, commit, push and sqaush (if necessary) the changes with detailed commit messages to your fork. From where you can proceed to making a pull request.
+
+Contributing to Source Code and Bug Fixes
+-----------------------------------------
+
+Provide PRs with appropriate tags for bug fixes or enhancements to the source code. Do follow the correct naming conventions and code styles when you work on and do try to implement all code reviews along the way.
+
+If you are looking for How to develop and debug the NNI source code, you can refer to `How to set up NNI developer environment doc <./SetupNniDeveloperEnvironment.rst>`__ file in the ``docs`` folder.
+
+Similarly for `Quick Start <QuickStart.rst>`__. For everything else, refer to `NNI Home page <http://nni.readthedocs.io>`__.
+
+Solve Existing Issues
+---------------------
+
+Head over to `issues <https://github.com/Microsoft/nni/issues>`__ to find issues where help is needed from contributors. You can find issues tagged with 'good-first-issue' or 'help-wanted' to contribute in.
+
+A person looking to contribute can take up an issue by claiming it as a comment/assign their Github ID to it. In case there is no PR or update in progress for a week on the said issue, then the issue reopens for anyone to take up again. We need to consider high priority issues/regressions where response time must be a day or so.
+
+Code Styles & Naming Conventions
+--------------------------------
+
+* We follow `PEP8 <https://www.python.org/dev/peps/pep-0008/>`__ for Python code and naming conventions, do try to adhere to the same when making a pull request or making a change. One can also take the help of linters such as ``flake8`` or ``pylint``
+* We also follow `NumPy Docstring Style <https://www.sphinx-doc.org/en/master/usage/extensions/example_numpy.html#example-numpy>`__ for Python Docstring Conventions. During the `documentation building <Contributing.rst#documentation>`__\ , we use `sphinx.ext.napoleon <https://www.sphinx-doc.org/en/master/usage/extensions/napoleon.html>`__ to generate Python API documentation from Docstring.
+* For docstrings, please refer to `numpydoc docstring guide <https://numpydoc.readthedocs.io/en/latest/format.html>`__ and `pandas docstring guide <https://python-sprints.github.io/pandas/guide/pandas_docstring.html>`__
+
+  * For function docstring, **description**, **Parameters**, and **Returns** **Yields** are mandatory.
+  * For class docstring, **description**, **Attributes** are mandatory.
+  * For docstring to describe ``dict``, which is commonly used in our hyper-param format description, please refer to `Internal Guideline on Writing Standards <https://ribokit.github.io/docs/text/>`__
+
+Documentation
+-------------
+
+Our documentation is built with :githublink:`sphinx <docs>`.
+
+* Before submitting the documentation change, please **build homepage locally**: ``cd docs/en_US && make html``, then you can see all the built documentation webpage under the folder ``docs/en_US/_build/html``. It's also highly recommended taking care of **every WARNING** during the build, which is very likely the signal of a **deadlink** and other annoying issues.
+
+* 
+  For links, please consider using **relative paths** first. However, if the documentation is written in reStructuredText format, and:
+
+
+  * It's an image link which needs to be formatted with embedded html grammar, please use global URL like ``https://user-images.githubusercontent.com/44491713/51381727-e3d0f780-1b4f-11e9-96ab-d26b9198ba65.png``, which can be automatically generated by dragging picture onto `Github Issue <https://github.com/Microsoft/nni/issues/new>`__ Box.
+  * It cannot be re-formatted by sphinx, such as source code, please use its global URL. For source code that links to our github repo, please use URLs rooted at ``https://github.com/Microsoft/nni/tree/master/`` (:githublink:`mnist.py <examples/trials/mnist-pytorch/mnist.py>` for example).

docs/en_US/Tutorial/ExperimentConfig.rst

+Experiment Config Reference (legacy)
+====================================
+
+This is the previous version (V1) of experiment configuration specification. It is still supported for now, but we recommend users to use `the new version of experiment configuration (V2) <../reference/experiment_config.rst>`_.
+
+A config file is needed when creating an experiment. The path of the config file is provided to ``nnictl``.
+The config file is in YAML format.
+This document describes the rules to write the config file, and provides some examples and templates.
+
+
+* `Experiment Config Reference <#experiment-config-reference>`__
+
+  * `Template <#template>`__
+  * `Configuration Spec <#configuration-spec>`__
+
+    * `authorName <#authorname>`__
+    * `experimentName <#experimentname>`__
+    * `trialConcurrency <#trialconcurrency>`__
+    * `maxExecDuration <#maxexecduration>`__
+    * `versionCheck <#versioncheck>`__
+    * `debug <#debug>`__
+    * `maxTrialNum <#maxtrialnum>`__
+    * `maxTrialDuration <#maxtrialduration>`__
+    * `trainingServicePlatform <#trainingserviceplatform>`__
+    * `searchSpacePath <#searchspacepath>`__
+    * `useAnnotation <#useannotation>`__
+    * `multiThread <#multithread>`__
+    * `nniManagerIp <#nnimanagerip>`__
+    * `logDir <#logdir>`__
+    * `logLevel <#loglevel>`__
+    * `logCollection <#logcollection>`__
+    * `tuner <#tuner>`__
+
+      * `builtinTunerName <#builtintunername>`__
+      * `codeDir <#codedir>`__
+      * `classFileName <#classfilename>`__
+      * `className <#classname>`__
+      * `classArgs <#classargs>`__
+      * `gpuIndices <#gpuindices>`__
+      * `includeIntermediateResults <#includeintermediateresults>`__
+
+    * `assessor <#assessor>`__
+
+      * `builtinAssessorName <#builtinassessorname>`__
+      * `codeDir <#codedir-1>`__
+      * `classFileName <#classfilename-1>`__
+      * `className <#classname-1>`__
+      * `classArgs <#classargs-1>`__
+
+    * `advisor <#advisor>`__
+
+      * `builtinAdvisorName <#builtinadvisorname>`__
+      * `codeDir <#codedir-2>`__
+      * `classFileName <#classfilename-2>`__
+      * `className <#classname-2>`__
+      * `classArgs <#classargs-2>`__
+      * `gpuIndices <#gpuindices-1>`__
+
+    * `trial <#trial>`__
+    * `localConfig <#localconfig>`__
+
+      * `gpuIndices <#gpuindices-2>`__
+      * `maxTrialNumPerGpu <#maxtrialnumpergpu>`__
+      * `useActiveGpu <#useactivegpu>`__
+
+    * `machineList <#machinelist>`__
+
+      * `ip <#ip>`__
+      * `port <#port>`__
+      * `username <#username>`__
+      * `passwd <#passwd>`__
+      * `sshKeyPath <#sshkeypath>`__
+      * `passphrase <#passphrase>`__
+      * `gpuIndices <#gpuindices-3>`__
+      * `maxTrialNumPerGpu <#maxtrialnumpergpu-1>`__
+      * `useActiveGpu <#useactivegpu-1>`__
+      * `pythonPath <#pythonPath>`__
+
+    * `kubeflowConfig <#kubeflowconfig>`__
+
+      * `operator <#operator>`__
+      * `storage <#storage>`__
+      * `nfs <#nfs>`__
+      * `keyVault <#keyvault>`__
+      * `azureStorage <#azurestorage>`__
+      * `uploadRetryCount <#uploadretrycount>`__
+
+    * `paiConfig <#paiconfig>`__
+
+      * `userName <#username>`__
+      * `password <#password>`__
+      * `token <#token>`__
+      * `host <#host>`__
+      * `reuse <#reuse>`__
+
+  * `Examples <#examples>`__
+
+    * `Local mode <#local-mode>`__
+    * `Remote mode <#remote-mode>`__
+    * `PAI mode <#pai-mode>`__
+    * `Kubeflow mode <#kubeflow-mode>`__
+    * `Kubeflow with azure storage <#kubeflow-with-azure-storage>`__
+
+Template
+--------
+
+
+* **Light weight (without Annotation and Assessor)**
+
+.. code-block:: yaml
+
+   authorName:
+   experimentName:
+   trialConcurrency:
+   maxExecDuration:
+   maxTrialNum:
+   #choice: local, remote, pai, kubeflow
+   trainingServicePlatform:
+   searchSpacePath:
+   #choice: true, false, default: false
+   useAnnotation:
+   #choice: true, false, default: false
+   multiThread:
+   tuner:
+     #choice: TPE, Random, Anneal, Evolution
+     builtinTunerName:
+     classArgs:
+       #choice: maximize, minimize
+       optimize_mode:
+     gpuIndices:
+   trial:
+     command:
+     codeDir:
+     gpuNum:
+   #machineList can be empty if the platform is local
+   machineList:
+     - ip:
+       port:
+       username:
+       passwd:
+
+
+* **Use Assessor**
+
+.. code-block:: yaml
+
+   authorName:
+   experimentName:
+   trialConcurrency:
+   maxExecDuration:
+   maxTrialNum:
+   #choice: local, remote, pai, kubeflow
+   trainingServicePlatform:
+   searchSpacePath:
+   #choice: true, false, default: false
+   useAnnotation:
+   #choice: true, false, default: false
+   multiThread:
+   tuner:
+     #choice: TPE, Random, Anneal, Evolution
+     builtinTunerName:
+     classArgs:
+       #choice: maximize, minimize
+       optimize_mode:
+     gpuIndices:
+   assessor:
+     #choice: Medianstop
+     builtinAssessorName:
+     classArgs:
+       #choice: maximize, minimize
+       optimize_mode:
+   trial:
+     command:
+     codeDir:
+     gpuNum:
+   #machineList can be empty if the platform is local
+   machineList:
+     - ip:
+       port:
+       username:
+       passwd:
+
+
+* **Use Annotation**
+
+.. code-block:: yaml
+
+   authorName:
+   experimentName:
+   trialConcurrency:
+   maxExecDuration:
+   maxTrialNum:
+   #choice: local, remote, pai, kubeflow
+   trainingServicePlatform:
+   #choice: true, false, default: false
+   useAnnotation:
+   #choice: true, false, default: false
+   multiThread:
+   tuner:
+     #choice: TPE, Random, Anneal, Evolution
+     builtinTunerName:
+     classArgs:
+       #choice: maximize, minimize
+       optimize_mode:
+     gpuIndices:
+   assessor:
+     #choice: Medianstop
+     builtinAssessorName:
+     classArgs:
+       #choice: maximize, minimize
+       optimize_mode:
+   trial:
+     command:
+     codeDir:
+     gpuNum:
+   #machineList can be empty if the platform is local
+   machineList:
+     - ip:
+       port:
+       username:
+       passwd:
+
+Configuration Spec
+------------------
+
+authorName
+^^^^^^^^^^
+
+Required. String.
+
+The name of the author who create the experiment.
+
+*TBD: add default value.*
+
+experimentName
+^^^^^^^^^^^^^^
+
+Required. String.
+
+The name of the experiment created.
+
+*TBD: add default value.*
+
+trialConcurrency
+^^^^^^^^^^^^^^^^
+
+Required. Integer between 1 and 99999.
+
+Specifies the max num of trial jobs run simultaneously.
+
+If trialGpuNum is bigger than the free gpu numbers, and the trial jobs running simultaneously can not reach **trialConcurrency** number, some trial jobs will be put into a queue to wait for gpu allocation.
+
+maxExecDuration
+^^^^^^^^^^^^^^^
+
+Optional. String. Default: 999d.
+
+**maxExecDuration** specifies the max duration time of an experiment. The unit of the time is {**s**\ , **m**\ , **h**\ , **d**\ }, which means {*seconds*\ , *minutes*\ , *hours*\ , *days*\ }.
+
+Note: The maxExecDuration spec set the time of an experiment, not a trial job. If the experiment reach the max duration time, the experiment will not stop, but could not submit new trial jobs any more.
+
+versionCheck
+^^^^^^^^^^^^
+
+Optional. Bool. Default: true.
+
+NNI will check the version of nniManager process and the version of trialKeeper in remote, pai and kubernetes platform. If you want to disable version check, you could set versionCheck be false.
+
+debug
+^^^^^
+
+Optional. Bool. Default: false.
+
+Debug mode will set versionCheck to false and set logLevel to be 'debug'.
+
+maxTrialNum
+^^^^^^^^^^^
+
+Optional. Integer between 1 and 99999. Default: 99999.
+
+Specifies the max number of trial jobs created by NNI, including succeeded and failed jobs.
+
+maxTrialDuration
+^^^^^^^^^^^^^^^^
+
+Optional. String. Default: 999d.
+
+**maxTrialDuration** specifies the max duration time of each trial job. The unit of the time is {**s**\ , **m**\ , **h**\ , **d**\ }, which means {*seconds*\ , *minutes*\ , *hours*\ , *days*\ }. If current trial job reach the max duration time, this trial job will stop.
+
+trainingServicePlatform
+^^^^^^^^^^^^^^^^^^^^^^^
+
+Required. String.
+
+Specifies the platform to run the experiment, including **local**\ , **remote**\ , **pai**\ , **kubeflow**\ , **frameworkcontroller**.
+
+
+* 
+  **local** run an experiment on local ubuntu machine.
+
+* 
+  **remote** submit trial jobs to remote ubuntu machines, and **machineList** field should be filed in order to set up SSH connection to remote machine.
+
+* 
+  **pai**  submit trial jobs to `OpenPAI <https://github.com/Microsoft/pai>`__ of Microsoft. For more details of pai configuration, please refer to `Guide to PAI Mode <../TrainingService/PaiMode.rst>`__
+
+* 
+  **kubeflow** submit trial jobs to `kubeflow <https://www.kubeflow.org/docs/about/kubeflow/>`__\ , NNI support kubeflow based on normal kubernetes and `azure kubernetes <https://azure.microsoft.com/en-us/services/kubernetes-service/>`__. For detail please refer to `Kubeflow Docs <../TrainingService/KubeflowMode.rst>`__
+
+* 
+  **adl** submit trial jobs to `AdaptDL <https://www.kubeflow.org/docs/about/kubeflow/>`__\ , NNI support AdaptDL on Kubernetes cluster. For detail please refer to `AdaptDL Docs <../TrainingService/AdaptDLMode.rst>`__
+
+* 
+  TODO: explain frameworkcontroller.
+
+searchSpacePath
+^^^^^^^^^^^^^^^
+
+Optional. Path to existing file.
+
+Specifies the path of search space file, which should be a valid path in the local linux machine.
+
+The only exception that **searchSpacePath** can be not fulfilled is when ``useAnnotation=True``.
+
+useAnnotation
+^^^^^^^^^^^^^
+
+Optional. Bool. Default: false.
+
+Use annotation to analysis trial code and generate search space.
+
+Note: if **useAnnotation** is true, the searchSpacePath field should be removed.
+
+multiThread
+^^^^^^^^^^^
+
+Optional. Bool. Default: false.
+
+Enable multi-thread mode for dispatcher. If multiThread is enabled, dispatcher will start a thread to process each command from NNI Manager.
+
+nniManagerIp
+^^^^^^^^^^^^
+
+Optional. String. Default: eth0 device IP.
+
+Set the IP address of the machine on which NNI manager process runs. This field is optional, and if it's not set, eth0 device IP will be used instead.
+
+Note: run ``ifconfig`` on NNI manager's machine to check if eth0 device exists. If not, **nniManagerIp** is recommended to set explicitly.
+
+logDir
+^^^^^^
+
+Optional. Path to a directory. Default: ``<user home directory>/nni-experiments``.
+
+Configures the directory to store logs and data of the experiment.
+
+logLevel
+^^^^^^^^
+
+Optional. String. Default: ``info``.
+
+Sets log level for the experiment. Available log levels are: ``trace``\ , ``debug``\ , ``info``\ , ``warning``\ , ``error``\ , ``fatal``.
+
+logCollection
+^^^^^^^^^^^^^
+
+Optional. ``http`` or ``none``. Default: ``none``.
+
+Set the way to collect log in remote, pai, kubeflow, frameworkcontroller platform. There are two ways to collect log, one way is from ``http``\ , trial keeper will post log content back from http request in this way, but this way may slow down the speed to process logs in trialKeeper. The other way is ``none``\ , trial keeper will not post log content back, and only post job metrics. If your log content is too big, you could consider setting this param be ``none``.
+
+tuner
+^^^^^
+
+Required.
+
+Specifies the tuner algorithm in the experiment, there are two kinds of ways to set tuner. One way is to use tuner provided by NNI sdk (built-in tuners), in which case you need to set **builtinTunerName** and **classArgs**. Another way is to use users' own tuner file, in which case **codeDirectory**\ , **classFileName**\ , **className** and **classArgs** are needed. *Users must choose exactly one way.*
+
+builtinTunerName
+^^^^^^^^^^^^^^^^
+
+Required if using built-in tuners. String.
+
+Specifies the name of system tuner, NNI sdk provides different tuners introduced `here <../Tuner/BuiltinTuner.rst>`__.
+
+codeDir
+^^^^^^^
+
+Required if using customized tuners. Path relative to the location of config file.
+
+Specifies the directory of tuner code.
+
+classFileName
+^^^^^^^^^^^^^
+
+Required if using customized tuners. File path relative to **codeDir**.
+
+Specifies the name of tuner file.
+
+className
+^^^^^^^^^
+
+Required if using customized tuners. String.
+
+Specifies the name of tuner class.
+
+classArgs
+^^^^^^^^^
+
+Optional. Key-value pairs. Default: empty.
+
+Specifies the arguments of tuner algorithm. Please refer to `this file <../Tuner/BuiltinTuner.rst>`__ for the configurable arguments of each built-in tuner.
+
+gpuIndices
+^^^^^^^^^^
+
+Optional. String. Default: empty.
+
+Specifies the GPUs that can be used by the tuner process. Single or multiple GPU indices can be specified. Multiple GPU indices are separated by comma ``,``. For example, ``1``\ , or ``0,1,3``. If the field is not set, no GPU will be visible to tuner (by setting ``HIP_VISIBLE_DEVICES`` to be an empty string).
+
+includeIntermediateResults
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Optional. Bool. Default: false.
+
+If **includeIntermediateResults** is true, the last intermediate result of the trial that is early stopped by assessor is sent to tuner as final result.
+
+assessor
+^^^^^^^^
+
+Specifies the assessor algorithm to run an experiment. Similar to tuners, there are two kinds of ways to set assessor. One way is to use assessor provided by NNI sdk. Users need to set **builtinAssessorName** and **classArgs**. Another way is to use users' own assessor file, and users need to set **codeDirectory**\ , **classFileName**\ , **className** and **classArgs**. *Users must choose exactly one way.*
+
+By default, there is no assessor enabled.
+
+builtinAssessorName
+^^^^^^^^^^^^^^^^^^^
+
+Required if using built-in assessors. String.
+
+Specifies the name of built-in assessor, NNI sdk provides different assessors introduced `here <../Assessor/BuiltinAssessor.rst>`__.
+
+codeDir
+^^^^^^^
+
+Required if using customized assessors. Path relative to the location of config file.
+
+Specifies the directory of assessor code.
+
+classFileName
+^^^^^^^^^^^^^
+
+Required if using customized assessors. File path relative to **codeDir**.
+
+Specifies the name of assessor file.
+
+className
+^^^^^^^^^
+
+Required if using customized assessors. String.
+
+Specifies the name of assessor class.
+
+classArgs
+^^^^^^^^^
+
+Optional. Key-value pairs. Default: empty.
+
+Specifies the arguments of assessor algorithm.
+
+advisor
+^^^^^^^
+
+Optional.
+
+Specifies the advisor algorithm in the experiment. Similar to tuners and assessors, there are two kinds of ways to specify advisor. One way is to use advisor provided by NNI sdk, need to set **builtinAdvisorName** and **classArgs**. Another way is to use users' own advisor file, and need to set **codeDirectory**\ , **classFileName**\ , **className** and **classArgs**.
+
+When advisor is enabled, settings of tuners and advisors will be bypassed.
+
+builtinAdvisorName
+^^^^^^^^^^^^^^^^^^
+
+Specifies the name of a built-in advisor. NNI sdk provides `BOHB <../Tuner/BohbAdvisor.rst>`__ and `Hyperband <../Tuner/HyperbandAdvisor.rst>`__.
+
+codeDir
+^^^^^^^
+
+Required if using customized advisors. Path relative to the location of config file.
+
+Specifies the directory of advisor code.
+
+classFileName
+^^^^^^^^^^^^^
+
+Required if using customized advisors. File path relative to **codeDir**.
+
+Specifies the name of advisor file.
+
+className
+^^^^^^^^^
+
+Required if using customized advisors. String.
+
+Specifies the name of advisor class.
+
+classArgs
+^^^^^^^^^
+
+Optional. Key-value pairs. Default: empty.
+
+Specifies the arguments of advisor.
+
+gpuIndices
+^^^^^^^^^^
+
+Optional. String. Default: empty.
+
+Specifies the GPUs that can be used. Single or multiple GPU indices can be specified. Multiple GPU indices are separated by comma ``,``. For example, ``1``\ , or ``0,1,3``. If the field is not set, no GPU will be visible to tuner (by setting ``HIP_VISIBLE_DEVICES`` to be an empty string).
+
+trial
+^^^^^
+
+Required. Key-value pairs.
+
+In local and remote mode, the following keys are required.
+
+
+* 
+  **command**\ : Required string. Specifies the command to run trial process.
+
+* 
+  **codeDir**\ : Required string. Specifies the directory of your own trial file. This directory will be automatically uploaded in remote mode.
+
+* 
+  **gpuNum**\ : Optional integer. Specifies the num of gpu to run the trial process. Default value is 0.
+
+In PAI mode, the following keys are required.
+
+
+* 
+  **command**\ : Required string. Specifies the command to run trial process.
+
+* 
+  **codeDir**\ : Required string. Specifies the directory of the own trial file. Files in the directory will be uploaded in PAI mode.
+
+* 
+  **gpuNum**\ : Required integer. Specifies the num of gpu to run the trial process. Default value is 0.
+
+* 
+  **cpuNum**\ : Required integer. Specifies the cpu number of cpu to be used in pai container.
+
+* 
+  **memoryMB**\ : Required integer. Set the memory size to be used in pai container, in megabytes.
+
+* 
+  **image**\ : Required string. Set the image to be used in pai.
+
+* 
+  **authFile**\ : Optional string. Used to provide Docker registry which needs authentication for image pull in PAI. `Reference <https://github.com/microsoft/pai/blob/2ea69b45faa018662bc164ed7733f6fdbb4c42b3/docs/faq.rst#q-how-to-use-private-docker-registry-job-image-when-submitting-an-openpai-job>`__.
+
+* 
+  **shmMB**\ : Optional integer. Shared memory size of container.
+
+* 
+  **portList**\ : List of key-values pairs with ``label``\ , ``beginAt``\ , ``portNumber``. See `job tutorial of PAI <https://github.com/microsoft/pai/blob/master/docs/job_tutorial.rst>`__ for details.
+
+.. cannot find `Reference <https://github.com/microsoft/pai/blob/2ea69b45faa018662bc164ed7733f6fdbb4c42b3/docs/faq.rst#q-how-to-use-private-docker-registry-job-image-when-submitting-an-openpai-job>`__  and `job tutorial of PAI <https://github.com/microsoft/pai/blob/master/docs/job_tutorial.rst>`__ 
+
+In Kubeflow mode, the following keys are required.
+
+
+* 
+  **codeDir**\ : The local directory where the code files are in.
+
+* 
+  **ps**\ : An optional configuration for kubeflow's tensorflow-operator, which includes
+
+
+  * 
+    **replicas**\ : The replica number of **ps** role.
+
+  * 
+    **command**\ : The run script in **ps**\ 's container.
+
+  * 
+    **gpuNum**\ : The gpu number to be used in **ps** container.
+
+  * 
+    **cpuNum**\ : The cpu number to be used in **ps** container.
+
+  * 
+    **memoryMB**\ : The memory size of the container.
+
+  * 
+    **image**\ : The image to be used in **ps**.
+
+* 
+  **worker**\ : An optional configuration for kubeflow's tensorflow-operator.
+
+
+  * 
+    **replicas**\ : The replica number of **worker** role.
+
+  * 
+    **command**\ : The run script in **worker**\ 's container.
+
+  * 
+    **gpuNum**\ : The gpu number to be used in **worker** container.
+
+  * 
+    **cpuNum**\ : The cpu number to be used in **worker** container.
+
+  * 
+    **memoryMB**\ : The memory size of the container.
+
+  * 
+    **image**\ : The image to be used in **worker**.
+
+localConfig
+^^^^^^^^^^^
+
+Optional in local mode. Key-value pairs.
+
+Only applicable if **trainingServicePlatform** is set to ``local``\ , otherwise there should not be **localConfig** section in configuration file.
+
+gpuIndices
+^^^^^^^^^^
+
+Optional. String. Default: none.
+
+Used to specify designated GPU devices for NNI, if it is set, only the specified GPU devices are used for NNI trial jobs. Single or multiple GPU indices can be specified. Multiple GPU indices should be separated with comma (\ ``,``\ ), such as ``1`` or  ``0,1,3``. By default, all GPUs available will be used.
+
+maxTrialNumPerGpu
+^^^^^^^^^^^^^^^^^
+
+Optional. Integer. Default: 1.
+
+Used to specify the max concurrency trial number on a GPU device.
+
+useActiveGpu
+^^^^^^^^^^^^
+
+Optional. Bool. Default: false.
+
+Used to specify whether to use a GPU if there is another process. By default, NNI will use the GPU only if there is no other active process in the GPU. If **useActiveGpu** is set to true, NNI will use the GPU regardless of another processes. This field is not applicable for NNI on Windows.
+
+machineList
+^^^^^^^^^^^
+
+Required in remote mode. A list of key-value pairs with the following keys.
+
+ip
+^^
+
+Required. IP address or host name that is accessible from the current machine.
+
+The IP address or host name of remote machine.
+
+port
+^^^^
+
+Optional. Integer. Valid port. Default: 22.
+
+The ssh port to be used to connect machine.
+
+username
+^^^^^^^^
+
+Required if authentication with username/password. String.
+
+The account of remote machine.
+
+passwd
+^^^^^^
+
+Required if authentication with username/password. String.
+
+Specifies the password of the account.
+
+sshKeyPath
+^^^^^^^^^^
+
+Required if authentication with ssh key. Path to private key file.
+
+If users use ssh key to login remote machine, **sshKeyPath** should be a valid path to a ssh key file.
+
+*Note: if users set passwd and sshKeyPath simultaneously, NNI will try passwd first.*
+
+passphrase
+^^^^^^^^^^
+
+Optional. String.
+
+Used to protect ssh key, which could be empty if users don't have passphrase.
+
+gpuIndices
+^^^^^^^^^^
+
+Optional. String. Default: none.
+
+Used to specify designated GPU devices for NNI, if it is set, only the specified GPU devices are used for NNI trial jobs. Single or multiple GPU indices can be specified. Multiple GPU indices should be separated with comma (\ ``,``\ ), such as ``1`` or  ``0,1,3``. By default, all GPUs available will be used.
+
+maxTrialNumPerGpu
+^^^^^^^^^^^^^^^^^
+
+Optional. Integer. Default: 1.
+
+Used to specify the max concurrency trial number on a GPU device.
+
+useActiveGpu
+^^^^^^^^^^^^
+
+Optional. Bool. Default: false.
+
+Used to specify whether to use a GPU if there is another process. By default, NNI will use the GPU only if there is no other active process in the GPU. If **useActiveGpu** is set to true, NNI will use the GPU regardless of another processes. This field is not applicable for NNI on Windows.
+
+pythonPath
+^^^^^^^^^^
+
+Optional. String.
+
+Users can configure the python path environment on remote machine by setting **pythonPath**.
+
+remoteConfig
+^^^^^^^^^^^^
+
+Optional field in remote mode. Users could set per machine information in ``machineList`` field, and set global configuration for remote mode in this field.
+
+reuse
+^^^^^
+
+Optional. Bool. default: ``false``. It's an experimental feature.
+
+If it's true, NNI will reuse remote jobs to run as many as possible trials. It can save time of creating new jobs. User needs to make sure each trial can run independent in same job, for example, avoid loading checkpoint from previous trials. 
+
+kubeflowConfig
+^^^^^^^^^^^^^^
+
+operator
+^^^^^^^^
+
+Required. String. Has to be ``tf-operator`` or ``pytorch-operator``.
+
+Specifies the kubeflow's operator to be used, NNI support ``tf-operator`` in current version.
+
+storage
+^^^^^^^
+
+Optional. String. Default. ``nfs``.
+
+Specifies the storage type of kubeflow, including ``nfs`` and ``azureStorage``.
+
+nfs
+^^^
+
+Required if using nfs. Key-value pairs.
+
+
+* 
+  **server** is the host of nfs server.
+
+* 
+  **path** is the mounted path of nfs.
+
+keyVault
+^^^^^^^^
+
+Required if using azure storage. Key-value pairs.
+
+Set **keyVault** to storage the private key of your azure storage account. Refer to `the doc <https://docs.microsoft.com/en-us/azure/key-vault/key-vault-manage-with-cli2>`__ .
+
+
+* 
+  **vaultName** is the value of ``--vault-name`` used in az command.
+
+* 
+  **name** is the value of ``--name`` used in az command.
+
+azureStorage
+^^^^^^^^^^^^
+
+Required if using azure storage. Key-value pairs.
+
+Set azure storage account to store code files.
+
+
+* 
+  **accountName** is the name of azure storage account.
+
+* 
+  **azureShare** is the share of the azure file storage.
+
+uploadRetryCount
+^^^^^^^^^^^^^^^^
+
+Required if using azure storage. Integer between 1 and 99999.
+
+If upload files to azure storage failed, NNI will retry the process of uploading, this field will specify the number of attempts to re-upload files.
+
+paiConfig
+^^^^^^^^^
+
+userName
+^^^^^^^^
+
+Required. String.
+
+The user name of your pai account.
+
+password
+^^^^^^^^
+
+Required if using password authentication. String.
+
+The password of the pai account.
+
+token
+^^^^^
+
+Required if using token authentication. String.
+
+Personal access token that can be retrieved from PAI portal.
+
+host
+^^^^
+
+Required. String.
+
+The hostname of IP address of PAI.
+
+reuse
+^^^^^
+
+Optional. Bool. default: ``false``. It's an experimental feature.
+
+If it's true, NNI will reuse OpenPAI jobs to run as many as possible trials. It can save time of creating new jobs. User needs to make sure each trial can run independent in same job, for example, avoid loading checkpoint from previous trials.
+
+sharedStorage
+^^^^^^^^^^^^^
+
+storageType
+^^^^^^^^^^^
+
+Required. String.
+
+The type of the storage, support ``NFS`` and ``AzureBlob``.
+
+localMountPoint
+^^^^^^^^^^^^^^^
+
+Required. String.
+
+The absolute or relative path that the storage has been or will be mounted in local. If the path does not exist, it will be created automatically. Recommended to use an absolute path. i.e. ``/tmp/nni-shared-storage``.
+
+remoteMountPoint
+^^^^^^^^^^^^^^^^
+
+Required. String.
+
+The absolute or relative path that the storage will be mounted in remote. If the path does not exist, it will be created automatically. Note that the directory must be empty if using AzureBlob. Recommended to use a relative path. i.e. ``./nni-shared-storage``.
+
+localMounted
+^^^^^^^^^^^^
+
+Required. String.
+
+One of ``usermount``, ``nnimount`` or ``nomount``. ``usermount`` means you have already mount this storage on localMountPoint. ``nnimount`` means nni will try to mount this storage on localMountPoint. ``nomount`` means storage will not mount in local machine, will support partial storages in the future.
+
+nfsServer
+^^^^^^^^^
+
+Optional. String.
+
+Required if using NFS storage. The NFS server host.
+
+exportedDirectory
+^^^^^^^^^^^^^^^^^
+
+Optional. String.
+
+Required if using NFS storage. The exported directory of NFS server.
+
+storageAccountName
+^^^^^^^^^^^^^^^^^^
+
+Optional. String.
+
+Required if using AzureBlob storage. The azure storage account name.
+
+storageAccountKey
+^^^^^^^^^^^^^^^^^
+
+Optional. String.
+
+Required if using AzureBlob storage. The azure storage account key.
+
+containerName
+^^^^^^^^^^^^^
+
+Optional. String.
+
+Required if using AzureBlob storage. The AzureBlob container name.
+
+Examples
+--------
+
+Local mode
+^^^^^^^^^^
+
+If users want to run trial jobs in local machine, and use annotation to generate search space, could use the following config:
+
+.. code-block:: yaml
+
+     authorName: test
+     experimentName: test_experiment
+     trialConcurrency: 3
+     maxExecDuration: 1h
+     maxTrialNum: 10
+     #choice: local, remote, pai, kubeflow
+     trainingServicePlatform: local
+     #choice: true, false
+     useAnnotation: true
+     tuner:
+       #choice: TPE, Random, Anneal, Evolution
+       builtinTunerName: TPE
+       classArgs:
+         #choice: maximize, minimize
+         optimize_mode: maximize
+     trial:
+       command: python3 mnist.py
+       codeDir: /nni/mnist
+       gpuNum: 0
+
+You can add assessor configuration.
+
+.. code-block:: yaml
+
+     authorName: test
+     experimentName: test_experiment
+     trialConcurrency: 3
+     maxExecDuration: 1h
+     maxTrialNum: 10
+     #choice: local, remote, pai, kubeflow
+     trainingServicePlatform: local
+     searchSpacePath: /nni/search_space.json
+     #choice: true, false
+     useAnnotation: false
+     tuner:
+       #choice: TPE, Random, Anneal, Evolution
+       builtinTunerName: TPE
+       classArgs:
+         #choice: maximize, minimize
+         optimize_mode: maximize
+     assessor:
+       #choice: Medianstop
+       builtinAssessorName: Medianstop
+       classArgs:
+         #choice: maximize, minimize
+         optimize_mode: maximize
+     trial:
+       command: python3 mnist.py
+       codeDir: /nni/mnist
+       gpuNum: 0
+
+Or you could specify your own tuner and assessor file as following,
+
+.. code-block:: yaml
+
+     authorName: test
+     experimentName: test_experiment
+     trialConcurrency: 3
+     maxExecDuration: 1h
+     maxTrialNum: 10
+     #choice: local, remote, pai, kubeflow
+     trainingServicePlatform: local
+     searchSpacePath: /nni/search_space.json
+     #choice: true, false
+     useAnnotation: false
+     tuner:
+       codeDir: /nni/tuner
+       classFileName: mytuner.py
+       className: MyTuner
+       classArgs:
+         #choice: maximize, minimize
+         optimize_mode: maximize
+     assessor:
+       codeDir: /nni/assessor
+       classFileName: myassessor.py
+       className: MyAssessor
+       classArgs:
+         #choice: maximize, minimize
+         optimize_mode: maximize
+     trial:
+       command: python3 mnist.py
+       codeDir: /nni/mnist
+       gpuNum: 0
+
+Remote mode
+^^^^^^^^^^^
+
+If run trial jobs in remote machine, users could specify the remote machine information as following format:
+
+.. code-block:: yaml
+
+     authorName: test
+     experimentName: test_experiment
+     trialConcurrency: 3
+     maxExecDuration: 1h
+     maxTrialNum: 10
+     #choice: local, remote, pai, kubeflow
+     trainingServicePlatform: remote
+     searchSpacePath: /nni/search_space.json
+     #choice: true, false
+     useAnnotation: false
+     tuner:
+       #choice: TPE, Random, Anneal, Evolution
+       builtinTunerName: TPE
+       classArgs:
+         #choice: maximize, minimize
+         optimize_mode: maximize
+     trial:
+       command: python3 mnist.py
+       codeDir: /nni/mnist
+       gpuNum: 0
+     #machineList can be empty if the platform is local
+     machineList:
+       - ip: 10.10.10.10
+         port: 22
+         username: test
+         passwd: test
+       - ip: 10.10.10.11
+         port: 22
+         username: test
+         passwd: test
+       - ip: 10.10.10.12
+         port: 22
+         username: test
+         sshKeyPath: /nni/sshkey
+         passphrase: qwert
+         # Below is an example of specifying python environment.
+         pythonPath: ${replace_to_python_environment_path_in_your_remote_machine}
+
+PAI mode
+^^^^^^^^
+
+.. code-block:: yaml
+
+     authorName: test
+     experimentName: nni_test1
+     trialConcurrency: 1
+     maxExecDuration:500h
+     maxTrialNum: 1
+     #choice: local, remote, pai, kubeflow
+     trainingServicePlatform: pai
+     searchSpacePath: search_space.json
+     #choice: true, false
+     useAnnotation: false
+     tuner:
+       #choice: TPE, Random, Anneal, Evolution, BatchTuner
+       #SMAC (SMAC should be installed through nnictl)
+       builtinTunerName: TPE
+       classArgs:
+         #choice: maximize, minimize
+         optimize_mode: maximize
+     trial:
+       command: python3 main.py
+       codeDir: .
+       gpuNum: 4
+       cpuNum: 2
+       memoryMB: 10000
+       #The docker image to run NNI job on pai
+       image: msranni/nni:latest
+     paiConfig:
+       #The username to login pai
+       userName: test
+       #The password to login pai
+       passWord: test
+       #The host of restful server of pai
+       host: 10.10.10.10
+
+Kubeflow mode
+^^^^^^^^^^^^^
+
+  kubeflow with nfs storage.
+
+.. code-block:: yaml
+
+     authorName: default
+     experimentName: example_mni
+     trialConcurrency: 1
+     maxExecDuration: 1h
+     maxTrialNum: 1
+     #choice: local, remote, pai, kubeflow
+     trainingServicePlatform: kubeflow
+     searchSpacePath: search_space.json
+     #choice: true, false
+     useAnnotation: false
+     tuner:
+       #choice: TPE, Random, Anneal, Evolution
+       builtinTunerName: TPE
+       classArgs:
+         #choice: maximize, minimize
+         optimize_mode: maximize
+     trial:
+       codeDir: .
+       worker:
+         replicas: 1
+         command: python3 mnist.py
+         gpuNum: 0
+         cpuNum: 1
+         memoryMB: 8192
+         image: msranni/nni:latest
+     kubeflowConfig:
+       operator: tf-operator
+       nfs:
+         server: 10.10.10.10
+         path: /var/nfs/general
+
+Kubeflow with azure storage
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. code-block:: yaml
+
+     authorName: default
+     experimentName: example_mni
+     trialConcurrency: 1
+     maxExecDuration: 1h
+     maxTrialNum: 1
+     #choice: local, remote, pai, kubeflow
+     trainingServicePlatform: kubeflow
+     searchSpacePath: search_space.json
+     #choice: true, false
+     useAnnotation: false
+     #nniManagerIp: 10.10.10.10
+     tuner:
+       #choice: TPE, Random, Anneal, Evolution
+       builtinTunerName: TPE
+       classArgs:
+         #choice: maximize, minimize
+         optimize_mode: maximize
+     assessor:
+       builtinAssessorName: Medianstop
+       classArgs:
+         optimize_mode: maximize
+     trial:
+       codeDir: .
+       worker:
+         replicas: 1
+         command: python3 mnist.py
+         gpuNum: 0
+         cpuNum: 1
+         memoryMB: 4096
+         image: msranni/nni:latest
+     kubeflowConfig:
+       operator: tf-operator
+       keyVault:
+         vaultName: Contoso-Vault
+         name: AzureStorageAccountKey
+       azureStorage:
+         accountName: storage
+         azureShare: share01

docs/en_US/Tutorial/FAQ.rst

+FAQ
+===
+
+This page is for frequent asked questions and answers.
+
+tmp folder fulled
+^^^^^^^^^^^^^^^^^
+
+nnictl will use tmp folder as a temporary folder to copy files under codeDir when executing experimentation creation.
+When met errors like below, try to clean up **tmp** folder first.
+
+..
+
+   OSError: [Errno 28] No space left on device
+
+
+Cannot get trials' metrics in OpenPAI mode
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+In OpenPAI training mode, we start a rest server which listens on 51189 port in NNI Manager to receive metrcis reported from trials running in OpenPAI cluster. If you didn't see any metrics from WebUI in OpenPAI mode, check your machine where NNI manager runs on to make sure 51189 port is turned on in the firewall rule.
+
+Segmentation Fault (core dumped) when installing
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. code-block:: text
+
+   make: *** [install-XXX] Segmentation fault (core dumped)
+
+
+Please try the following solutions in turn:
+
+
+* Update or reinstall you current python's pip like ``python3 -m pip install -U pip``
+* Install NNI with ``--no-cache-dir`` flag like ``python3 -m pip install nni --no-cache-dir``
+
+Job management error: getIPV4Address() failed because os.networkInterfaces().eth0 is undefined.
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Your machine don't have eth0 device, please set `nniManagerIp <ExperimentConfig.rst>`__ in your config file manually.
+
+Exceed the MaxDuration but didn't stop
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+When the duration of experiment reaches the maximum duration, nniManager will not create new trials, but the existing trials will continue unless user manually stop the experiment.
+
+Could not stop an experiment using ``nnictl stop``
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+If you upgrade your NNI or you delete some config files of NNI when there is an experiment running, this kind of issue may happen because the loss of config file. You could use ``ps -ef | grep node`` to find the PID of your experiment, and use ``kill -9 {pid}`` to kill it manually.
+
+Could not get ``default metric`` in webUI of virtual machines
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Config the network mode to bridge mode or other mode that could make virtual machine's host accessible from external machine, and make sure the port of virtual machine is not forbidden by firewall.
+
+Could not open webUI link
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Unable to open the WebUI may have the following reasons:
+
+
+* ``http://127.0.0.1``\ , ``http://172.17.0.1`` and ``http://10.0.0.15`` are referred to localhost, if you start your experiment on the server or remote machine. You can replace the IP to your server IP to view the WebUI, like ``http://[your_server_ip]:8080``
+* If you still can't see the WebUI after you use the server IP, you can check the proxy and the firewall of your machine. Or use the browser on the machine where you start your NNI experiment.
+* Another reason may be your experiment is failed and NNI may fail to get the experiment information. You can check the log of NNIManager in the following directory: ``~/nni-experiments/[your_experiment_id]`` ``/log/nnimanager.log``
+
+Restful server start failed
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Probably it's a problem with your network config. Here is a checklist.
+
+
+* You might need to link ``127.0.0.1`` with ``localhost``. Add a line ``127.0.0.1 localhost`` to ``/etc/hosts``.
+* It's also possible that you have set some proxy config. Check your environment for variables like ``HTTP_PROXY`` or ``HTTPS_PROXY`` and unset if they are set.
+
+NNI on Windows problems
+^^^^^^^^^^^^^^^^^^^^^^^
+
+Please refer to `NNI on Windows <InstallationWin.rst>`__
+
+More FAQ issues
+^^^^^^^^^^^^^^^
+
+`NNI Issues with FAQ labels <https://github.com/microsoft/nni/labels/FAQ>`__
+
+Help us improve
+^^^^^^^^^^^^^^^
+
+Please inquiry the problem in https://github.com/Microsoft/nni/issues to see whether there are other people already reported the problem, create a new one if there are no existing issues been created.