add dtk24.04 code

docs/_removed/Tuner/CustomizeAdvisor.rst

+**How To** - Customize Your Own Advisor
+=======================================
+
+*Warning: API is subject to change in future releases.*
+
+Advisor targets the scenario that the automl algorithm wants the methods of both tuner and assessor. Advisor is similar to tuner on that it receives trial parameters request, final results, and generate trial parameters. Also, it is similar to assessor on that it receives intermediate results, trial's end state, and could send trial kill command. Note that, if you use Advisor, tuner and assessor are not allowed to be used at the same time.
+
+If a user want to implement a customized Advisor, she/he only needs to:
+
+**1. Define an Advisor inheriting from the MsgDispatcherBase class.** For example:
+
+.. code-block:: python
+
+   from nni.runtime.msg_dispatcher_base import MsgDispatcherBase
+
+   class CustomizedAdvisor(MsgDispatcherBase):
+       def __init__(self, ...):
+           ...
+
+**2. Implement the methods with prefix "handle_" except "handle_request""**
+
+You might find `docs <../autotune_ref.rst#Advisor>`__ for ``MsgDispatcherBase`` helpful.
+
+**3. Configure your customized Advisor in experiment YAML config file.**
+
+Similar to tuner and assessor. NNI needs to locate your customized Advisor class and instantiate the class, so you need to specify the location of the customized Advisor class and pass literal values as parameters to the ``__init__`` constructor.
+
+.. code-block:: yaml
+
+   advisor:
+     codeDir: /home/abc/myadvisor
+     classFileName: my_customized_advisor.py
+     className: CustomizedAdvisor
+     # Any parameter need to pass to your advisor class __init__ constructor
+     # can be specified in this optional classArgs field, for example
+     classArgs:
+       arg1: value1
+
+**Note that** The working directory of your advisor is ``<home>/nni-experiments/<experiment_id>/log``, which can be retrieved with environment variable ``NNI_LOG_DIRECTORY``.
+
+Example
+-------
+
+Here we provide an :githublink:`example <examples/tuners/mnist_keras_customized_advisor>`.

docs/_removed/Tuner/DngoTuner.rst

+DNGO Tuner
+==========
+
+Usage
+-----
+
+Installation
+^^^^^^^^^^^^
+
+classArgs requirements
+^^^^^^^^^^^^^^^^^^^^^^
+
+* **optimize_mode** (*'maximize' or 'minimize'*) - If 'maximize', the tuner will target to maximize metrics. If 'minimize', the tuner will target to minimize metrics.
+* **sample_size** (*int, default = 1000*) - Number of samples to select in each iteration. The best one will be picked from the samples as the next trial.
+* **trials_per_update** (*int, default = 20*) - Number of trials to collect before updating the model.
+* **num_epochs_per_training** (*int, default = 500*) - Number of epochs to train DNGO model.
+
+Example Configuration
+^^^^^^^^^^^^^^^^^^^^^
+
+.. code-block:: yaml
+
+   # config.yml
+   tuner:
+     name: DNGOTuner
+     classArgs:
+       optimize_mode: maximize

docs/_removed/Tuner/EvolutionTuner.rst

+Naive Evolution Tuner
+=====================
+
+Naive Evolution comes from `Large-Scale Evolution of Image Classifiers <https://arxiv.org/pdf/1703.01041.pdf>`__. It randomly initializes a population based on the search space. For each generation, it chooses better ones and does some mutation (e.g., changes a hyperparameter, adds/removes one layer, etc.) on them to get the next generation. Naive Evolution requires many trials to works but it's very simple and it's easily expanded with new features.
+
+Usage
+-----
+
+classArgs Requirements
+^^^^^^^^^^^^^^^^^^^^^^
+
+* 
+  **optimize_mode** (*maximize or minimize, optional, default = maximize*) - If 'maximize', the tuner will try to maximize metrics. If 'minimize', the tuner will try to minimize metrics.
+
+* 
+  **population_size** (*int value (should > 0), optional, default = 20*) - the initial size of the population (trial num) in the evolution tuner. It's suggested that ``population_size`` be much larger than ``concurrency`` so users can get the most out of the algorithm (and at least ``concurrency``, or the tuner will fail on its first generation of parameters).
+
+Example Configuration
+^^^^^^^^^^^^^^^^^^^^^
+
+.. code-block:: yaml
+
+   # config.yml
+   tuner:
+     name: Evolution
+     classArgs:
+       optimize_mode: maximize
+       population_size: 100
+

docs/_removed/Tuner/GPTuner.rst

+GP Tuner
+========
+
+Bayesian optimization works by constructing a posterior distribution of functions (a Gaussian Process) that best describes the function you want to optimize. As the number of observations grows, the posterior distribution improves, and the algorithm becomes more certain of which regions in parameter space are worth exploring and which are not.
+
+GP Tuner is designed to minimize/maximize the number of steps required to find a combination of parameters that are close to the optimal combination. To do so, this method uses a proxy optimization problem (finding the maximum of the acquisition function) that, albeit still a hard problem, is cheaper (in the computational sense) to solve, and it's amenable to common tools. Therefore, Bayesian Optimization is suggested for situations where sampling the function to be optimized is very expensive.
+
+Note that the only acceptable types within the search space are ``randint``, ``uniform``, ``quniform``, ``loguniform``, ``qloguniform``, and numerical ``choice``.
+
+This optimization approach is described in Section 3 of `Algorithms for Hyper-Parameter Optimization <https://papers.nips.cc/paper/4443-algorithms-for-hyper-parameter-optimization.pdf>`__.
+
+Usage
+-----
+
+classArgs requirements
+^^^^^^^^^^^^^^^^^^^^^^
+
+* **optimize_mode** (*'maximize' or 'minimize', optional, default = 'maximize'*) - If 'maximize', the tuner will try to maximize metrics. If 'minimize', the tuner will try to minimize metrics.
+* **utility** (*'ei', 'ucb' or 'poi', optional, default = 'ei'*) - The utility function (acquisition function). 'ei', 'ucb', and 'poi' correspond to 'Expected Improvement', 'Upper Confidence Bound', and 'Probability of Improvement', respectively.
+* **kappa** (*float, optional, default = 5*) - Used by the 'ucb' utility function. The bigger ``kappa`` is, the more exploratory the tuner will be.
+* **xi** (*float, optional, default = 0*) - Used by the 'ei' and 'poi' utility functions. The bigger ``xi`` is, the more exploratory the tuner will be.
+* **nu** (*float, optional, default = 2.5*) - Used to specify the Matern kernel. The smaller nu, the less smooth the approximated function is.
+* **alpha** (*float, optional, default = 1e-6*) - Used to specify the Gaussian Process Regressor. Larger values correspond to an increased noise level in the observations.
+* **cold_start_num** (*int, optional, default = 10*) - Number of random explorations to perform before the Gaussian Process. Random exploration can help by diversifying the exploration space.
+* **selection_num_warm_up** (*int, optional, default = 1e5*) - Number of random points to evaluate when getting the point which maximizes the acquisition function.
+* **selection_num_starting_points** (*int, optional, default = 250*) - Number of times to run L-BFGS-B from a random starting point after the warmup.
+
+Example Configuration
+^^^^^^^^^^^^^^^^^^^^^
+
+.. code-block:: yaml
+
+   # config.yml
+   tuner:
+     name: GPTuner
+     classArgs:
+       optimize_mode: maximize
+       utility: 'ei'
+       kappa: 5.0
+       xi: 0.0
+       nu: 2.5
+       alpha: 1e-6
+       cold_start_num: 10
+       selection_num_warm_up: 100000
+       selection_num_starting_points: 250

docs/_removed/Tuner/HyperbandAdvisor.rst

+Hyperband Advisor
+=================
+
+`Hyperband <https://arxiv.org/pdf/1603.06560.pdf>`__ is a popular autoML algorithm. The basic idea of Hyperband is to create several buckets, each having ``n`` randomly generated hyperparameter configurations, each configuration using ``r`` resources (e.g., epoch number, batch number). After the ``n`` configurations are finished, it chooses the top ``n/eta`` configurations and runs them using increased ``r*eta`` resources. At last, it chooses the best configuration it has found so far.
+
+Implementation with full parallelism
+------------------------------------
+
+First, this is an example of how to write an autoML algorithm based on MsgDispatcherBase, rather than Tuner and Assessor. Hyperband is implemented in this way because it integrates the functions of both Tuner and Assessor, thus, we call it Advisor.
+
+Second, this implementation fully leverages Hyperband's internal parallelism. Specifically, the next bucket is not started strictly after the current bucket. Instead, it starts when there are available resources. If you want to use full parallelism mode, set ``exec_mode`` with ``parallelism``. 
+
+Or if you want to set ``exec_mode`` with ``serial`` according to the original algorithm. In this mode, the next bucket will start strictly after the current bucket.
+
+``parallelism`` mode may lead to multiple unfinished buckets, and there is at most one unfinished bucket under ``serial`` mode. The advantage of ``parallelism`` mode is to make full use of resources, which may reduce the experiment duration multiple times. The following two pictures are the results of quick verification using `nas-bench-201 <../NAS/Benchmarks.rst>`__, picture above is in ``parallelism`` mode, picture below is in ``serial`` mode.
+
+
+.. image:: ../../img/hyperband_parallelism.png
+   :target: ../../img/hyperband_parallelism.png
+   :alt: parallelism mode
+
+
+
+.. image:: ../../img/hyperband_serial.png
+   :target: ../../img/hyperband_serial.png
+   :alt: serial mode
+
+
+If you want to reproduce these results, refer to the example under ``examples/trials/benchmarking/`` for details.
+
+Usage
+-----
+
+Config file
+^^^^^^^^^^^
+
+To use Hyperband, you should add the following spec in your experiment's YAML config file:
+
+.. code-block:: bash
+
+   advisor:
+     #choice: Hyperband
+     builtinAdvisorName: Hyperband
+     classArgs:
+       #R: the maximum trial budget
+       R: 100
+       #eta: proportion of discarded trials
+       eta: 3
+       #choice: maximize, minimize
+       optimize_mode: maximize
+       #choice: serial, parallelism
+       exec_mode: parallelism
+
+Note that once you use Advisor, you are not allowed to add a Tuner and Assessor spec in the config file. If you use Hyperband, among the hyperparameters (i.e., key-value pairs) received by a trial, there will be one more key called ``TRIAL_BUDGET`` defined by user. **By using this ``TRIAL_BUDGET``, the trial can control how long it runs**.
+
+For ``report_intermediate_result(metric)`` and ``report_final_result(metric)`` in your trial code, **``metric`` should be either a number or a dict which has a key ``default`` with a number as its value**. This number is the one you want to maximize or minimize, for example, accuracy or loss.
+
+``R`` and ``eta`` are the parameters of Hyperband that you can change. ``R`` means the maximum trial budget that can be allocated to a configuration. Here, trial budget could mean the number of epochs or mini-batches. This ``TRIAL_BUDGET`` should be used by the trial to control how long it runs. Refer to the example under ``examples/trials/mnist-advisor/`` for details.
+
+``eta`` means ``n/eta`` configurations from ``n`` configurations will survive and rerun using more budgets.
+
+Here is a concrete example of ``R=81`` and ``eta=3``:
+
+.. list-table::
+   :header-rows: 1
+   :widths: auto
+
+   * -
+     - s=4
+     - s=3
+     - s=2
+     - s=1
+     - s=0
+   * - i
+     - n r
+     - n r
+     - n r
+     - n r
+     - n r
+   * - 0
+     - 81 1
+     - 27 3
+     - 9 9
+     - 6 27
+     - 5 81
+   * - 1
+     - 27 3
+     - 9 9
+     - 3 27
+     - 2 81
+     -
+   * - 2
+     - 9 9
+     - 3 27
+     - 1 81
+     -
+     -
+   * - 3
+     - 3 27
+     - 1 81
+     -
+     -
+     -
+   * - 4
+     - 1 81
+     -
+     -
+     -
+     -
+
+
+``s`` means bucket, ``n`` means the number of configurations that are generated, the corresponding ``r`` means how many budgets these configurations run. ``i`` means round, for example, bucket 4 has 5 rounds, bucket 3 has 4 rounds.
+
+For information about writing trial code, please refer to the instructions under ``examples/trials/mnist-hyperband/``.
+
+classArgs requirements
+^^^^^^^^^^^^^^^^^^^^^^
+
+
+* **optimize_mode** (*maximize or minimize, optional, default = maximize*) - If 'maximize', the tuner will try to maximize metrics. If 'minimize', the tuner will try to minimize metrics.
+* **R** (*int, optional, default = 60*) - the maximum budget given to a trial (could be the number of mini-batches or epochs). Each trial should use TRIAL_BUDGET to control how long they run.
+* **eta** (*int, optional, default = 3*) - ``(eta-1)/eta`` is the proportion of discarded trials.
+* **exec_mode** (*serial or parallelism, optional, default = parallelism*) - If 'parallelism', the tuner will try to use available resources to start new bucket immediately. If 'serial', the tuner will only start new bucket after the current bucket is done.
+
+Example Configuration
+^^^^^^^^^^^^^^^^^^^^^
+
+.. code-block:: yaml
+
+   # config.yml
+   advisor:
+     builtinAdvisorName: Hyperband
+     classArgs:
+       optimize_mode: maximize
+       R: 60
+       eta: 3
+
+Future improvements
+-------------------
+
+The current implementation of Hyperband can be further improved by supporting a simple early stop algorithm since it's possible that not all the configurations in the top ``n/eta`` perform well. Any unpromising configurations should be stopped early.
+
+In the current implementation, configurations are generated randomly which follows the design in the `paper <https://arxiv.org/pdf/1603.06560.pdf>`__. As an improvement, configurations could be generated more wisely by leveraging advanced algorithms.

docs/_removed/Tuner/MetisTuner.rst

+Metis Tuner
+===========
+
+`Metis <https://www.microsoft.com/en-us/research/publication/metis-robustly-tuning-tail-latencies-cloud-systems/>`__ offers several benefits over other tuning algorithms. While most tools only predict the optimal configuration, Metis gives you two outputs, a prediction for the optimal configuration and a suggestion for the next trial. No more guess work!
+
+While most tools assume training datasets do not have noisy data, Metis actually tells you if you need to resample a particular hyper-parameter.
+
+While most tools have problems of being exploitation-heavy, Metis' search strategy balances exploration, exploitation, and (optional) resampling.
+
+Metis belongs to the class of sequential model-based optimization (SMBO) algorithms and it is based on the Bayesian Optimization framework. To model the parameter-vs-performance space, Metis uses both a Gaussian Process and GMM. Since each trial can impose a high time cost, Metis heavily trades inference computations with naive trials. At each iteration, Metis does two tasks:
+
+
+* 
+  It finds the global optimal point in the Gaussian Process space. This point represents the optimal configuration.
+
+* 
+  It identifies the next hyper-parameter candidate. This is achieved by inferring the potential information gain of exploration, exploitation, and resampling.
+
+Note that the only acceptable types within the search space are ``quniform``, ``uniform``, ``randint``, and numerical ``choice``.
+
+More details can be found in our `paper <https://www.microsoft.com/en-us/research/publication/metis-robustly-tuning-tail-latencies-cloud-systems/>`__.
+
+Usage
+-----
+
+classArgs requirements
+^^^^^^^^^^^^^^^^^^^^^^
+
+* **optimize_mode** (*'maximize' or 'minimize', optional, default = 'maximize'*) - If 'maximize', the tuner will try to maximize metrics. If 'minimize', the tuner will try to minimize metrics.
+
+Example Configuration
+^^^^^^^^^^^^^^^^^^^^^
+
+.. code-block:: yaml
+
+   # config.yml
+   tuner:
+     name: MetisTuner
+     classArgs:
+       optimize_mode: maximize

docs/_removed/Tuner/PBTTuner.rst

+PBT Tuner
+=========
+
+Population Based Training (PBT) comes from `Population Based Training of Neural Networks <https://arxiv.org/abs/1711.09846v1>`__. It's a simple asynchronous optimization algorithm which effectively utilizes a fixed computational budget to jointly optimize a population of models and their hyperparameters to maximize performance. Importantly, PBT discovers a schedule of hyperparameter settings rather than following the generally sub-optimal strategy of trying to find a single fixed set to use for the whole course of training. 
+
+
+.. image:: ../../img/pbt.jpg
+   :target: ../../img/pbt.jpg
+   :alt: 
+
+
+PBTTuner initializes a population with several trials (i.e., ``population_size``). There are four steps in the above figure, each trial only runs by one step. How long is one step is controlled by trial code, e.g., one epoch. When a trial starts, it loads a checkpoint specified by PBTTuner and continues to run one step, then saves checkpoint to a directory specified by PBTTuner and exits. The trials in a population run steps synchronously, that is, after all the trials finish the ``i``-th step, the ``(i+1)``-th step can be started. Exploitation and exploration of PBT are executed between two consecutive steps.
+
+Usage
+-----
+
+Provide checkpoint directory
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Since some trials need to load other trial's checkpoint, users should provide a directory (i.e., ``all_checkpoint_dir``) which is accessible by every trial. It is easy for local mode, users could directly use the default directory or specify any directory on the local machine. For other training services, users should follow `the document of those training services <../TrainingService/Overview.rst>`__ to provide a directory in a shared storage, such as NFS, Azure storage.
+
+Modify your trial code
+^^^^^^^^^^^^^^^^^^^^^^
+
+Before running a step, a trial needs to load a checkpoint, the checkpoint directory is specified in hyper-parameter configuration generated by PBTTuner, i.e., ``params['load_checkpoint_dir']``. Similarly, the directory for saving checkpoint is also included in the configuration, i.e., ``params['save_checkpoint_dir']``. Here, ``all_checkpoint_dir`` is base folder of ``load_checkpoint_dir`` and ``save_checkpoint_dir`` whose format is ``all_checkpoint_dir/<population-id>/<step>``.
+
+.. code-block:: python
+
+   params = nni.get_next_parameter()
+   # the path of the checkpoint to load
+   load_path = os.path.join(params['load_checkpoint_dir'], 'model.pth')
+   # load checkpoint from `load_path`
+   ...
+   # run one step
+   ...
+   # the path for saving a checkpoint
+   save_path = os.path.join(params['save_checkpoint_dir'], 'model.pth')
+   # save checkpoint to `save_path`
+   ...
+
+The complete example code can be found :githublink:`here <examples/trials/mnist-pbt-tuner-pytorch>`.
+
+classArgs requirements
+^^^^^^^^^^^^^^^^^^^^^^
+
+* **optimize_mode** (*'maximize' or 'minimize'*) - If 'maximize', the tuner will target to maximize metrics. If 'minimize', the tuner will target to minimize metrics.
+* **all_checkpoint_dir** (*str, optional, default = None*) - Directory for trials to load and save checkpoint, if not specified, the directory would be "~/nni/checkpoint/\ :raw-html:`<exp-id>`\ ". Note that if the experiment is not local mode, users should provide a path in a shared storage which can be accessed by all the trials.
+* **population_size** (*int, optional, default = 10*) - Number of trials in a population. Each step has this number of trials. In our implementation, one step is running each trial by specific training epochs set by users.
+* **factors** (*tuple, optional, default = (1.2, 0.8)*) - Factors for perturbation of hyperparameters.
+* **fraction** (*float, optional, default = 0.2*) - Fraction for selecting bottom and top trials.
+
+Experiment config
+^^^^^^^^^^^^^^^^^
+
+Below is an exmaple of PBTTuner configuration in experiment config file. **Note that Assessor is not allowed if PBTTuner is used.**
+
+.. code-block:: yaml
+
+   # config.yml
+   tuner:
+     name: PBTTuner
+     classArgs:
+       optimize_mode: maximize
+       all_checkpoint_dir: /the/path/to/store/checkpoints
+       population_size: 10
+
+Example Configuration
+^^^^^^^^^^^^^^^^^^^^^
+
+.. code-block:: yaml
+
+   # config.yml
+   tuner:
+     name: PBTTuner
+     classArgs:
+       optimize_mode: maximize

docs/_removed/Tuner/SmacTuner.rst

+SMAC Tuner
+==========
+
+`SMAC <https://www.cs.ubc.ca/~hutter/papers/10-TR-SMAC.pdf>`__ is based on Sequential Model-Based Optimization (SMBO). It adapts the most prominent previously used model class (Gaussian stochastic process models) and introduces the model class of random forests to SMBO in order to handle categorical parameters. The SMAC supported by nni is a wrapper on `the SMAC3 github repo <https://github.com/automl/SMAC3>`__.
+
+Note that SMAC on nni only supports a subset of the types in the `search space spec <../Tutorial/SearchSpaceSpec.rst>`__: ``choice``, ``randint``, ``uniform``, ``loguniform``, and ``quniform``.
+
+Usage
+-----
+
+Installation
+^^^^^^^^^^^^
+
+SMAC has dependencies that need to be installed by following command before the first usage. As a reminder, ``swig`` is required for SMAC: for Ubuntu ``swig`` can be installed with ``apt``.
+
+.. code-block:: bash
+
+   pip install nni[SMAC]
+
+classArgs requirements
+^^^^^^^^^^^^^^^^^^^^^^
+
+* **optimize_mode** (*maximize or minimize, optional, default = maximize*) - If 'maximize', the tuner will try to maximize metrics. If 'minimize', the tuner will try to minimize metrics.
+* **config_dedup** (*True or False, optional, default = False*) - If True, the tuner will not generate a configuration that has been already generated. If False, a configuration may be generated twice, but it is rare for a relatively large search space.
+
+Example Configuration
+^^^^^^^^^^^^^^^^^^^^^
+
+.. code-block:: yaml
+
+   # config.yml
+   tuner:
+     name: SMAC
+     classArgs:
+       optimize_mode: maximize

docs/_removed/Tutorial/NNSpider.md

+<!-- <style>
+  table, tr, td{
+    border: none;
+  }
+  div{
+    width: 300px;
+    height: 200px;
+    border: 1px solid grey;
+    background: #ccc;
+    box-sizing: border-box;
+  }
+  img{
+    width: 260px;
+    height: 160px;
+    margin: 20px;
+  }
+</style> -->
+
+<table>
+  <tr>
+    <td>
+      <div>
+        <img style="
+          width: 300px;
+        "
+        src="../../img/emoicons/NoBug.png"/>
+      </div>
+    </td>
+    <td>
+        <div>
+        <img style="
+          width: 300px;
+        " src="../../img/emoicons/Holiday.png"/>
+      </div>  
+    </td>
+    <td>
+        <div>
+        <img style="
+          width: 300px;
+          height: 180px;
+        " src="../../img/emoicons/Error.png"/>
+      </div>  
+    </td>
+  </tr>
+  <tr>
+    <td align="center">No bug</td>
+    <td align="center">Holiday</td>
+    <td align="center">Error</td>
+  </tr>
+  <tr>
+    <td>
+      <div>
+        <img style="
+          width: 300px;
+          height: 210px;
+        " src="../../img/emoicons/Working.png"/>
+      </div>
+    </td>
+    <td >
+        <div>
+        <img style="
+          width: 300px;
+        " src="../../img/emoicons/Sign.png"/>
+      </div>  
+    </td>
+    <td>
+        <div>
+        <img style="
+          width: 300px;
+        " src="../../img/emoicons/Crying.png"/>
+      </div>  
+    </td>
+  </tr>
+  <tr>
+    <td align="center" >Working</td>
+    <td align="center" >Sign</td>
+    <td align="center" >Crying</td>
+  </tr>
+  <tr>
+    <td>
+      <div>
+        <img style="
+          width: 300px;
+          height: 190px;
+        " src="../../img/emoicons/Cut.png"/>
+      </div>
+    </td>
+    <td>
+        <div>
+        <img style="
+          width: 300px;
+        " src="../../img/emoicons/Weaving.png"/>
+      </div>  
+    </td>
+    <td>
+        <div>
+        <img style="
+          width: 300px;
+        " src="../../img/emoicons/Comfort.png"/>
+      </div>  
+    </td>
+  </tr>
+  <tr>
+    <td align="center">Cut</td>
+    <td align="center">Weaving</td>
+    <td align="center">Comfort</td>
+  </tr>
+  <tr>
+    <td>
+      <div>
+        <img style="
+          width: 300px;
+        " src="../../img/emoicons/Sweat.png"/>
+      </div>
+    </td>
+    <td></td>
+    <td></td>
+  </tr>
+  <tr>
+    <td align="center">Sweat</td>
+    <td align="center"></td>
+    <td align="center"></td>
+  </tr>
+</table>

docs/_removed/Tutorial/python_api_connect.ipynb

+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "white-electron",
+   "metadata": {},
+   "source": [
+    "## Connect and Manage an Exist Experiment"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "recent-italic",
+   "metadata": {},
+   "source": [
+    "### 1. Connect Experiment"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "statistical-repair",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "[2021-03-05 12:18:28] Connect to port 8080 success, experiment id is DH8pVfXc, status is RUNNING.\n"
+     ]
+    }
+   ],
+   "source": [
+    "from nni.experiment import Experiment\n",
+    "experiment = Experiment.connect(8080)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "defensive-scratch",
+   "metadata": {},
+   "source": [
+    "### 2. Experiment View & Control"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "independent-touch",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'id': 'DH8pVfXc',\n",
+       " 'revision': 4,\n",
+       " 'execDuration': 10,\n",
+       " 'logDir': '/home/ningshang/nni-experiments/DH8pVfXc',\n",
+       " 'nextSequenceId': 1,\n",
+       " 'params': {'authorName': 'default',\n",
+       "  'experimentName': 'example_sklearn-classification',\n",
+       "  'trialConcurrency': 1,\n",
+       "  'maxExecDuration': 3600,\n",
+       "  'maxTrialNum': 100,\n",
+       "  'searchSpace': '{\"C\": {\"_type\": \"uniform\", \"_value\": [0.1, 1]}, \"kernel\": {\"_type\": \"choice\", \"_value\": [\"linear\", \"rbf\", \"poly\", \"sigmoid\"]}, \"degree\": {\"_type\": \"choice\", \"_value\": [1, 2, 3, 4]}, \"gamma\": {\"_type\": \"uniform\", \"_value\": [0.01, 0.1]}, \"coef0\": {\"_type\": \"uniform\", \"_value\": [0.01, 0.1]}}',\n",
+       "  'trainingServicePlatform': 'local',\n",
+       "  'tuner': {'builtinTunerName': 'TPE',\n",
+       "   'classArgs': {'optimize_mode': 'maximize'},\n",
+       "   'checkpointDir': '/home/ningshang/nni-experiments/DH8pVfXc/checkpoint'},\n",
+       "  'versionCheck': True,\n",
+       "  'clusterMetaData': [{'key': 'trial_config',\n",
+       "    'value': {'command': 'python3 main.py',\n",
+       "     'codeDir': '/home/ningshang/nni/examples/trials/sklearn/classification/.',\n",
+       "     'gpuNum': 0}}]},\n",
+       " 'startTime': 1614946699989}"
+      ]
+     },
+     "execution_count": 2,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "experiment.get_experiment_profile()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "printable-bookmark",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "[2021-03-05 12:18:32] (root) Successfully update maxTrialNum.\n"
+     ]
+    }
+   ],
+   "source": [
+    "experiment.update_max_trial_number(200)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "marine-serial",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'id': 'DH8pVfXc',\n",
+       " 'revision': 5,\n",
+       " 'execDuration': 14,\n",
+       " 'logDir': '/home/ningshang/nni-experiments/DH8pVfXc',\n",
+       " 'nextSequenceId': 1,\n",
+       " 'params': {'authorName': 'default',\n",
+       "  'experimentName': 'example_sklearn-classification',\n",
+       "  'trialConcurrency': 1,\n",
+       "  'maxExecDuration': 3600,\n",
+       "  'maxTrialNum': 200,\n",
+       "  'searchSpace': '{\"C\": {\"_type\": \"uniform\", \"_value\": [0.1, 1]}, \"kernel\": {\"_type\": \"choice\", \"_value\": [\"linear\", \"rbf\", \"poly\", \"sigmoid\"]}, \"degree\": {\"_type\": \"choice\", \"_value\": [1, 2, 3, 4]}, \"gamma\": {\"_type\": \"uniform\", \"_value\": [0.01, 0.1]}, \"coef0\": {\"_type\": \"uniform\", \"_value\": [0.01, 0.1]}}',\n",
+       "  'trainingServicePlatform': 'local',\n",
+       "  'tuner': {'builtinTunerName': 'TPE',\n",
+       "   'classArgs': {'optimize_mode': 'maximize'},\n",
+       "   'checkpointDir': '/home/ningshang/nni-experiments/DH8pVfXc/checkpoint'},\n",
+       "  'versionCheck': True,\n",
+       "  'clusterMetaData': [{'key': 'trial_config',\n",
+       "    'value': {'command': 'python3 main.py',\n",
+       "     'codeDir': '/home/ningshang/nni/examples/trials/sklearn/classification/.',\n",
+       "     'gpuNum': 0}}]},\n",
+       " 'startTime': 1614946699989}"
+      ]
+     },
+     "execution_count": 4,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "experiment.get_experiment_profile()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "opened-lounge",
+   "metadata": {},
+   "source": [
+    "### 3. Stop Experiment"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "emotional-machinery",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "[2021-03-05 12:18:36] Stopping experiment, please wait...\n",
+      "[2021-03-05 12:18:38] Experiment stopped\n"
+     ]
+    }
+   ],
+   "source": [
+    "experiment.stop()"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "nni-dev",
+   "language": "python",
+   "name": "nni-dev"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.7.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/_removed/Tutorial/python_api_start.ipynb

+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "technological-script",
+   "metadata": {},
+   "source": [
+    "## Start and Manage a New Experiment"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "reported-somerset",
+   "metadata": {},
+   "source": [
+    "### 1. Configure Search Space"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "potential-williams",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "search_space = {\n",
+    "    \"C\": {\"_type\":\"quniform\",\"_value\":[0.1, 1, 0.1]},\n",
+    "    \"kernel\": {\"_type\":\"choice\",\"_value\":[\"linear\", \"rbf\", \"poly\", \"sigmoid\"]},\n",
+    "    \"degree\": {\"_type\":\"choice\",\"_value\":[1, 2, 3, 4]},\n",
+    "    \"gamma\": {\"_type\":\"quniform\",\"_value\":[0.01, 0.1, 0.01]},\n",
+    "    \"coef0\": {\"_type\":\"quniform\",\"_value\":[0.01, 0.1, 0.01]}\n",
+    "}"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "greek-archive",
+   "metadata": {},
+   "source": [
+    "### 2. Configure Experiment "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "fiscal-expansion",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from nni.experiment import Experiment\n",
+    "experiment = Experiment('local')\n",
+    "experiment.config.experiment_name = 'Example'\n",
+    "experiment.config.trial_concurrency = 2\n",
+    "experiment.config.max_trial_number = 10\n",
+    "experiment.config.search_space = search_space\n",
+    "experiment.config.trial_command = 'python3 main.py'\n",
+    "experiment.config.trial_code_directory = './'\n",
+    "experiment.config.tuner.name = 'TPE'\n",
+    "experiment.config.tuner.class_args['optimize_mode'] = 'maximize'\n",
+    "experiment.config.training_service.use_active_gpu = True"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "received-tattoo",
+   "metadata": {},
+   "source": [
+    "### 3. Start Experiment"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "pleasant-patent",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "[2021-03-05 12:12:19] Creating experiment, Experiment ID: wdt0le3v\n",
+      "[2021-03-05 12:12:19] Starting web server...\n",
+      "[2021-03-05 12:12:20] Setting up...\n",
+      "[2021-03-05 12:12:20] Web UI URLs: http://127.0.0.1:8080 http://10.0.1.5:8080 http://172.17.0.1:8080\n"
+     ]
+    }
+   ],
+   "source": [
+    "experiment.start(8080)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "miniature-prison",
+   "metadata": {},
+   "source": [
+    "### 4. Experiment View & Control"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "animated-english",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "'RUNNING'"
+      ]
+     },
+     "execution_count": 4,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "experiment.get_status()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "id": "alpha-ottawa",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "[TrialResult(parameter={'C': 0.30000000000000004, 'kernel': 'linear', 'degree': 3, 'gamma': 0.03, 'coef0': 0.07}, value=0.9888888888888889, trialJobId='VLqU9'),\n",
+       " TrialResult(parameter={'C': 0.5, 'kernel': 'sigmoid', 'degree': 1, 'gamma': 0.03, 'coef0': 0.07}, value=0.8888888888888888, trialJobId='DLo6r')]"
+      ]
+     },
+     "execution_count": 5,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "experiment.export_data()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "unique-rendering",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "{'DLo6r': [TrialMetricData(timestamp=1614946351592, trialJobId='DLo6r', parameterId='1', type='FINAL', sequence=0, data=0.8888888888888888)],\n",
+       " 'VLqU9': [TrialMetricData(timestamp=1614946351607, trialJobId='VLqU9', parameterId='0', type='FINAL', sequence=0, data=0.9888888888888889)]}"
+      ]
+     },
+     "execution_count": 6,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "experiment.get_job_metrics()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "welsh-difference",
+   "metadata": {},
+   "source": [
+    "### 5. Stop Experiment"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "technological-cleanup",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "[2021-03-05 12:12:40] Stopping experiment, please wait...\n",
+      "[2021-03-05 12:12:42] Experiment stopped\n"
+     ]
+    }
+   ],
+   "source": [
+    "experiment.stop()"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "nni-dev",
+   "language": "python",
+   "name": "nni-dev"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.7.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

docs/_removed/WebUI.rst

+WebUI
+=====
+
+Experiments managerment
+-----------------------
+
+Click the tab ``All experiments`` on the nav bar.
+
+.. image:: ../../img/webui-img/managerExperimentList/experimentListNav.png
+   :target: ../../img/webui-img/managerExperimentList/experimentListNav.png
+   :alt: ExperimentList nav
+
+
+
+* On the ``All experiments`` page, you can see all the experiments on your machine. 
+
+.. image:: ../../img/webui-img/managerExperimentList/expList.png
+   :target: ../../img/webui-img/managerExperimentList/expList.png
+   :alt: Experiments list
+
+
+
+* When you want to see more details about an experiment you could click the trial id, look that:
+
+.. image:: ../../img/webui-img/managerExperimentList/toAnotherExp.png
+   :target: ../../img/webui-img/managerExperimentList/toAnotherExp.png
+   :alt: See this experiment detail
+
+
+
+* If has many experiments on the table, you can use the ``filter`` button.
+
+.. image:: ../../img/webui-img/managerExperimentList/expFilter.png
+   :target: ../../img/webui-img/managerExperimentList/expFilter.png
+   :alt: filter button
+
+
+
+View summary page
+-----------------
+
+Click the tab ``Overview``.
+
+
+* On the overview tab, you can see the experiment information and status and the performance of ``top trials``.
+
+
+.. image:: ../../img/webui-img/full-oview.png
+   :target: ../../img/webui-img/full-oview.png
+   :alt: overview
+
+
+
+* If you want to see experiment search space and config, please click the right button ``Search space`` and ``Config`` (when you hover on this button).
+
+   1. Search space file:
+
+
+      .. image:: ../../img/webui-img/searchSpace.png
+         :target: ../../img/webui-img/searchSpace.png
+         :alt: searchSpace
+
+
+
+   2. Config file:
+
+
+      .. image:: ../../img/webui-img/config.png
+         :target: ../../img/webui-img/config.png
+         :alt: config
+
+
+
+* You can view and download ``nni-manager/dispatcher log files`` on here.
+
+
+.. image:: ../../img/webui-img/review-log.png
+   :target: ../../img/webui-img/review-log.png
+   :alt: logfile
+
+
+
+* If your experiment has many trials, you can change the refresh interval here.
+
+
+.. image:: ../../img/webui-img/refresh-interval.png
+   :target: ../../img/webui-img/refresh-interval.png
+   :alt: refresh
+
+
+
+
+* You can review and download the experiment results(``experiment config``, ``trial message`` and ``intermeidate metrics``) when you click the button ``Experiment summary``.
+
+
+.. image:: ../../img/webui-img/summary.png
+   :target: ../../img/webui-img/summary.png
+   :alt: summary
+
+
+
+* You can change some experiment configurations such as ``maxExecDuration``, ``maxTrialNum`` and ``trial concurrency`` on here.
+
+
+.. image:: ../../img/webui-img/edit-experiment-param.png
+   :target: ../../img/webui-img/edit-experiment-param.png
+   :alt: editExperimentParams
+
+
+
+* You can click the icon to see specific error message and ``nni-manager/dispatcher log files`` by clicking ``Learn about`` link.
+
+
+.. image:: ../../img/webui-img/experimentError.png
+   :target: ../../img/webui-img/experimentError.png
+   :alt: experimentError
+
+
+
+
+* You can click ``About`` to see the version and report any questions.
+
+View job default metric
+-----------------------
+
+
+* Click the tab ``Default metric`` to see the point graph of all trials. Hover to see its specific default metric and search space message.
+
+
+.. image:: ../../img/webui-img/default-metric.png
+   :target: ../../img/webui-img/default-metric.png
+   :alt: defaultMetricGraph
+
+
+
+* Turn on the switch named ``Optimization curve`` to see the experiment's optimization curve.
+
+
+.. image:: ../../img/webui-img/best-curve.png
+   :target: ../../img/webui-img/best-curve.png
+   :alt: bestCurveGraph
+
+
+View hyper parameter
+--------------------
+
+Click the tab ``Hyper-parameter`` to see the parallel graph.
+
+
+* You can click the ``add/remove`` button to add or remove axes.
+* Drag the axes to swap axes on the chart.
+* You can select the percentage to see top trials.
+
+
+.. image:: ../../img/webui-img/hyperPara.png
+   :target: ../../img/webui-img/hyperPara.png
+   :alt: hyperParameterGraph
+
+
+
+View Trial Duration
+-------------------
+
+Click the tab ``Trial Duration`` to see the bar graph.
+
+
+.. image:: ../../img/webui-img/trial_duration.png
+   :target: ../../img/webui-img/trial_duration.png
+   :alt: trialDurationGraph
+
+
+
+View Trial Intermediate Result Graph
+------------------------------------
+
+Click the tab ``Intermediate Result`` to see the line graph.
+
+
+.. image:: ../../img/webui-img/trials_intermeidate.png
+   :target: ../../img/webui-img/trials_intermeidate.png
+   :alt: trialIntermediateGraph
+
+
+
+The trial may have many intermediate results in the training process. In order to see the trend of some trials more clearly, we set a filtering function for the intermediate result graph.
+
+You may find that these trials will get better or worse at an intermediate result. This indicates that it is an important and relevant intermediate result. To take a closer look at the point here, you need to enter its corresponding X-value at #Intermediate. Then input the range of metrics on this intermedia result. In the picture below, we choose the No. 4 intermediate result and set the range of metrics to 0.8-1.
+
+
+.. image:: ../../img/webui-img/filter-intermediate.png
+   :target: ../../img/webui-img/filter-intermediate.png
+   :alt: filterIntermediateGraph
+
+
+
+View trials status
+------------------
+
+Click the tab ``Trials Detail`` to see the status of all trials. Specifically:
+
+
+* Trial detail: trial's id, trial's duration, start time, end time, status, accuracy, and search space file.
+
+
+.. image:: ../../img/webui-img/detail-local.png
+   :target: ../../img/webui-img/detail-local.png
+   :alt: detailLocalImage
+
+
+
+* Support searching for a specific trial by its id, status, Trial No. and trial parameters.
+
+1. Trial id: 
+
+.. image:: ../../img/webui-img/detail/searchId.png
+   :target: ../../img/webui-img/detail/searchId.png
+   :alt: searchTrialId
+
+
+2. Trial No.: 
+
+.. image:: ../../img/webui-img/detail/searchNo.png
+   :target: ../../img/webui-img/detail/searchNo.png
+   :alt: searchTrialNo.
+
+
+3. Trial status:
+
+.. image:: ../../img/webui-img/detail/searchStatus.png
+   :target: ../../img/webui-img/detail/searchStatus.png
+   :alt: searchStatus
+
+4. Trial parameters:
+
+(1) parameters whose type is choice:
+
+.. image:: ../../img/webui-img/detail/searchParameterChoice.png
+   :target: ../../img/webui-img/detail/searchParameterChoice.png
+   :alt: searchParameterChoice
+
+(2) parameters whose type is not choice:
+
+.. image:: ../../img/webui-img/detail/searchParameterRange.png
+   :target: ../../img/webui-img/detail/searchParameterRange.png
+   :alt: searchParameterRange
+
+
+* The button named ``Add column`` can select which column to show on the table. If you run an experiment whose final result is a dict, you can see other keys in the table. You can choose the column ``Intermediate count`` to watch the trial's progress.
+
+
+.. image:: ../../img/webui-img/addColumn.png
+   :target: ../../img/webui-img/addColumn.png
+   :alt: addColumnGraph
+
+
+
+* If you want to compare some trials, you can select them and then click ``Compare`` to see the results.
+
+
+.. image:: ../../img/webui-img/select-trial.png
+   :target: ../../img/webui-img/select-trial.png
+   :alt: selectTrialGraph
+
+
+.. image:: ../../img/webui-img/compare.png
+   :target: ../../img/webui-img/compare.png
+   :alt: compareTrialsGraph
+
+
+* ``Tensorboard`` please refer `doc <Tensorboard.rst>`__.
+
+
+* You can use the button named ``Copy as python`` to copy the trial's parameters.
+
+
+.. image:: ../../img/webui-img/copyParameter.png
+   :target: ../../img/webui-img/copyParameter.png
+   :alt: copyTrialParameters
+
+
+
+* You could see trial logs on the tab of ``Log``. There are three buttons ``View trial log``, ``View trial error`` and ``View trial stdout`` on local mode. If you run on the OpenPAI or Kubeflow platform, you could see trial stdout and nfs log.
+
+1. local mode:
+
+.. image:: ../../img/webui-img/detail/log-local.png
+   :target: ../../img/webui-img/detail/log-local.png
+   :alt: logOnLocal
+
+
+2. OpenPAI, Kubeflow and other mode:
+
+.. image:: ../../img/webui-img/detail-pai.png
+   :target: ../../img/webui-img/detail-pai.png
+   :alt: detailPai
+
+
+* Intermediate Result Graph: you can see the default metric in this graph by clicking the intermediate button.
+
+
+.. image:: ../../img/webui-img/intermediate.png
+   :target: ../../img/webui-img/intermediate.png
+   :alt: intermeidateGraph
+
+
+
+* Kill: you can kill a job that status is running.
+
+
+.. image:: ../../img/webui-img/kill-running.png
+   :target: ../../img/webui-img/kill-running.png
+   :alt: killTrial
+
+
+
+* Customized trial: you can change this trial parameters and then submit it to the experiment. If you want to rerun a failed trial you could submit the same parameters to the experiment.
+
+.. image:: ../../img/webui-img/detail/customizedTrialButton.png
+   :target: ../../img/webui-img/detail/customizedTrialButton.png
+   :alt: customizedTrialButton
+
+
+
+.. image:: ../../img/webui-img/detail/customizedTrial.png
+   :target: ../../img/webui-img/detail/customizedTrial.png
+   :alt: customizedTrial

docs/_removed/faq.rst

+FAQ
+===
+
+This page is for frequent asked questions and answers.
+
+tmp folder fulled
+^^^^^^^^^^^^^^^^^
+
+nnictl will use tmp folder as a temporary folder to copy files under codeDir when executing experimentation creation.
+When met errors like below, try to clean up **tmp** folder first.
+
+..
+
+   OSError: [Errno 28] No space left on device
+
+
+Cannot get trials' metrics in OpenPAI mode
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+In OpenPAI training mode, we start a rest server which listens on 51189 port in NNI Manager to receive metrcis reported from trials running in OpenPAI cluster. If you didn't see any metrics from WebUI in OpenPAI mode, check your machine where NNI manager runs on to make sure 51189 port is turned on in the firewall rule.
+
+Segmentation Fault (core dumped) when installing
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. code-block:: text
+
+   make: *** [install-XXX] Segmentation fault (core dumped)
+
+
+Please try the following solutions in turn:
+
+
+* Update or reinstall you current python's pip like ``python3 -m pip install -U pip``
+* Install NNI with ``--no-cache-dir`` flag like ``python3 -m pip install nni --no-cache-dir``
+
+Job management error: getIPV4Address() failed because os.networkInterfaces().eth0 is undefined.
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Your machine don't have eth0 device, please set `nniManagerIp <ExperimentConfig.rst>`__ in your config file manually.
+
+Exceed the MaxDuration but didn't stop
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+When the duration of experiment reaches the maximum duration, nniManager will not create new trials, but the existing trials will continue unless user manually stop the experiment.
+
+Could not stop an experiment using ``nnictl stop``
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+If you upgrade your NNI or you delete some config files of NNI when there is an experiment running, this kind of issue may happen because the loss of config file. You could use ``ps -ef | grep node`` to find the PID of your experiment, and use ``kill -9 {pid}`` to kill it manually.
+
+Could not get ``default metric`` in webUI of virtual machines
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Config the network mode to bridge mode or other mode that could make virtual machine's host accessible from external machine, and make sure the port of virtual machine is not forbidden by firewall.
+
+Could not open webUI link
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Unable to open the WebUI may have the following reasons:
+
+
+* ``http://127.0.0.1``\ , ``http://172.17.0.1`` and ``http://10.0.0.15`` are referred to localhost, if you start your experiment on the server or remote machine. You can replace the IP to your server IP to view the WebUI, like ``http://[your_server_ip]:8080``
+* If you still can't see the WebUI after you use the server IP, you can check the proxy and the firewall of your machine. Or use the browser on the machine where you start your NNI experiment.
+* Another reason may be your experiment is failed and NNI may fail to get the experiment information. You can check the log of NNIManager in the following directory: ``~/nni-experiments/[your_experiment_id]`` ``/log/nnimanager.log``
+
+Restful server start failed
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Probably it's a problem with your network config. Here is a checklist.
+
+
+* You might need to link ``127.0.0.1`` with ``localhost``. Add a line ``127.0.0.1 localhost`` to ``/etc/hosts``.
+* It's also possible that you have set some proxy config. Check your environment for variables like ``HTTP_PROXY`` or ``HTTPS_PROXY`` and unset if they are set.
+
+NNI on Windows problems
+^^^^^^^^^^^^^^^^^^^^^^^
+
+Please refer to `NNI on Windows <InstallationWin.rst>`__
+
+More FAQ issues
+^^^^^^^^^^^^^^^
+
+`NNI Issues with FAQ labels <https://github.com/microsoft/nni/labels/FAQ>`__
+
+Help us improve
+^^^^^^^^^^^^^^^
+
+Please inquiry the problem in https://github.com/Microsoft/nni/issues to see whether there are other people already reported the problem, create a new one if there are no existing issues been created.

docs/_removed/installation.rst

+############
+Installation
+############
+
+Currently we support installation on Linux, Mac and Windows. We also allow you to use docker.
+
+..  toctree::
+    :maxdepth: 2
+
+    Linux & Mac <Tutorial/InstallationLinux>
+    Windows <Tutorial/InstallationWin>
+    Use Docker <Tutorial/HowToUseDocker>
\ No newline at end of file

docs/_removed/reference/python_api.rst

+:orphan:
+
+Python API Reference
+====================
+
+.. autosummary::
+   :toctree: _modules
+   :recursive:
+
+   nni

docs/_removed/training_services.rst

+Introduction to NNI Training Services
+=====================================
+
+..  toctree::
+    Overview <./TrainingService/Overview>
+    Remote<./TrainingService/RemoteMachineMode>
+    OpenPAI<./TrainingService/PaiMode>
+    Kubeflow<./TrainingService/KubeflowMode>
+    FrameworkController<./TrainingService/FrameworkControllerMode>
+    AML<./TrainingService/AMLMode>
+    Hybrid<./TrainingService/HybridMode>

docs/extension/cardlinkitem.py

+"""
+Directive "cardlinkitem" used in tutorials navigation page.
+"""
+
+import os
+
+from docutils.parsers.rst import Directive, directives
+from docutils.statemachine import StringList
+from docutils import nodes
+from sphinx.addnodes import pending_xref
+
+TAG_TEMPLATE = """<span class="card-link-tag">{tag}</span>"""
+
+TAGS_TEMPLATE = """
+    <p class="card-link-summary">{tags}</p>
+"""
+
+CARD_HEADER = """
+.. raw:: html
+
+    <div class="card-link admonition">
+
+    <a class="card-link-clickable" href="#">
+
+    <div class="card-link-body">
+
+    <div class="card-link-text">
+
+    <div class="card-link-title-container">
+        <h4>{header}</h4>
+    </div>
+
+    <p class="card-link-summary">{description}</p>
+
+    {tags}
+
+    </div>
+
+    <div class="card-link-icon circle {image_background}">
+
+.. image:: {image}
+
+.. raw:: html
+
+    </div>
+
+    </div>
+
+    </a>
+"""
+
+CARD_FOOTER = """
+.. raw:: html
+
+    </div>
+"""
+
+
+class CustomCardItemDirective(Directive):
+    option_spec = {
+        'header': directives.unchanged,
+        'image': directives.unchanged,
+        'background': directives.unchanged,
+        'link': directives.unchanged,
+        'description': directives.unchanged,
+        'tags': directives.unchanged
+    }
+
+    def run(self):
+        env = self.state.document.settings.env
+
+        try:
+            if 'header' in self.options:
+                header = self.options['header']
+            else:
+                raise ValueError('header not found')
+
+            if 'link' in self.options:
+                link = directives.uri(self.options['link'])
+            else:
+                raise ValueError('link not found')
+
+            if 'image' in self.options:
+                image = directives.uri(self.options['image'])
+            else:
+                image = os.path.join(os.path.relpath(env.app.srcdir, env.app.confdir), '../img/thumbnails/nni_icon_white.png')
+
+            image_background = self.options.get('background', 'indigo')
+            description = self.options.get('description', '')
+
+            tags = self.options.get('tags', '').strip().split('/')
+            tags = [t.strip() for t in tags if t.strip()]
+
+        except ValueError as e:
+            print(e)
+            raise
+
+        if tags:
+            tags_rst = TAGS_TEMPLATE.format(tags=''.join([TAG_TEMPLATE.format(tag=tag) for tag in tags]))
+        else:
+            tags_rst = ''
+
+        card_rst = CARD_HEADER.format(
+            header=header,
+            image=image,
+            image_background=image_background,
+            link=link,
+            description=description,
+            tags=tags_rst)
+        card = nodes.paragraph()
+        self.state.nested_parse(StringList(card_rst.split('\n')), self.content_offset, card)
+
+        # This needs to corporate with javascript: propagate_card_link.
+        # because sphinx can't correctly handle image in a `pending_xref` after `keepformat`.
+        link_node = pending_xref('<a/>',
+                                 reftype='doc',
+                                 refdomain='std',
+                                 reftarget=link,
+                                 refexplicit=False,
+                                 refwarn=True)
+        link_node += nodes.paragraph(header)
+        link_node['classes'] = ['card-link-anchor']
+        card += link_node
+
+        self.state.nested_parse(StringList(CARD_FOOTER.split('\n')), self.content_offset, card)
+
+        return [card]
+
+
+def setup(app):
+    app.add_directive('cardlinkitem', CustomCardItemDirective)

docs/extension/codesnippetcard.py

+"""
+Code snippet card, used in index page.
+"""
+
+from docutils.parsers.rst import Directive, directives
+from docutils.statemachine import StringList
+from docutils import nodes
+
+from sphinx.addnodes import pending_xref
+
+
+CARD_TEMPLATE_HEADER = """
+.. raw:: html
+
+    <div class="codesnippet-card admonition">
+
+    <div class="codesnippet-card-body">
+
+    <div class="codesnippet-card-title-container">
+
+    <div class="codesnippet-card-icon">
+
+.. image:: {icon}
+
+.. raw:: html
+
+    </div>
+
+    <h4>{title}</h4>
+    </div>
+
+"""
+
+CARD_TEMPLATE_FOOTER = """
+.. raw:: html
+
+    </div>
+"""
+
+CARD_TEMPLATE_LINK_CONTAINER_HEADER = """
+.. raw:: html
+
+    <div class="codesnippet-card-footer">
+"""
+
+CARD_TEMPLATE_LINK = """
+.. raw:: html
+
+    <div class="codesnippet-card-link">
+    {seemore}
+    <span class="material-icons right">arrow_forward</span>
+    </div>
+"""
+
+
+class CodeSnippetCardDirective(Directive):
+    option_spec = {
+        'icon': directives.unchanged,
+        'title': directives.unchanged,
+        'link': directives.unchanged,
+        'seemore': directives.unchanged,
+    }
+
+    has_content = True
+
+    def run(self):
+        anchor_node = nodes.paragraph()
+
+        try:
+            title = self.options['title']
+            link = directives.uri(self.options['link'])
+            icon = directives.uri(self.options['icon'])
+            seemore = self.options.get('seemore', 'For a full tutorial, please go here.')
+        except ValueError as e:
+            print(e)
+            raise
+
+        # header, title, icon...
+        card_rst = CARD_TEMPLATE_HEADER.format(title=title, icon=icon)
+        card_list = StringList(card_rst.split('\n'))
+        self.state.nested_parse(card_list, self.content_offset, anchor_node)
+
+        # code snippet
+        self.state.nested_parse(self.content, self.content_offset, anchor_node)
+
+        # close body
+        self.state.nested_parse(StringList(CARD_TEMPLATE_FOOTER.split('\n')), self.content_offset, anchor_node)
+
+        # start footer
+        self.state.nested_parse(StringList(CARD_TEMPLATE_LINK_CONTAINER_HEADER.split('\n')), self.content_offset, anchor_node)
+
+        # full tutorial link
+        link_node = pending_xref(CARD_TEMPLATE_LINK,
+                                 reftype='doc',
+                                 refdomain='std',
+                                 reftarget=link,
+                                 refexplicit=False,
+                                 refwarn=True,
+                                 refkeepformat=True)
+        # refkeepformat is handled in `patch_autodoc.py`
+        self.state.nested_parse(StringList(CARD_TEMPLATE_LINK.format(seemore=seemore).split('\n')), self.content_offset, link_node)
+        anchor_node += link_node
+
+        # close footer
+        self.state.nested_parse(StringList(CARD_TEMPLATE_FOOTER.split('\n')), self.content_offset, anchor_node)
+
+        # close whole
+        self.state.nested_parse(StringList(CARD_TEMPLATE_FOOTER.split('\n')), self.content_offset, anchor_node)
+
+        return [anchor_node]
+
+
+def setup(app):
+    app.add_directive('codesnippetcard', CodeSnippetCardDirective)

docs/extension/getpartialtext.py

+"""
+Basically same as
+`sphinx gettext buidler <https://www.sphinx-doc.org/en/master/_modules/sphinx/builders/gettext.html>`_,
+but only get texts from files in a whitelist.
+"""
+
+import re
+
+from docutils import nodes
+from sphinx.application import Sphinx
+from sphinx.builders.gettext import MessageCatalogBuilder
+
+
+class PartialMessageCatalogBuilder(MessageCatalogBuilder):
+    name = 'getpartialtext'
+
+    def init(self):
+        super().init()
+
+        self.whitelist_docs = [re.compile(x) for x in self.config.gettext_documents]
+
+    def write_doc(self, docname: str, doctree: nodes.document) -> None:
+        for doc_re in self.whitelist_docs:
+            if doc_re.match(docname):
+                return super().write_doc(docname, doctree)
+
+
+def setup(app: Sphinx):
+    app.add_builder(PartialMessageCatalogBuilder)
+    app.add_config_value('gettext_documents', [], 'gettext')

docs/extension/inplace_translation.py

+"""
+Sphinx inplace translation.
+Please put `xxx_zh_CN.rst` alongside `xxx.rst`. When language is set to `zh_CN`,
+`xxx_zh_CN.rst` will be used in place of `xxx.rst`.
+If translation does not exist, it will automatically fallback to the original files, without warning.
+
+I write this based on the example of:
+https://github.com/readthedocs/sphinxcontrib-multisrc/blob/master/sphinxcontrib/multisrc.py
+"""
+
+import os
+import types
+
+
+def builder_inited(app):
+    """Event listener to set up multiple environments."""
+    patch_doc2path(app.env, app.config.language)
+
+
+def patch_doc2path(env, language):
+    # patch doc2path so that it resolves to the correct language.
+    override_doc2path = env.doc2path
+
+    def doc2path(env, docname: str, base: bool = True):
+        path = override_doc2path(docname, base)
+        if language not in (None, 'en'):
+            # The language is set to another one
+            new_docname = f'{docname}_{language}'
+            new_path = override_doc2path(new_docname, base)
+            if os.path.exists(new_path):
+                return new_path
+        return path
+
+    env.doc2path = types.MethodType(doc2path, env)
+
+
+def setup(app):
+    app.connect('builder-inited', builder_inited)