create branch for v2.9

docs/source/feature_engineering/toctree.rst

+Feature Engineering
+===================
+
+..  toctree::
+    :maxdepth: 2
+
+    Overview <overview>
+    GradientFeatureSelector <gradient_feature_selector>
+    GBDTSelector <gbdt_selector>

docs/source/hpo/advanced_usage.rst

+Advanced Usage
+==============
+
+.. toctree::
+    :hidden:
+
+    Command Line Tool Example </tutorials/hpo_nnictl/nnictl>
+    Implement Custom Tuners and Assessors <custom_algorithm>
+    Install Custom or 3rd-party Tuners and Assessors <custom_algorithm_installation>
+    Tuner Benchmark <hpo_benchmark>
+    Tuner Benchmark Example Statistics <hpo_benchmark_stats>

docs/source/hpo/assessors.rst

+Assessor: Early Stopping
+========================
+
+In HPO, some hyperparameter sets may have obviously poor performance and it will be unnecessary to finish the evaluation.
+This is called *early stopping*, and in NNI early stopping algorithms are called *assessors*.
+
+An assessor monitors *intermediate results* of each *trial*.
+If a trial is predicted to produce suboptimal final result, the assessor will stop that trial immediately,
+to save computing resources for other hyperparameter sets.
+
+As introduced in quickstart tutorial, a trial is the evaluation process of a hyperparameter set,
+and intermediate results are reported with :func:`nni.report_intermediate_result` API in trial code.
+Typically, intermediate results are accuracy or loss metrics of each epoch.
+
+Using an assessor will increase the efficiency of computing resources,
+but may slightly reduce the predicition accuracy of tuners.
+It is recommended to use an assessor when computing resources are insufficient.
+
+Common Usage
+------------
+
+The usage of assessors are similar to tuners.
+
+To use a built-in assessor you need to specify its name and arguments:
+
+.. code-block:: python
+
+    config.assessor.name = 'Medianstop'
+    config.tuner.class_args = {'optimize_mode': 'maximize'}
+
+Built-in Assessors
+------------------
+
+.. list-table::
+    :header-rows: 1
+    :widths: auto
+ 
+    * - Assessor
+      - Brief Introduction of Algorithm
+ 
+    * - :class:`Median Stop <nni.algorithms.hpo.medianstop_assessor.MedianstopAssessor>`
+      - Stop if the hyperparameter set performs worse than median at any step.
+
+    * - :class:`Curve Fitting <nni.algorithms.hpo.curvefitting_assessor.CurvefittingAssessor>`
+      - Stop if the learning curve will likely converge to suboptimal result.

docs/source/hpo/custom_algorithm.rst

+Customizing Algorithms
+======================
+
+Customize Tuner
+---------------
+
+NNI provides state-of-the-art tuning algorithm in builtin-tuners. NNI supports to build a tuner by yourself for tuning demand.
+
+If you want to implement your own tuning algorithm, you can implement a customized Tuner, there are three things to do:
+
+
+#. Inherit the base Tuner class
+#. Implement receive_trial_result, generate_parameter and update_search_space function
+#. Configure your customized tuner in experiment YAML config file
+
+Here is an example:
+
+**1. Inherit the base Tuner class**
+
+.. code-block:: python
+
+   from nni.tuner import Tuner
+
+   class CustomizedTuner(Tuner):
+       def __init__(self, *args, **kwargs):
+           ...
+
+**2. Implement receive_trial_result, generate_parameter and update_search_space function**
+
+.. code-block:: python
+
+   from nni.tuner import Tuner
+
+   class CustomizedTuner(Tuner):
+       def __init__(self, *args, **kwargs):
+           ...
+
+       def receive_trial_result(self, parameter_id, parameters, value, **kwargs):
+           '''
+           Receive trial's final result.
+           parameter_id: int
+           parameters: object created by 'generate_parameters()'
+           value: final metrics of the trial, including default metric
+           '''
+           # your code implements here.
+       ...
+
+       def generate_parameters(self, parameter_id, **kwargs):
+           '''
+           Returns a set of trial (hyper-)parameters, as a serializable object
+           parameter_id: int
+           '''
+           # your code implements here.
+           return your_parameters
+       ...
+
+       def update_search_space(self, search_space):
+           '''
+           Tuners are advised to support updating search space at run-time.
+           If a tuner can only set search space once before generating first hyper-parameters,
+           it should explicitly document this behaviour.
+           search_space: JSON object created by experiment owner
+           '''
+           # your code implements here.
+       ...
+
+``receive_trial_result`` will receive the ``parameter_id, parameters, value`` as parameters input. Also, Tuner will receive the ``value`` object are exactly same value that Trial send.
+
+The ``your_parameters`` return from ``generate_parameters`` function, will be package as json object by NNI SDK. NNI SDK will unpack json object so the Trial will receive the exact same ``your_parameters`` from Tuner.
+
+For example:
+If the you implement the ``generate_parameters`` like this:
+
+.. code-block:: python
+
+   def generate_parameters(self, parameter_id, **kwargs):
+       '''
+       Returns a set of trial (hyper-)parameters, as a serializable object
+       parameter_id: int
+       '''
+       # your code implements here.
+       return {"dropout": 0.3, "learning_rate": 0.4}
+
+It means your Tuner will always generate parameters ``{"dropout": 0.3, "learning_rate": 0.4}``. Then Trial will receive ``{"dropout": 0.3, "learning_rate": 0.4}`` by calling API ``nni.get_next_parameter()``. Once the trial ends with a result (normally some kind of metrics), it can send the result to Tuner by calling API ``nni.report_final_result()``, for example ``nni.report_final_result(0.93)``. Then your Tuner's ``receive_trial_result`` function will receied the result like：
+
+.. code-block:: python
+
+   parameter_id = 82347
+   parameters = {"dropout": 0.3, "learning_rate": 0.4}
+   value = 0.93
+
+**Note that** The working directory of your tuner is ``<home>/nni-experiments/<experiment_id>/log``, which can be retrieved with environment variable ``NNI_LOG_DIRECTORY``, therefore, if you want to access a file (e.g., ``data.txt``) in the directory of your own tuner, you cannot use ``open('data.txt', 'r')``. Instead, you should use the following:
+
+.. code-block:: python
+
+   _pwd = os.path.dirname(__file__)
+   _fd = open(os.path.join(_pwd, 'data.txt'), 'r')
+
+This is because your tuner is not executed in the directory of your tuner (i.e., ``pwd`` is not the directory of your own tuner).
+
+**3. Configure your customized tuner in experiment YAML config file**
+
+NNI needs to locate your customized tuner class and instantiate the class, so you need to specify the location of the customized tuner class and pass literal values as parameters to the __init__ constructor.
+
+.. code-block:: yaml
+
+   tuner:
+     codeDir: /home/abc/mytuner
+     classFileName: my_customized_tuner.py
+     className: CustomizedTuner
+     # Any parameter need to pass to your tuner class __init__ constructor
+     # can be specified in this optional classArgs field, for example
+     classArgs:
+       arg1: value1
+
+More detail example you could see:
+
+..
+
+   * :githublink:`evolution-tuner <nni/algorithms/hpo/evolution_tuner.py>`
+   * :githublink:`hyperopt-tuner <nni/algorithms/hpo/hyperopt_tuner.py>`
+   * :githublink:`evolution-based-customized-tuner <examples/tuners/ga_customer_tuner>`
+
+
+Write a more advanced automl algorithm
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The methods above are usually enough to write a general tuner. However, users may also want more methods, for example, intermediate results, trials' state (e.g., the methods in assessor), in order to have a more powerful automl algorithm. Therefore, we have another concept called ``advisor`` which directly inherits from ``MsgDispatcherBase`` in :githublink:`msg_dispatcher_base.py <nni/runtime/msg_dispatcher_base.py>`.
+
+Customize Assessor
+------------------
+
+NNI supports to build an assessor by yourself for tuning demand.
+
+If you want to implement a customized Assessor, there are three things to do:
+
+
+#. Inherit the base Assessor class
+#. Implement assess_trial function
+#. Configure your customized Assessor in experiment YAML config file
+
+**1. Inherit the base Assessor class**
+
+.. code-block:: python
+
+   from nni.assessor import Assessor
+
+   class CustomizedAssessor(Assessor):
+       def __init__(self, *args, **kwargs):
+           ...
+
+**2. Implement assess trial function**
+
+.. code-block:: python
+
+   from nni.assessor import Assessor, AssessResult
+
+   class CustomizedAssessor(Assessor):
+       def __init__(self, *args, **kwargs):
+           ...
+
+       def assess_trial(self, trial_history):
+           """
+           Determines whether a trial should be killed. Must override.
+           trial_history: a list of intermediate result objects.
+           Returns AssessResult.Good or AssessResult.Bad.
+           """
+           # you code implement here.
+           ...
+
+**3. Configure your customized Assessor in experiment YAML config file**
+
+NNI needs to locate your customized Assessor class and instantiate the class, so you need to specify the location of the customized Assessor class and pass literal values as parameters to the __init__ constructor.
+
+.. code-block:: yaml
+
+   assessor:
+     codeDir: /home/abc/myassessor
+     classFileName: my_customized_assessor.py
+     className: CustomizedAssessor
+     # Any parameter need to pass to your Assessor class __init__ constructor
+     # can be specified in this optional classArgs field, for example
+     classArgs:
+       arg1: value1
+
+Please noted in **2**. The object ``trial_history`` are exact the object that Trial send to Assessor by using SDK ``report_intermediate_result`` function.
+
+The working directory of your assessor is ``<home>/nni-experiments/<experiment_id>/log``\ , which can be retrieved with environment variable ``NNI_LOG_DIRECTORY``\ ,
+
+More detail example you could see:
+
+* :githublink:`medianstop-assessor <nni/algorithms/hpo/medianstop_assessor.py>`
+* :githublink:`curvefitting-assessor <nni/algorithms/hpo/curvefitting_assessor/>`

docs/source/hpo/custom_algorithm_installation.rst

+How to register customized algorithms as builtin tuners, assessors and advisors
+===============================================================================
+
+Overview
+--------
+
+NNI provides a lot of :doc:`builtin tuners <tuners>`, and :doc:`assessors <assessors>` can be used directly for Hyper Parameter Optimization, and some extra algorithms can be registered via ``nnictl algo register --meta <path_to_meta_file>`` after NNI is installed. You can check builtin algorithms via ``nnictl algo list`` command.
+
+NNI also provides the ability to build your own customized tuners, advisors and assessors. To use the customized algorithm, users can simply follow the spec in experiment config file to properly reference the algorithm, which has been illustrated in the tutorials of :doc:`customized algorithms <custom_algorithm>`.
+
+NNI also allows users to install the customized algorithm as a builtin algorithm, in order for users to use the algorithm in the same way as NNI builtin tuners/advisors/assessors. More importantly, it becomes much easier for users to share or distribute their implemented algorithm to others. Customized tuners/advisors/assessors can be installed into NNI as builtin algorithms, once they are installed into NNI, you can use your customized algorithms the same way as builtin tuners/advisors/assessors in your experiment configuration file. For example, you built a customized tuner and installed it into NNI using a builtin name ``mytuner``, then you can use this tuner in your configuration file like below:
+
+.. code-block:: yaml
+
+   tuner:
+     builtinTunerName: mytuner
+
+Register customized algorithms like builtin tuners, assessors and advisors
+--------------------------------------------------------------------------
+
+You can follow below steps to build a customized tuner/assessor/advisor, and register it into NNI as builtin algorithm.
+
+1. Create a customized tuner/assessor/advisor
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Reference following instruction: :doc:`custom_algorithm`
+
+2. (Optional) Create a validator to validate classArgs
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+NNI provides a ``ClassArgsValidator`` interface for customized algorithms author to validate the classArgs parameters in experiment configuration file which are passed to customized algorithms constructors.
+The ``ClassArgsValidator`` interface is defined as:
+
+.. code-block:: python
+
+   class ClassArgsValidator(object):
+       def validate_class_args(self, **kwargs):
+           """
+           The classArgs fields in experiment configuration are packed as a dict and
+           passed to validator as kwargs.
+           """
+           pass
+
+For example, you can implement your validator such as:
+
+.. code-block:: python
+
+   from schema import Schema, Optional
+   from nni import ClassArgsValidator
+
+   class MedianstopClassArgsValidator(ClassArgsValidator):
+       def validate_class_args(self, **kwargs):
+           Schema({
+               Optional('optimize_mode'): self.choices('optimize_mode', 'maximize', 'minimize'),
+               Optional('start_step'): self.range('start_step', int, 0, 9999),
+           }).validate(kwargs)
+
+The validator will be invoked before experiment is started to check whether the classArgs fields are valid for your customized algorithms.
+
+3. Install your customized algorithms into python environment
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Firstly, the customized algorithms need to be prepared as a python package. Then you can install the package into python environment via:
+
+
+* Run command ``python setup.py develop`` from the package directory, this command will install the package in development mode, this is recommended if your algorithm is under development.
+* Run command ``python setup.py bdist_wheel`` from the package directory, this command build a whl file which is a pip installation source. Then run ``pip install <wheel file>`` to install it.
+
+4. Prepare meta file
+^^^^^^^^^^^^^^^^^^^^
+
+Create a yaml file with following keys as meta file:
+
+
+* ``algoType``: type of algorithms, could be one of ``tuner``, ``assessor``, ``advisor``
+* ``builtinName``: builtin name used in experiment configuration file
+* `className`: tuner class name, including its module name, for example: ``demo_tuner.DemoTuner``
+* `classArgsValidator`: class args validator class name, including its module name, for example: ``demo_tuner.MyClassArgsValidator``
+
+Following is an example of the yaml file:
+
+.. code-block:: yaml
+
+   algoType: tuner
+   builtinName: demotuner
+   className: demo_tuner.DemoTuner
+   classArgsValidator: demo_tuner.MyClassArgsValidator
+
+5. Register customized algorithms into NNI
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Run following command to register the customized algorithms as builtin algorithms in NNI:
+
+.. code-block:: bash
+
+   nnictl algo register --meta PATH_TO_META_FILE
+
+The ``PATH_TO_META_FILE`` is the path to the yaml file your created in above section.
+
+Reference `customized tuner example <#example-register-a-customized-tuner-as-a-builtin-tuner>`_ for a full example.
+
+Use the installed builtin algorithms in experiment
+--------------------------------------------------
+
+Once your customized algorithms is installed, you can use it in experiment configuration file the same way as other builtin tuners/assessors/advisors, for example:
+
+.. code-block:: yaml
+
+   tuner:
+     builtinTunerName: demotuner
+     classArgs:
+       #choice: maximize, minimize
+       optimize_mode: maximize
+
+Manage builtin algorithms using ``nnictl algo``
+-----------------------------------------------
+
+List builtin algorithms
+^^^^^^^^^^^^^^^^^^^^^^^
+
+Run following command to list the registered builtin algorithms:
+
+.. code-block:: text
+
+   nnictl algo list
+   +-----------------+------------+-----------+--------=-------------+------------------------------------------+
+   |      Name       |    Type    | Source    |      Class Name      |               Module Name                |
+   +-----------------+------------+-----------+----------------------+------------------------------------------+
+   | TPE             | tuners     | nni       | HyperoptTuner        | nni.hyperopt_tuner.hyperopt_tuner        |
+   | Random          | tuners     | nni       | HyperoptTuner        | nni.hyperopt_tuner.hyperopt_tuner        |
+   | Anneal          | tuners     | nni       | HyperoptTuner        | nni.hyperopt_tuner.hyperopt_tuner        |
+   | Evolution       | tuners     | nni       | EvolutionTuner       | nni.evolution_tuner.evolution_tuner      |
+   | BatchTuner      | tuners     | nni       | BatchTuner           | nni.batch_tuner.batch_tuner              |
+   | GridSearch      | tuners     | nni       | GridSearchTuner      | nni.gridsearch_tuner.gridsearch_tuner    |
+   | NetworkMorphism | tuners     | nni       | NetworkMorphismTuner | nni.networkmorphism_tuner.networkmo...   |
+   | MetisTuner      | tuners     | nni       | MetisTuner           | nni.metis_tuner.metis_tuner              |
+   | GPTuner         | tuners     | nni       | GPTuner              | nni.gp_tuner.gp_tuner                    |
+   | PBTTuner        | tuners     | nni       | PBTTuner             | nni.pbt_tuner.pbt_tuner                  |
+   | SMAC            | tuners     | nni       | SMACTuner            | nni.smac_tuner.smac_tuner                |
+   | PPOTuner        | tuners     | nni       | PPOTuner             | nni.ppo_tuner.ppo_tuner                  |
+   | Medianstop      | assessors  | nni       | MedianstopAssessor   | nni.medianstop_assessor.medianstop_...   |
+   | Curvefitting    | assessors  | nni       | CurvefittingAssessor | nni.curvefitting_assessor.curvefitt...   |
+   | Hyperband       | advisors   | nni       | Hyperband            | nni.hyperband_advisor.hyperband_adv...   |
+   | BOHB            | advisors   | nni       | BOHB                 | nni.bohb_advisor.bohb_advisor            |
+   +-----------------+------------+-----------+----------------------+------------------------------------------+
+
+Unregister builtin algorithms
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Run following command to uninstall an installed package:
+
+``nnictl algo unregister <builtin name>``
+
+For example:
+
+``nnictl algo unregister demotuner``
+
+
+Porting customized algorithms from v1.x to v2.x
+-----------------------------------------------
+
+All that needs to be modified is to delete ``NNI Package :: tuner`` metadata in ``setup.py`` and add a meta file mentioned in `4. Prepare meta file`_.
+Then you can follow `Register customized algorithms like builtin tuners, assessors and advisors`_ to register your customized algorithms.
+
+Example: Register a customized tuner as a builtin tuner
+-------------------------------------------------------
+
+You can following below steps to register a customized tuner in ``nni/examples/tuners/customized_tuner`` as a builtin tuner.
+
+Install the customized tuner package into python environment
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+There are 2 options to install the package into python environment:
+
+Option 1: install from directory
+""""""""""""""""""""""""""""""""
+
+From ``nni/examples/tuners/customized_tuner`` directory, run:
+
+``python setup.py develop``
+
+This command will build the ``nni/examples/tuners/customized_tuner`` directory as a pip installation source.
+
+Option 2: install from whl file
+"""""""""""""""""""""""""""""""
+
+Step 1: From ``nni/examples/tuners/customized_tuner`` directory, run:
+
+``python setup.py bdist_wheel``
+
+This command build a whl file which is a pip installation source.
+
+Step 2: Run command:
+
+``pip install dist/demo_tuner-0.1-py3-none-any.whl``
+
+Register the customized tuner as builtin tuner:
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Run following command:
+
+``nnictl algo register --meta meta_file.yml``
+
+Check the registered builtin algorithms
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Then run command ``nnictl algo list``\ , you should be able to see that demotuner is installed:
+
+.. code-block:: text
+
+   +-----------------+------------+-----------+--------=-------------+------------------------------------------+
+   |      Name       |    Type    |   source  |      Class Name      |               Module Name                |
+   +-----------------+------------+-----------+----------------------+------------------------------------------+
+   | demotuner       | tuners     |    User   | DemoTuner            | demo_tuner                               |
+   +-----------------+------------+-----------+----------------------+------------------------------------------+

docs/source/hpo/hpo_benchmark.rst

+HPO Benchmarks
+==============
+
+We provide a benchmarking tool to compare the performances of tuners provided by NNI (and users' custom tuners) on different
+types of tasks. This tool uses the `automlbenchmark repository <https://github.com/openml/automlbenchmark>`_  to run different *benchmarks* on the NNI *tuners*.
+The tool is located in ``examples/trials/benchmarking/automlbenchmark``. This document provides a brief introduction to the tool, its usage, and currently available benchmarks.
+
+Overview and Terminologies
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Ideally, an **HPO Benchmark** provides a tuner with a search space, calls the tuner repeatedly, and evaluates how the tuner probes
+the search space and approaches to good solutions. In addition, inside the benchmark, an evaluator should be associated to
+each search space for evaluating the score of points in this search space to give feedbacks to the tuner. For instance,
+the search space could be the space of hyperparameters for a neural network. Then the evaluator should contain train data,
+test data, and a criterion. To evaluate a point in the search space, the evaluator will train the network on the train data
+and report the score of the model on the test data as the score for the point.
+
+However, a **benchmark** provided by the automlbenchmark repository only provides part of the functionality of the evaluator.
+More concretely, it assumes that it is evaluating a **framework**. Different from a tuner, given train data, a **framework**
+can directly solve a **task** and predict on the test set. The **benchmark** from the automlbenchmark repository directly provides
+train and test datasets to a **framework**, evaluates the prediction on the test set, and reports this score as the final score.
+Therefore, to implement **HPO Benchmark** using automlbenchmark, we pair up a tuner with a search space to form a **framework**,
+and handle the repeated trial-evaluate-feedback loop in the **framework** abstraction. In other words, each **HPO Benchmark**
+contains two main components: a **benchmark** from the automlbenchmark library, and an **architecture** which defines the search
+space and the evaluator. To further clarify, we provide the definition for the terminologies used in this document.
+
+* **tuner**\ : a :doc:`tuner or advisor provided by NNI <tuners>`, or a custom tuner provided by the user.
+* **task**\ : an abstraction used by automlbenchmark. A task can be thought of as a tuple (dataset, metric). It provides train and test datasets to the frameworks. Then, based on the returns predictions on the test set, the task evaluates the metric (e.g., mse for regression, f1 for classification) and reports the score.
+* **benchmark**\ : an abstraction used by automlbenchmark. A benchmark is a set of tasks, along with other external constraints such as time limits.
+* **framework**\ : an abstraction used by automlbenchmark. Given a task, a framework solves the proposed regression or classification problem using train data and produces predictions on the test set. In our implementation, each framework is an architecture, which defines a search space. To evaluate a task given by the benchmark on a specific tuner, we let the tuner continuously tune the hyperparameters (by giving it cross-validation score on the train data as feedback) until the time or trial limit is reached. Then, the architecture is retrained on the entire train set using the best set of hyperparameters.
+* **architecture**\ : an architecture is a specific method for solving the tasks, along with a set of hyperparameters to optimize (i.e., the search space). See ``./nni/extensions/NNI/architectures`` for examples.
+
+Supported HPO Benchmarks
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+From the previous discussion, we can see that to define an **HPO Benchmark**, we need to specify a **benchmark** and an **architecture**.
+
+Currently, the only architectures we support are random forest and MLP. We use the
+`scikit-learn implementation <https://scikit-learn.org/stable/modules/classes.html#>`_. Typically, there are a number of
+hyperparameters that may directly affect the performances of random forest and MLP models. We design the search
+spaces to be the following.
+
+Search Space for Random Forest:
+
+.. code-block:: json
+
+   {
+       "n_estimators": {"_type":"randint", "_value": [4, 2048]},
+       "max_depth": {"_type":"choice", "_value": [4, 8, 16, 32, 64, 128, 256, 0]},
+       "min_samples_leaf": {"_type":"randint", "_value": [1, 8]},
+       "min_samples_split": {"_type":"randint", "_value": [2, 16]},
+       "max_leaf_nodes": {"_type":"randint", "_value": [0, 4096]}
+    }
+
+Search Space for MLP:
+
+.. code-block:: json
+
+    {
+       "hidden_layer_sizes": {"_type":"choice", "_value": [[16], [64], [128], [256], [16, 16], [64, 64], [128, 128], [256, 256], [16, 16, 16], [64, 64, 64], [128, 128, 128], [256, 256, 256], [256, 128, 64, 16], [128, 64, 16], [64, 16], [16, 64, 128, 256], [16, 64, 128], [16, 64]]},
+       "learning_rate_init": {"_type":"choice", "_value": [0.1, 0.05, 0.01, 0.005, 0.001, 0.0005, 0.0001, 0.00005, 0.00001]},
+       "alpha": {"_type":"choice", "_value": [0.1, 0.05, 0.01, 0.005, 0.001, 0.0005, 0.0001]},
+       "momentum": {"_type":"uniform","_value":[0, 1]},
+       "beta_1": {"_type":"uniform","_value":[0, 1]},
+       "tol": {"_type":"choice", "_value": [0.001, 0.0005, 0.0001, 0.00005, 0.00001]},
+       "max_iter": {"_type":"randint", "_value": [2, 256]}
+    }
+
+In addition, we write the search space in different ways (e.g., using "choice" or "randint" or "loguniform").
+The architecture implementation and search space definition can be found in ``./nni/extensions/NNI/architectures/``.
+You may replace the search space definition in this file to experiment different search spaces.
+
+For the automlbenchmarks, in addition to the built-in benchmarks provided by automl
+(defined in ``/examples/trials/benchmarking/automlbenchmark/automlbenchmark/resources/benchmarks/``), we design several
+additional benchmarks, defined in ``/examples/trials/benchmarking/automlbenchmark/nni/benchmarks``.
+One example of larger benchmarks is "nnismall", which consists of 8 regression tasks, 8 binary classification tasks, and
+8 multi-class classification tasks. We also provide three separate 8-task benchmarks "nnismall-regression", "nnismall-binary", and "nnismall-multiclass"
+corresponding to the three types of tasks in nnismall. These tasks are suitable to solve with random forest and MLP.
+
+The following table summarizes the benchmarks we provide. For ``nnismall``, please check ``/examples/trials/benchmarking/automlbenchmark/automlbenchmark/resources/benchmarks/``
+for a more detailed description for each task. Also, since all tasks are from the OpenML platform, you can find the descriptions
+of all datasets at `this webpage <https://www.openml.org/search?type=data>`_.
+
+.. list-table::
+   :header-rows: 1
+   :widths: 1 2 2 2
+
+   * - Benchmark name
+     - Description
+     - Task List
+     - Location
+   * - nnivalid
+     - A three-task benchmark to validate benchmark installation.
+     - ``kc2, iris, cholesterol``
+     - ``/examples/trials/benchmarking/automlbenchmark/nni/benchmarks/``
+   * - nnismall-regression
+     - An eight-task benchmark consisting of **regression** tasks only.
+     - ``cholesterol, liver-disorders, kin8nm, cpu_small, titanic_2, boston, stock, space_ga``
+     - ``/examples/trials/benchmarking/automlbenchmark/nni/benchmarks/``
+   * - nnismall-binary
+     - An eight-task benchmark consisting of **binary classification** tasks only.
+     - ``Australian, blood-transfusion, christine, credit-g, kc1, kr-vs-kp, phoneme, sylvine``
+     - ``/examples/trials/benchmarking/automlbenchmark/nni/benchmarks/``
+   * - nnismall-multiclass
+     - An eight-task benchmark consisting of **multi-class classification** tasks only.
+     - ``car, cnae-9, dilbert, fabert, jasmine, mfeat-factors, segment, vehicle``
+     - ``/examples/trials/benchmarking/automlbenchmark/nni/benchmarks/``
+   * - nnismall
+     - A 24-task benchmark that is the superset of nnismall-regression, nnismall-binary, and nnismall-multiclass.
+     - ``cholesterol, liver-disorders, kin8nm, cpu_small, titanic_2, boston, stock, space_ga, Australian, blood-transfusion, christine, credit-g, kc1, kr-vs-kp, phoneme, sylvine, car, cnae-9, dilbert, fabert, jasmine, mfeat-factors, segment, vehicle``
+     - ``/examples/trials/benchmarking/automlbenchmark/nni/benchmarks/``
+
+Setup
+^^^^^
+
+Due to some incompatibilities between automlbenchmark and python 3.8, python 3.7 is recommended for running experiments contained in this folder. First, run the following shell script to clone the automlbenchmark repository. Note: it is recommended to perform the following steps in a separate virtual environment, as the setup code may install several packages. 
+
+.. code-block:: bash
+
+   ./setup.sh
+
+Run predefined benchmarks on existing tuners
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. code-block:: bash
+
+   ./runbenchmark_nni.sh [tuner-names]
+
+This script runs the benchmark 'nnivalid', which consists of a regression task, a binary classification task, and a
+multi-class classification task. After the script finishes, you can find a summary of the results in the folder results_[time]/reports/.
+To run on other predefined benchmarks, change the ``benchmark`` variable in ``runbenchmark_nni.sh``. To change to another
+search space (by using another architecture), chang the `arch_type` parameter in ``./nni/frameworks.yaml``. Note that currently,
+we only support ``random_forest`` or ``mlp`` as the `arch_type`. To experiment on other search spaces with the same
+architecture, please change the search space defined in ``./nni/extensions/NNI/architectures/run_[architecture].py``.
+
+The ``./nni/frameworks.yaml`` is the actual configuration file for the HPO Benchmark. The ``limit_type`` parameter specifies
+the limits for running the benchmark on one tuner. If ``limit_type`` is set to `ntrials`, then the tuner is called for
+`trial_limit` times and then stopped. If ``limit_type`` is set to `time`, then the tuner is continuously called until
+timeout for the benchmark is reached. The timeout for the benchmarks can be changed in the each benchmark file located
+in ``./nni/benchmarks``.
+
+By default, the script runs the benchmark on all embedded tuners in NNI. If provided a list of tuners in [tuner-names],
+it only runs the tuners in the list. Currently, the following tuner names are supported: "TPE", "Random", "Anneal",
+"Evolution", "SMAC", "GPTuner", "MetisTuner", "DNGOTuner", "Hyperband", "BOHB". It is also possible to run the benchmark
+on custom tuners. See the next sections for details.
+
+By default, the script runs the specified tuners against the specified benchmark one by one. To run the experiment for
+all tuners simultaneously in the background, set the "serialize" flag to false in ``runbenchmark_nni.sh``.
+
+Note: the SMAC tuner, DNGO tuner, and the BOHB advisor has to be manually installed before running benchmarks on them.
+Please refer to :doc:`this page <tuners>` for more details
+on installation.
+
+Run customized benchmarks on existing tuners
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+You can design your own benchmarks and evaluate the performance of NNI tuners on them. To run customized benchmarks,
+add a benchmark_name.yaml file in the folder ``./nni/benchmarks``, and change the ``benchmark`` variable in ``runbenchmark_nni.sh``.
+See ``./automlbenchmark/resources/benchmarks/`` for some examples of defining a custom benchmark.
+
+Run benchmarks on custom tuners
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+You may also use the benchmark to compare a custom tuner written by yourself with the NNI built-in tuners. To use custom
+tuners, first make sure that the tuner inherits from ``nni.tuner.Tuner`` and correctly implements the required APIs. For
+more information on implementing a custom tuner, please refer to :doc:`here <custom_algorithm>`.
+Next, perform the following steps:
+
+#. Install the custom tuner via the command ``nnictl algo register``. Check :doc:`this document <../reference/nnictl>` for details. 
+#. In ``./nni/frameworks.yaml``\ , add a new framework extending the base framework NNI. Make sure that the parameter ``tuner_type`` corresponds to the "builtinName" of tuner installed in step 1.
+#. Run the following command
+
+.. code-block:: bash
+
+      ./runbenchmark_nni.sh new-tuner-builtinName
+
+The benchmark will automatically find and match the tuner newly added to your NNI installation.

docs/source/hpo/hpo_benchmark_stats.rst

+HPO Benchmark Example Statistics
+================================
+
+A Benchmark Example
+^^^^^^^^^^^^^^^^^^^
+
+As an example, we ran the "nnismall" benchmark with the random forest search space on the following 8 tuners: "TPE",
+"Random", "Anneal", "Evolution", "SMAC", "GPTuner", "MetisTuner", "DNGOTuner". For convenience of reference, we also list
+the search space we experimented on here. Note that the way in which the search space is written may significantly affect
+hyperparameter optimization performance, and we plan to conduct further experiments on how well NNI built-in tuners adapt
+to different search space formulations using this benchmarking tool.
+
+.. code-block:: json
+
+   {
+       "n_estimators": {"_type":"randint", "_value": [8, 512]},
+       "max_depth": {"_type":"choice", "_value": [4, 8, 16, 32, 64, 128, 256, 0]},
+       "min_samples_leaf": {"_type":"randint", "_value": [1, 8]},
+       "min_samples_split": {"_type":"randint", "_value": [2, 16]},
+       "max_leaf_nodes": {"_type":"randint", "_value": [0, 4096]}
+    }
+
+As some of the tasks contains a considerable amount of training data, it took about 2 days to run the whole benchmark on
+one tuner. For a more detailed description of the tasks, please check
+``/examples/trials/benchmarking/automlbenchmark/nni/benchmarks/nnismall_description.txt``. For binary and multi-class
+classification tasks, the metric "auc" and "logloss" were used for evaluation, while for regression, "r2" and "rmse" were used.
+
+After the script finishes, the final scores of each tuner are summarized in the file ``results[time]/reports/performances.txt``.
+Since the file is large, we only show the following screenshot and summarize other important statistics instead.
+
+.. image:: ../../img/hpo_benchmark/performances.png
+   :target: ../../img/hpo_benchmark/performances.png
+   :alt:
+
+When the results are parsed, the tuners are also ranked based on their final performance. The following three tables show
+the average ranking of the tuners for each metric (logloss, rmse, auc).
+
+Also, for every tuner, their performance for each type of metric is summarized (another view of the same data).
+We present this statistics in the fourth table. Note that this information can be found at ``results[time]/reports/rankings.txt``.
+
+Average rankings for metric rmse (for regression tasks). We found that Anneal performs the best among all NNI built-in tuners.
+
+.. list-table::
+   :header-rows: 1
+
+   * - Tuner Name
+     - Average Ranking
+   * - Anneal
+     - 3.75
+   * - Random
+     - 4.00
+   * - Evolution
+     - 4.44
+   * - DNGOTuner
+     - 4.44
+   * - SMAC
+     - 4.56
+   * - TPE
+     - 4.94
+   * - GPTuner
+     - 4.94
+   * - MetisTuner
+     - 4.94
+
+Average rankings for metric auc (for classification tasks). We found that SMAC performs the best among all NNI built-in tuners.
+
+.. list-table::
+   :header-rows: 1
+
+   * - Tuner Name
+     - Average Ranking
+   * - SMAC
+     - 3.67
+   * - GPTuner
+     - 4.00
+   * - Evolution
+     - 4.22
+   * - Anneal
+     - 4.39
+   * - MetisTuner
+     - 4.39
+   * - TPE
+     - 4.67
+   * - Random
+     - 5.33
+   * - DNGOTuner
+     - 5.33
+
+Average rankings for metric logloss (for classification tasks). We found that Random performs the best among all NNI built-in tuners.
+
+.. list-table::
+   :header-rows: 1
+
+   * - Tuner Name
+     - Average Ranking
+   * - Random
+     - 3.36
+   * - DNGOTuner
+     - 3.50
+   * - SMAC
+     - 3.93
+   * - GPTuner
+     - 4.64
+   * - TPE
+     - 4.71
+   * - Anneal
+     - 4.93
+   * - Evolution
+     - 5.00
+   * - MetisTuner
+     - 5.93
+
+To view the same data in another way, for each tuner, we present the average rankings on different types of metrics. From the table, we can find that, for example, the DNGOTuner performs better for the tasks whose metric is "logloss" than for the tasks with metric "auc". We hope this information can to some extent guide the choice of tuners given some knowledge of task types.
+
+.. list-table::
+   :header-rows: 1
+
+   * - Tuner Name
+     - rmse
+     - auc
+     - logloss
+   * - TPE
+     - 4.94
+     - 4.67
+     - 4.71
+   * - Random
+     - 4.00
+     - 5.33
+     - 3.36
+   * - Anneal
+     - 3.75
+     - 4.39
+     - 4.93
+   * - Evolution
+     - 4.44
+     - 4.22
+     - 5.00
+   * - GPTuner
+     - 4.94
+     - 4.00
+     - 4.64
+   * - MetisTuner
+     - 4.94
+     - 4.39
+     - 5.93
+   * - SMAC
+     - 4.56
+     - 3.67
+     - 3.93
+   * - DNGOTuner
+     - 4.44
+     - 5.33
+     - 3.50
+
+Besides these reports, our script also generates two graphs for each fold of each task: one graph presents the best score received by each tuner until trial x, and another graph shows the score that each tuner receives in trial x. These two graphs can give some information regarding how the tuners are "converging" to their final solution. We found that for "nnismall", tuners on the random forest model with search space defined in ``/examples/trials/benchmarking/automlbenchmark/nni/extensions/NNI/architectures/run_random_forest.py`` generally converge to the final solution after 40 to 60 trials. As there are too much graphs to incldue in a single report (96 graphs in total), we only present 10 graphs here.
+
+.. image:: ../../img/hpo_benchmark/car_fold1_1.jpg
+   :target: ../../img/hpo_benchmark/car_fold1_1.jpg
+   :alt:
+
+
+.. image:: ../../img/hpo_benchmark/car_fold1_2.jpg
+   :target: ../../img/hpo_benchmark/car_fold1_2.jpg
+   :alt:
+
+The previous two graphs are generated for fold 1 of the task "car". In the first graph, we observe that most tuners find a relatively good solution within 40 trials. In this experiment, among all tuners, the DNGOTuner converges fastest to the best solution (within 10 trials). Its best score improved for three times in the entire experiment. In the second graph, we observe that most tuners have their score flucturate between 0.8 and 1 throughout the experiment. However, it seems that the Anneal tuner (green line) is more unstable (having more fluctuations) while the GPTuner has a more stable pattern. This may be interpreted as the Anneal tuner explores more aggressively than the GPTuner and thus its scores for different trials vary a lot. Regardless, although this pattern can to some extent hint a tuner's position on the explore-exploit tradeoff, it is not a comprehensive evaluation of a tuner's effectiveness.
+
+.. image:: ../../img/hpo_benchmark/christine_fold0_1.jpg
+   :target: ../../img/hpo_benchmark/christine_fold0_1.jpg
+   :alt:
+
+
+.. image:: ../../img/hpo_benchmark/christine_fold0_2.jpg
+   :target: ../../img/hpo_benchmark/christine_fold0_2.jpg
+   :alt:
+
+
+.. image:: ../../img/hpo_benchmark/cnae-9_fold0_1.jpg
+   :target: ../../img/hpo_benchmark/cnae-9_fold0_1.jpg
+   :alt:
+
+
+.. image:: ../../img/hpo_benchmark/cnae-9_fold0_2.jpg
+   :target: ../../img/hpo_benchmark/cnae-9_fold0_2.jpg
+   :alt:
+
+
+.. image:: ../../img/hpo_benchmark/credit-g_fold1_1.jpg
+   :target: ../../img/hpo_benchmark/credit-g_fold1_1.jpg
+   :alt:
+
+
+.. image:: ../../img/hpo_benchmark/credit-g_fold1_2.jpg
+   :target: ../../img/hpo_benchmark/credit-g_fold1_2.jpg
+   :alt:
+
+
+.. image:: ../../img/hpo_benchmark/titanic_2_fold1_1.jpg
+   :target: ../../img/hpo_benchmark/titanic_2_fold1_1.jpg
+   :alt:
+
+
+.. image:: ../../img/hpo_benchmark/titanic_2_fold1_2.jpg
+   :target: ../../img/hpo_benchmark/titanic_2_fold1_2.jpg
+   :alt:

docs/source/hpo/nni_annotation.rst

+:orphan:
+
+NNI Annotation
+==============
+
+Overview
+--------
+
+To improve user experience and reduce user effort, we design an annotation grammar. Using NNI annotation, users can adapt their code to NNI just by adding some standalone annotating strings, which does not affect the execution of the original code.
+
+Below is an example:
+
+.. code-block:: python
+
+   '''@nni.variable(nni.choice(0.1, 0.01, 0.001), name=learning_rate)'''
+   learning_rate = 0.1
+
+The meaning of this example is that NNI will choose one of several values (0.1, 0.01, 0.001) to assign to the learning_rate variable. Specifically, this first line is an NNI annotation, which is a single string. Following is an assignment statement. What nni does here is to replace the right value of this assignment statement according to the information provided by the annotation line.
+
+In this way, users could either run the python code directly or launch NNI to tune hyper-parameter in this code, without changing any codes.
+
+Types of Annotation:
+--------------------
+
+In NNI, there are mainly four types of annotation:
+
+1. Annotate variables
+^^^^^^^^^^^^^^^^^^^^^
+
+   ``'''@nni.variable(sampling_algo, name)'''``
+
+``@nni.variable`` is used in NNI to annotate a variable.
+
+**Arguments**
+
+
+* **sampling_algo**\ : Sampling algorithm that specifies a search space. User should replace it with a built-in NNI sampling function whose name consists of an ``nni.`` identification and a search space type specified in :doc:`SearchSpaceSpec <search_space>` such as ``choice`` or ``uniform``.
+* **name**\ : The name of the variable that the selected value will be assigned to. Note that this argument should be the same as the left value of the following assignment statement.
+
+There are 10 types to express your search space as follows:
+
+
+* ``@nni.variable(nni.choice(option1,option2,...,optionN),name=variable)``
+  Which means the variable value is one of the options, which should be a list The elements of options can themselves be stochastic expressions
+* ``@nni.variable(nni.randint(lower, upper),name=variable)``
+  Which means the variable value is a value like round(uniform(low, high)). For now, the type of chosen value is float. If you want to use integer value, please convert it explicitly.
+* ``@nni.variable(nni.uniform(low, high),name=variable)``
+  Which means the variable value is a value uniformly between low and high.
+* ``@nni.variable(nni.quniform(low, high, q),name=variable)``
+  Which means the variable value is a value like clip(round(uniform(low, high) / q) * q, low, high), where the clip operation is used to constraint the generated value in the bound.
+* ``@nni.variable(nni.loguniform(low, high),name=variable)``
+  Which means the variable value is a value drawn according to exp(uniform(low, high)) so that the logarithm of the return value is uniformly distributed.
+* ``@nni.variable(nni.qloguniform(low, high, q),name=variable)``
+  Which means the variable value is a value like clip(round(loguniform(low, high) / q) * q, low, high), where the clip operation is used to constraint the generated value in the bound.
+* ``@nni.variable(nni.normal(mu, sigma),name=variable)``
+  Which means the variable value is a real value that's normally-distributed with mean mu and standard deviation sigma.
+* ``@nni.variable(nni.qnormal(mu, sigma, q),name=variable)``
+  Which means the variable value is a value like round(normal(mu, sigma) / q) * q
+* ``@nni.variable(nni.lognormal(mu, sigma),name=variable)``
+  Which means the variable value is a value drawn according to exp(normal(mu, sigma))
+* ``@nni.variable(nni.qlognormal(mu, sigma, q),name=variable)``
+  Which means the variable value is a value like round(exp(normal(mu, sigma)) / q) * q
+
+Below is an example:
+
+.. code-block:: python
+
+   '''@nni.variable(nni.choice(0.1, 0.01, 0.001), name=learning_rate)'''
+   learning_rate = 0.1
+
+2. Annotate functions
+^^^^^^^^^^^^^^^^^^^^^
+
+   ``'''@nni.function_choice(*functions, name)'''``
+
+``@nni.function_choice`` is used to choose one from several functions.
+
+**Arguments**
+
+
+* **functions**\ : Several functions that are waiting to be selected from. Note that it should be a complete function call with arguments. Such as ``max_pool(hidden_layer, pool_size)``.
+* **name**\ : The name of the function that will be replaced in the following assignment statement.
+
+An example here is:
+
+.. code-block:: python
+
+   """@nni.function_choice(max_pool(hidden_layer, pool_size), avg_pool(hidden_layer, pool_size), name=max_pool)"""
+   h_pooling = max_pool(hidden_layer, pool_size)
+
+3. Annotate intermediate result
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+   ``'''@nni.report_intermediate_result(metrics)'''``
+
+``@nni.report_intermediate_result`` is used to report intermediate result, whose usage is the same as :func:`nni.report_intermediate_result`.
+
+4. Annotate final result
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+   ``'''@nni.report_final_result(metrics)'''``
+
+``@nni.report_final_result`` is used to report the final result of the current trial, whose usage is the same as :func:`nni.report_final_result`.

docs/source/hpo/overview.rst

+Hyperparameter Optimization Overview
+====================================
+
+Auto hyperparameter optimization (HPO), or auto tuning, is one of the key features of NNI.
+
+Introduction to HPO
+-------------------
+
+In machine learning, a hyperparameter is a parameter whose value is used to control learning process,
+and HPO is the problem of choosing a set of optimal hyperparameters for a learning algorithm.
+(`From <https://en.wikipedia.org/wiki/Hyperparameter_(machine_learning)>`__
+`Wikipedia <https://en.wikipedia.org/wiki/Hyperparameter_optimization>`__)
+
+Following code snippet demonstrates a naive HPO process:
+
+.. code-block:: python
+
+    best_hyperparameters = None
+    best_accuracy = 0
+
+    for learning_rate in [0.1, 0.01, 0.001, 0.0001]:
+        for momentum in [i / 10 for i in range(10)]:
+            for activation_type in ['relu', 'tanh', 'sigmoid']:
+                model = build_model(activation_type)
+                train_model(model, learning_rate, momentum)
+                accuracy = evaluate_model(model)
+
+                if accuracy > best_accuracy:
+                    best_accuracy = accuracy
+                    best_hyperparameters = (learning_rate, momentum, activation_type)
+
+    print('Best hyperparameters:', best_hyperparameters)
+
+You may have noticed, the example will train 4×10×3=120 models in total.
+Since it consumes so much computing resources, you may want to:
+
+1. :ref:`Find the best hyperparameter set with less iterations. <hpo-overview-tuners>`
+2. :ref:`Train the models on distributed platforms. <hpo-overview-platforms>`
+3. :ref:`Have a portal to monitor and control the process. <hpo-overview-portal>`
+
+NNI will do them for you.
+
+Key Features of NNI HPO
+-----------------------
+
+.. _hpo-overview-tuners:
+
+Tuning Algorithms
+^^^^^^^^^^^^^^^^^
+
+NNI provides *tuners* to speed up the process of finding best hyperparameter set.
+
+A tuner, or a tuning algorithm, decides the order in which hyperparameter sets are evaluated.
+Based on the results of historical hyperparameter sets, an efficient tuner can predict where the best hyperparameters locates around,
+and finds them in much fewer attempts.
+
+The naive example above evaluates all possible hyperparameter sets in constant order, ignoring the historical results.
+This is the brute-force tuning algorithm called *grid search*.
+
+NNI has out-of-the-box support for a variety of popular tuners.
+It includes naive algorithms like random search and grid search, Bayesian-based algorithms like TPE and SMAC,
+RL based algorithms like PPO, and much more.
+
+Main article: :doc:`tuners`
+
+.. _hpo-overview-platforms:
+
+Training Platforms
+^^^^^^^^^^^^^^^^^^
+
+If you are not interested in distributed platforms, you can simply run NNI HPO with current computer,
+just like any ordinary Python library.
+
+And when you want to leverage more computing resources, NNI provides built-in integration for training platforms
+from simple on-premise servers to scalable commercial clouds.
+
+With NNI you can write one piece of model code, and concurrently evaluate hyperparameter sets on local machine, SSH servers,
+Kubernetes-based clusters, AzureML service, and much more.
+
+Main article: :doc:`/experiment/training_service/overview`
+
+.. _hpo-overview-portal:
+
+Web Portal
+^^^^^^^^^^
+
+NNI provides a web portal to monitor training progress, to visualize hyperparameter performance,
+to manually customize hyperparameters, and to manage multiple HPO experiments.
+
+Main article: :doc:`/experiment/web_portal/web_portal`
+
+.. image:: ../../static/img/webui.gif
+    :width: 100%
+
+Tutorials
+---------
+
+To start using NNI HPO, choose the quickstart tutorial of your favorite framework:
+
+* :doc:`PyTorch tutorial </tutorials/hpo_quickstart_pytorch/main>`
+* :doc:`TensorFlow tutorial </tutorials/hpo_quickstart_tensorflow/main>`
+
+Extra Features
+--------------
+
+After you are familiar with basic usage, you can explore more HPO features:
+
+* :doc:`Use command line tool to create and manage experiments (nnictl) </reference/nnictl>`
+
+  * :doc:`nnictl example </tutorials/hpo_nnictl/nnictl>`
+
+* :doc:`Early stop non-optimal models (assessor) <assessors>`
+* :doc:`TensorBoard integration </experiment/web_portal/tensorboard>`
+* :doc:`Implement your own algorithm <custom_algorithm>`
+* :doc:`Benchmark tuners <hpo_benchmark>`

docs/source/hpo/overview_zh.rst

+.. c74f6d072f5f8fa93eadd214bba992b4
+
+超参调优
+========
+
+自动超参调优（hyperparameter optimization, HPO）是 NNI 的主要功能之一。
+
+超参调优简介
+------------
+
+在机器学习中，用来控制学习过程的参数被称为“超参数”或“超参”，而为一种机器学习算法选择最优超参组合的问题被称为“超参调优”。
+
+以下代码片段演示了一次朴素的超参调优：
+
+.. code-block:: python
+
+    best_hyperparameters = None
+    best_accuracy = 0
+
+    for learning_rate in [0.1, 0.01, 0.001, 0.0001]:
+        for momentum in [i / 10 for i in range(10)]:
+            for activation_type in ['relu', 'tanh', 'sigmoid']:
+                model = build_model(activation_type)
+                train_model(model, learning_rate, momentum)
+                accuracy = evaluate_model(model)
+
+                if accuracy > best_accuracy:
+                    best_accuracy = accuracy
+                    best_hyperparameters = (learning_rate, momentum, activation_type)
+
+    print('最优超参：', best_hyperparameters)
+
+可以看到，这段超参调优代码总计训练4×10×3=120个模型，要消耗大量的计算资源，因此您可能会有以下需求：
+
+1. :ref:`通过较少的尝试次数找到最优超参组合 <zh-hpo-overview-tuners>`
+2. :ref:`利用分布式平台进行训练 <zh-hpo-overview-platforms>`
+3. :ref:`使用网页控制台来监控调参过程 <zh-hpo-overview-portal>`
+
+NNI 可以满足您的这些需求。
+
+NNI 超参调优的主要功能
+----------------------
+
+.. _zh-hpo-overview-tuners:
+
+调优算法
+^^^^^^^^
+
+NNI 通过调优算法来更快地找到最优超参组合，这些算法被称为“tuner”（调参器）。
+
+调优算法会决定需要运行、评估哪些超参组合，以及应该以何种顺序评估超参组合。
+高效的算法可以通过已评估超参组合的结果去预测最优超参的取值，从而减少找到最优超参所需的评估次数。
+
+开头的示例以固定顺序评估所有可能的超参组合，无视了超参的评估结果，这种朴素方法被称为“grid search”（网格搜索）。
+
+NNI 内建了很多流行的调优算法，包括朴素算法如随机搜索、网格搜索，贝叶斯优化类算法如 TPE、SMAC，强化学习算法如 PPO 等等。
+
+完整内容： :doc:`tuners`
+
+.. _zh-hpo-overview-platforms:
+
+训练平台
+^^^^^^^^
+
+如果您不准备使用分布式训练平台，您可以像使用普通 Python 函数库一样，在自己的电脑上直接运行 NNI 超参调优。
+
+如果想利用更多计算资源加速调优过程，您也可以使用 NNI 内建的训练平台集成，从简单的 SSH 服务器到可扩容的 Kubernetes 集群 NNI 都提供支持。
+
+完整内容： :doc:`/experiment/training_service/overview`
+
+.. _zh-hpo-overview-portal:
+
+网页控制台
+^^^^^^^^^^
+
+您可以使用 NNI 的网页控制台来监控超参调优实验，它支持实时显示实验进度、对超参性能进行可视化、人工修改超参数值、同时管理多个实验等诸多功能。
+
+完整内容： :doc:`/experiment/web_portal/web_portal`
+
+.. image:: ../../static/img/webui.gif
+    :width: 100%
+
+教程
+----
+
+我们提供了以下教程帮助您上手 NNI 超参调优，您可以选择最熟悉的机器学习框架：
+
+* :doc:`使用PyTorch的超参调优教程 </tutorials/hpo_quickstart_pytorch/main>`
+* :doc:`使用TensorFlow的超参调优教程（英文） </tutorials/hpo_quickstart_tensorflow/main>`
+
+更多功能
+--------
+
+在掌握了 NNI 超参调优的基础用法之后，您可以尝试以下更多功能：
+
+* :doc:`Use command line tool to create and manage experiments (nnictl) </reference/nnictl>`
+
+  * :doc:`nnictl example </tutorials/hpo_nnictl/nnictl>`
+
+* :doc:`Early stop non-optimal models (assessor) <assessors>`
+* :doc:`TensorBoard integration </experiment/web_portal/tensorboard>`
+* :doc:`Implement your own algorithm <custom_algorithm>`
+* :doc:`Benchmark tuners <hpo_benchmark>`

docs/source/hpo/quickstart.rst

+Quickstart
+==========
+
+.. toctree::
+
+    PyTorch </tutorials/hpo_quickstart_pytorch/main>
+    TensorFlow </tutorials/hpo_quickstart_tensorflow/main>
+
+.. toctree::
+    :hidden:
+
+    /tutorials/hpo_quickstart_pytorch/index
+    /tutorials/hpo_quickstart_tensorflow/index

docs/source/hpo/search_space.rst

+Search Space
+============
+
+Overview
+--------
+
+In NNI, tuner will sample hyperparameters according to the search space.
+
+To define a search space, users should define the name of the variable, the type of sampling strategy and its parameters.
+
+* An example of a search space definition in JSON format is as follow:
+
+.. code-block:: json 
+
+   {
+       "dropout_rate": {"_type": "uniform", "_value": [0.1, 0.5]},
+       "conv_size": {"_type": "choice", "_value": [2, 3, 5, 7]},
+       "hidden_size": {"_type": "choice", "_value": [124, 512, 1024]},
+       "batch_size": {"_type": "choice", "_value": [50, 250, 500]},
+       "learning_rate": {"_type": "uniform", "_value": [0.0001, 0.1]}
+   }
+
+Take the first line as an example.
+``dropout_rate`` is defined as a variable whose prior distribution is a uniform distribution with a range from ``0.1`` to ``0.5``.
+
+.. attention::
+
+    The available sampling strategies within a search space depend on the tuner you want to use.
+    We list the supported types for each built-in tuner :ref:`below <hpo-space-support>`.
+
+    For a customized tuner, you don't have to follow our convention and you will have the flexibility to define any type you want.
+
+Types
+-----
+
+All types of sampling strategies and their parameter are listed here:
+
+choice
+^^^^^^
+
+.. code-block:: python 
+
+    {"_type": "choice", "_value": options}
+
+* The variable's value is one of the options. Here ``options`` should be a list of **numbers** or a list of **strings**. Using arbitrary objects as members of this list (like sublists, a mixture of numbers and strings, or null values) should work in most cases, but may trigger undefined behaviors.
+* ``options`` can also be a nested sub-search-space, this sub-search-space takes effect only when the corresponding element is chosen. The variables in this sub-search-space can be seen as conditional variables. Here is an simple :githublink:`example of nested search space definition <examples/trials/mnist-nested-search-space/search_space.json>`. If an element in the options list is a dict, it is a sub-search-space, and for our built-in tuners you have to add a ``_name`` key in this dict, which helps you to identify which element is chosen. Accordingly, here is a :githublink:`sample <examples/trials/mnist-nested-search-space/sample.json>` which users can get from nni with nested search space definition. See the table below for the tuners which support nested search spaces.
+
+randint
+^^^^^^^
+
+.. code-block:: python 
+
+    {"_type": "randint", "_value": [lower, upper]}
+
+* Choosing a random integer between ``lower`` (inclusive) and ``upper`` (exclusive).
+* Note: Different tuners may interpret ``randint`` differently. Some (e.g., TPE, GridSearch) treat integers from lower
+  to upper as unordered ones, while others respect the ordering (e.g., SMAC). If you want all the tuners to respect
+  the ordering, please use ``quniform`` with ``q=1``.
+
+uniform
+^^^^^^^
+
+.. code-block:: python 
+
+    {"_type": "uniform", "_value": [low, high]}
+
+* The variable value is uniformly sampled between low and high.
+* When optimizing, this variable is constrained to a two-sided interval.
+
+quniform
+^^^^^^^^
+
+.. code-block:: python 
+
+    {"_type": "quniform", "_value": [low, high, q]}
+
+* The variable value is determined using ``clip(round(uniform(low, high) / q) * q, low, high)``\ , where the clip operation is used to constrain the generated value within the bounds. For example, for ``_value`` specified as [0, 10, 2.5], possible values are [0, 2.5, 5.0, 7.5, 10.0]; For ``_value`` specified as [2, 10, 5], possible values are [2, 5, 10].
+* Suitable for a discrete value with respect to which the objective is still somewhat "smooth", but which should be bounded both above and below. If you want to uniformly choose an integer from a range [low, high], you can write ``_value`` like this: ``[low, high, 1]``.
+
+loguniform
+^^^^^^^^^^
+
+.. code-block:: python 
+
+    {"_type": "loguniform", "_value": [low, high]}
+
+* The variable value is drawn from a range [low, high] according to a loguniform distribution like exp(uniform(log(low), log(high))), so that the logarithm of the return value is uniformly distributed.
+* When optimizing, this variable is constrained to be positive.
+
+qloguniform
+^^^^^^^^^^^
+
+.. code-block:: python 
+
+    {"_type": "qloguniform", "_value": [low, high, q]}
+
+* The variable value is determined using ``clip(round(loguniform(low, high) / q) * q, low, high)``\ , where the clip operation is used to constrain the generated value within the bounds.
+* Suitable for a discrete variable with respect to which the objective is "smooth" and gets smoother with the size of the value, but which should be bounded both above and below.
+
+normal
+^^^^^^
+
+.. code-block:: python 
+
+    {"_type": "normal", "_value": [mu, sigma]}
+
+* The variable value is a real value that's normally-distributed with mean mu and standard deviation sigma. When optimizing, this is an unconstrained variable.
+
+qnormal
+^^^^^^^
+
+.. code-block:: python 
+
+    {"_type": "qnormal", "_value": [mu, sigma, q]}
+
+* The variable value is determined using ``round(normal(mu, sigma) / q) * q``
+* Suitable for a discrete variable that probably takes a value around mu, but is fundamentally unbounded.
+
+lognormal
+^^^^^^^^^
+
+.. code-block:: python 
+
+    {"_type": "lognormal", "_value": [mu, sigma]}
+
+* The variable value is drawn according to ``exp(normal(mu, sigma))`` so that the logarithm of the return value is normally distributed. When optimizing, this variable is constrained to be positive.
+
+qlognormal
+^^^^^^^^^^
+
+.. code-block:: python 
+
+    {"_type": "qlognormal", "_value": [mu, sigma, q]}
+
+* The variable value is determined using ``round(exp(normal(mu, sigma)) / q) * q``
+* Suitable for a discrete variable with respect to which the objective is smooth and gets smoother with the size of the variable, which is bounded from one side.
+
+.. _hpo-space-support:
+
+Search Space Types Supported by Each Tuner
+------------------------------------------
+
+.. list-table::
+   :header-rows: 1
+   :widths: auto
+
+   * - 
+     - choice
+     - choice(nested)
+     - randint
+     - uniform
+     - quniform
+     - loguniform
+     - qloguniform
+     - normal
+     - qnormal
+     - lognormal
+     - qlognormal
+
+   * - :class:`TPE <nni.algorithms.hpo.tpe_tuner.TpeTuner>`
+     - ✓
+     - ✓
+     - ✓
+     - ✓
+     - ✓
+     - ✓
+     - ✓
+     - ✓
+     - ✓
+     - ✓
+     - ✓
+
+   * - :class:`Random <nni.algorithms.hpo.random_tuner.RandomTuner>`
+     - ✓
+     - ✓
+     - ✓
+     - ✓
+     - ✓
+     - ✓
+     - ✓
+     - ✓
+     - ✓
+     - ✓
+     - ✓
+
+   * - :class:`Grid Search <nni.algorithms.hpo.gridsearch_tuner.GridSearchTuner>`
+     - ✓
+     - ✓
+     - ✓
+     - ✓
+     - ✓
+     - ✓
+     - ✓
+     - ✓
+     - ✓
+     - ✓
+     - ✓
+
+   * - :class:`Anneal <nni.algorithms.hpo.hyperopt_tuner.HyperoptTuner>`
+     - ✓
+     - ✓
+     - ✓
+     - ✓
+     - ✓
+     - ✓
+     - ✓
+     - ✓
+     - ✓
+     - ✓
+     - ✓
+
+   * - :class:`Evolution <nni.algorithms.hpo.evolution_tuner.EvolutionTuner>`
+     - ✓
+     - ✓
+     - ✓
+     - ✓
+     - ✓
+     - ✓
+     - ✓
+     - ✓
+     - ✓
+     - ✓
+     - ✓
+
+   * - :class:`SMAC <nni.algorithms.hpo.smac_tuner.SMACTuner>`
+     - ✓
+     - 
+     - ✓
+     - ✓
+     - ✓
+     - ✓
+     - 
+     - 
+     - 
+     - 
+     - 
+
+   * - :class:`Batch <nni.algorithms.hpo.batch_tuner.BatchTuner>`
+     - ✓
+     - 
+     - 
+     - 
+     - 
+     - 
+     - 
+     - 
+     - 
+     - 
+     - 
+
+   * - :class:`Hyperband <nni.algorithms.hpo.hyperband_advisor.Hyperband>`
+     - ✓
+     - 
+     - ✓
+     - ✓
+     - ✓
+     - ✓
+     - ✓
+     - ✓
+     - ✓
+     - ✓
+     - ✓
+
+   * - :class:`Metis <nni.algorithms.hpo.metis_tuner.MetisTuner>`
+     - ✓
+     - 
+     - ✓
+     - ✓
+     - ✓
+     - 
+     - 
+     - 
+     - 
+     - 
+     - 
+
+   * - :class:`BOHB <nni.algorithms.hpo.bohb_advisor.BOHB>`
+     - ✓
+     -
+     - ✓
+     - ✓
+     - ✓
+     - ✓
+     - ✓
+     - ✓
+     - ✓
+     - ✓
+     - ✓
+
+   * - :class:`GP <nni.algorithms.hpo.gp_tuner.GPTuner>`
+     - ✓
+     - 
+     - ✓
+     - ✓
+     - ✓
+     - ✓
+     - ✓
+     - 
+     - 
+     - 
+     - 
+
+   * - :class:`PBT <nni.algorithms.hpo.pbt_tuner.PBTTuner>`
+     - ✓
+     -
+     - ✓
+     - ✓
+     - ✓
+     - ✓
+     - ✓
+     - ✓
+     - ✓
+     - ✓
+     - ✓
+
+   * - :class:`DNGO <nni.algorithms.hpo.dngo_tuner.DNGOTuner>`
+     - ✓
+     - 
+     - ✓
+     - ✓
+     - ✓
+     - ✓
+     - ✓
+     - 
+     - 
+     - 
+     - 
+
+
+Known Limitations:
+
+* GP Tuner, Metis Tuner and DNGO tuner support only **numerical values** in search space
+  (``choice`` type values can be no-numerical with other tuners, e.g. string values).
+  Both GP Tuner and Metis Tuner use Gaussian Process Regressor(GPR).
+  GPR make predictions based on a kernel function and the 'distance' between different points,
+  it's hard to get the true distance between no-numerical values.
+
+* Note that for nested search space:
+
+  * Only TPE/Random/Grid Search/Anneal/Evolution tuners support nested search space.

docs/source/hpo/toctree.rst

+Hyperparameter Optimization
+===========================
+
+.. toctree::
+    :hidden:
+
+    Overview <overview>
+    quickstart
+    Search Space <search_space>
+    Tuners <tuners>
+    Assessors <assessors>
+    advanced_usage

docs/source/hpo/toctree_zh.rst

+.. 21e9c3e0f6b182cf42a99a7f6c4ecf98
+
+超参调优
+========
+
+.. toctree::
+    :hidden:
+
+    概述 <overview>
+    教程 <quickstart>
+    搜索空间 <search_space>
+    Tuners <tuners>
+    Assessors <assessors>
+    高级用法 <advanced_usage>

docs/source/hpo/tuners.rst

+Tuner: Tuning Algorithms
+========================
+
+The tuner decides which hyperparameter sets will be evaluated. It is a most important part of NNI HPO.
+
+A tuner works like following pseudocode:
+
+.. code-block:: python
+
+    space = get_search_space()
+    history = []
+    while not experiment_end:
+        hp = suggest_hyperparameter_set(space, history)
+        result = run_trial(hp)
+        history.append((hp, result))
+
+NNI has out-of-the-box support for many popular tuning algorithms. 
+They should be sufficient to cover most typical machine learning scenarios.
+
+However, if you have a very specific demand, or if you have designed an algorithm yourself,
+you can also implement your own tuner: :doc:`custom_algorithm`
+
+Common Usage
+------------
+
+All built-in tuners have similar usage.
+
+To use a built-in tuner, you need to specify its name and arguments in experiment config,
+and provides a standard :doc:`search_space`.
+Some tuners, like SMAC and DNGO, have extra dependencies that need to be installed separately.
+Please check each tuner's reference page for what arguments it supports and whether it needs extra dependencies.
+
+As a general example, random tuner can be configured as follow:
+
+.. code-block:: python
+
+    config.search_space = {
+        'x': {'_type': 'uniform', '_value': [0, 1]},
+        'y': {'_type': 'choice', '_value': ['a', 'b', 'c']}
+    }
+    config.tuner.name = 'random'
+    config.tuner.class_args = {'seed': 0}
+
+Built-in Tuners
+---------------
+
+.. list-table::
+    :header-rows: 1
+    :widths: auto
+
+    * - Tuner
+      - Category
+      - Brief Introduction
+
+    * - :class:`TPE <nni.algorithms.hpo.tpe_tuner.TpeTuner>`
+      - Bayesian
+      - Tree-structured Parzen Estimator, a classic Bayesian optimization algorithm.
+        (`paper <https://papers.nips.cc/paper/4443-algorithms-for-hyper-parameter-optimization.pdf>`__)
+
+        TPE is a lightweight tuner that has no extra dependency and supports all search space types.
+        Good to start with.
+
+        The drawback is that TPE cannot discover relationship between different hyperparameters.
+
+    * - :class:`Random <nni.algorithms.hpo.random_tuner.RandomTuner>`
+      - Basic
+      - Naive random search, the baseline. It supports all search space types.
+
+    * - :class:`Grid Search <nni.algorithms.hpo.gridsearch_tuner.GridSearchTuner>`
+      - Basic
+      - Divides search space into evenly spaced grid, and performs brute-force traverse. Another baseline.
+
+        It supports all search space types.
+        Recommended when the search space is small, and when you want to find the strictly optimal hyperparameters.
+
+    * - :class:`Anneal <nni.algorithms.hpo.hyperopt_tuner.HyperoptTuner>`
+      - Heuristic
+      - This simple annealing algorithm begins by sampling from the prior, but tends over time to sample from points closer and closer to the best ones observed. This algorithm is a simple variation on the random search that leverages smoothness in the response surface. The annealing rate is not adaptive.
+
+    * - :class:`Evolution <nni.algorithms.hpo.evolution_tuner.EvolutionTuner>`
+      - Heuristic
+      - Naive Evolution comes from Large-Scale Evolution of Image Classifiers. It randomly initializes a population-based on search space. For each generation, it chooses better ones and does some mutation (e.g., change a hyperparameter, add/remove one layer) on them to get the next generation. Naïve Evolution requires many trials to work, but it's very simple and easy to expand new features. `Reference paper <https://arxiv.org/pdf/1703.01041.pdf>`__
+
+    * - :class:`SMAC <nni.algorithms.hpo.smac_tuner.SMACTuner>`
+      - Bayesian
+      - SMAC is based on Sequential Model-Based Optimization (SMBO). It adapts the most prominent previously used model class (Gaussian stochastic process models) and introduces the model class of random forests to SMBO, in order to handle categorical parameters. The SMAC supported by NNI is a wrapper on the SMAC3 GitHub repo.
+
+        Notice, SMAC needs to be installed by ``pip install nni[SMAC]`` command. `Reference Paper, <https://www.cs.ubc.ca/~hutter/papers/10-TR-SMAC.pdf>`__ `GitHub Repo <https://github.com/automl/SMAC3>`__
+
+    * - :class:`Batch <nni.algorithms.hpo.batch_tuner.BatchTuner>`
+      - Basic
+      - Batch tuner allows users to simply provide several configurations (i.e., choices of hyper-parameters) for their trial code. After finishing all the configurations, the experiment is done. Batch tuner only supports the type choice in search space spec.
+
+    * - :class:`Hyperband <nni.algorithms.hpo.hyperband_advisor.Hyperband>`
+      - Heuristic
+      - Hyperband tries to use limited resources to explore as many configurations as possible and returns the most promising ones as a final result. The basic idea is to generate many configurations and run them for a small number of trials. The half least-promising configurations are thrown out, the remaining are further trained along with a selection of new configurations. The size of these populations is sensitive to resource constraints (e.g. allotted search time). `Reference Paper <https://arxiv.org/pdf/1603.06560.pdf>`__
+
+    * - :class:`Metis <nni.algorithms.hpo.metis_tuner.MetisTuner>`
+      - Bayesian
+      - Metis offers the following benefits when it comes to tuning parameters: While most tools only predict the optimal configuration, Metis gives you two outputs: (a) current prediction of optimal configuration, and (b) suggestion for the next trial. No more guesswork. While most tools assume training datasets do not have noisy data, Metis actually tells you if you need to re-sample a particular hyper-parameter. `Reference Paper <https://www.microsoft.com/en-us/research/publication/metis-robustly-tuning-tail-latencies-cloud-systems/>`__
+
+    * - :class:`BOHB <nni.algorithms.hpo.bohb_advisor.BOHB>`
+      - Bayesian
+      - BOHB is a follow-up work to Hyperband. It targets the weakness of Hyperband that new configurations are generated randomly without leveraging finished trials. For the name BOHB, HB means Hyperband, BO means Bayesian Optimization. BOHB leverages finished trials by building multiple TPE models, a proportion of new configurations are generated through these models. `Reference Paper <https://arxiv.org/abs/1807.01774>`__
+
+    * - :class:`GP <nni.algorithms.hpo.gp_tuner.GPTuner>`
+      - Bayesian
+      - Gaussian Process Tuner is a sequential model-based optimization (SMBO) approach with Gaussian Process as the surrogate. `Reference Paper <https://papers.nips.cc/paper/4443-algorithms-for-hyper-parameter-optimization.pdf>`__, `Github Repo <https://github.com/fmfn/BayesianOptimization>`__
+
+    * - :class:`PBT <nni.algorithms.hpo.pbt_tuner.PBTTuner>`
+      - Heuristic
+      - PBT Tuner is a simple asynchronous optimization algorithm which effectively utilizes a fixed computational budget to jointly optimize a population of models and their hyperparameters to maximize performance. `Reference Paper <https://arxiv.org/abs/1711.09846v1>`__
+
+    * - :class:`DNGO <nni.algorithms.hpo.dngo_tuner.DNGOTuner>`
+      - Bayesian
+      - Use of neural networks as an alternative to GPs to model distributions over functions in bayesian optimization.
+
+Comparison
+----------
+
+These articles have compared built-in tuners' performance on some different tasks:
+
+:doc:`hpo_benchmark_stats`
+
+:doc:`/sharings/hpo_comparison`

docs/source/index.rst

+NNI Documentation
+=================
+
+.. toctree::
+   :maxdepth: 2
+   :caption: Get Started
+   :hidden:
+
+   installation
+   quickstart
+
+.. toctree::
+   :maxdepth: 2
+   :caption: User Guide
+   :hidden:
+
+   hpo/toctree
+   nas/toctree
+   Model Compression <compression/toctree>
+   feature_engineering/toctree
+   experiment/toctree
+
+.. toctree::
+   :maxdepth: 2
+   :caption: References
+   :hidden:
+
+   Python API <reference/python_api>
+   reference/experiment_config
+   reference/nnictl
+
+.. toctree::
+   :maxdepth: 2
+   :caption: Misc
+   :hidden:
+
+   examples
+   sharings/community_sharings
+   notes/research_publications
+   notes/build_from_source
+   notes/contributing
+   release
+
+**NNI (Neural Network Intelligence)** is a lightweight but powerful toolkit to help users **automate**:
+
+* :doc:`Hyperparameter Optimization </hpo/overview>`
+* :doc:`Neural Architecture Search </nas/overview>`
+* :doc:`Model Compression </compression/overview>`
+* :doc:`Feature Engineering </feature_engineering/overview>`
+
+Get Started
+-----------
+
+To install the current release:
+
+.. code-block:: bash
+
+   $ pip install nni
+
+See the :doc:`installation guide </installation>` if you need additional help on installation.
+
+Try your first NNI experiment
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. code-block:: shell
+
+   $ nnictl hello
+
+.. note:: You need to have `PyTorch <https://pytorch.org/>`_ (as well as `torchvision <https://pytorch.org/vision/stable/index.html>`_) installed to run this experiment.
+
+To start your journey now, please follow the :doc:`absolute quickstart of NNI <quickstart>`!
+
+Why choose NNI?
+---------------
+
+NNI makes AutoML techniques plug-and-play
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. raw:: html
+
+   <div class="codesnippet-card-container">
+
+.. codesnippetcard::
+   :icon: ../img/thumbnails/hpo-small.svg
+   :title: Hyperparameter Tuning
+   :link: tutorials/hpo_quickstart_pytorch/main
+
+   .. code-block::
+
+      params = nni.get_next_parameter()
+
+      class Net(nn.Module):
+          ...
+
+      model = Net()
+      optimizer = optim.SGD(model.parameters(),
+                            params['lr'],
+                            params['momentum'])
+
+      for epoch in range(10):
+          train(...)
+
+      accuracy = test(model)
+      nni.report_final_result(accuracy)
+
+.. codesnippetcard::
+   :icon: ../img/thumbnails/pruning-small.svg
+   :title: Model Pruning
+   :link: tutorials/pruning_quick_start_mnist
+
+   .. code-block::
+
+      # define a config_list
+      config = [{
+          'sparsity': 0.8,
+          'op_types': ['Conv2d']
+      }]
+
+      # generate masks for simulated pruning
+      wrapped_model, masks = \
+          L1NormPruner(model, config). \
+          compress()
+
+      # apply the masks for real speedup
+      ModelSpeedup(unwrapped_model, input, masks). \
+          speedup_model()
+
+.. codesnippetcard::
+   :icon: ../img/thumbnails/quantization-small.svg
+   :title: Quantization
+   :link: tutorials/quantization_quick_start_mnist
+
+   .. code-block::
+
+      # define a config_list
+      config = [{
+          'quant_types': ['input', 'weight'],
+          'quant_bits': {'input': 8, 'weight': 8},
+          'op_types': ['Conv2d']
+      }]
+
+      # in case quantizer needs a extra training
+      quantizer = QAT_Quantizer(model, config)
+      quantizer.compress()
+      # Training...
+
+      # export calibration config and
+      # generate TensorRT engine for real speedup
+      calibration_config = quantizer.export_model(
+          model_path, calibration_path)
+      engine = ModelSpeedupTensorRT(
+          model, input_shape, config=calib_config)
+      engine.compress()
+
+.. codesnippetcard::
+   :icon: ../img/thumbnails/multi-trial-nas-small.svg
+   :title: Neural Architecture Search
+   :link: tutorials/hello_nas
+
+   .. code-block:: python
+
+      # define model space
+      class Model(nn.Module):
+          self.conv2 = nn.LayerChoice([
+              nn.Conv2d(32, 64, 3, 1),
+              DepthwiseSeparableConv(32, 64)
+          ])
+      model_space = Model()
+      # search strategy + evaluator
+      strategy = RegularizedEvolution()
+      evaluator = FunctionalEvaluator(
+          train_eval_fn)
+
+      # run experiment
+      RetiariiExperiment(model_space,
+          evaluator, strategy).run()
+
+.. codesnippetcard::
+   :icon: ../img/thumbnails/one-shot-nas-small.svg
+   :title: One-shot NAS
+   :link: nas/exploration_strategy
+
+   .. code-block::
+
+      # define model space
+      space = AnySearchSpace()
+
+      # get a darts trainer
+      trainer = DartsTrainer(space, loss, metrics)
+      trainer.fit()
+
+      # get final searched architecture
+      arch = trainer.export()
+
+.. codesnippetcard::
+   :icon: ../img/thumbnails/feature-engineering-small.svg
+   :title: Feature Engineering
+   :link: feature_engineering/overview
+
+   .. code-block::
+
+      selector = GBDTSelector()
+      selector.fit(
+          X_train, y_train,
+          lgb_params=lgb_params,
+          eval_ratio=eval_ratio,
+          early_stopping_rounds=10,
+          importance_type='gain',
+          num_boost_round=1000)
+
+      # get selected features
+      features = selector.get_selected_features()
+
+.. End of code snippet card
+
+.. raw:: html
+
+   </div>
+
+NNI eases the effort to scale and manage AutoML experiments
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. codesnippetcard::
+   :icon: ../img/thumbnails/training-service-small.svg
+   :title: Training Service
+   :link: experiment/training_service/overview
+   :seemore: See more here.
+
+   An AutoML experiment requires many trials to explore feasible and potentially good-performing models.
+   **Training service** aims to make the tuning process easily scalable in a distributed platforms.
+   It provides a unified user experience for diverse computation resources (e.g., local machine, remote servers, AKS).
+   Currently, NNI supports **more than 9** kinds of training services.
+
+.. codesnippetcard::
+   :icon: ../img/thumbnails/web-portal-small.svg
+   :title: Web Portal
+   :link: experiment/web_portal/web_portal
+   :seemore: See more here.
+
+   Web portal visualizes the tuning process, exposing the ability to inspect, monitor and control the experiment.
+
+   .. image:: ../static/img/webui.gif
+      :width: 100%
+
+.. codesnippetcard::
+   :icon: ../img/thumbnails/experiment-management-small.svg
+   :title: Experiment Management
+   :link: experiment/experiment_management
+   :seemore: See more here.
+
+   The DNN model tuning often requires more than one experiment.
+   Users might try different tuning algorithms, fine-tune their search space, or switch to another training service.
+   **Experiment management** provides the power to aggregate and compare tuning results from multiple experiments,
+   so that the tuning workflow becomes clean and organized.
+
+Get Support and Contribute Back
+-------------------------------
+
+NNI is maintained on the `NNI GitHub repository <https://github.com/microsoft/nni>`_. We collect feedbacks and new proposals/ideas on GitHub. You can:
+
+* Open a `GitHub issue <https://github.com/microsoft/nni/issues>`_ for bugs and feature requests.
+* Open a `pull request <https://github.com/microsoft/nni/pulls>`_ to contribute code (make sure to read the :doc:`contribution guide <notes/contributing>` before doing this).
+* Participate in `NNI Discussion <https://github.com/microsoft/nni/discussions>`_ for general questions and new ideas.
+* Join the following IM groups.
+
+.. list-table::
+   :header-rows: 1
+   :widths: auto
+
+   * - Gitter
+     - WeChat
+   * -
+       .. image:: https://user-images.githubusercontent.com/39592018/80665738-e0574a80-8acc-11ea-91bc-0836dc4cbf89.png
+     -
+       .. image:: https://github.com/scarlett2018/nniutil/raw/master/wechat.png
+
+Citing NNI
+----------
+
+If you use NNI in a scientific publication, please consider citing NNI in your references.
+
+   Microsoft. Neural Network Intelligence (version |release|). https://github.com/microsoft/nni
+
+Bibtex entry (please replace the version with the particular version you are using): ::
+
+   @software{nni2021,
+      author = {{Microsoft}},
+      month = {1},
+      title = {{Neural Network Intelligence}},
+      url = {https://github.com/microsoft/nni},
+      version = {2.0},
+      year = {2021}
+   }

docs/source/index_zh.rst

+.. dbd41cab307bcd76cc747b3d478709b8
+
+
+NNI 文档
+=================
+
+.. toctree::
+   :maxdepth: 2
+   :caption: 开始使用
+   :hidden:
+
+   安装 <installation>
+   快速入门 <quickstart>
+
+.. toctree::
+   :maxdepth: 2
+   :caption: 用户指南
+   :hidden:
+
+   超参调优 <hpo/toctree>
+   架构搜索 <nas/toctree>
+   模型压缩 <compression/toctree>
+   特征工程 <feature_engineering/toctree>
+   实验管理 <experiment/toctree>
+
+.. toctree::
+   :maxdepth: 2
+   :caption: 参考
+   :hidden:
+
+   Python API <reference/python_api>
+   实验配置 <reference/experiment_config>
+   nnictl 命令 <reference/nnictl>
+
+.. toctree::
+   :maxdepth: 2
+   :caption: 杂项
+   :hidden:
+
+   示例 <examples>
+   社区分享 <sharings/community_sharings>
+   研究发布 <notes/research_publications>
+   源码安装 <notes/build_from_source>
+   贡献指南 <notes/contributing>
+   版本说明 <release>
+
+**NNI (Neural Network Intelligence)** 是一个轻量而强大的工具，可以帮助用户 **自动化**：
+
+* :doc:`超参调优 </hpo/overview>`
+* :doc:`架构搜索 </nas/overview>`
+* :doc:`模型压缩 </compression/overview>`
+* :doc:`特征工程 </feature_engineering/overview>`
+
+开始使用
+-----------
+
+安装最新的版本，可执行以下命令：
+
+.. code-block:: bash
+
+   $ pip install nni
+
+如果在安装上遇到问题，可参考 :doc:`安装指南 </installation>`。
+
+开始你的第一个 NNI 实验
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. code-block:: shell
+
+   $ nnictl hello
+
+.. note:: 你需要预先安装 `PyTorch <https://pytorch.org/>`_ （以及 `torchvision <https://pytorch.org/vision/stable/index.html>`_ ）才能运行这个实验。
+
+请阅读 :doc:`NNI 快速入门 <quickstart>` 以开启你的 NNI 旅程！
+
+为什么选择 NNI？
+--------------------
+
+NNI 使得自动机器学习技术即插即用
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. raw:: html
+
+   <div class="codesnippet-card-container">
+
+.. codesnippetcard::
+   :icon: ../img/thumbnails/hpo-small.svg
+   :title: 超参调优
+   :link: tutorials/hpo_quickstart_pytorch/main
+   :seemore: 点这里阅读完整教程
+
+   .. code-block::
+
+      params = nni.get_next_parameter()
+
+      class Net(nn.Module):
+          ...
+
+      model = Net()
+      optimizer = optim.SGD(model.parameters(),
+                            params['lr'],
+                            params['momentum'])
+
+      for epoch in range(10):
+          train(...)
+
+      accuracy = test(model)
+      nni.report_final_result(accuracy)
+
+.. codesnippetcard::
+   :icon: ../img/thumbnails/pruning-small.svg
+   :title: 模型剪枝
+   :link: tutorials/pruning_quick_start_mnist
+   :seemore: 点这里阅读完整教程
+
+   .. code-block::
+
+      # define a config_list
+      config = [{
+          'sparsity': 0.8,
+          'op_types': ['Conv2d']
+      }]
+
+      # generate masks for simulated pruning
+      wrapped_model, masks = \
+          L1NormPruner(model, config). \
+          compress()
+
+      # apply the masks for real speedup
+      ModelSpeedup(unwrapped_model, input, masks). \
+          speedup_model()
+
+.. codesnippetcard::
+   :icon: ../img/thumbnails/quantization-small.svg
+   :title: 模型量化
+   :link: tutorials/quantization_speedup
+   :seemore: 点这里阅读完整教程
+
+   .. code-block::
+
+      # define a config_list
+      config = [{
+          'quant_types': ['input', 'weight'],
+          'quant_bits': {'input': 8, 'weight': 8},
+          'op_types': ['Conv2d']
+      }]
+
+      # in case quantizer needs a extra training
+      quantizer = QAT_Quantizer(model, config)
+      quantizer.compress()
+      # Training...
+
+      # export calibration config and
+      # generate TensorRT engine for real speedup
+      calibration_config = quantizer.export_model(
+          model_path, calibration_path)
+      engine = ModelSpeedupTensorRT(
+          model, input_shape, config=calib_config)
+      engine.compress()
+
+.. codesnippetcard::
+   :icon: ../img/thumbnails/multi-trial-nas-small.svg
+   :title: 神经网络架构搜索
+   :link: tutorials/hello_nas
+   :seemore: 点这里阅读完整教程
+
+   .. code-block::
+
+      # define model space
+      -   self.conv2 = nn.Conv2d(32, 64, 3, 1)
+      +   self.conv2 = nn.LayerChoice([
+      +       nn.Conv2d(32, 64, 3, 1),
+      +       DepthwiseSeparableConv(32, 64)
+      +   ])
+      # search strategy + evaluator
+      strategy = RegularizedEvolution()
+      evaluator = FunctionalEvaluator(
+          train_eval_fn)
+
+      # run experiment
+      RetiariiExperiment(model_space,
+          evaluator, strategy).run()
+
+.. codesnippetcard::
+   :icon: ../img/thumbnails/one-shot-nas-small.svg
+   :title: 单尝试 (One-shot) NAS
+   :link: nas/exploration_strategy
+   :seemore: 点这里阅读完整教程
+
+   .. code-block::
+
+      # define model space
+      space = AnySearchSpace()
+
+      # get a darts trainer
+      trainer = DartsTrainer(space, loss, metrics)
+      trainer.fit()
+
+      # get final searched architecture
+      arch = trainer.export()
+
+.. codesnippetcard::
+   :icon: ../img/thumbnails/feature-engineering-small.svg
+   :title: 特征工程
+   :link: feature_engineering/overview
+   :seemore: 点这里阅读完整教程
+
+   .. code-block::
+
+      selector = GBDTSelector()
+      selector.fit(
+          X_train, y_train,
+          lgb_params=lgb_params,
+          eval_ratio=eval_ratio,
+          early_stopping_rounds=10,
+          importance_type='gain',
+          num_boost_round=1000)
+
+      # get selected features
+      features = selector.get_selected_features()
+
+.. End of code snippet card
+
+.. raw:: html
+
+   </div>
+
+NNI 可降低自动机器学习实验管理的成本
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. codesnippetcard::
+   :icon: ../img/thumbnails/training-service-small.svg
+   :title: 训练平台
+   :link: experiment/training_service/overview
+   :seemore: 点这里了解更多
+
+   一个自动机器学习实验通常需要很多次尝试，来找到合适且具有潜力的模型。
+   **训练平台** 的目标便是让整个调优过程可以轻松的扩展到分布式平台上，为不同的计算资源（例如本地机器、远端服务器、集群等）提供的统一的用户体验。
+   目前，NNI 已经支持 **超过九种** 训练平台。
+
+.. codesnippetcard::
+   :icon: ../img/thumbnails/web-portal-small.svg
+   :title: 网页控制台
+   :link: experiment/web_portal/web_portal
+   :seemore: 点这里了解更多
+
+   网页控制台提供了可视化调优过程的能力，让你可以轻松检查、跟踪、控制实验流程。
+
+   .. image:: ../static/img/webui.gif
+      :width: 100%
+
+.. codesnippetcard::
+   :icon: ../img/thumbnails/experiment-management-small.svg
+   :title: 多实验管理
+   :link: experiment/experiment_management
+   :seemore: 点这里了解更多
+
+   深度学习模型往往需要多个实验不断迭代，例如用户可能想尝试不同的调优算法，优化他们的搜索空间，或者切换到其他的计算资源。
+   **多实验管理** 提供了对多个实验的结果进行聚合和比较的强大能力，极大程度上简化了开发者的开发流程。
+
+获取帮助或参与贡献
+-------------------------------
+
+NNI 使用 `NNI GitHub 仓库 <https://github.com/microsoft/nni>`_ 进行维护。我们在 GitHub 上收集反馈，以及新需求和想法。你可以：
+
+* 新建一个 `GitHub issue <https://github.com/microsoft/nni/issues>`_ 反馈一个 bug 或者需求。
+* 新建一个 `pull request <https://github.com/microsoft/nni/pulls>`_ 以贡献代码（在此之前，请务必确保你已经阅读过 :doc:`贡献指南 <notes/contributing>`）。
+* 如果你有任何问题，都可以加入 `NNI 讨论 <https://github.com/microsoft/nni/discussions>`_。
+* 加入即时聊天群组：
+
+.. list-table::
+   :header-rows: 1
+   :widths: auto
+
+   * - Gitter
+     - 微信
+   * -
+       .. image:: https://user-images.githubusercontent.com/39592018/80665738-e0574a80-8acc-11ea-91bc-0836dc4cbf89.png
+     -
+       .. image:: https://github.com/scarlett2018/nniutil/raw/master/wechat.png
+
+引用 NNI
+----------
+
+如果你在你的文献中用到了 NNI，请考虑引用我们：
+
+   Microsoft. Neural Network Intelligence (version |release|). https://github.com/microsoft/nni
+
+Bibtex 格式如下（请将版本号替换成你在使用的特定版本）： ::
+
+   @software{nni2021,
+      author = {{Microsoft}},
+      month = {1},
+      title = {{Neural Network Intelligence}},
+      url = {https://github.com/microsoft/nni},
+      version = {2.0},
+      year = {2021}
+   }

docs/source/installation.rst

+Install NNI
+===========
+
+NNI requires Python >= 3.7.
+It is tested and supported on Ubuntu >= 18.04,
+Windows 10 >= 21H2, and macOS >= 11.
+
+There are 3 ways to install NNI:
+
+* :ref:`Using pip <installation-pip>`
+* :ref:`Build source code <installation-source>`
+* :ref:`Using Docker <installation-docker>`
+
+.. _installation-pip:
+
+Using pip
+---------
+
+NNI provides official packages for x86-64 CPUs. They can be installed with pip:
+
+.. code-block:: text
+
+    pip install nni
+
+Or to upgrade to latest version:
+
+.. code-block:: text
+
+    pip install --latest nni
+
+You can check installation with:
+
+.. code-block:: text
+
+    nnictl --version
+
+On Linux systems without Conda, you may encounter ``bash: nnictl: command not found`` error.
+In this case you need to add pip script directory to ``PATH``:
+
+.. code-block:: bash
+
+    echo 'export PATH=${PATH}:${HOME}/.local/bin' >> ~/.bashrc
+    source ~/.bashrc
+
+.. _installation-source:
+
+Installing from Source Code
+---------------------------
+
+NNI hosts source code on `GitHub <https://github.com/microsoft/nni>`__.
+
+NNI has experimental support for ARM64 CPUs, including Apple M1.
+It requires to install from source code.
+
+See :doc:`/notes/build_from_source`.
+
+.. _installation-docker:
+
+Using Docker
+------------
+
+NNI provides official Docker image on `Docker Hub <https://hub.docker.com/r/msranni/nni>`__.
+
+.. code-block:: text
+
+    docker pull msranni/nni
+
+Installing Extra Dependencies
+-----------------------------
+
+Some built-in algorithms of NNI requires extra packages.
+Use ``nni[<algorithm-name>]`` to install their dependencies.
+
+For example, to install dependencies of :class:`DNGO tuner<nni.algorithms.hpo.dngo_tuner.DNGOTuner>` :
+
+.. code-block:: text
+
+    pip install nni[DNGO]
+
+This command will not reinstall NNI itself, even if it was installed in development mode.
+
+Alternatively, you may install all extra dependencies at once:
+
+.. code-block:: text
+
+    pip install nni[all]
+
+**NOTE**: SMAC tuner depends on swig3, which requires a manual downgrade on Ubuntu:
+
+.. code-block:: bash
+
+    sudo apt install swig3.0
+    sudo rm /usr/bin/swig
+    sudo ln -s swig3.0 /usr/bin/swig

docs/source/installation_zh.rst

+.. b4703fc8c8e8dc1babdb38ba9ebcd4a6
+
+安装 NNI
+========
+
+NNI 依赖于 Python 3.7 或以上版本。
+
+您可以通过以下三种方式之一安装 NNI：
+
+* :ref:`通过 pip 安装<zh-installation-pip>`
+* :ref:`从源代码编译安装<zh-installation-source>`
+* :ref:`使用 Docker 容器<zh-installation-docker>`
+
+.. _zh-installation-pip:
+
+pip 安装
+--------
+
+NNI 为 x86-64 平台提供预编译的安装包，您可以使用 pip 进行安装：
+
+.. code-block:: text
+
+    pip install nni
+
+您也可以升级已安装的旧版本 NNI：
+
+.. code-block:: text
+
+    pip install --latest nni
+
+安装完成后，请运行以下命令进行检查：
+
+.. code-block:: text
+
+    nnictl --version
+
+如果您使用的是 Linux 系统并且没有使用 Conda，您可能会遇到 ``bash: nnictl: command not found`` 错误，
+此时您需要将 pip 安装的可执行文件添加到 ``PATH`` 环境变量：
+
+.. code-block:: bash
+
+    echo 'export PATH=${PATH}:${HOME}/.local/bin' >> ~/.bashrc
+    source ~/.bashrc
+
+.. _zh-installation-source:
+
+编译安装
+--------
+
+NNI 项目使用 `GitHub <https://github.com/microsoft/nni>`__ 托管源代码。
+
+NNI 对 ARM64 平台（包括苹果 M1）提供实验性支持，如果您希望在此类平台上使用 NNI，请从源代码编译安装。
+
+编译步骤请参见英文文档： :doc:`/notes/build_from_source`
+
+.. _zh-installation-docker:
+
+Docker 镜像
+-----------
+
+NNI 在 `Docker Hub <https://hub.docker.com/r/msranni/nni>`__ 上提供了官方镜像。
+
+.. code-block:: text
+
+    docker pull msranni/nni
+
+安装额外依赖
+------------
+
+有一些算法依赖于额外的 pip 包，在使用前需要先指定 ``nni[算法名]`` 安装依赖。以 DNGO 算法为例，使用前请运行以下命令：
+
+.. code-block:: text
+
+    pip install nni[DNGO]
+
+如果您已经通过任一种方式安装了 NNI，以上命令不会重新安装或改变 NNI 版本，只会安装 DNGO 算法的额外依赖。
+
+您也可以一次性安装所有可选依赖：
+
+.. code-block:: text
+
+    pip install nni[all]
+
+**注意**：SMAC 算法依赖于 swig3，在 Ubuntu 系统中需要手动进行降级：
+
+.. code-block:: bash
+
+    sudo apt install swig3.0
+    sudo rm /usr/bin/swig
+    sudo ln -s swig3.0 /usr/bin/swig

docs/source/locales/zh/LC_MESSAGES/compression.po

+# SOME DESCRIPTIVE TITLE.
+# Copyright (C) 2022, Microsoft
+# This file is distributed under the same license as the NNI package.
+# FIRST AUTHOR <EMAIL@ADDRESS>, 2022.
+#
+#, fuzzy
+msgid ""
+msgstr ""
+"Project-Id-Version: NNI \n"
+"Report-Msgid-Bugs-To: \n"
+"POT-Creation-Date: 2022-04-13 03:14+0000\n"
+"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
+"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
+"Language-Team: LANGUAGE <LL@li.org>\n"
+"MIME-Version: 1.0\n"
+"Content-Type: text/plain; charset=utf-8\n"
+"Content-Transfer-Encoding: 8bit\n"
+"Generated-By: Babel 2.9.1\n"
+
+#: ../../source/compression/overview.rst:2
+msgid "Overview of NNI Model Compression"
+msgstr ""
+
+#: ../../source/compression/overview.rst:4
+msgid ""
+"Deep neural networks (DNNs) have achieved great success in many tasks "
+"like computer vision, nature launguage processing, speech processing. "
+"However, typical neural networks are both computationally expensive and "
+"energy-intensive, which can be difficult to be deployed on devices with "
+"low computation resources or with strict latency requirements. Therefore,"
+" a natural thought is to perform model compression to reduce model size "
+"and accelerate model training/inference without losing performance "
+"significantly. Model compression techniques can be divided into two "
+"categories: pruning and quantization. The pruning methods explore the "
+"redundancy in the model weights and try to remove/prune the redundant and"
+" uncritical weights. Quantization refers to compress models by reducing "
+"the number of bits required to represent weights or activations. We "
+"further elaborate on the two methods, pruning and quantization, in the "
+"following chapters. Besides, the figure below visualizes the difference "
+"between these two methods."
+msgstr ""
+
+#: ../../source/compression/overview.rst:19
+msgid ""
+"NNI provides an easy-to-use toolkit to help users design and use model "
+"pruning and quantization algorithms. For users to compress their models, "
+"they only need to add several lines in their code. There are some popular"
+" model compression algorithms built-in in NNI. On the other hand, users "
+"could easily customize their new compression algorithms using NNI’s "
+"interface."
+msgstr ""
+
+#: ../../source/compression/overview.rst:24
+msgid "There are several core features supported by NNI model compression:"
+msgstr ""
+
+#: ../../source/compression/overview.rst:26
+msgid "Support many popular pruning and quantization algorithms."
+msgstr ""
+
+#: ../../source/compression/overview.rst:27
+msgid ""
+"Automate model pruning and quantization process with state-of-the-art "
+"strategies and NNI's auto tuning power."
+msgstr ""
+
+#: ../../source/compression/overview.rst:28
+msgid ""
+"Speedup a compressed model to make it have lower inference latency and "
+"also make it smaller."
+msgstr ""
+
+#: ../../source/compression/overview.rst:29
+msgid ""
+"Provide friendly and easy-to-use compression utilities for users to dive "
+"into the compression process and results."
+msgstr ""
+
+#: ../../source/compression/overview.rst:30
+msgid "Concise interface for users to customize their own compression algorithms."
+msgstr ""
+
+#: ../../source/compression/overview.rst:34
+msgid "Compression Pipeline"
+msgstr ""
+
+#: ../../source/compression/overview.rst:42
+msgid ""
+"The overall compression pipeline in NNI is shown above. For compressing a"
+" pretrained model, pruning and quantization can be used alone or in "
+"combination. If users want to apply both, a sequential mode is "
+"recommended as common practise."
+msgstr ""
+
+#: ../../source/compression/overview.rst:46
+msgid ""
+"Note that NNI pruners or quantizers are not meant to physically compact "
+"the model but for simulating the compression effect. Whereas NNI speedup "
+"tool can truly compress model by changing the network architecture and "
+"therefore reduce latency. To obtain a truly compact model, users should "
+"conduct :doc:`pruning speedup <../tutorials/pruning_speedup>` or "
+":doc:`quantizaiton speedup <../tutorials/quantization_speedup>`. The "
+"interface and APIs are unified for both PyTorch and TensorFlow. Currently"
+" only PyTorch version has been supported, and TensorFlow version will be "
+"supported in future."
+msgstr ""
+
+#: ../../source/compression/overview.rst:52
+msgid "Model Speedup"
+msgstr ""
+
+#: ../../source/compression/overview.rst:54
+msgid ""
+"The final goal of model compression is to reduce inference latency and "
+"model size. However, existing model compression algorithms mainly use "
+"simulation to check the performance (e.g., accuracy) of compressed model."
+" For example, using masks for pruning algorithms, and storing quantized "
+"values still in float32 for quantization algorithms. Given the output "
+"masks and quantization bits produced by those algorithms, NNI can really "
+"speedup the model."
+msgstr ""
+
+#: ../../source/compression/overview.rst:59
+msgid "The following figure shows how NNI prunes and speeds up your models."
+msgstr ""
+
+#: ../../source/compression/overview.rst:67
+msgid ""
+"The detailed tutorial of Speedup Model with Mask can be found :doc:`here "
+"<../tutorials/pruning_speedup>`. The detailed tutorial of Speedup Model "
+"with Calibration Config can be found :doc:`here "
+"<../tutorials/quantization_speedup>`."
+msgstr ""
+
+#: ../../source/compression/overview.rst:72
+msgid ""
+"NNI's model pruning framework has been upgraded to a more powerful "
+"version (named pruning v2 before nni v2.6). The old version (`named "
+"pruning before nni v2.6 "
+"<https://nni.readthedocs.io/en/v2.6/Compression/pruning.html>`_) will be "
+"out of maintenance. If for some reason you have to use the old pruning, "
+"v2.6 is the last nni version to support old pruning version."
+msgstr ""
+