Merge master into dev-retiarii (#3178)

docs/en_US/FeatureEngineering/Overview.rst

+Feature Engineering with NNI
+============================
+
+We are glad to announce the alpha release for Feature Engineering toolkit on top of NNI, it's still in the experiment phase which might evolve based on user feedback. We'd like to invite you to use, feedback and even contribute.
+
+For now, we support the following feature selector:
+
+
+* `GradientFeatureSelector <./GradientFeatureSelector.rst>`__
+* `GBDTSelector <./GBDTSelector.rst>`__
+
+These selectors are suitable for tabular data(which means it doesn't include image, speech and text data).
+
+In addition, those selector only for feature selection. If you want to:
+1) generate high-order combined features on nni while doing feature selection;
+2) leverage your distributed resources;
+you could try this :githublink:`example <examples/feature_engineering/auto-feature-engineering>`.
+
+How to use?
+-----------
+
+.. code-block:: python
+
+   from nni.feature_engineering.gradient_selector import FeatureGradientSelector
+   # from nni.feature_engineering.gbdt_selector import GBDTSelector
+
+   # load data
+   ...
+   X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)
+
+   # initlize a selector
+   fgs = FeatureGradientSelector(...)
+   # fit data
+   fgs.fit(X_train, y_train)
+   # get improtant features
+   # will return the index with important feature here.
+   print(fgs.get_selected_features(...))
+
+   ...
+
+When using the built-in Selector, you first need to ``import`` a feature selector, and ``initialize`` it. You could call the function ``fit`` in the selector to pass the data to the selector. After that, you could use ``get_seleteced_features`` to get important features. The function parameters in different selectors might be different, so you need to check the docs before using it. 
+
+How to customize?
+-----------------
+
+NNI provides *state-of-the-art* feature selector algorithm in the builtin-selector. NNI also supports to build a feature selector by yourself.
+
+If you want to implement a customized feature selector, you need to:
+
+
+#. Inherit the base FeatureSelector class
+#. Implement *fit* and _get_selected*features* function
+#. Integrate with sklearn (Optional)
+
+Here is an example:
+
+**1. Inherit the base Featureselector Class**
+
+.. code-block:: python
+
+   from nni.feature_engineering.feature_selector import FeatureSelector
+
+   class CustomizedSelector(FeatureSelector):
+       def __init__(self, ...):
+       ...
+
+**2. Implement *fit* and _get_selected*features* Function**
+
+.. code-block:: python
+
+   from nni.tuner import Tuner
+
+   from nni.feature_engineering.feature_selector import FeatureSelector
+
+   class CustomizedSelector(FeatureSelector):
+       def __init__(self, ...):
+       ...
+
+       def fit(self, X, y, **kwargs):
+           """
+           Fit the training data to FeatureSelector
+
+           Parameters
+           ------------
+           X : array-like numpy matrix
+           The training input samples, which shape is [n_samples, n_features].
+           y: array-like numpy matrix
+           The target values (class labels in classification, real numbers in regression). Which shape is [n_samples].
+           """
+           self.X = X
+           self.y = y
+           ...
+
+       def get_selected_features(self):
+           """
+           Get important feature
+
+           Returns
+           -------
+           list :
+           Return the index of the important feature.
+           """
+           ...
+           return self.selected_features_
+
+       ...
+
+**3. Integrate with Sklearn**
+
+``sklearn.pipeline.Pipeline`` can connect models in series, such as feature selector, normalization, and classification/regression to form a typical machine learning problem workflow. 
+The following step could help us to better integrate with sklearn, which means we could treat the customized feature selector as a module of the pipeline.
+
+
+#. Inherit the calss *sklearn.base.BaseEstimator*
+#. Implement _get\ *params* and _set*params* function in *BaseEstimator*
+#. Inherit the class _sklearn.feature\ *selection.base.SelectorMixin*
+#. Implement _get\ *support*\ , *transform* and _inverse*transform* Function in *SelectorMixin*
+
+Here is an example:
+
+**1. Inherit the BaseEstimator Class and its Function**
+
+.. code-block:: python
+
+   from sklearn.base import BaseEstimator
+   from nni.feature_engineering.feature_selector import FeatureSelector
+
+   class CustomizedSelector(FeatureSelector, BaseEstimator):
+       def __init__(self, ...):
+       ...
+
+       def get_params(self, ...):
+           """
+           Get parameters for this estimator.
+           """
+           params = self.__dict__
+           params = {key: val for (key, val) in params.items()
+           if not key.endswith('_')}
+           return params
+
+       def set_params(self, **params):
+           """
+           Set the parameters of this estimator.
+           """
+           for param in params:
+           if hasattr(self, param):
+           setattr(self, param, params[param])
+           return self
+
+**2. Inherit the SelectorMixin Class and its Function**
+
+.. code-block:: python
+
+   from sklearn.base import BaseEstimator
+   from sklearn.feature_selection.base import SelectorMixin
+
+   from nni.feature_engineering.feature_selector import FeatureSelector
+
+   class CustomizedSelector(FeatureSelector, BaseEstimator, SelectorMixin):
+       def __init__(self, ...):
+           ...
+
+       def get_params(self, ...):
+           """
+           Get parameters for this estimator.
+           """
+           params = self.__dict__
+           params = {key: val for (key, val) in params.items()
+           if not key.endswith('_')}
+           return params
+
+       def set_params(self, **params):
+           """
+           Set the parameters of this estimator.
+           """
+           for param in params:
+           if hasattr(self, param):
+           setattr(self, param, params[param])
+           return self
+
+       def get_support(self, indices=False):
+           """
+           Get a mask, or integer index, of the features selected.
+
+           Parameters
+           ----------
+           indices : bool
+           Default False. If True, the return value will be an array of integers, rather than a boolean mask.
+
+           Returns
+           -------
+           list :
+           returns support: An index that selects the retained features from a feature vector.
+           If indices are False, this is a boolean array of shape [# input features], in which an element is True iff its corresponding feature is selected for retention.
+           If indices are True, this is an integer array of shape [# output features] whose values
+           are indices into the input feature vector.
+           """
+           ...
+           return mask
+
+
+       def transform(self, X):
+           """Reduce X to the selected features.
+
+           Parameters
+           ----------
+           X : array
+           which shape is [n_samples, n_features]
+
+           Returns
+           -------
+           X_r : array
+           which shape is [n_samples, n_selected_features]
+           The input samples with only the selected features.
+           """
+           ...
+           return X_r
+
+
+       def inverse_transform(self, X):
+           """
+           Reverse the transformation operation
+
+           Parameters
+           ----------
+           X : array
+           shape is [n_samples, n_selected_features]
+
+           Returns
+           -------
+           X_r : array
+           shape is [n_samples, n_original_features]
+           """
+           ...
+           return X_r
+
+After integrating with Sklearn, we could use the feature selector as follows:
+
+.. code-block:: python
+
+   from sklearn.linear_model import LogisticRegression
+
+   # load data
+   ...
+   X_train, y_train = ...
+
+   # build a ppipeline
+   pipeline = make_pipeline(XXXSelector(...), LogisticRegression())
+   pipeline = make_pipeline(SelectFromModel(ExtraTreesClassifier(n_estimators=50)), LogisticRegression())
+   pipeline.fit(X_train, y_train)
+
+   # score
+   print("Pipeline Score: ", pipeline.score(X_train, y_train))
+
+Benchmark
+---------
+
+``Baseline`` means without any feature selection, we directly pass the data to LogisticRegression. For this benchmark, we only use 10% data from the train as test data. For the GradientFeatureSelector, we only take the top20 features. The metric is the mean accuracy on the given test data and labels.
+
+.. list-table::
+   :header-rows: 1
+   :widths: auto
+
+   * - Dataset
+     - All Features + LR (acc, time, memory)
+     - GradientFeatureSelector + LR (acc, time, memory)
+     - TreeBasedClassifier + LR (acc, time, memory)
+     - #Train
+     - #Feature
+   * - colon-cancer
+     - 0.7547, 890ms, 348MiB
+     - 0.7368, 363ms, 286MiB
+     - 0.7223, 171ms, 1171 MiB
+     - 62
+     - 2,000
+   * - gisette
+     - 0.9725, 215ms, 584MiB
+     - 0.89416, 446ms, 397MiB
+     - 0.9792, 911ms, 234MiB
+     - 6,000
+     - 5,000
+   * - avazu
+     - 0.8834, N/A, N/A
+     - N/A, N/A, N/A
+     - N/A, N/A, N/A
+     - 40,428,967
+     - 1,000,000
+   * - rcv1
+     - 0.9644, 557ms, 241MiB
+     - 0.7333, 401ms, 281MiB
+     - 0.9615, 752ms, 284MiB
+     - 20,242
+     - 47,236
+   * - news20.binary
+     - 0.9208, 707ms, 361MiB
+     - 0.6870, 565ms, 371MiB
+     - 0.9070, 904ms, 364MiB
+     - 19,996
+     - 1,355,191
+   * - real-sim
+     - 0.9681, 433ms, 274MiB
+     - 0.7969, 251ms, 274MiB
+     - 0.9591, 643ms, 367MiB
+     - 72,309
+     - 20,958
+
+
+The dataset of benchmark could be download in `here <https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/>`__
+
+The code could be refenrence ``/examples/feature_engineering/gradient_feature_selector/benchmark_test.py``.
+
+Reference and Feedback
+----------------------
+
+
+* To `report a bug <https://github.com/microsoft/nni/issues/new?template=bug-report.rst>`__ for this feature in GitHub;
+* To `file a feature or improvement request <https://github.com/microsoft/nni/issues/new?template=enhancement.rst>`__ for this feature in GitHub;
+* To know more about :githublink:`Neural Architecture Search with NNI <docs/en_US/NAS/Overview.rst>`\ ;
+* To know more about :githublink:`Model Compression with NNI <docs/en_US/Compression/Overview.rst>`\ ;
+* To know more about :githublink:`Hyperparameter Tuning with NNI <docs/en_US/Tuner/BuiltinTuner.rst>`\ ;

docs/en_US/NAS/Advanced.rst

+Customize a NAS Algorithm
+=========================
+
+Extend the Ability of One-Shot Trainers
+---------------------------------------
+
+Users might want to do multiple things if they are using the trainers on real tasks, for example, distributed training, half-precision training, logging periodically, writing tensorboard, dumping checkpoints and so on. As mentioned previously, some trainers do have support for some of the items listed above; others might not. Generally, there are two recommended ways to add anything you want to an existing trainer: inherit an existing trainer and override, or copy an existing trainer and modify.
+
+Either way, you are walking into the scope of implementing a new trainer. Basically, implementing a one-shot trainer is no different from any traditional deep learning trainer, except that a new concept called mutator will reveal itself. So that the implementation will be different in at least two places:
+
+
+* Initialization
+
+.. code-block:: python
+
+   model = Model()
+   mutator = MyMutator(model)
+
+
+* Training
+
+.. code-block:: python
+
+   for _ in range(epochs):
+       for x, y in data_loader:
+           mutator.reset()  # reset all the choices in model
+           out = model(x)  # like traditional model
+           loss = criterion(out, y)
+           loss.backward()
+           # no difference below
+
+To demonstrate what mutators are for, we need to know how one-shot NAS normally works. Usually, one-shot NAS "co-optimize model weights and architecture weights". It repeatedly: sample an architecture or combination of several architectures from the supernet, train the chosen architectures like traditional deep learning model, update the trained parameters to the supernet, and use the metrics or loss as some signal to guide the architecture sampler. The mutator, is the architecture sampler here, often defined to be another deep-learning model. Therefore, you can treat it as any model, by defining parameters in it and optimizing it with optimizers. One mutator is initialized with exactly one model. Once a mutator is binded to a model, it cannot be rebinded to another model.
+
+``mutator.reset()`` is the core step. That's where all the choices in the model are finalized. The reset result will be always effective, until the next reset flushes the data. After the reset, the model can be seen as a traditional model to do forward-pass and backward-pass.
+
+Finally, mutators provide a method called ``mutator.export()`` that export a dict with architectures to the model. Note that currently this dict this a mapping from keys of mutables to tensors of selection. So in order to dump to json, users need to convert the tensors explicitly into python list.
+
+Meanwhile, NNI provides some useful tools so that users can implement trainers more easily. See `Trainers <./NasReference.rst>`__ for details.
+
+Implement New Mutators
+----------------------
+
+To start with, here is the pseudo-code that demonstrates what happens on ``mutator.reset()`` and ``mutator.export()``.
+
+.. code-block:: python
+
+   def reset(self):
+       self.apply_on_model(self.sample_search())
+
+.. code-block:: python
+
+   def export(self):
+       return self.sample_final()
+
+On reset, a new architecture is sampled with ``sample_search()`` and applied on the model. Then the model is trained for one or more steps in search phase. On export, a new architecture is sampled with ``sample_final()`` and **do nothing to the model**. This is either for checkpoint or exporting the final architecture.
+
+The requirements of return values of ``sample_search()`` and ``sample_final()`` are the same: a mapping from mutable keys to tensors. The tensor can be either a BoolTensor (true for selected, false for negative), or a FloatTensor which applies weight on each candidate. The selected branches will then be computed (in ``LayerChoice``\ , modules will be called; in ``InputChoice``\ , it's just tensors themselves), and reduce with the reduction operation specified in the choices. For most algorithms only worrying about the former part, here is an example of your mutator implementation.
+
+.. code-block:: python
+
+   class RandomMutator(Mutator):
+       def __init__(self, model):
+           super().__init__(model)  # don't forget to call super
+           # do something else
+
+       def sample_search(self):
+           result = dict()
+           for mutable in self.mutables:  # this is all the mutable modules in user model
+               # mutables share the same key will be de-duplicated
+               if isinstance(mutable, LayerChoice):
+                   # decided that this mutable should choose `gen_index`
+                   gen_index = np.random.randint(mutable.length)
+                   result[mutable.key] = torch.tensor([i == gen_index for i in range(mutable.length)], 
+                                                      dtype=torch.bool)
+               elif isinstance(mutable, InputChoice):
+                   if mutable.n_chosen is None:  # n_chosen is None, then choose any number
+                       result[mutable.key] = torch.randint(high=2, size=(mutable.n_candidates,)).view(-1).bool()
+                   # else do something else
+           return result
+
+       def sample_final(self):
+           return self.sample_search()  # use the same logic here. you can do something different
+
+The complete example of random mutator can be found :githublink:`here <src/sdk/pynni/nni/nas/pytorch/random/mutator.py>`.
+
+For advanced usages, e.g., users want to manipulate the way modules in ``LayerChoice`` are executed, they can inherit ``BaseMutator``\ , and overwrite ``on_forward_layer_choice`` and ``on_forward_input_choice``\ , which are the callback implementation of ``LayerChoice`` and ``InputChoice`` respectively. Users can still use property ``mutables`` to get all ``LayerChoice`` and ``InputChoice`` in the model code. For details, please refer to :githublink:`reference <src/sdk/pynni/nni/nas/pytorch>` here to learn more.
+
+.. tip::
+    A useful application of random mutator is for debugging. Use
+
+    .. code-block:: python
+
+        mutator = RandomMutator(model)
+        mutator.reset()
+
+    will immediately set one possible candidate in the search space as the active one.
+
+Implemented a Distributed NAS Tuner
+-----------------------------------
+
+Before learning how to write a distributed NAS tuner, users should first learn how to write a general tuner. read `Customize Tuner <../Tuner/CustomizeTuner.rst>`__ for tutorials.
+
+When users call "\ `nnictl ss_gen <../Tutorial/Nnictl.rst>`__\ " to generate search space file, a search space file like this will be generated:
+
+.. code-block:: json
+
+   {
+       "key_name": {
+           "_type": "layer_choice",
+           "_value": ["op1_repr", "op2_repr", "op3_repr"]
+       },
+       "key_name": {
+           "_type": "input_choice",
+           "_value": {
+               "candidates": ["in1_key", "in2_key", "in3_key"],
+               "n_chosen": 1
+           }
+       }
+   }
+
+This is the exact search space tuners will receive in ``update_search_space``. It's then tuners' responsibility to interpret the search space and generate new candidates in ``generate_parameters``. A valid "parameters" will be in the following format:
+
+.. code-block:: json
+
+   {
+       "key_name": {
+           "_value": "op1_repr",
+           "_idx": 0
+       },
+       "key_name": {
+           "_value": ["in2_key"],
+           "_idex": [1]
+       }
+   }
+
+Send it through ``generate_parameters``\ , and the tuner would look like any HPO tuner. Refer to `SPOS <./SPOS.rst>`__ example code for an example.

docs/en_US/NAS/Benchmarks.rst

+NAS Benchmarks
+==============
+
+..  toctree::
+    :hidden:
+
+    Example Usages <BenchmarksExample>
+
+Introduction
+------------
+
+To imporve the reproducibility of NAS algorithms as well as reducing computing resource requirements, researchers proposed a series of NAS benchmarks such as `NAS-Bench-101 <https://arxiv.org/abs/1902.09635>`__\ , `NAS-Bench-201 <https://arxiv.org/abs/2001.00326>`__\ , `NDS <https://arxiv.org/abs/1905.13214>`__\ , etc. NNI provides a query interface for users to acquire these benchmarks. Within just a few lines of code, researcher are able to evaluate their NAS algorithms easily and fairly by utilizing these benchmarks.
+
+Prerequisites
+-------------
+
+
+* Please prepare a folder to household all the benchmark databases. By default, it can be found at ``${HOME}/.nni/nasbenchmark``. You can place it anywhere you like, and specify it in ``NASBENCHMARK_DIR`` via ``export NASBENCHMARK_DIR=/path/to/your/nasbenchmark`` before importing NNI.
+* Please install ``peewee`` via ``pip3 install peewee``\ , which NNI uses to connect to database.
+
+Data Preparation
+----------------
+
+To avoid storage and legality issues, we do not provide any prepared databases. Please follow the following steps.
+
+
+#. 
+   Clone NNI to your machine and enter ``examples/nas/benchmarks`` directory.
+
+   .. code-block:: bash
+
+      git clone -b ${NNI_VERSION} https://github.com/microsoft/nni
+      cd nni/examples/nas/benchmarks
+
+   Replace ``${NNI_VERSION}`` with a released version name or branch name, e.g., ``v1.9``.
+
+#. 
+   Install dependencies via ``pip3 install -r xxx.requirements.txt``. ``xxx`` can be ``nasbench101``\ , ``nasbench201`` or ``nds``.
+
+#. Generate the database via ``./xxx.sh``. The directory that stores the benchmark file can be configured with ``NASBENCHMARK_DIR`` environment variable, which defaults to ``~/.nni/nasbenchmark``. Note that the NAS-Bench-201 dataset will be downloaded from a google drive.
+
+Please make sure there is at least 10GB free disk space and note that the conversion process can take up to hours to complete.
+
+Example Usages
+--------------
+
+Please refer to `examples usages of Benchmarks API <./BenchmarksExample>`__.
+
+NAS-Bench-101
+-------------
+
+`Paper link <https://arxiv.org/abs/1902.09635>`__ &nbsp; &nbsp; `Open-source <https://github.com/google-research/nasbench>`__
+
+NAS-Bench-101 contains 423,624 unique neural networks, combined with 4 variations in number of epochs (4, 12, 36, 108), each of which is trained 3 times. It is a cell-wise search space, which constructs and stacks a cell by enumerating DAGs with at most 7 operators, and no more than 9 connections. All operators can be chosen from ``CONV3X3_BN_RELU``\ , ``CONV1X1_BN_RELU`` and ``MAXPOOL3X3``\ , except the first operator (always ``INPUT``\ ) and last operator (always ``OUTPUT``\ ).
+
+Notably, NAS-Bench-101 eliminates invalid cells (e.g., there is no path from input to output, or there is redundant computation). Furthermore, isomorphic cells are de-duplicated, i.e., all the remaining cells are computationally unique.
+
+API Documentation
+^^^^^^^^^^^^^^^^^
+
+.. autofunction:: nni.nas.benchmarks.nasbench101.query_nb101_trial_stats
+
+.. autoattribute:: nni.nas.benchmarks.nasbench101.INPUT
+
+.. autoattribute:: nni.nas.benchmarks.nasbench101.OUTPUT
+
+.. autoattribute:: nni.nas.benchmarks.nasbench101.CONV3X3_BN_RELU
+
+.. autoattribute:: nni.nas.benchmarks.nasbench101.CONV1X1_BN_RELU
+
+.. autoattribute:: nni.nas.benchmarks.nasbench101.MAXPOOL3X3
+
+.. autoclass:: nni.nas.benchmarks.nasbench101.Nb101TrialConfig
+
+.. autoclass:: nni.nas.benchmarks.nasbench101.Nb101TrialStats
+
+.. autoclass:: nni.nas.benchmarks.nasbench101.Nb101IntermediateStats
+
+.. autofunction:: nni.nas.benchmarks.nasbench101.graph_util.nasbench_format_to_architecture_repr
+
+.. autofunction:: nni.nas.benchmarks.nasbench101.graph_util.infer_num_vertices
+
+.. autofunction:: nni.nas.benchmarks.nasbench101.graph_util.hash_module
+
+NAS-Bench-201
+-------------
+
+`Paper link <https://arxiv.org/abs/2001.00326>`__ &nbsp; &nbsp; `Open-source API <https://github.com/D-X-Y/NAS-Bench-201>`__ &nbsp; &nbsp;\ `Implementations <https://github.com/D-X-Y/AutoDL-Projects>`__
+
+NAS-Bench-201 is a cell-wise search space that views nodes as tensors and edges as operators. The search space contains all possible densely-connected DAGs with 4 nodes, resulting in 15,625 candidates in total. Each operator (i.e., edge) is selected from a pre-defined operator set (\ ``NONE``\ , ``SKIP_CONNECT``\ , ``CONV_1X1``\ , ``CONV_3X3`` and ``AVG_POOL_3X3``\ ). Training appraoches vary in the dataset used (CIFAR-10, CIFAR-100, ImageNet) and number of epochs scheduled (12 and 200). Each combination of architecture and training approach is repeated 1 - 3 times with different random seeds.
+
+API Documentation
+^^^^^^^^^^^^^^^^^
+
+.. autofunction:: nni.nas.benchmarks.nasbench201.query_nb201_trial_stats
+
+.. autoattribute:: nni.nas.benchmarks.nasbench201.NONE
+
+.. autoattribute:: nni.nas.benchmarks.nasbench201.SKIP_CONNECT
+
+.. autoattribute:: nni.nas.benchmarks.nasbench201.CONV_1X1
+
+.. autoattribute:: nni.nas.benchmarks.nasbench201.CONV_3X3
+
+.. autoattribute:: nni.nas.benchmarks.nasbench201.AVG_POOL_3X3
+
+.. autoclass:: nni.nas.benchmarks.nasbench201.Nb201TrialConfig
+
+.. autoclass:: nni.nas.benchmarks.nasbench201.Nb201TrialStats
+
+.. autoclass:: nni.nas.benchmarks.nasbench201.Nb201IntermediateStats
+
+NDS
+---
+
+`Paper link <https://arxiv.org/abs/1905.13214>`__ &nbsp; &nbsp; `Open-source <https://github.com/facebookresearch/nds>`__
+
+*On Network Design Spaces for Visual Recognition* released trial statistics of over 100,000 configurations (models + hyper-parameters) sampled from multiple model families, including vanilla (feedforward network loosely inspired by VGG), ResNet and ResNeXt (residual basic block and residual bottleneck block) and NAS cells (following popular design from NASNet, Ameoba, PNAS, ENAS and DARTS). Most configurations are trained only once with a fixed seed, except a few that are trained twice or three times.
+
+Instead of storing results obtained with different configurations in separate files, we dump them into one single database to enable comparison in multiple dimensions. Specifically, we use ``model_family`` to distinguish model types, ``model_spec`` for all hyper-parameters needed to build this model, ``cell_spec`` for detailed information on operators and connections if it is a NAS cell, ``generator`` to denote the sampling policy through which this configuration is generated. Refer to API documentation for details.
+
+Available Operators
+-------------------
+
+Here is a list of available operators used in NDS.
+
+.. autoattribute:: nni.nas.benchmarks.nds.constants.NONE
+
+.. autoattribute:: nni.nas.benchmarks.nds.constants.SKIP_CONNECT
+
+.. autoattribute:: nni.nas.benchmarks.nds.constants.AVG_POOL_3X3
+
+.. autoattribute:: nni.nas.benchmarks.nds.constants.MAX_POOL_3X3
+
+.. autoattribute:: nni.nas.benchmarks.nds.constants.MAX_POOL_5X5
+
+.. autoattribute:: nni.nas.benchmarks.nds.constants.MAX_POOL_7X7
+
+.. autoattribute:: nni.nas.benchmarks.nds.constants.CONV_1X1
+
+.. autoattribute:: nni.nas.benchmarks.nds.constants.CONV_3X3
+
+.. autoattribute:: nni.nas.benchmarks.nds.constants.CONV_3X1_1X3
+
+.. autoattribute:: nni.nas.benchmarks.nds.constants.CONV_7X1_1X7
+
+.. autoattribute:: nni.nas.benchmarks.nds.constants.DIL_CONV_3X3
+
+.. autoattribute:: nni.nas.benchmarks.nds.constants.DIL_CONV_5X5
+
+.. autoattribute:: nni.nas.benchmarks.nds.constants.SEP_CONV_3X3
+
+.. autoattribute:: nni.nas.benchmarks.nds.constants.SEP_CONV_5X5
+
+.. autoattribute:: nni.nas.benchmarks.nds.constants.SEP_CONV_7X7
+
+.. autoattribute:: nni.nas.benchmarks.nds.constants.DIL_SEP_CONV_3X3
+
+API Documentation
+^^^^^^^^^^^^^^^^^
+
+.. autofunction:: nni.nas.benchmarks.nds.query_nds_trial_stats
+
+.. autoclass:: nni.nas.benchmarks.nds.NdsTrialConfig
+
+.. autoclass:: nni.nas.benchmarks.nds.NdsTrialStats
+
+.. autoclass:: nni.nas.benchmarks.nds.NdsIntermediateStats

docs/en_US/NAS/CDARTS.rst

+CDARTS
+======
+
+Introduction
+------------
+
+`CDARTS <https://arxiv.org/pdf/2006.10724.pdf>`__ builds a cyclic feedback mechanism between the search and evaluation networks. First, the search network generates an initial topology for evaluation, so that the weights of the evaluation network can be optimized. Second, the architecture topology in the search network is further optimized by the label supervision in classification, as well as the regularization from the evaluation network through feature distillation. Repeating the above cycle results in a joint optimization of the search and evaluation networks, and thus enables the evolution of the topology to fit the final evaluation network.
+
+In implementation of ``CdartsTrainer``\ , it first instantiates two models and two mutators (one for each). The first model is the so-called "search network", which is mutated with a ``RegularizedDartsMutator`` -- a mutator with subtle differences with ``DartsMutator``. The second model is the "evaluation network", which is mutated with a discrete mutator that leverages the previous search network mutator, to sample a single path each time. Trainers train models and mutators alternatively. Users can refer to `paper <https://arxiv.org/pdf/2006.10724.pdf>`__ if they are interested in more details on these trainers and mutators.
+
+Reproduction Results
+--------------------
+
+This is CDARTS based on the NNI platform, which currently supports CIFAR10 search and retrain. ImageNet search and retrain should also be supported, and we provide corresponding interfaces. Our reproduced results on NNI are slightly lower than the paper, but much higher than the original DARTS. Here we show the results of three independent experiments on CIFAR10.
+
+.. list-table::
+   :header-rows: 1
+   :widths: auto
+
+   * - Runs
+     - Paper
+     - NNI
+   * - 1
+     - 97.52
+     - 97.44
+   * - 2
+     - 97.53
+     - 97.48
+   * - 3
+     - 97.58
+     - 97.56
+
+
+Examples
+--------
+
+`Example code <https://github.com/microsoft/nni/tree/master/examples/nas/cdarts>`__
+
+.. code-block:: bash
+
+   # In case NNI code is not cloned. If the code is cloned already, ignore this line and enter code folder.
+   git clone https://github.com/Microsoft/nni.git
+
+   # install apex for distributed training.
+   git clone https://github.com/NVIDIA/apex
+   cd apex
+   python setup.py install --cpp_ext --cuda_ext
+
+   # search the best architecture
+   cd examples/nas/cdarts
+   bash run_search_cifar.sh
+
+   # train the best architecture.
+   bash run_retrain_cifar.sh
+
+Reference
+---------
+
+PyTorch
+^^^^^^^
+
+..  autoclass:: nni.algorithms.nas.pytorch.cdarts.CdartsTrainer
+    :members:
+
+..  autoclass:: nni.algorithms.nas.pytorch.cdarts.RegularizedDartsMutator
+    :members:
+
+..  autoclass:: nni.algorithms.nas.pytorch.cdarts.DartsDiscreteMutator
+    :members:
+
+..  autoclass:: nni.algorithms.nas.pytorch.cdarts.RegularizedMutatorParallel
+    :members:

docs/en_US/NAS/ClassicNas.rst

+.. role:: raw-html(raw)
+   :format: html
+
+
+Classic NAS Algorithms
+======================
+
+In classic NAS algorithms, each architecture is trained as a trial and the NAS algorithm acts as a tuner. Thus, this training mode naturally fits within the NNI hyper-parameter tuning framework, where Tuner generates new architecture for the next trial and trials run in the training service.
+
+Quick Start
+-----------
+
+The following example shows how to use classic NAS algorithms. You can see it is quite similar to NNI hyper-parameter tuning.
+
+.. code-block:: python
+
+   model = Net()
+
+   # get the chosen architecture from tuner and apply it on model
+   get_and_apply_next_architecture(model)
+   train(model)  # your code for training the model
+   acc = test(model)  # test the trained model
+   nni.report_final_result(acc)  # report the performance of the chosen architecture
+
+First, instantiate the model. Search space has been defined in this model through ``LayerChoice`` and ``InputChoice``. After that, user should invoke ``get_and_apply_next_architecture(model)`` to settle down to a specific architecture. This function receives the architecture from tuner (i.e., the classic NAS algorithm) and applies the architecture to ``model``. At this point, ``model`` becomes a specific architecture rather than a search space. Then users are free to train this model just like training a normal PyTorch model. After get the accuracy of this model, users should invoke ``nni.report_final_result(acc)`` to report the result to the tuner.
+
+At this point, trial code is ready. Then, we can prepare an NNI experiment, i.e., search space file and experiment config file. Different from NNI hyper-parameter tuning, search space file is automatically generated from the trial code by running the command (the detailed usage of this command can be found `here <../Tutorial/Nnictl.rst>`__\ ):
+
+``nnictl ss_gen --trial_command="the command for running your trial code"``
+
+A file named ``nni_auto_gen_search_space.json`` is generated by this command. Then put the path of the generated search space in the field ``searchSpacePath`` of the experiment config file. The other fields of the config file can be filled by referring `this tutorial <../Tutorial/QuickStart.rst>`__.
+
+Currently, we only support :githublink:`PPO Tuner <examples/tuners/random_nas_tuner>` for classic NAS. More classic NAS algorithms will be supported soon.
+
+The complete examples can be found :githublink:`here <examples/nas/classic_nas>` for PyTorch and :githublink:`here <examples/nas/classic_nas-tf>` for TensorFlow.
+
+Standalone mode for easy debugging
+----------------------------------
+
+We support a standalone mode for easy debugging, where you can directly run the trial command without launching an NNI experiment. This is for checking whether your trial code can correctly run. The first candidate(s) are chosen for ``LayerChoice`` and ``InputChoice`` in this standalone mode.
+
+:raw-html:`<a name="regulaized-evolution-tuner"></a>`
+
+Regularized Evolution Tuner
+---------------------------
+
+This is a tuner geared for NNI’s Neural Architecture Search (NAS) interface. It uses the `evolution algorithm <https://arxiv.org/pdf/1802.01548.pdf>`__.
+
+The tuner first randomly initializes the number of ``population`` models and evaluates them. After that, every time to produce a new architecture, the tuner randomly chooses the number of ``sample`` architectures from ``population``\ , then mutates the best model in ``sample``\ , the parent model, to produce the child model. The mutation includes the hidden mutation and the op mutation. The hidden state mutation consists of replacing a hidden state with another hidden state from within the cell, subject to the constraint that no loops are formed. The op mutation behaves like the hidden state mutation as far as replacing one op with another op from the op set. Note that keeping the child model the same as its parent is not allowed. After evaluating the child model, it is added to the tail of the ``population``\ , then pops the front one.
+
+Note that **trial concurrency should be less than the population of the model**\ , otherwise NO_MORE_TRIAL exception will be raised.
+
+The whole procedure is summarized by the pseudocode below.
+
+
+.. image:: ../../img/EvoNasTuner.png
+   :target: ../../img/EvoNasTuner.png
+   :alt: 
+

docs/en_US/NAS/Cream.rst

+.. role:: raw-html(raw)
+   :format: html
+
+
+Cream of the Crop: Distilling Prioritized Paths For One-Shot Neural Architecture Search
+=======================================================================================
+
+ **`[Paper] <https://papers.nips.cc/paper/2020/file/d072677d210ac4c03ba046120f0802ec-Paper.pdf>`__ `[Models-Google Drive] <https://drive.google.com/drive/folders/1NLGAbBF9bA1IUAxKlk2VjgRXhr6RHvRW?usp=sharing>`__\ `[Models-Baidu Disk (PWD: wqw6)] <https://pan.baidu.com/s/1TqQNm2s14oEdyNPimw3T9g>`__ `[BibTex] <https://scholar.googleusercontent.com/scholar.bib?q=info:ICWVXc_SsKAJ:scholar.google.com/&output=citation&scisdr=CgUmooXfEMfTi0cV5aU:AAGBfm0AAAAAX7sQ_aXoamdKRaBI12tAVN8REq1VKNwM&scisig=AAGBfm0AAAAAX7sQ_RdYtp6BSro3zgbXVJU2MCgsG730&scisf=4&ct=citation&cd=-1&hl=ja>`__**   :raw-html:`<br/>`
+
+In this work, we present a simple yet effective architecture distillation method. The central idea is that subnetworks can learn collaboratively and teach each other throughout the training process, aiming to boost the convergence of individual models. We introduce the concept of prioritized path, which refers to the architecture candidates exhibiting superior performance during training. Distilling knowledge from the prioritized paths is able to boost the training of subnetworks. Since the prioritized paths are changed on the fly depending on their performance and complexity, the final obtained paths are the cream of the crop. The discovered architectures achieve superior performance compared to the recent `MobileNetV3 <https://arxiv.org/abs/1905.02244>`__ and `EfficientNet <https://arxiv.org/abs/1905.11946>`__ families under aligned settings.
+
+:raw-html:`<div ><img src="https://github.com/microsoft/Cream/blob/main/demo/intro.jpg" width="800"/></div>`
+Reproduced Results
+------------------
+
+Top-1 Accuracy on ImageNet. The top-1 accuracy of Cream search algorithm surpasses MobileNetV3 and EfficientNet-B0/B1 on ImageNet.
+The training with 16 Gpus is a little bit superior than 8 Gpus, as below.
+
+.. list-table::
+   :header-rows: 1
+   :widths: auto
+
+   * - Model (M Flops)
+     - 8Gpus
+     - 16Gpus
+   * - 14M
+     - 53.7
+     - 53.8
+   * - 43M
+     - 65.8
+     - 66.5
+   * - 114M
+     - 72.1
+     - 72.8
+   * - 287M
+     - 76.7
+     - 77.6
+   * - 481M
+     - 78.9
+     - 79.2
+   * - 604M
+     - 79.4
+     - 80.0
+
+
+
+.. raw:: html
+
+   <table style="border: none">
+       <th><img src="./../../img/cream_flops100.jpg" alt="drawing" width="400"/></th>
+       <th><img src="./../../img/cream_flops600.jpg" alt="drawing" width="400"/></th>
+   </table>
+
+
+Examples
+--------
+
+`Example code <https://github.com/microsoft/nni/tree/master/examples/nas/cream>`__
+
+Please run the following scripts in the example folder.
+
+Data Preparation
+----------------
+
+You need to first download the `ImageNet-2012 <http://www.image-net.org/>`__ to the folder ``./data/imagenet`` and move the validation set to the subfolder ``./data/imagenet/val``. To move the validation set, you cloud use the following script: https://raw.githubusercontent.com/soumith/imagenetloader.torch/master/valprep.sh 
+
+Put the imagenet data in ``./data``. It should be like following:
+
+.. code-block:: bash
+
+   ./data/imagenet/train
+   ./data/imagenet/val
+   ...
+
+Quick Start
+-----------
+
+I. Search
+^^^^^^^^^
+
+First, build environments for searching.
+
+.. code-block:: bash
+
+   pip install -r ./requirements
+
+   git clone https://github.com/NVIDIA/apex.git
+   cd apex
+   python setup.py install --cpp_ext --cuda_ext
+
+To search for an architecture, you need to configure the parameters ``FLOPS_MINIMUM`` and ``FLOPS_MAXIMUM`` to specify the desired model flops, such as [0,600]MB flops. You can specify the flops interval by changing these two parameters in ``./configs/train.yaml``
+
+.. code-block:: bash
+
+   FLOPS_MINIMUM: 0 # Minimum Flops of Architecture
+   FLOPS_MAXIMUM: 600 # Maximum Flops of Architecture
+
+For example, if you expect to search an architecture with model flops <= 200M, please set the ``FLOPS_MINIMUM`` and ``FLOPS_MAXIMUM`` to be ``0`` and ``200``.
+
+After you specify the flops of the architectures you would like to search, you can search an architecture now by running:
+
+.. code-block:: bash
+
+   python -m torch.distributed.launch --nproc_per_node=8 ./train.py --cfg ./configs/train.yaml
+
+The searched architectures need to be retrained and obtain the final model. The final model is saved in ``.pth.tar`` format. Retraining code will be released soon.
+
+II. Retrain
+^^^^^^^^^^^
+
+To train searched architectures, you need to configure the parameter ``MODEL_SELECTION`` to specify the model Flops. To specify which model to train, you should add ``MODEL_SELECTION`` in ``./configs/retrain.yaml``. You can select one from [14,43,112,287,481,604], which stands for different Flops(MB).
+
+.. code-block:: bash
+
+   MODEL_SELECTION: 43 # Retrain 43m model
+   MODEL_SELECTION: 481 # Retrain 481m model
+   ......
+
+To train random architectures, you need specify ``MODEL_SELECTION`` to ``-1`` and configure the parameter ``INPUT_ARCH``\ :
+
+.. code-block:: bash
+
+   MODEL_SELECTION: -1 # Train random architectures
+   INPUT_ARCH: [[0], [3], [3, 3], [3, 1, 3], [3, 3, 3, 3], [3, 3, 3], [0]] # Random Architectures
+   ......
+
+After adding ``MODEL_SELECTION`` in ``./configs/retrain.yaml``\ , you need to use the following command to train the model.
+
+.. code-block:: bash
+
+   python -m torch.distributed.launch --nproc_per_node=8 ./retrain.py --cfg ./configs/retrain.yaml
+
+III. Test
+^^^^^^^^^
+
+To test our trained of models, you need to use ``MODEL_SELECTION`` in ``./configs/test.yaml`` to specify which model to test.
+
+.. code-block:: bash
+
+   MODEL_SELECTION: 43 # test 43m model
+   MODEL_SELECTION: 481 # test 470m model
+   ......
+
+After specifying the flops of the model, you need to write the path to the resume model in ``./test.sh``.
+
+.. code-block:: bash
+
+   RESUME_PATH: './43.pth.tar'
+   RESUME_PATH: './481.pth.tar'
+   ......
+
+We provide 14M/43M/114M/287M/481M/604M pretrained models in `google drive <https://drive.google.com/drive/folders/1CQjyBryZ4F20Rutj7coF8HWFcedApUn2>`__ or `[Models-Baidu Disk (password: wqw6)] <https://pan.baidu.com/s/1TqQNm2s14oEdyNPimw3T9g>`__ .
+
+After downloading the pretrained models and adding ``MODEL_SELECTION`` and ``RESUME_PATH`` in './configs/test.yaml', you need to use the following command to test the model.
+
+.. code-block:: bash
+
+   python -m torch.distributed.launch --nproc_per_node=8 ./test.py --cfg ./configs/test.yaml

docs/en_US/NAS/DARTS.rst

+DARTS
+=====
+
+Introduction
+------------
+
+The paper `DARTS: Differentiable Architecture Search <https://arxiv.org/abs/1806.09055>`__ addresses the scalability challenge of architecture search by formulating the task in a differentiable manner. Their method is based on the continuous relaxation of the architecture representation, allowing efficient search of the architecture using gradient descent.
+
+Authors' code optimizes the network weights and architecture weights alternatively in mini-batches. They further explore the possibility that uses second order optimization (unroll) instead of first order, to improve the performance.
+
+Implementation on NNI is based on the `official implementation <https://github.com/quark0/darts>`__ and a `popular 3rd-party repo <https://github.com/khanrc/pt.darts>`__. DARTS on NNI is designed to be general for arbitrary search space. A CNN search space tailored for CIFAR10, same as the original paper, is implemented as a use case of DARTS.
+
+Reproduction Results
+--------------------
+
+The above-mentioned example is meant to reproduce the results in the paper, we do experiments with first and second order optimization. Due to the time limit, we retrain *only the best architecture* derived from the search phase and we repeat the experiment *only once*. Our results is currently on par with the results reported in paper. We will add more results later when ready.
+
+.. list-table::
+   :header-rows: 1
+   :widths: auto
+
+   * - 
+     - In paper
+     - Reproduction
+   * - First order (CIFAR10)
+     - 3.00 +/- 0.14
+     - 2.78
+   * - Second order (CIFAR10)
+     - 2.76 +/- 0.09
+     - 2.80
+
+
+Examples
+--------
+
+CNN Search Space
+^^^^^^^^^^^^^^^^
+
+:githublink:`Example code <examples/nas/darts>`
+
+.. code-block:: bash
+
+   # In case NNI code is not cloned. If the code is cloned already, ignore this line and enter code folder.
+   git clone https://github.com/Microsoft/nni.git
+
+   # search the best architecture
+   cd examples/nas/darts
+   python3 search.py
+
+   # train the best architecture
+   python3 retrain.py --arc-checkpoint ./checkpoints/epoch_49.json
+
+Reference
+---------
+
+PyTorch
+^^^^^^^
+
+..  autoclass:: nni.algorithms.nas.pytorch.darts.DartsTrainer
+    :members:
+
+..  autoclass:: nni.algorithms.nas.pytorch.darts.DartsMutator
+    :members:
+
+Limitations
+-----------
+
+
+* DARTS doesn't support DataParallel and needs to be customized in order to support DistributedDataParallel.

docs/en_US/NAS/ENAS.rst

+ENAS
+====
+
+Introduction
+------------
+
+The paper `Efficient Neural Architecture Search via Parameter Sharing <https://arxiv.org/abs/1802.03268>`__ uses parameter sharing between child models to accelerate the NAS process. In ENAS, a controller learns to discover neural network architectures by searching for an optimal subgraph within a large computational graph. The controller is trained with policy gradient to select a subgraph that maximizes the expected reward on the validation set. Meanwhile the model corresponding to the selected subgraph is trained to minimize a canonical cross entropy loss.
+
+Implementation on NNI is based on the `official implementation in Tensorflow <https://github.com/melodyguan/enas>`__\ , including a general-purpose Reinforcement-learning controller and a trainer that trains target network and this controller alternatively. Following paper, we have also implemented macro and micro search space on CIFAR10 to demonstrate how to use these trainers. Since code to train from scratch on NNI is not ready yet, reproduction results are currently unavailable.
+
+Examples
+--------
+
+CIFAR10 Macro/Micro Search Space
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+:githublink:`Example code <examples/nas/enas>`
+
+.. code-block:: bash
+
+   # In case NNI code is not cloned. If the code is cloned already, ignore this line and enter code folder.
+   git clone https://github.com/Microsoft/nni.git
+
+   # search the best architecture
+   cd examples/nas/enas
+
+   # search in macro search space
+   python3 search.py --search-for macro
+
+   # search in micro search space
+   python3 search.py --search-for micro
+
+   # view more options for search
+   python3 search.py -h
+
+Reference
+---------
+
+PyTorch
+^^^^^^^
+
+..  autoclass:: nni.algorithms.nas.pytorch.enas.EnasTrainer
+    :members:
+
+..  autoclass:: nni.algorithms.nas.pytorch.enas.EnasMutator
+    :members:

docs/en_US/NAS/NasGuide.rst

+One-shot NAS algorithms
+=======================
+
+Besides `classic NAS algorithms <./ClassicNas.rst>`__\ , users also apply more advanced one-shot NAS algorithms to find better models from a search space. There are lots of related works about one-shot NAS algorithms, such as `SMASH <https://arxiv.org/abs/1708.05344>`__\ , `ENAS <https://arxiv.org/abs/1802.03268>`__\ , `DARTS <https://arxiv.org/abs/1808.05377>`__\ , `FBNet <https://arxiv.org/abs/1812.03443>`__\ , `ProxylessNAS <https://arxiv.org/abs/1812.00332>`__\ , `SPOS <https://arxiv.org/abs/1904.00420>`__\ , `Single-Path NAS <https://arxiv.org/abs/1904.02877>`__\ ,  `Understanding One-shot <http://proceedings.mlr.press/v80/bender18a>`__ and `GDAS <https://arxiv.org/abs/1910.04465>`__. One-shot NAS algorithms usually build a supernet containing every candidate in the search space as its subnetwork, and in each step, a subnetwork or combination of several subnetworks is trained.
+
+Currently, several one-shot NAS methods are supported on NNI. For example, ``DartsTrainer``\ , which uses SGD to train architecture weights and model weights iteratively, and ``ENASTrainer``\ , which `uses a controller to train the model <https://arxiv.org/abs/1802.03268>`__. New and more efficient NAS trainers keep emerging in research community and some will be implemented in future releases of NNI.
+
+Search with One-shot NAS Algorithms
+-----------------------------------
+
+Each one-shot NAS algorithm implements a trainer, for which users can find usage details in the description of each algorithm. Here is a simple example, demonstrating how users can use ``EnasTrainer``.
+
+.. code-block:: python
+
+   # this is exactly same as traditional model training
+   model = Net()
+   dataset_train = CIFAR10(root="./data", train=True, download=True, transform=train_transform)
+   dataset_valid = CIFAR10(root="./data", train=False, download=True, transform=valid_transform)
+   criterion = nn.CrossEntropyLoss()
+   optimizer = torch.optim.SGD(model.parameters(), 0.05, momentum=0.9, weight_decay=1.0E-4)
+
+   # use NAS here
+   def top1_accuracy(output, target):
+       # this is the function that computes the reward, as required by ENAS algorithm
+       batch_size = target.size(0)
+       _, predicted = torch.max(output.data, 1)
+       return (predicted == target).sum().item() / batch_size
+
+   def metrics_fn(output, target):
+       # metrics function receives output and target and computes a dict of metrics
+       return {"acc1": top1_accuracy(output, target)}
+
+   from nni.algorithms.nas.pytorch import enas
+   trainer = enas.EnasTrainer(model,
+                              loss=criterion,
+                              metrics=metrics_fn,
+                              reward_function=top1_accuracy,
+                              optimizer=optimizer,
+                              batch_size=128
+                              num_epochs=10,  # 10 epochs
+                              dataset_train=dataset_train,
+                              dataset_valid=dataset_valid,
+                              log_frequency=10)  # print log every 10 steps
+   trainer.train()  # training
+   trainer.export(file="model_dir/final_architecture.json")  # export the final architecture to file
+
+``model`` is the one with `user defined search space <./WriteSearchSpace.rst>`__. Then users should prepare training data and model evaluation metrics. To search from the defined search space, a one-shot algorithm is instantiated, called trainer (e.g., EnasTrainer). The trainer exposes a few arguments that you can customize. For example, the loss function, the metrics function, the optimizer, and the datasets. These should satisfy most usage requirements and we do our best to make sure our built-in trainers work on as many models, tasks, and datasets as possible.
+
+**Note that** when using one-shot NAS algorithms, there is no need to start an NNI experiment. Users can directly run this Python script (i.e., ``train.py``\ ) through ``python3 train.py`` without ``nnictl``. After training, users can export the best one of the found models through ``trainer.export()``.
+
+Each trainer in NNI has its targeted scenario and usage. Some trainers have the assumption that the task is a classification task; some trainers might have a different definition of "epoch" (e.g., an ENAS epoch = some child steps + some controller steps). Most trainers do not have support for distributed training: they won't wrap your model with ``DataParallel`` or ``DistributedDataParallel`` to do that. So after a few tryouts, if you want to actually use the trainers on your very customized applications, you might need to `customize your trainer <./Advanced.rst#extend-the-ability-of-one-shot-trainers>`__.
+
+Furthermore, one-shot NAS can be visualized with our NAS UI. `See more details. <./Visualization.rst>`__
+
+Retrain with Exported Architecture
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+After the search phase, it's time to train the found architecture. Unlike many open-source NAS algorithms who write a whole new model specifically for retraining. We found that the search model and retraining model are usually very similar, and therefore you can construct your final model with the exact same model code. For example
+
+.. code-block:: python
+
+   model = Net()
+   apply_fixed_architecture(model, "model_dir/final_architecture.json")
+
+The JSON is simply a mapping from mutable keys to choices. Choices can be expressed in:
+
+
+* A string: select the candidate with corresponding name.
+* A number: select the candidate with corresponding index.
+* A list of string: select the candidates with corresponding names.
+* A list of number: select the candidates with corresponding indices.
+* A list of boolean values: a multi-hot array.
+
+For example,
+
+.. code-block:: json
+
+   {
+       "LayerChoice1": "conv5x5",
+       "LayerChoice2": 6,
+       "InputChoice3": ["layer1", "layer3"],
+       "InputChoice4": [1, 2],
+       "InputChoice5": [false, true, false, false, true]
+   }
+
+After applying, the model is then fixed and ready for final training. The model works as a single model, and unused parameters and modules are pruned.
+
+Also, refer to `DARTS <./DARTS.rst>`__ for code exemplifying retraining.

docs/en_US/NAS/NasReference.rst

+NAS Reference
+=============
+
+.. contents::
+
+Mutables
+--------
+
+..  autoclass:: nni.nas.pytorch.mutables.Mutable
+    :members:
+
+..  autoclass:: nni.nas.pytorch.mutables.LayerChoice
+    :members:
+
+..  autoclass:: nni.nas.pytorch.mutables.InputChoice
+    :members:
+
+..  autoclass:: nni.nas.pytorch.mutables.MutableScope
+    :members:
+
+Utilities
+^^^^^^^^^
+
+..  autofunction:: nni.nas.pytorch.utils.global_mutable_counting
+
+Mutators
+--------
+
+..  autoclass:: nni.nas.pytorch.base_mutator.BaseMutator
+    :members:
+
+..  autoclass:: nni.nas.pytorch.mutator.Mutator
+    :members:
+
+Random Mutator
+^^^^^^^^^^^^^^
+
+..  autoclass:: nni.algorithms.nas.pytorch.random.RandomMutator
+    :members:
+
+Utilities
+^^^^^^^^^
+
+..  autoclass:: nni.nas.pytorch.utils.StructuredMutableTreeNode
+    :members:
+
+Trainers
+--------
+
+Trainer
+^^^^^^^
+
+..  autoclass:: nni.nas.pytorch.base_trainer.BaseTrainer
+    :members:
+
+..  autoclass:: nni.nas.pytorch.trainer.Trainer
+    :members:
+
+Retrain
+^^^^^^^
+
+..  autofunction:: nni.nas.pytorch.fixed.apply_fixed_architecture
+
+..  autoclass:: nni.nas.pytorch.fixed.FixedArchitecture
+    :members:
+
+Distributed NAS
+^^^^^^^^^^^^^^^
+
+..  autofunction:: nni.algorithms.nas.pytorch.classic_nas.get_and_apply_next_architecture
+
+..  autoclass:: nni.algorithms.nas.pytorch.classic_nas.mutator.ClassicMutator
+    :members:
+
+Callbacks
+^^^^^^^^^
+
+..  autoclass:: nni.nas.pytorch.callbacks.Callback
+    :members:
+
+..  autoclass:: nni.nas.pytorch.callbacks.LRSchedulerCallback
+    :members:
+
+..  autoclass:: nni.nas.pytorch.callbacks.ArchitectureCheckpoint
+    :members:
+
+..  autoclass:: nni.nas.pytorch.callbacks.ModelCheckpoint
+    :members:
+
+Utilities
+^^^^^^^^^
+
+..  autoclass:: nni.nas.pytorch.utils.AverageMeterGroup
+    :members:
+
+..  autoclass:: nni.nas.pytorch.utils.AverageMeter
+    :members:
+
+..  autofunction:: nni.nas.pytorch.utils.to_device

docs/en_US/NAS/Overview.rst

+Neural Architecture Search (NAS) on NNI
+=======================================
+
+.. contents::
+
+Overview
+--------
+
+Automatic neural architecture search is taking an increasingly important role in finding better models. Recent research has proved the feasibility of automatic NAS and has lead to models that beat many manually designed and tuned models. Some representative works are `NASNet <https://arxiv.org/abs/1707.07012>`__\ , `ENAS <https://arxiv.org/abs/1802.03268>`__\ , `DARTS <https://arxiv.org/abs/1806.09055>`__\ , `Network Morphism <https://arxiv.org/abs/1806.10282>`__\ , and `Evolution <https://arxiv.org/abs/1703.01041>`__. Further, new innovations keep emerging.
+
+However, it takes a great effort to implement NAS algorithms, and it's hard to reuse the code base of existing algorithms for new ones. To facilitate NAS innovations (e.g., the design and implementation of new NAS models, the comparison of different NAS models side-by-side, etc.), an easy-to-use and flexible programming interface is crucial.
+
+With this motivation, our ambition is to provide a unified architecture in NNI, accelerate innovations on NAS, and apply state-of-the-art algorithms to real-world problems faster.
+
+With the unified interface, there are two different modes for architecture search. `One <#supported-one-shot-nas-algorithms>`__ is the so-called one-shot NAS, where a super-net is built based on a search space and one-shot training is used to generate a good-performing child model. `The other <#supported-classic-nas-algorithms>`__ is the traditional search-based approach, where each child model within the search space runs as an independent trial. We call it classic NAS.
+
+NNI also provides dedicated `visualization tool <#nas-visualization>`__ for users to check the status of the neural architecture search process.
+
+Supported Classic NAS Algorithms
+--------------------------------
+
+The procedure of classic NAS algorithms is similar to hyper-parameter tuning, users use ``nnictl`` to start experiments and each model runs as a trial. The difference is that search space file is automatically generated from user model (with search space in it) by running ``nnictl ss_gen``. The following table listed supported tuning algorihtms for classic NAS mode. More algorihtms will be supported in future release.
+
+.. list-table::
+   :header-rows: 1
+   :widths: auto
+
+   * - Name
+     - Brief Introduction of Algorithm
+   * - :githublink:`Random Search <examples/tuners/random_nas_tuner>`
+     - Randomly pick a model from search space
+   * - `PPO Tuner </Tuner/BuiltinTuner.html#PPOTuner>`__
+     - PPO Tuner is a Reinforcement Learning tuner based on PPO algorithm. `Reference Paper <https://arxiv.org/abs/1707.06347>`__
+
+
+Please refer to `here <ClassicNas.rst>`__ for the usage of classic NAS algorithms.
+
+Supported One-shot NAS Algorithms
+---------------------------------
+
+NNI currently supports the one-shot NAS algorithms listed below and is adding more. Users can reproduce an algorithm or use it on their own dataset. We also encourage users to implement other algorithms with `NNI API <#use-nni-api>`__\ , to benefit more people.
+
+.. list-table::
+   :header-rows: 1
+   :widths: auto
+
+   * - Name
+     - Brief Introduction of Algorithm
+   * - `ENAS </NAS/ENAS.html>`__
+     - `Efficient Neural Architecture Search via Parameter Sharing <https://arxiv.org/abs/1802.03268>`__. In ENAS, a controller learns to discover neural network architectures by searching for an optimal subgraph within a large computational graph. It uses parameter sharing between child models to achieve fast speed and excellent performance.
+   * - `DARTS </NAS/DARTS.html>`__
+     - `DARTS: Differentiable Architecture Search <https://arxiv.org/abs/1806.09055>`__ introduces a novel algorithm for differentiable network architecture search on bilevel optimization.
+   * - `P-DARTS </NAS/PDARTS.html>`__
+     - `Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluation <https://arxiv.org/abs/1904.12760>`__ is based on DARTS. It introduces an efficient algorithm which allows the depth of searched architectures to grow gradually during the training procedure.
+   * - `SPOS </NAS/SPOS.html>`__
+     - `Single Path One-Shot Neural Architecture Search with Uniform Sampling <https://arxiv.org/abs/1904.00420>`__ constructs a simplified supernet trained with a uniform path sampling method and applies an evolutionary algorithm to efficiently search for the best-performing architectures.
+   * - `CDARTS </NAS/CDARTS.html>`__
+     - `Cyclic Differentiable Architecture Search <https://arxiv.org/abs/****>`__ builds a cyclic feedback mechanism between the search and evaluation networks. It introduces a cyclic differentiable architecture search framework which integrates the two networks into a unified architecture.
+   * - `ProxylessNAS </NAS/Proxylessnas.html>`__
+     - `ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware <https://arxiv.org/abs/1812.00332>`__. It removes proxy, directly learns the architectures for large-scale target tasks and target hardware platforms.
+   * - `TextNAS </NAS/TextNAS.html>`__
+     - `TextNAS: A Neural Architecture Search Space tailored for Text Representation <https://arxiv.org/pdf/1912.10729.pdf>`__. It is a neural architecture search algorithm tailored for text representation.
+
+
+One-shot algorithms run **standalone without nnictl**. NNI supports both PyTorch and Tensorflow 2.X.
+
+Here are some common dependencies to run the examples. PyTorch needs to be above 1.2 to use ``BoolTensor``.
+
+
+* tensorboard
+* PyTorch 1.2+
+* git
+
+Please refer to `here <NasGuide.rst>`__ for the usage of one-shot NAS algorithms.
+
+One-shot NAS can be visualized with our visualization tool. Learn more details `here <./Visualization.rst>`__.
+
+Search Space Zoo
+----------------
+
+NNI provides some predefined search space which can be easily reused. By stacking the extracted cells, user can quickly reproduce those NAS models.
+
+Search Space Zoo contains the following NAS cells:
+
+
+* `DartsCell <./SearchSpaceZoo.rst#DartsCell>`__
+* `ENAS micro <./SearchSpaceZoo.rst#ENASMicroLayer>`__
+* `ENAS macro <./SearchSpaceZoo.rst#ENASMacroLayer>`__
+* `NAS Bench 201 <./SearchSpaceZoo.rst#nas-bench-201>`__
+
+Using NNI API to Write Your Search Space
+----------------------------------------
+
+The programming interface of designing and searching a model is often demanded in two scenarios.
+
+
+#. When designing a neural network, there may be multiple operation choices on a layer, sub-model, or connection, and it's undetermined which one or combination performs best. So, it needs an easy way to express the candidate layers or sub-models.
+#. When applying NAS on a neural network, it needs a unified way to express the search space of architectures, so that it doesn't need to update trial code for different search algorithms.
+
+For using NNI NAS, we suggest users to first go through `the tutorial of NAS API for building search space <./WriteSearchSpace.rst>`__.
+
+NAS Visualization
+-----------------
+
+To help users track the process and status of how the model is searched under specified search space, we developed a visualization tool. It visualizes search space as a super-net and shows importance of subnets and layers/operations, as well as how the importance changes along with the search process. Please refer to `the document of NAS visualization <./Visualization.rst>`__ for how to use it.
+
+Reference and Feedback
+----------------------
+
+
+* To `report a bug <https://github.com/microsoft/nni/issues/new?template=bug-report.rst>`__ for this feature in GitHub;
+* To `file a feature or improvement request <https://github.com/microsoft/nni/issues/new?template=enhancement.rst>`__ for this feature in GitHub.

docs/en_US/NAS/PDARTS.rst

+P-DARTS
+=======
+
+Examples
+--------
+
+:githublink:`Example code <examples/nas/pdarts>`
+
+.. code-block:: bash
+
+   # In case NNI code is not cloned. If the code is cloned already, ignore this line and enter code folder.
+   git clone https://github.com/Microsoft/nni.git
+
+   # search the best architecture
+   cd examples/nas/pdarts
+   python3 search.py
+
+   # train the best architecture, it's the same progress as darts.
+   cd ../darts
+   python3 retrain.py --arc-checkpoint ../pdarts/checkpoints/epoch_2.json

docs/en_US/NAS/Proxylessnas.rst

+ProxylessNAS on NNI
+===================
+
+Introduction
+------------
+
+The paper `ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware <https://arxiv.org/pdf/1812.00332.pdf>`__ removes proxy, it directly learns the architectures for large-scale target tasks and target hardware platforms. They address high memory consumption issue of differentiable NAS and reduce the computational cost to the same level of regular training while still allowing a large candidate set. Please refer to the paper for the details.
+
+Usage
+-----
+
+To use ProxylessNAS training/searching approach, users need to specify search space in their model using `NNI NAS interface <NasGuide.rst>`__\ , e.g., ``LayerChoice``\ , ``InputChoice``. After defining and instantiating the model, the following work can be leaved to ProxylessNasTrainer by instantiating the trainer and passing the model to it.
+
+.. code-block:: python
+
+   trainer = ProxylessNasTrainer(model,
+                                 model_optim=optimizer,
+                                 train_loader=data_provider.train,
+                                 valid_loader=data_provider.valid,
+                                 device=device,
+                                 warmup=True,
+                                 ckpt_path=args.checkpoint_path,
+                                 arch_path=args.arch_path)
+   trainer.train()
+   trainer.export(args.arch_path)
+
+The complete example code can be found :githublink:`here <examples/nas/proxylessnas>`.
+
+**Input arguments of ProxylessNasTrainer**
+
+
+* **model** (*PyTorch model, required*\ ) - The model that users want to tune/search. It has mutables to specify search space.
+* **model_optim** (*PyTorch optimizer, required*\ ) - The optimizer users want to train the model.
+* **device** (*device, required*\ ) - The devices that users provide to do the train/search. The trainer applies data parallel on the model for users.
+* **train_loader** (*PyTorch data loader, required*\ ) - The data loader for training set.
+* **valid_loader** (*PyTorch data loader, required*\ ) - The data loader for validation set.
+* **label_smoothing** (*float, optional, default = 0.1*\ ) - The degree of label smoothing.
+* **n_epochs** (*int, optional, default = 120*\ ) - The number of epochs to train/search.
+* **init_lr** (*float, optional, default = 0.025*\ ) - The initial learning rate for training the model.
+* **binary_mode** (*'two', 'full', or 'full_v2', optional, default = 'full_v2'*\ ) - The forward/backward mode for the binary weights in mutator. 'full' means forward all the candidate ops, 'two' means only forward two sampled ops, 'full_v2' means recomputing the inactive ops during backward.
+* **arch_init_type** (*'normal' or 'uniform', optional, default = 'normal'*\ ) - The way to init architecture parameters.
+* **arch_init_ratio** (*float, optional, default = 1e-3*\ ) - The ratio to init architecture parameters.
+* **arch_optim_lr** (*float, optional, default = 1e-3*\ ) - The learning rate of the architecture parameters optimizer.
+* **arch_weight_decay** (*float, optional, default = 0*\ ) - Weight decay of the architecture parameters optimizer.
+* **grad_update_arch_param_every** (*int, optional, default = 5*\ ) - Update architecture weights every this number of minibatches.
+* **grad_update_steps** (*int, optional, default = 1*\ ) - During each update of architecture weights, the number of steps to train architecture weights.
+* **warmup** (*bool, optional, default = True*\ ) - Whether to do warmup.
+* **warmup_epochs** (*int, optional, default = 25*\ ) - The number of epochs to do during warmup.
+* **arch_valid_frequency** (*int, optional, default = 1*\ ) - The frequency of printing validation result.
+* **load_ckpt** (*bool, optional, default = False*\ ) - Whether to load checkpoint.
+* **ckpt_path** (*str, optional, default = None*\ ) - checkpoint path, if load_ckpt is True, ckpt_path cannot be None.
+* **arch_path** (*str, optional, default = None*\ ) - The path to store chosen architecture.
+
+Implementation
+--------------
+
+The implementation on NNI is based on the `offical implementation <https://github.com/mit-han-lab/ProxylessNAS>`__. The official implementation supports two training approaches: gradient descent and RL based, and support different targeted hardware, including 'mobile', 'cpu', 'gpu8', 'flops'. In our current implementation on NNI, gradient descent training approach is supported, but has not supported different hardwares. The complete support is ongoing.
+
+Below we will describe implementation details. Like other one-shot NAS algorithms on NNI, ProxylessNAS is composed of two parts: *search space* and *training approach*. For users to flexibly define their own search space and use built-in ProxylessNAS training approach, we put the specified search space in :githublink:`example code <examples/nas/proxylessnas>` using :githublink:`NNI NAS interface <src/sdk/pynni/nni/nas/pytorch/proxylessnas>`.
+
+
+.. image:: ../../img/proxylessnas.png
+   :target: ../../img/proxylessnas.png
+   :alt: 
+
+
+ProxylessNAS training approach is composed of ProxylessNasMutator and ProxylessNasTrainer. ProxylessNasMutator instantiates MixedOp for each mutable (i.e., LayerChoice), and manage architecture weights in MixedOp. **For DataParallel**\ , architecture weights should be included in user model. Specifically, in ProxylessNAS implementation, we add MixedOp to the corresponding mutable (i.e., LayerChoice) as a member variable. The mutator also exposes two member functions, i.e., ``arch_requires_grad``\ , ``arch_disable_grad``\ , for the trainer to control the training of architecture weights.
+
+ProxylessNasMutator also implements the forward logic of the mutables (i.e., LayerChoice).
+
+Reproduce Results
+-----------------
+
+To reproduce the result, we first run the search, we found that though it runs many epochs the chosen architecture converges at the first several epochs. This is probably induced by hyper-parameters or the implementation, we are working on it. The test accuracy of the found architecture is top1: 72.31, top5: 90.26.

docs/en_US/NAS/SPOS.rst

+Single Path One-Shot (SPOS)
+===========================
+
+Introduction
+------------
+
+Proposed in `Single Path One-Shot Neural Architecture Search with Uniform Sampling <https://arxiv.org/abs/1904.00420>`__ is a one-shot NAS method that addresses the difficulties in training One-Shot NAS models by constructing a simplified supernet trained with an uniform path sampling method, so that all underlying architectures (and their weights) get trained fully and equally. An evolutionary algorithm is then applied to efficiently search for the best-performing architectures without any fine tuning.
+
+Implementation on NNI is based on `official repo <https://github.com/megvii-model/SinglePathOneShot>`__. We implement a trainer that trains the supernet and a evolution tuner that leverages the power of NNI framework that speeds up the evolutionary search phase. We have also shown 
+
+Examples
+--------
+
+Here is a use case, which is the search space in paper, and the way to use flops limit to perform uniform sampling.
+
+:githublink:`Example code <examples/nas/spos>`
+
+Requirements
+^^^^^^^^^^^^
+
+NVIDIA DALI >= 0.16 is needed as we use DALI to accelerate the data loading of ImageNet. `Installation guide <https://docs.nvidia.com/deeplearning/sdk/dali-developer-guide/docs/installation.html>`__
+
+Download the flops lookup table from `here <https://1drv.ms/u/s!Am_mmG2-KsrnajesvSdfsq_cN48?e=aHVppN>`__ (maintained by `Megvii <https://github.com/megvii-model>`__\ ).
+Put ``op_flops_dict.pkl`` and ``checkpoint-150000.pth.tar`` (if you don't want to retrain the supernet) under ``data`` directory.
+
+Prepare ImageNet in the standard format (follow the script `here <https://gist.github.com/BIGBALLON/8a71d225eff18d88e469e6ea9b39cef4>`__\ ). Linking it to ``data/imagenet`` will be more convenient.
+
+After preparation, it's expected to have the following code structure:
+
+.. code-block:: bash
+
+   spos
+   ├── architecture_final.json
+   ├── blocks.py
+   ├── config_search.yml
+   ├── data
+   │   ├── imagenet
+   │   │   ├── train
+   │   │   └── val
+   │   └── op_flops_dict.pkl
+   ├── dataloader.py
+   ├── network.py
+   ├── readme.md
+   ├── scratch.py
+   ├── supernet.py
+   ├── tester.py
+   ├── tuner.py
+   └── utils.py
+
+Step 1. Train Supernet
+^^^^^^^^^^^^^^^^^^^^^^
+
+.. code-block:: bash
+
+   python supernet.py
+
+Will export the checkpoint to ``checkpoints`` directory, for the next step.
+
+NOTE: The data loading used in the official repo is `slightly different from usual <https://github.com/megvii-model/SinglePathOneShot/issues/5>`__\ , as they use BGR tensor and keep the values between 0 and 255 intentionally to align with their own DL framework. The option ``--spos-preprocessing`` will simulate the behavior used originally and enable you to use the checkpoints pretrained.
+
+Step 2. Evolution Search
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+Single Path One-Shot leverages evolution algorithm to search for the best architecture. The tester, which is responsible for testing the sampled architecture, recalculates all the batch norm for a subset of training images, and evaluates the architecture on the full validation set.
+
+In order to make the tuner aware of the flops limit and have the ability to calculate the flops, we created a new tuner called ``EvolutionWithFlops`` in ``tuner.py``\ , inheriting the tuner in SDK.
+
+To have a search space ready for NNI framework, first run
+
+.. code-block:: bash
+
+   nnictl ss_gen -t "python tester.py"
+
+This will generate a file called ``nni_auto_gen_search_space.json``\ , which is a serialized representation of your search space.
+
+By default, it will use ``checkpoint-150000.pth.tar`` downloaded previously. In case you want to use the checkpoint trained by yourself from the last step, specify ``--checkpoint`` in the command in ``config_search.yml``.
+
+Then search with evolution tuner.
+
+.. code-block:: bash
+
+   nnictl create --config config_search.yml
+
+The final architecture exported from every epoch of evolution can be found in ``checkpoints`` under the working directory of your tuner, which, by default, is ``$HOME/nni-experiments/your_experiment_id/log``.
+
+Step 3. Train from Scratch
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. code-block:: bash
+
+   python scratch.py
+
+By default, it will use ``architecture_final.json``. This architecture is provided by the official repo (converted into NNI format). You can use any architecture (e.g., the architecture found in step 2) with ``--fixed-arc`` option.
+
+Reference
+---------
+
+PyTorch
+^^^^^^^
+
+..  autoclass:: nni.algorithms.nas.pytorch.spos.SPOSEvolution
+    :members:
+
+..  autoclass:: nni.algorithms.nas.pytorch.spos.SPOSSupernetTrainer
+    :members:
+
+..  autoclass:: nni.algorithms.nas.pytorch.spos.SPOSSupernetTrainingMutator
+    :members:
+
+Known Limitations
+-----------------
+
+
+* Block search only. Channel search is not supported yet.
+* Only GPU version is provided here.
+
+Current Reproduction Results
+----------------------------
+
+Reproduction is still undergoing. Due to the gap between official release and original paper, we compare our current results with official repo (our run) and paper.
+
+
+* Evolution phase is almost aligned with official repo. Our evolution algorithm shows a converging trend and reaches ~65% accuracy at the end of search. Nevertheless, this result is not on par with paper. For details, please refer to `this issue <https://github.com/megvii-model/SinglePathOneShot/issues/6>`__.
+* Retrain phase is not aligned. Our retraining code, which uses the architecture released by the authors, reaches 72.14% accuracy, still having a gap towards 73.61% by official release and 74.3% reported in original paper.

docs/en_US/NAS/SearchSpaceZoo.rst

+.. role:: raw-html(raw)
+   :format: html
+
+
+Search Space Zoo
+================
+
+DartsCell
+---------
+
+DartsCell is extracted from :githublink:`CNN model <examples/nas/darts>`. A DartsCell is a directed acyclic graph containing an ordered sequence of N nodes and each node stands for a latent representation (e.g. feature map in a convolutional network). Directed edges from Node 1 to Node 2 are associated with some operations that transform Node 1 and the result is stored on Node 2. The `Candidate operators <#predefined-operations-darts>`__ between nodes is predefined and unchangeable. One edge represents an operation that chosen from the predefined ones to be applied to the starting node of the edge. One cell contains two input nodes, a single output node, and other ``n_node`` nodes. The input nodes are defined as the cell outputs in the previous two layers. The output of the cell is obtained by applying a reduction operation (e.g. concatenation) to all the intermediate nodes. To make the search space continuous, the categorical choice of a particular operation is relaxed to a softmax over all possible operations. By adjusting the weight of softmax on every node, the operation with the highest probability is chosen to be part of the final structure. A CNN model can be formed by stacking several cells together, which builds a search space. Note that, in DARTS paper all cells in the model share the same structure.
+
+One structure in the Darts search space is shown below. Note that, NNI merges the last one of the four intermediate nodes and the output node.
+
+
+.. image:: ../../img/NAS_Darts_cell.svg
+   :target: ../../img/NAS_Darts_cell.svg
+   :alt: 
+
+
+The predefined operators are shown `here <#predefined-operations-darts>`__.
+
+..  autoclass:: nni.nas.pytorch.search_space_zoo.DartsCell
+    :members:
+
+Example code
+^^^^^^^^^^^^
+
+:githublink:`example code <examples/nas/search_space_zoo/darts_example.py>`
+
+.. code-block:: bash
+
+   git clone https://github.com/Microsoft/nni.git
+   cd nni/examples/nas/search_space_zoo
+   # search the best structure
+   python3 darts_example.py
+
+:raw-html:`<a name="predefined-operations-darts"></a>`
+
+Candidate operators
+^^^^^^^^^^^^^^^^^^^
+
+All supported operators for Darts are listed below.
+
+
+* 
+  MaxPool / AvgPool
+
+
+  * MaxPool: Call ``torch.nn.MaxPool2d``. This operation applies a 2D max pooling over all input channels. Its parameters ``kernel_size=3`` and ``padding=1`` are fixed. The pooling result will pass through a BatchNorm2d then return as the result.
+  * 
+    AvgPool: Call ``torch.nn.AvgPool2d``. This operation applies a 2D average pooling over all input channels. Its parameters ``kernel_size=3`` and ``padding=1`` are fixed. The pooling result will pass through a BatchNorm2d then return as the result.
+
+    MaxPool / AvgPool with ``kernel_size=3`` and ``padding=1`` followed by BatchNorm2d
+
+..  autoclass:: nni.nas.pytorch.search_space_zoo.darts_ops.PoolBN
+
+* 
+  SkipConnect
+
+    There is no operation between two nodes. Call ``torch.nn.Identity`` to forward what it gets to the output.
+
+* 
+  Zero operation
+
+    There is no connection between two nodes.
+
+* 
+  DilConv3x3 / DilConv5x5
+
+    :raw-html:`<a name="DilConv"></a>`\ DilConv3x3: (Dilated) depthwise separable Conv. It's a 3x3 depthwise convolution with ``C_in`` groups, followed by a 1x1 pointwise convolution. It reduces the amount of parameters. Input is first passed through relu, then DilConv and finally batchNorm2d. **Note that the operation is not Dilated Convolution, but we follow the convention in NAS papers to name it DilConv.** 3x3 DilConv has parameters ``kernel_size=3``\ , ``padding=1`` and 5x5 DilConv has parameters ``kernel_size=5``\ , ``padding=4``.
+
+  ..  autoclass:: nni.nas.pytorch.search_space_zoo.darts_ops.DilConv
+
+* 
+  SepConv3x3 / SepConv5x5
+
+    Composed of two DilConvs with fixed ``kernel_size=3``\ , ``padding=1`` or ``kernel_size=5``\ , ``padding=2`` sequentially.
+
+  ..  autoclass:: nni.nas.pytorch.search_space_zoo.darts_ops.SepConv
+
+ENASMicroLayer
+--------------
+
+This layer is extracted from the model designed :githublink:`here <examples/nas/enas>`. A model contains several blocks that share the same architecture. A block is made up of some normal layers and reduction layers, ``ENASMicroLayer`` is a unified implementation of the two types of layers. The only difference between the two layers is that reduction layers apply all operations with ``stride=2``.
+
+ENAS Micro employs a DAG with N nodes in one cell, where the nodes represent local computations, and the edges represent the flow of information between the N nodes. One cell contains two input nodes and a single output node. The following nodes choose two previous nodes as input and apply two operations from `predefined ones <#predefined-operations-enas>`__ then add them as the output of this node. For example, Node 4 chooses Node 1 and Node 3 as inputs then applies ``MaxPool`` and ``AvgPool`` on the inputs respectively, then adds and sums them as the output of Node 4. Nodes that are not served as input for any other node are viewed as the output of the layer. If there are multiple output nodes, the model will calculate the average of these nodes as the layer output.
+
+The ENAS micro search space is shown below.
+
+
+.. image:: ../../img/NAS_ENAS_micro.svg
+   :target: ../../img/NAS_ENAS_micro.svg
+   :alt: 
+ 
+
+The predefined operators can be seen `here <#predefined-operations-enas>`__.
+
+..  autoclass:: nni.nas.pytorch.search_space_zoo.ENASMicroLayer
+    :members:
+
+The Reduction Layer is made up of two Conv operations followed by BatchNorm, each of them will output ``C_out//2`` channels and concat them in channels as the output. The Convolution has ``kernel_size=1`` and ``stride=2``\ , and they perform alternate sampling on the input to reduce the resolution without loss of information. This layer is wrapped in ``ENASMicroLayer``.
+
+Example code
+^^^^^^^^^^^^
+
+:githublink:`example code <examples/nas/search_space_zoo/enas_micro_example.py>`
+
+.. code-block:: bash
+
+   git clone https://github.com/Microsoft/nni.git
+   cd nni/examples/nas/search_space_zoo
+   # search the best cell structure
+   python3 enas_micro_example.py
+
+:raw-html:`<a name="predefined-operations-enas"></a>`
+
+Candidate operators
+^^^^^^^^^^^^^^^^^^^
+
+All supported operators for ENAS micro search are listed below.
+
+
+* 
+  MaxPool / AvgPool
+
+
+  * MaxPool: Call ``torch.nn.MaxPool2d``. This operation applies a 2D max pooling over all input channels followed by BatchNorm2d. Its parameters are fixed to ``kernel_size=3``\ , ``stride=1`` and ``padding=1``.
+  * AvgPool: Call ``torch.nn.AvgPool2d``. This operation applies a 2D average pooling over all input channels followed by BatchNorm2d. Its parameters are fixed to ``kernel_size=3``\ , ``stride=1`` and ``padding=1``.
+
+..  autoclass:: nni.nas.pytorch.search_space_zoo.enas_ops.Pool
+
+* 
+  SepConv
+
+
+  * SepConvBN3x3: ReLU followed by a `DilConv <#DilConv>`__ and BatchNorm. Convolution parameters are ``kernel_size=3``\ , ``stride=1`` and ``padding=1``.
+  * 
+    SepConvBN5x5: Do the same operation as the previous one but it has different kernel sizes and paddings, which is set to 5 and 2 respectively.
+
+..  autoclass:: nni.nas.pytorch.search_space_zoo.enas_ops.SepConvBN
+
+* 
+  SkipConnect
+
+    Call ``torch.nn.Identity`` to connect directly to the next cell.
+
+ENASMacroLayer
+--------------
+
+In Macro search, the controller makes two decisions for each layer: i) the `operation <#macro-operations>`__ to perform on the result of the previous layer, ii) which the previous layer to connect to for SkipConnects. ENAS uses a controller to design the whole model architecture instead of one of its components. The output of operations is going to concat with the tensor of the chosen layer for SkipConnect. NNI provides `predefined operators <#macro-operations>`__ for macro search, which are listed in `Candidate operators <#macro-operations>`__.
+
+Part of one structure in the ENAS macro search space is shown below.
+
+
+.. image:: ../../img/NAS_ENAS_macro.svg
+   :target: ../../img/NAS_ENAS_macro.svg
+   :alt: 
+
+
+..  autoclass:: nni.nas.pytorch.search_space_zoo.ENASMacroLayer
+    :members:
+
+To describe the whole search space, NNI provides a model, which is built by stacking the layers.
+
+..  autoclass:: nni.nas.pytorch.search_space_zoo.ENASMacroGeneralModel
+    :members:
+
+Example code
+^^^^^^^^^^^^
+
+:githublink:`example code <examples/nas/search_space_zoo/enas_macro_example.py>`
+
+.. code-block:: bash
+
+   git clone https://github.com/Microsoft/nni.git
+   cd nni/examples/nas/search_space_zoo
+   # search the best cell structure
+   python3 enas_macro_example.py
+
+:raw-html:`<a name="macro-operations"></a>`
+
+Candidate operators
+^^^^^^^^^^^^^^^^^^^
+
+All supported operators for ENAS macro search are listed below.
+
+
+* 
+  ConvBranch
+
+    All input first passes into a StdConv, which is made up of a 1x1Conv followed by BatchNorm2d and ReLU. Then the intermediate result goes through one of the operations listed below. The final result is calculated through a BatchNorm2d and ReLU as post-procedure.
+
+
+  * Separable Conv3x3: If ``separable=True``\ , the cell will use `SepConv <#DilConv>`__ instead of normal Conv operation. SepConv's ``kernel_size=3``\ , ``stride=1`` and ``padding=1``.
+  * Separable Conv5x5: SepConv's ``kernel_size=5``\ , ``stride=1`` and ``padding=2``.
+  * Normal Conv3x3: If ``separable=False``\ , the cell will use a normal Conv operations with ``kernel_size=3``\ , ``stride=1`` and ``padding=1``.
+  * 
+    Normal Conv5x5: Conv's ``kernel_size=5``\ , ``stride=1`` and ``padding=2``.
+
+..  autoclass:: nni.nas.pytorch.search_space_zoo.enas_ops.ConvBranch
+
+* 
+  PoolBranch
+
+    All input first passes into a StdConv, which is made up of a 1x1Conv followed by BatchNorm2d and ReLU. Then the intermediate goes through pooling operation followed by BatchNorm.
+
+
+  * AvgPool: Call ``torch.nn.AvgPool2d``. This operation applies a 2D average pooling over all input channels. Its parameters are fixed to ``kernel_size=3``\ , ``stride=1`` and ``padding=1``.
+  * 
+    MaxPool: Call ``torch.nn.MaxPool2d``. This operation applies a 2D max pooling over all input channels. Its parameters are fixed to ``kernel_size=3``\ , ``stride=1`` and ``padding=1``.
+
+..  autoclass:: nni.nas.pytorch.search_space_zoo.enas_ops.PoolBranch
+
+NAS-Bench-201
+-------------
+
+NAS Bench 201 defines a unified search space, which is algorithm agnostic. The predefined skeleton consists of a stack of cells that share the same architecture. Every cell contains four nodes and a DAG is formed by connecting edges among them, where the node represents the sum of feature maps and the edge stands for an operation transforming a tensor from the source node to the target node. The predefined candidate operators can be found in `Candidate operators <#nas-bench-201-reference>`__.
+
+The search space of NAS Bench 201 is shown below.
+
+
+.. image:: ../../img/NAS_Bench_201.svg
+   :target: ../../img/NAS_Bench_201.svg
+   :alt: 
+
+
+..  autoclass:: nni.nas.pytorch.nasbench201.NASBench201Cell
+    :members:
+
+Example code
+^^^^^^^^^^^^
+
+:githublink:`example code <examples/nas/search_space_zoo/nas_bench_201.py>`
+
+.. code-block:: bash
+
+   # for structure searching
+   git clone https://github.com/Microsoft/nni.git
+   cd nni/examples/nas/search_space_zoo
+   python3 nas_bench_201.py
+
+:raw-html:`<a name="nas-bench-201-reference"></a>`
+
+Candidate operators
+^^^^^^^^^^^^^^^^^^^
+
+All supported operators for NAS Bench 201 are listed below.
+
+
+* 
+  AvgPool
+
+  If the number of input channels is not equal to the number of output channels, the input will first pass through a ``ReLUConvBN`` layer with ``kernel_size=1``\ , ``stride=1``\ , ``padding=0``\ , and ``dilation=0``.
+  Call ``torch.nn.AvgPool2d``. This operation applies a 2D average pooling over all input channels followed by BatchNorm2d. Its parameters are fixed to ``kernel_size=3`` and ``padding=1``.
+
+..  autoclass:: nni.nas.pytorch.nasbench201.nasbench201_ops.Pooling
+    :members:
+
+* 
+  Conv
+
+
+  * Conv1x1: Consist of a sequence of ReLU, ``nn.Cinv2d`` and BatchNorm. The Conv operation's parameter is fixed to ``kernal_size=1``\ , ``padding=0``\ , and ``dilation=1``.
+  * Conv3x3: Consist of a sequence of ReLU, ``nn.Cinv2d`` and BatchNorm. The Conv operation's parameter is fixed to ``kernal_size=3``\ , ``padding=1``\ , and ``dilation=1``.
+
+..  autoclass:: nni.nas.pytorch.nasbench201.nasbench201_ops.ReLUConvBN
+    :members:
+
+* 
+  SkipConnect
+
+  Call ``torch.nn.Identity`` to connect directly to the next cell.
+
+* 
+  Zeroize
+
+  Generate zero tensors indicating there is no connection from the source node to the target node.
+
+..  autoclass:: nni.nas.pytorch.nasbench201.nasbench201_ops.Zero
+    :members:

docs/en_US/NAS/TextNAS.rst

+TextNAS
+=======
+
+Introduction
+------------
+
+This is the implementation of the TextNAS algorithm proposed in the paper `TextNAS: A Neural Architecture Search Space tailored for Text Representation <https://arxiv.org/pdf/1912.10729.pdf>`__. TextNAS is a neural architecture search algorithm tailored for text representation, more specifically, TextNAS is based on a novel search space consists of operators widely adopted to solve various NLP tasks, and TextNAS also supports multi-path ensemble within a single network to balance the width and depth of the architecture. 
+
+The search space of TextNAS contains: 
+
+.. code-block:: bash
+
+   * 1-D convolutional operator with filter size 1, 3, 5, 7 
+   * recurrent operator (bi-directional GRU) 
+   * self-attention operator
+   * pooling operator (max/average)
+
+
+Following the ENAS algorithm, TextNAS also utilizes parameter sharing to accelerate the search speed and adopts a reinforcement-learning controller for the architecture sampling and generation. Please refer to the paper for more details of TextNAS.
+
+Preparation
+-----------
+
+Prepare the word vectors and SST dataset, and organize them in data directory as shown below:
+
+.. code-block:: bash
+
+   textnas
+   ├── data
+   │   ├── sst
+   │   │   └── trees
+   │   │       ├── dev.txt
+   │   │       ├── test.txt
+   │   │       └── train.txt
+   │   └── glove.840B.300d.txt
+   ├── dataloader.py
+   ├── model.py
+   ├── ops.py
+   ├── README.md
+   ├── search.py
+   └── utils.py
+
+The following link might be helpful for finding and downloading the corresponding dataset:
+
+
+* `GloVe: Global Vectors for Word Representation <https://nlp.stanford.edu/projects/glove/>`__
+
+  * `glove.840B.300d.txt <http://nlp.stanford.edu/data/glove.840B.300d.zip>`__
+
+* `Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank <https://nlp.stanford.edu/sentiment/>`__
+
+  * `trainDevTestTrees_PTB.zip <https://nlp.stanford.edu/sentiment/trainDevTestTrees_PTB.zip>`__
+
+Examples
+--------
+
+Search Space
+^^^^^^^^^^^^
+
+:githublink:`Example code <examples/nas/textnas>`
+
+.. code-block:: bash
+
+   # In case NNI code is not cloned. If the code is cloned already, ignore this line and enter code folder.
+   git clone https://github.com/Microsoft/nni.git
+
+   # search the best architecture
+   cd examples/nas/textnas
+
+   # view more options for search
+   python3 search.py -h
+
+After each search epoch, 10 sampled architectures will be tested directly. Their performances are expected to be 40% - 42% after 10 epochs.
+
+By default, 20 sampled architectures will be exported into ``checkpoints`` directory for next step.
+
+retrain
+^^^^^^^
+
+.. code-block:: bash
+
+   # In case NNI code is not cloned. If the code is cloned already, ignore this line and enter code folder.
+   git clone https://github.com/Microsoft/nni.git
+
+   # search the best architecture
+   cd examples/nas/textnas
+
+   # default to retrain on sst-2
+   sh run_retrain.sh
+
+Reference
+---------
+
+TextNAS directly uses EnasTrainer, please refer to `ENAS <./ENAS.rst>`__ for the trainer APIs.

docs/en_US/NAS/Visualization.rst

+NAS Visualization (Experimental)
+================================
+
+Built-in Trainers Support
+-------------------------
+
+Currently, only ENAS and DARTS support visualization. Examples of `ENAS <./ENAS.md>`__ and `DARTS <./DARTS.rst>`__ has demonstrated how to enable visualization in your code, namely, adding this before ``trainer.train()``\ :
+
+.. code-block:: python
+
+   trainer.enable_visualization()
+
+This will create a directory ``logs/<current_time_stamp>`` in your working folder, in which you will find two files ``graph.json`` and ``log``.
+
+You don't have to wait until your program finishes to launch NAS UI, but it's important that these two files have been already created. Launch NAS UI with
+
+.. code-block:: bash
+
+   nnictl webui nas --logdir logs/<current_time_stamp> --port <port>
+
+Visualize a Customized Trainer
+------------------------------
+
+If you are interested in how to customize a trainer, please read this `doc <./Advanced.rst#extend-the-ability-of-one-shot-trainers>`__.
+
+You should do two modifications to an existing trainer to enable visualization:
+
+
+#. Export your graph before training, with
+
+.. code-block:: python
+
+   vis_graph = self.mutator.graph(inputs)
+   # `inputs` is a dummy input to your model. For example, torch.randn((1, 3, 32, 32)).cuda()
+   # If your model has multiple inputs, it should be a tuple.
+   with open("/path/to/your/logdir/graph.json", "w") as f:
+       json.dump(vis_graph, f)
+
+
+#. Logging the choices you've made. You can do it once per epoch, once per mini-batch or whatever frequency you'd like.
+
+.. code-block:: python
+
+   def __init__(self):
+       # ...
+       self.status_writer = open("/path/to/your/logdir/log", "w")  # create a writer
+
+   def train(self):
+       # ...
+       print(json.dumps(self.mutator.status()), file=self.status_writer, flush=True)  # dump a record of status
+
+If you are implementing your customized trainer inheriting ``Trainer``. We have provided ``enable_visualization()`` and ``_write_graph_status()`` for easy-to-use purposes. All you need to do is calling ``trainer.enable_visualization()`` before start, and ``trainer._write_graph_status()`` each time you want to do the logging. But remember both of these APIs are experimental and subject to change in future.
+
+Last but not least, invode NAS UI with
+
+.. code-block:: bash
+
+   nnictl webui nas --logdir /path/to/your/logdir
+
+NAS UI Preview
+--------------
+
+
+.. image:: ../../img/nasui-1.png
+   :target: ../../img/nasui-1.png
+   :alt: 
+
+
+
+.. image:: ../../img/nasui-2.png
+   :target: ../../img/nasui-2.png
+   :alt: 
+
+
+Limitations
+-----------
+
+
+* NAS visualization only works with PyTorch >=1.4. We've tested it on PyTorch 1.3.1 and it doesn't work.
+* We rely on PyTorch support for tensorboard for graph export, which relies on ``torch.jit``. It will not work if your model doesn't support ``jit``.
+* There are known performance issues when loading a moderate-size graph with many op choices (like DARTS search space).
+
+Feedback
+--------
+
+NAS UI is currently experimental. We welcome your feedback. `Here <https://github.com/microsoft/nni/pull/2085>`__ we have listed all the to-do items of NAS UI in the future. Feel free to comment (or `submit a new issue <https://github.com/microsoft/nni/issues/new?template=enhancement.rst>`__\ ) if you have other suggestions.

docs/en_US/NAS/WriteSearchSpace.rst

+Write A .. role:: raw-html(raw)
+   :format: html
+
+Search Space
+====================
+
+Genrally, a search space describes candiate architectures from which users want to find the best one. Different search algorithms, no matter classic NAS or one-shot NAS, can be applied on the search space. NNI provides APIs to unified the expression of neural architecture search space.
+
+A search space can be built on a base model. This is also a common practice when a user wants to apply NAS on an existing model. Take `MNIST on PyTorch <https://github.com/pytorch/examples/blob/master/mnist/main.py>`__ as an example. Note that NNI provides the same APIs for expressing search space on PyTorch and TensorFlow.
+
+.. code-block:: python
+
+   from nni.nas.pytorch import mutables
+
+   class Net(nn.Module):
+       def __init__(self):
+           super(Net, self).__init__()
+           self.conv1 = mutables.LayerChoice([
+               nn.Conv2d(1, 32, 3, 1),
+               nn.Conv2d(1, 32, 5, 3)
+           ])  # try 3x3 kernel and 5x5 kernel
+           self.conv2 = nn.Conv2d(32, 64, 3, 1)
+           self.dropout1 = nn.Dropout2d(0.25)
+           self.dropout2 = nn.Dropout2d(0.5)
+           self.fc1 = nn.Linear(9216, 128)
+           self.fc2 = nn.Linear(128, 10)
+
+       def forward(self, x):
+           x = self.conv1(x)
+           x = F.relu(x)
+           # ... same as original ...
+           return output
+
+The example above adds an option of choosing conv5x5 at conv1. The modification is as simple as declaring a ``LayerChoice`` with the original conv3x3 and a new conv5x5 as its parameter. That's it! You don't have to modify the forward function in any way. You can imagine conv1 as any other module without NAS.
+
+So how about the possibilities of connections? This can be done using ``InputChoice``. To allow for a skip connection on the MNIST example, we add another layer called conv3. In the following example, a possible connection from conv2 is added to the output of conv3.
+
+.. code-block:: python
+
+   from nni.nas.pytorch import mutables
+
+   class Net(nn.Module):
+       def __init__(self):
+           # ... same ...
+           self.conv2 = nn.Conv2d(32, 64, 3, 1)
+           self.conv3 = nn.Conv2d(64, 64, 1, 1)
+           # declaring that there is exactly one candidate to choose from
+           # search strategy will choose one or None
+           self.skipcon = mutables.InputChoice(n_candidates=1)
+           # ... same ...
+
+       def forward(self, x):
+           x = self.conv1(x)
+           x = F.relu(x)
+           x = self.conv2(x)
+           x0 = self.skipcon([x])  # choose one or none from [x]
+           x = self.conv3(x)
+           if x0 is not None:  # skipconnection is open
+               x += x0
+           x = F.max_pool2d(x, 2)
+           # ... same ...
+           return output
+
+Input choice can be thought of as a callable module that receives a list of tensors and outputs the concatenation/sum/mean of some of them (sum by default), or ``None`` if none is selected. Like layer choices, input choices should be **initialized in ``__init__`` and called in ``forward``**. This is to allow search algorithms to identify these choices and do necessary preparations.
+
+``LayerChoice`` and ``InputChoice`` are both **mutables**. Mutable means "changeable". As opposed to traditional deep learning layers/modules which have fixed operation types once defined, models with mutable are essentially a series of possible models.
+
+Users can specify a **key** for each mutable. By default, NNI will assign one for you that is globally unique, but in case users want to share choices (for example, there are two ``LayerChoice``\ s with the same candidate operations and you want them to have the same choice, i.e., if first one chooses the i-th op, the second one also chooses the i-th op), they can give them the same key. The key marks the identity for this choice and will be used in the dumped checkpoint. So if you want to increase the readability of your exported architecture, manually assigning keys to each mutable would be a good idea. For advanced usage on mutables (e.g., ``LayerChoice`` and ``InputChoice``\ ), see `Mutables <./NasReference.rst>`__.
+
+With search space defined, the next step is searching for the best model from it. Please refer to `classic NAS algorithms <./ClassicNas.md>`__ and `one-shot NAS algorithms <./NasGuide.rst>`__ for how to search from your defined search space.

docs/en_US/Overview.rst

+Overview
+========
+
+NNI (Neural Network Intelligence) is a toolkit to help users design and tune machine learning models (e.g., hyperparameters), neural network architectures, or complex system's parameters, in an efficient and automatic way. NNI has several appealing properties: ease-of-use, scalability, flexibility, and efficiency.
+
+
+* **Ease-of-use**\ : NNI can be easily installed through python pip. Only several lines need to be added to your code in order to use NNI's power. You can use both the commandline tool and WebUI to work with your experiments.
+* **Scalability**\ : Tuning hyperparameters or the neural architecture often demands a large number of computational resources, while NNI is designed to fully leverage different computation resources, such as remote machines, training platforms (e.g., OpenPAI, Kubernetes). Hundreds of trials could run in parallel by depending on the capacity of your configured training platforms.
+* **Flexibility**\ : Besides rich built-in algorithms, NNI allows users to customize various hyperparameter tuning algorithms, neural architecture search algorithms, early stopping algorithms, etc. Users can also extend NNI with more training platforms, such as virtual machines, kubernetes service on the cloud. Moreover, NNI can connect to external environments to tune special applications/models on them.
+* **Efficiency**\ : We are intensively working on more efficient model tuning on both the system and algorithm level. For example, we leverage early feedback to speedup the tuning procedure.
+
+The figure below shows high-level architecture of NNI.
+
+
+.. raw:: html
+
+   <p align="center">
+   <img src="https://user-images.githubusercontent.com/16907603/92089316-94147200-ee00-11ea-9944-bf3c4544257f.png" alt="drawing" width="700"/>
+   </p>
+
+
+Key Concepts
+------------
+
+
+* 
+  *Experiment*\ : One task of, for example, finding out the best hyperparameters of a model, finding out the best neural network architecture, etc. It consists of trials and AutoML algorithms.
+
+* 
+  *Search Space*\ : The feasible region for tuning the model. For example, the value range of each hyperparameter.
+
+* 
+  *Configuration*\ : An instance from the search space, that is, each hyperparameter has a specific value.
+
+* 
+  *Trial*\ : An individual attempt at applying a new configuration (e.g., a set of hyperparameter values, a specific neural architecture, etc.). Trial code should be able to run with the provided configuration.
+
+* 
+  *Tuner*\ : An AutoML algorithm, which generates a new configuration for the next try. A new trial will run with this configuration.
+
+* 
+  *Assessor*\ : Analyze a trial's intermediate results (e.g., periodically evaluated accuracy on test dataset) to tell whether this trial can be early stopped or not.
+
+* 
+  *Training Platform*\ : Where trials are executed. Depending on your experiment's configuration, it could be your local machine, or remote servers, or large-scale training platform (e.g., OpenPAI, Kubernetes).
+
+Basically, an experiment runs as follows: Tuner receives search space and generates configurations. These configurations will be submitted to training platforms, such as the local machine, remote machines, or training clusters. Their performances are reported back to Tuner. Then, new configurations are generated and submitted.
+
+For each experiment, the user only needs to define a search space and update a few lines of code, and then leverage NNI built-in Tuner/Assessor and training platforms to search the best hyperparameters and/or neural architecture. There are basically 3 steps:
+
+..
+
+   Step 1: `Define search space <Tutorial/SearchSpaceSpec.rst>`__
+
+   Step 2: `Update model codes <TrialExample/Trials.rst>`__
+
+   Step 3: `Define Experiment <Tutorial/ExperimentConfig.rst>`__
+
+
+
+.. raw:: html
+
+   <p align="center">
+   <img src="https://user-images.githubusercontent.com/23273522/51816627-5d13db80-2302-11e9-8f3e-627e260203d5.jpg" alt="drawing"/>
+   </p>
+
+
+For more details about how to run an experiment, please refer to `Get Started <Tutorial/QuickStart.rst>`__.
+
+Core Features
+-------------
+
+NNI provides a key capacity to run multiple instances in parallel to find the best combinations of parameters. This feature can be used in various domains, like finding the best hyperparameters for a deep learning model or finding the best configuration for database and other complex systems with real data.
+
+NNI also provides algorithm toolkits for machine learning and deep learning, especially neural architecture search (NAS) algorithms, model compression algorithms, and feature engineering algorithms.
+
+Hyperparameter Tuning
+^^^^^^^^^^^^^^^^^^^^^
+
+This is a core and basic feature of NNI, we provide many popular `automatic tuning algorithms <Tuner/BuiltinTuner.md>`__ (i.e., tuner) and `early stop algorithms <Assessor/BuiltinAssessor.md>`__ (i.e., assessor). You can follow `Quick Start <Tutorial/QuickStart.rst>`__ to tune your model (or system). Basically, there are the above three steps and then starting an NNI experiment.
+
+General NAS Framework
+^^^^^^^^^^^^^^^^^^^^^
+
+This NAS framework is for users to easily specify candidate neural architectures, for example, one can specify multiple candidate operations (e.g., separable conv, dilated conv) for a single layer, and specify possible skip connections. NNI will find the best candidate automatically. On the other hand, the NAS framework provides a simple interface for another type of user (e.g., NAS algorithm researchers) to implement new NAS algorithms. A detailed description of NAS and its usage can be found `here <NAS/Overview.rst>`__.
+
+NNI has support for many one-shot NAS algorithms such as ENAS and DARTS through NNI trial SDK. To use these algorithms you do not have to start an NNI experiment. Instead, import an algorithm in your trial code and simply run your trial code. If you want to tune the hyperparameters in the algorithms or want to run multiple instances, you can choose a tuner and start an NNI experiment.
+
+Other than one-shot NAS, NAS can also run in a classic mode where each candidate architecture runs as an independent trial job. In this mode, similar to hyperparameter tuning, users have to start an NNI experiment and choose a tuner for NAS.
+
+Model Compression
+^^^^^^^^^^^^^^^^^
+
+NNI provides an easy-to-use model compression framework to compress deep neural networks, the compressed networks typically have much smaller model size and much faster
+inference speed without losing performance significantlly. Model compression on NNI includes pruning algorithms and quantization algorithms. NNI provides many pruning and
+quantization algorithms through NNI trial SDK. Users can directly use them in their trial code and run the trial code without starting an NNI experiment. Users can also use NNI model compression framework to customize their own pruning and quantization algorithms.
+
+A detailed description of model compression and its usage can be found `here <Compression/Overview.rst>`__.
+
+Automatic Feature Engineering
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Automatic feature engineering is for users to find the best features for their tasks. A detailed description of automatic feature engineering and its usage can be found `here <FeatureEngineering/Overview.rst>`__. It is supported through NNI trial SDK, which means you do not have to create an NNI experiment. Instead, simply import a built-in auto-feature-engineering algorithm in your trial code and directly run your trial code. 
+
+The auto-feature-engineering algorithms usually have a bunch of hyperparameters themselves. If you want to automatically tune those hyperparameters, you can leverage hyperparameter tuning of NNI, that is, choose a tuning algorithm (i.e., tuner) and start an NNI experiment for it.
+
+Learn More
+----------
+
+
+* `Get started <Tutorial/QuickStart.rst>`__
+* `How to adapt your trial code on NNI? <TrialExample/Trials.rst>`__
+* `What are tuners supported by NNI? <Tuner/BuiltinTuner.rst>`__
+* `How to customize your own tuner? <Tuner/CustomizeTuner.rst>`__
+* `What are assessors supported by NNI? <Assessor/BuiltinAssessor.rst>`__
+* `How to customize your own assessor? <Assessor/CustomizeAssessor.rst>`__
+* `How to run an experiment on local? <TrainingService/LocalMode.rst>`__
+* `How to run an experiment on multiple machines? <TrainingService/RemoteMachineMode.rst>`__
+* `How to run an experiment on OpenPAI? <TrainingService/PaiMode.rst>`__
+* `Examples <TrialExample/MnistExamples.rst>`__
+* `Neural Architecture Search on NNI <NAS/Overview.rst>`__
+* `Model Compression on NNI <Compression/Overview.rst>`__
+* `Automatic feature engineering on NNI <FeatureEngineering/Overview.rst>`__

docs/en_US/Release.rst

+.. role:: raw-html(raw)
+   :format: html
+
+
+ChangeLog
+=========
+
+Release 1.9 - 10/22/2020
+========================
+
+Major updates
+-------------
+
+Neural architecture search
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+
+* Support regularized evolution algorithm for NAS scenario (#2802)
+* Add NASBench201 in search space zoo (#2766)
+
+Model compression
+^^^^^^^^^^^^^^^^^
+
+
+* AMC pruner improvement: support resnet, support reproduction of the experiments (default parameters in our example code) in AMC paper (#2876 #2906)
+* Support constraint-aware on some of our pruners to improve model compression efficiency (#2657)
+* Support "tf.keras.Sequential" in model compression for TensorFlow (#2887)
+* Support customized op in the model flops counter (#2795)
+* Support quantizing bias in QAT quantizer (#2914)
+
+Training service
+^^^^^^^^^^^^^^^^
+
+
+* Support configuring python environment using "preCommand" in remote mode (#2875)
+* Support AML training service in Windows (#2882)
+* Support reuse mode for remote training service (#2923)
+
+WebUI & nnictl
+^^^^^^^^^^^^^^
+
+
+* The "Overview" page on WebUI is redesigned with new layout (#2914)
+* Upgraded node, yarn and FabricUI, and enabled Eslint (#2894 #2873 #2744)
+* Add/Remove columns in hyper-parameter chart and trials table in "Trials detail" page (#2900)
+* JSON format utility beautify on WebUI (#2863)
+* Support nnictl command auto-completion (#2857)
+
+UT & IT
+-------
+
+
+* Add integration test for experiment import and export (#2878)
+* Add integration test for user installed builtin tuner (#2859)
+* Add unit test for nnictl (#2912)
+
+Documentation
+-------------
+
+
+* Refactor of the document for model compression (#2919)
+
+Bug fixes
+---------
+
+
+* Bug fix of naïve evolution tuner, correctly deal with trial fails (#2695)
+* Resolve the warning "WARNING (nni.protocol) IPC pipeline not exists, maybe you are importing tuner/assessor from trial code?" (#2864)
+* Fix search space issue in experiment save/load (#2886)
+* Fix bug in experiment import data (#2878)
+* Fix annotation in remote mode (python 3.8 ast update issue) (#2881)
+* Support boolean type for "choice" hyper-parameter when customizing trial configuration on WebUI (#3003)
+
+Release 1.8 - 8/27/2020
+=======================
+
+Major updates
+-------------
+
+Training service
+^^^^^^^^^^^^^^^^
+
+
+* Access trial log directly on WebUI (local mode only) (#2718)
+* Add OpenPAI trial job detail link (#2703)
+* Support GPU scheduler in reusable environment (#2627) (#2769)
+* Add timeout for ``web_channel`` in ``trial_runner`` (#2710)
+* Show environment error message in AzureML mode (#2724)
+* Add more log information when copying data in OpenPAI mode (#2702)
+
+WebUI, nnictl and nnicli
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+
+* Improve hyper-parameter parallel coordinates plot (#2691) (#2759)
+* Add pagination for trial job list (#2738) (#2773)
+* Enable panel close when clicking overlay region (#2734)
+* Remove support for Multiphase on WebUI (#2760)
+* Support save and restore experiments (#2750)
+* Add intermediate results in export result (#2706)
+* Add `command <https://github.com/microsoft/nni/blob/v1.8/docs/en_US/Tutorial/Nnictl.rst#nnictl-trial>`__ to list trial results with highest/lowest metrics (#2747)
+* Improve the user experience of `nnicli <https://github.com/microsoft/nni/blob/v1.8/docs/en_US/nnicli_ref.rst>`__ with `examples <https://github.com/microsoft/nni/blob/v1.8/examples/notebooks/retrieve_nni_info_with_python.ipynb>`__ (#2713)
+
+Neural architecture search
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+
+* `Search space zoo: ENAS and DARTS <https://github.com/microsoft/nni/blob/v1.8/docs/en_US/NAS/SearchSpaceZoo.rst>`__ (#2589)
+* API to query intermediate results in NAS benchmark (#2728)
+
+Model compression
+^^^^^^^^^^^^^^^^^
+
+
+* Support the List/Tuple Construct/Unpack operation for TorchModuleGraph (#2609)
+* Model speedup improvement: Add support of DenseNet and InceptionV3 (#2719)
+* Support the multiple successive tuple unpack operations (#2768)
+* `Doc of comparing the performance of supported pruners <https://github.com/microsoft/nni/blob/v1.8/docs/en_US/CommunitySharings/ModelCompressionComparison.rst>`__ (#2742)
+* New pruners: `Sensitivity pruner <https://github.com/microsoft/nni/blob/v1.8/docs/en_US/Compressor/Pruner.md#sensitivity-pruner>`__ (#2684) and `AMC pruner <https://github.com/microsoft/nni/blob/v1.8/docs/en_US/Compressor/Pruner.rst>`__ (#2573) (#2786)
+* TensorFlow v2 support in model compression (#2755)
+
+Backward incompatible changes
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+
+* Update the default experiment folder from ``$HOME/nni/experiments`` to ``$HOME/nni-experiments``. If you want to view the experiments created by previous NNI releases, you can move the experiments folders from  ``$HOME/nni/experiments`` to ``$HOME/nni-experiments`` manually. (#2686) (#2753)
+* Dropped support for Python 3.5 and scikit-learn 0.20 (#2778) (#2777) (2783) (#2787) (#2788) (#2790)
+
+Others
+^^^^^^
+
+
+* Upgrade TensorFlow version in Docker image (#2732) (#2735) (#2720)
+
+Examples
+--------
+
+
+* Remove gpuNum in assessor examples (#2641)
+
+Documentation
+-------------
+
+
+* Improve customized tuner documentation (#2628)
+* Fix several typos and grammar mistakes in documentation (#2637 #2638, thanks @tomzx)
+* Improve AzureML training service documentation (#2631)
+* Improve CI of Chinese translation (#2654)
+* Improve OpenPAI training service documenation (#2685)
+* Improve documentation of community sharing (#2640)
+* Add tutorial of Colab support (#2700)
+* Improve documentation structure for model compression (#2676)
+
+Bug fixes
+---------
+
+
+* Fix mkdir error in training service (#2673)
+* Fix bug when using chmod in remote training service (#2689)
+* Fix dependency issue by making ``_graph_utils`` imported inline (#2675)
+* Fix mask issue in ``SimulatedAnnealingPruner`` (#2736)
+* Fix intermediate graph zooming issue (#2738)
+* Fix issue when dict is unordered when querying NAS benchmark (#2728)
+* Fix import issue for gradient selector dataloader iterator (#2690)
+* Fix support of adding tens of machines in remote training service (#2725)
+* Fix several styling issues in WebUI (#2762 #2737)
+* Fix support of unusual types in metrics including NaN and Infinity (#2782)
+* Fix nnictl experiment delete (#2791)
+
+Release 1.7 - 7/8/2020
+======================
+
+Major Features
+--------------
+
+Training Service
+^^^^^^^^^^^^^^^^
+
+
+* Support AML(Azure Machine Learning) platform as NNI training service.
+* OpenPAI job can be reusable. When a trial is completed, the OpenPAI job won't stop, and wait next trial. `refer to reuse flag in OpenPAI config <https://github.com/microsoft/nni/blob/v1.7/docs/en_US/TrainingService/PaiMode.rst#openpai-configurations>`__.
+* `Support ignoring files and folders in code directory with .nniignore when uploading code directory to training service <https://github.com/microsoft/nni/blob/v1.7/docs/en_US/TrainingService/Overview.rst#how-to-use-training-service>`__.
+
+Neural Architecture Search (NAS)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+
+* 
+  `Provide NAS Open Benchmarks (NasBench101, NasBench201, NDS) with friendly APIs <https://github.com/microsoft/nni/blob/v1.7/docs/en_US/NAS/Benchmarks.rst>`__.
+
+* 
+  `Support Classic NAS (i.e., non-weight-sharing mode) on TensorFlow 2.X <https://github.com/microsoft/nni/blob/v1.7/docs/en_US/NAS/ClassicNas.rst>`__.
+
+Model Compression
+^^^^^^^^^^^^^^^^^
+
+
+* Improve Model Speedup: track more dependencies among layers and automatically resolve mask conflict, support the speedup of pruned resnet.
+* Added new pruners, including three auto model pruning algorithms: `NetAdapt Pruner <https://github.com/microsoft/nni/blob/v1.7/docs/en_US/Compressor/Pruner.md#netadapt-pruner>`__\ , `SimulatedAnnealing Pruner <https://github.com/microsoft/nni/blob/v1.7/docs/en_US/Compressor/Pruner.md#simulatedannealing-pruner>`__\ , `AutoCompress Pruner <https://github.com/microsoft/nni/blob/v1.7/docs/en_US/Compressor/Pruner.md#autocompress-pruner>`__\ , and `ADMM Pruner <https://github.com/microsoft/nni/blob/v1.7/docs/en_US/Compressor/Pruner.rst#admm-pruner>`__.
+* Added `model sensitivity analysis tool <https://github.com/microsoft/nni/blob/v1.7/docs/en_US/Compressor/CompressionUtils.rst>`__ to help users find the sensitivity of each layer to the pruning.
+* 
+  `Easy flops calculation for model compression and NAS <https://github.com/microsoft/nni/blob/v1.7/docs/en_US/Compressor/CompressionUtils.rst#model-flops-parameters-counter>`__.
+
+* 
+  Update lottery ticket pruner to export winning ticket.
+
+Examples
+^^^^^^^^
+
+
+* Automatically optimize tensor operators on NNI with a new `customized tuner OpEvo <https://github.com/microsoft/nni/blob/v1.7/docs/en_US/TrialExample/OpEvoExamples.rst>`__.
+
+Built-in tuners/assessors/advisors
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+
+* `Allow customized tuners/assessor/advisors to be installed as built-in algorithms <https://github.com/microsoft/nni/blob/v1.7/docs/en_US/Tutorial/InstallCustomizedAlgos.rst>`__.
+
+WebUI
+^^^^^
+
+
+* Support visualizing nested search space more friendly.
+* Show trial's dict keys in hyper-parameter graph.
+* Enhancements to trial duration display.
+
+Others
+^^^^^^
+
+
+* Provide utility function to merge parameters received from NNI
+* Support setting paiStorageConfigName in pai mode
+
+Documentation
+-------------
+
+
+* Improve `documentation for model compression <https://github.com/microsoft/nni/blob/v1.7/docs/en_US/Compressor/Overview.rst>`__
+* Improve `documentation <https://github.com/microsoft/nni/blob/v1.7/docs/en_US/NAS/Benchmarks.rst>`__
+  and `examples <https://github.com/microsoft/nni/blob/v1.7/docs/en_US/NAS/BenchmarksExample.ipynb>`__ for NAS benchmarks.
+* Improve `documentation for AzureML training service <https://github.com/microsoft/nni/blob/v1.7/docs/en_US/TrainingService/AMLMode.rst>`__
+* Homepage migration to readthedoc.
+
+Bug Fixes
+---------
+
+
+* Fix bug for model graph with shared nn.Module
+* Fix nodejs OOM when ``make build``
+* Fix NASUI bugs
+* Fix duration and intermediate results pictures update issue.
+* Fix minor WebUI table style issues.
+
+Release 1.6 - 5/26/2020
+-----------------------
+
+Major Features
+^^^^^^^^^^^^^^
+
+New Features and improvement
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+
+* Improve IPC limitation to 100W
+* improve code storage upload logic among trials in non-local platform
+* support ``__version__`` for SDK version
+* support windows dev intall
+
+Web UI
+^^^^^^
+
+
+* Show trial error message
+* finalize homepage layout
+* Refactor overview's best trials module
+* Remove multiphase from webui
+* add tooltip for trial concurrency in the overview page
+* Show top trials for hyper-parameter graph
+
+HPO Updates
+^^^^^^^^^^^
+
+
+* Improve PBT on failure handling and support experiment resume for PBT
+
+NAS Updates
+^^^^^^^^^^^
+
+
+* NAS support for TensorFlow 2.0 (preview) `TF2.0 NAS examples <https://github.com/microsoft/nni/tree/v1.9/examples/nas/naive-tf>`__
+* Use OrderedDict for LayerChoice
+* Prettify the format of export
+* Replace layer choice with selected module after applied fixed architecture
+
+Model Compression Updates
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+
+* Model compression PyTorch 1.4 support
+
+Training Service Updates
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+
+* update pai yaml merge logic
+* support windows as remote machine in remote mode `Remote Mode <https://github.com/microsoft/nni/blob/v1.9/docs/en_US/TrainingService/RemoteMachineMode.rst#windows>`__
+
+Bug Fix
+^^^^^^^
+
+
+* fix dev install
+* SPOS example crash when the checkpoints do not have state_dict
+* Fix table sort issue when experiment had failed trial
+* Support multi python env (conda, pyenv etc)
+
+Release 1.5 - 4/13/2020
+-----------------------
+
+New Features and Documentation
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Hyper-Parameter Optimizing
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+
+* New tuner: `Population Based Training (PBT) <https://github.com/microsoft/nni/blob/v1.9/docs/en_US/Tuner/PBTTuner.rst>`__
+* Trials can now report infinity and NaN as result
+
+Neural Architecture Search
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+
+* New NAS algorithm: `TextNAS <https://github.com/microsoft/nni/blob/v1.9/docs/en_US/NAS/TextNAS.rst>`__
+* ENAS and DARTS now support `visualization <https://github.com/microsoft/nni/blob/v1.9/docs/en_US/NAS/Visualization.rst>`__ through web UI.
+
+Model Compression
+^^^^^^^^^^^^^^^^^
+
+
+* New Pruner: `GradientRankFilterPruner <https://github.com/microsoft/nni/blob/v1.9/docs/en_US/Compressor/Pruner.rst#gradientrankfilterpruner>`__
+* Compressors will validate configuration by default
+* Refactor: Adding optimizer as an input argument of pruner, for easy support of DataParallel and more efficient iterative pruning. This is a broken change for the usage of iterative pruning algorithms.
+* Model compression examples are refactored and improved
+* Added documentation for `implementing compressing algorithm <https://github.com/microsoft/nni/blob/v1.9/docs/en_US/Compressor/Framework.rst>`__
+
+Training Service
+^^^^^^^^^^^^^^^^
+
+
+* Kubeflow now supports pytorchjob crd v1 (thanks external contributor @jiapinai)
+* Experimental `DLTS <https://github.com/microsoft/nni/blob/v1.9/docs/en_US/TrainingService/DLTSMode.rst>`__ support
+
+Overall Documentation Improvement
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+
+* Documentation is significantly improved on grammar, spelling, and wording (thanks external contributor @AHartNtkn)
+
+Fixed Bugs
+^^^^^^^^^^
+
+
+* ENAS cannot have more than one LSTM layers (thanks external contributor @marsggbo)
+* NNI manager's timers will never unsubscribe (thanks external contributor @guilhermehn)
+* NNI manager may exhaust head memory (thanks external contributor @Sundrops)
+* Batch tuner does not support customized trials (#2075)
+* Experiment cannot be killed if it failed on start (#2080)
+* Non-number type metrics break web UI (#2278)
+* A bug in lottery ticket pruner
+* Other minor glitches
+
+Release 1.4 - 2/19/2020
+-----------------------
+
+Major Features
+^^^^^^^^^^^^^^
+
+Neural Architecture Search
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+
+* Support `C-DARTS <https://github.com/microsoft/nni/blob/v1.4/docs/en_US/NAS/CDARTS.rst>`__ algorithm and add `the example <https://github.com/microsoft/nni/tree/v1.4/examples/nas/cdarts>`__ using it
+* Support a preliminary version of `ProxylessNAS <https://github.com/microsoft/nni/blob/v1.4/docs/en_US/NAS/Proxylessnas.rst>`__ and the corresponding `example <https://github.com/microsoft/nni/tree/v1.4/examples/nas/proxylessnas>`__
+* Add unit tests for the NAS framework
+
+Model Compression
+^^^^^^^^^^^^^^^^^
+
+
+* Support DataParallel for compressing models, and provide `an example <https://github.com/microsoft/nni/blob/v1.4/examples/model_compress/multi_gpu.py>`__ of using DataParallel
+* Support `model speedup <https://github.com/microsoft/nni/blob/v1.4/docs/en_US/Compressor/ModelSpeedup.rst>`__ for compressed models, in Alpha version
+
+Training Service
+^^^^^^^^^^^^^^^^
+
+
+* Support complete PAI configurations by allowing users to specify PAI config file path
+* Add example config yaml files for the new PAI mode (i.e., paiK8S)
+* Support deleting experiments using sshkey in remote mode (thanks external contributor @tyusr)
+
+WebUI
+^^^^^
+
+
+* WebUI refactor: adopt fabric framework
+
+Others
+^^^^^^
+
+
+* Support running `NNI experiment at foreground <https://github.com/microsoft/nni/blob/v1.4/docs/en_US/Tutorial/Nnictl#manage-an-experiment>`__\ , i.e., ``--foreground`` argument in ``nnictl create/resume/view``
+* Support canceling the trials in UNKNOWN state
+* Support large search space whose size could be up to 50mb (thanks external contributor @Sundrops)
+
+Documentation
+^^^^^^^^^^^^^
+
+
+* Improve `the index structure <https://nni.readthedocs.io/en/latest/>`__ of NNI readthedocs
+* Improve `documentation for NAS <https://github.com/microsoft/nni/blob/v1.4/docs/en_US/NAS/NasGuide.rst>`__
+* Improve documentation for `the new PAI mode <https://github.com/microsoft/nni/blob/v1.4/docs/en_US/TrainingService/PaiMode.rst>`__
+* Add QuickStart guidance for `NAS <https://github.com/microsoft/nni/blob/v1.4/docs/en_US/NAS/QuickStart.md>`__ and `model compression <https://github.com/microsoft/nni/blob/v1.4/docs/en_US/Compressor/QuickStart.rst>`__
+* Improve documentation for `the supported EfficientNet <https://github.com/microsoft/nni/blob/v1.4/docs/en_US/TrialExample/EfficientNet.rst>`__
+
+Bug Fixes
+^^^^^^^^^
+
+
+* Correctly support NaN in metric data, JSON compliant
+* Fix the out-of-range bug of ``randint`` type in search space
+* Fix the bug of wrong tensor device when exporting onnx model in model compression
+* Fix incorrect handling of nnimanagerIP in the new PAI mode (i.e., paiK8S)
+
+Release 1.3 - 12/30/2019
+------------------------
+
+Major Features
+^^^^^^^^^^^^^^
+
+Neural Architecture Search Algorithms Support
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+
+* `Single Path One Shot <https://github.com/microsoft/nni/tree/v1.3/examples/nas/spos/>`__ algorithm and the example using it
+
+Model Compression Algorithms Support
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+
+* `Knowledge Distillation <https://github.com/microsoft/nni/blob/v1.3/docs/en_US/TrialExample/KDExample.rst>`__ algorithm and the example using itExample
+* Pruners
+
+  * `L2Filter Pruner <https://github.com/microsoft/nni/blob/v1.3/docs/en_US/Compressor/Pruner.rst#3-l2filter-pruner>`__
+  * `ActivationAPoZRankFilterPruner <https://github.com/microsoft/nni/blob/v1.3/docs/en_US/Compressor/Pruner.rst#1-activationapozrankfilterpruner>`__
+  * `ActivationMeanRankFilterPruner <https://github.com/microsoft/nni/blob/v1.3/docs/en_US/Compressor/Pruner.rst#2-activationmeanrankfilterpruner>`__
+
+* `BNN Quantizer <https://github.com/microsoft/nni/blob/v1.3/docs/en_US/Compressor/Quantizer.rst#bnn-quantizer>`__
+  #### Training Service
+* 
+  NFS Support for PAI
+
+    Instead of using HDFS as default storage, since OpenPAI v0.11, OpenPAI can have NFS or AzureBlob or other storage as default storage. In this release, NNI extended the support for this recent change made by OpenPAI, and could integrate with OpenPAI v0.11 or later version with various default storage.
+
+* 
+  Kubeflow update adoption
+
+    Adopted the Kubeflow 0.7's new supports for tf-operator.
+
+Engineering (code and build automation)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+
+* Enforced `ESLint <https://eslint.org/>`__ on static code analysis.
+
+Small changes & Bug Fixes
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+
+* correctly recognize builtin tuner and customized tuner
+* logging in dispatcher base
+* fix the bug where tuner/assessor's failure sometimes kills the experiment.
+* Fix local system as remote machine `issue <https://github.com/microsoft/nni/issues/1852>`__
+* de-duplicate trial configuration in smac tuner `ticket <https://github.com/microsoft/nni/issues/1364>`__
+
+Release 1.2 - 12/02/2019
+------------------------
+
+Major Features
+^^^^^^^^^^^^^^
+
+
+* `Feature Engineering <https://github.com/microsoft/nni/blob/v1.2/docs/en_US/FeatureEngineering/Overview.rst>`__
+
+  * New feature engineering interface
+  * Feature selection algorithms: `Gradient feature selector <https://github.com/microsoft/nni/blob/v1.2/docs/en_US/FeatureEngineering/GradientFeatureSelector.md>`__ & `GBDT selector <https://github.com/microsoft/nni/blob/v1.2/docs/en_US/FeatureEngineering/GBDTSelector.rst>`__
+  * `Examples for feature engineering <https://github.com/microsoft/nni/tree/v1.2/examples/feature_engineering>`__
+
+* Neural Architecture Search (NAS) on NNI
+
+  * `New NAS interface <https://github.com/microsoft/nni/blob/v1.2/docs/en_US/NAS/NasInterface.rst>`__
+  * NAS algorithms: `ENAS <https://github.com/microsoft/nni/blob/v1.2/docs/en_US/NAS/Overview.md#enas>`__\ , `DARTS <https://github.com/microsoft/nni/blob/v1.2/docs/en_US/NAS/Overview.md#darts>`__\ , `P-DARTS <https://github.com/microsoft/nni/blob/v1.2/docs/en_US/NAS/Overview.rst#p-darts>`__ (in PyTorch)
+  * NAS in classic mode (each trial runs independently)
+
+* Model compression
+
+  * `New model pruning algorithms <https://github.com/microsoft/nni/blob/v1.2/docs/en_US/Compressor/Overview.rst>`__\ : lottery ticket pruning approach, L1Filter pruner, Slim pruner, FPGM pruner
+  * `New model quantization algorithms <https://github.com/microsoft/nni/blob/v1.2/docs/en_US/Compressor/Overview.rst>`__\ : QAT quantizer, DoReFa quantizer
+  * Support the API for exporting compressed model.
+
+* Training Service
+
+  * Support OpenPAI token authentication
+
+* Examples:
+
+  * `An example to automatically tune rocksdb configuration with NNI <https://github.com/microsoft/nni/tree/v1.2/examples/trials/systems/rocksdb-fillrandom>`__.
+  * `A new MNIST trial example supports tensorflow 2.0 <https://github.com/microsoft/nni/tree/v1.2/examples/trials/mnist-tfv2>`__.
+
+* Engineering Improvements
+
+  * For remote training service,  trial jobs require no GPU are now scheduled with round-robin policy instead of random.
+  * Pylint rules added to check pull requests, new pull requests need to comply with these `pylint rules <https://github.com/microsoft/nni/blob/v1.2/pylintrc>`__.
+
+* Web Portal & User Experience
+
+  * Support user to add customized trial.
+  * User can zoom out/in in detail graphs, except Hyper-parameter.
+
+* Documentation
+
+  * Improved NNI API documentation with more API docstring.
+
+Bug fix
+^^^^^^^
+
+
+* Fix the table sort issue when failed trials haven't metrics. -Issue #1773
+* Maintain selected status(Maximal/Minimal) when the page switched. -PR#1710
+* Make hyper-parameters graph's default metric yAxis more accurate. -PR#1736
+* Fix GPU script permission issue. -Issue #1665
+
+Release 1.1 - 10/23/2019
+------------------------
+
+Major Features
+^^^^^^^^^^^^^^
+
+
+* New tuner: `PPO Tuner <https://github.com/microsoft/nni/blob/v1.1/docs/en_US/Tuner/PPOTuner.rst>`__
+* `View stopped experiments <https://github.com/microsoft/nni/blob/v1.1/docs/en_US/Tutorial/Nnictl.rst#view>`__
+* Tuners can now use dedicated GPU resource (see ``gpuIndices`` in `tutorial <https://github.com/microsoft/nni/blob/v1.1/docs/en_US/Tutorial/ExperimentConfig.rst>`__ for details)
+* Web UI improvements
+
+  * Trials detail page can now list hyperparameters of each trial, as well as their start and end time (via "add column")
+  * Viewing huge experiment is now less laggy
+
+* More examples
+
+  * `EfficientNet PyTorch example <https://github.com/ultmaster/EfficientNet-PyTorch>`__
+  * `Cifar10 NAS example <https://github.com/microsoft/nni/blob/v1.1/examples/trials/nas_cifar10/README.rst>`__
+
+* `Model compression toolkit - Alpha release <https://github.com/microsoft/nni/blob/v1.1/docs/en_US/Compressor/Overview.rst>`__\ : We are glad to announce the alpha release for model compression toolkit on top of NNI, it's still in the experiment phase which might evolve based on usage feedback. We'd like to invite you to use, feedback and even contribute
+
+Fixed Bugs
+^^^^^^^^^^
+
+
+* Multiphase job hangs when search space exhuasted (issue #1204)
+* ``nnictl`` fails when log not available (issue #1548)
+
+Release 1.0 - 9/2/2019
+----------------------
+
+Major Features
+^^^^^^^^^^^^^^
+
+
+* 
+  Tuners and Assessors
+
+
+  * Support Auto-Feature generator & selection    -Issue#877  -PR #1387
+
+    * Provide auto feature interface
+    * Tuner based on beam search
+    * `Add Pakdd example <https://github.com/microsoft/nni/tree/v1.9/examples/trials/auto-feature-engineering>`__
+
+  * Add a parallel algorithm to improve the performance of TPE with large concurrency.  -PR #1052
+  * Support multiphase for hyperband    -PR #1257
+
+* 
+  Training Service
+
+
+  * Support private docker registry   -PR #755
+
+
+  * Engineering Improvements
+
+    * Python wrapper for rest api, support retrieve the values of the metrics in a programmatic way  PR #1318
+    * New python API : get_experiment_id(), get_trial_id()  -PR #1353   -Issue #1331 & -Issue#1368
+    * Optimized NAS Searchspace  -PR #1393
+
+      * Unify NAS search space with _type -- "mutable_type"e
+      * Update random search tuner
+
+    * Set gpuNum as optional      -Issue #1365
+    * Remove outputDir and dataDir configuration in PAI mode   -Issue #1342
+    * When creating a trial in Kubeflow mode, codeDir will no longer be copied to logDir   -Issue #1224
+
+* 
+  Web Portal & User Experience
+
+
+  * Show the best metric curve during search progress in WebUI  -Issue #1218
+  * Show the current number of parameters list in multiphase experiment   -Issue1210  -PR #1348
+  * Add "Intermediate count" option in AddColumn.      -Issue #1210
+  * Support search parameters value in WebUI     -Issue #1208
+  * Enable automatic scaling of axes for metric value  in default metric graph   -Issue #1360
+  * Add a detailed documentation link to the nnictl command in the command prompt    -Issue #1260
+  * UX improvement for showing Error log   -Issue #1173
+
+* 
+  Documentation
+
+
+  * Update the docs structure  -Issue #1231
+  * (deprecated) Multi phase document improvement   -Issue #1233  -PR #1242
+
+    * Add configuration example
+
+  * `WebUI description improvement <Tutorial/WebUI.rst>`__  -PR #1419
+
+Bug fix
+^^^^^^^
+
+
+* (Bug fix)Fix the broken links in 0.9 release  -Issue #1236
+* (Bug fix)Script for auto-complete
+* (Bug fix)Fix pipeline issue that it only check exit code of last command in a script.  -PR #1417
+* (Bug fix)quniform fors tuners    -Issue #1377
+* (Bug fix)'quniform' has different meaning beween GridSearch and other tuner.   -Issue #1335
+* (Bug fix)"nnictl experiment list" give the status of a "RUNNING" experiment as "INITIALIZED" -PR #1388
+* (Bug fix)SMAC cannot be installed if nni is installed in dev mode    -Issue #1376
+* (Bug fix)The filter button of the intermediate result cannot be clicked   -Issue #1263
+* (Bug fix)API "/api/v1/nni/trial-jobs/xxx" doesn't show a trial's all parameters in multiphase experiment    -Issue #1258
+* (Bug fix)Succeeded trial doesn't have final result but webui show ×××(FINAL)  -Issue #1207
+* (Bug fix)IT for nnictl stop -Issue #1298
+* (Bug fix)fix security warning
+* (Bug fix)Hyper-parameter page broken  -Issue #1332
+* (Bug fix)Run flake8 tests to find Python syntax errors and undefined names -PR #1217
+
+Release 0.9 - 7/1/2019
+----------------------
+
+Major Features
+^^^^^^^^^^^^^^
+
+
+* General NAS programming interface
+
+  * Add ``enas-mode``  and ``oneshot-mode`` for NAS interface: `PR #1201 <https://github.com/microsoft/nni/pull/1201#issue-291094510>`__
+
+* 
+  `Gaussian Process Tuner with Matern kernel <Tuner/GPTuner.rst>`__
+
+* 
+  (deprecated) Multiphase experiment supports
+
+
+  * Added new training service support for multiphase experiment: PAI mode supports multiphase experiment since v0.9.
+  * Added multiphase capability for the following builtin tuners:
+
+    * TPE, Random Search, Anneal, Naïve Evolution, SMAC, Network Morphism, Metis Tuner.
+
+* 
+  Web Portal
+
+
+  * Enable trial comparation in Web Portal. For details, refer to `View trials status <Tutorial/WebUI.rst>`__
+  * Allow users to adjust rendering interval of Web Portal. For details, refer to `View Summary Page <Tutorial/WebUI.rst>`__
+  * show intermediate results more friendly. For details, refer to `View trials status <Tutorial/WebUI.rst>`__
+
+* `Commandline Interface <Tutorial/Nnictl.rst>`__
+
+  * ``nnictl experiment delete``\ : delete one or all experiments, it includes log, result, environment information and cache. It uses to delete useless experiment result, or save disk space.
+  * ``nnictl platform clean``\ : It uses to clean up disk on a target platform. The provided YAML file includes the information of target platform, and it follows the same schema as the NNI configuration file.
+    ### Bug fix and other changes
+
+* Tuner Installation Improvements: add `sklearn <https://scikit-learn.org/stable/>`__ to nni dependencies.
+* (Bug Fix) Failed to connect to PAI http code - `Issue #1076 <https://github.com/microsoft/nni/issues/1076>`__
+* (Bug Fix) Validate file name for PAI platform - `Issue #1164 <https://github.com/microsoft/nni/issues/1164>`__
+* (Bug Fix) Update GMM evaluation in Metis Tuner
+* (Bug Fix) Negative time number rendering in Web Portal - `Issue #1182 <https://github.com/microsoft/nni/issues/1182>`__\ , `Issue #1185 <https://github.com/microsoft/nni/issues/1185>`__
+* (Bug Fix) Hyper-parameter not shown correctly in WebUI when there is only one hyper parameter - `Issue #1192 <https://github.com/microsoft/nni/issues/1192>`__
+
+Release 0.8 - 6/4/2019
+----------------------
+
+Major Features
+^^^^^^^^^^^^^^
+
+
+* Support NNI on Windows for OpenPAI/Remote mode
+
+  * NNI running on windows for remote mode
+  * NNI running on windows for OpenPAI mode
+
+* Advanced features for using GPU
+
+  * Run multiple trial jobs on the same GPU for local and remote mode
+  * Run trial jobs on the GPU running non-NNI jobs
+
+* Kubeflow v1beta2 operator
+
+  * Support Kubeflow TFJob/PyTorchJob v1beta2
+
+* `General NAS programming interface <https://github.com/microsoft/nni/blob/v0.8/docs/en_US/GeneralNasInterfaces.rst>`__
+
+  * Provide NAS programming interface for users to easily express their neural architecture search space through NNI annotation
+  * Provide a new command ``nnictl trial codegen`` for debugging the NAS code
+  * Tutorial of NAS programming interface, example of NAS on MNIST, customized random tuner for NAS
+
+* Support resume tuner/advisor's state for experiment resume
+* For experiment resume, tuner/advisor will be resumed by replaying finished trial data
+* Web Portal
+
+  * Improve the design of copying trial's parameters
+  * Support 'randint' type in hyper-parameter graph
+  * Use should ComponentUpdate to avoid unnecessary render
+
+Bug fix and other changes
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+
+* Bug fix that ``nnictl update`` has inconsistent command styles
+* Support import data for SMAC tuner
+* Bug fix that experiment state transition from ERROR back to RUNNING
+* Fix bug of table entries
+* Nested search space refinement
+* Refine 'randint' type and support lower bound
+* `Comparison of different hyper-parameter tuning algorithm <CommunitySharings/HpoComparison.rst>`__
+* `Comparison of NAS algorithm <CommunitySharings/NasComparison.rst>`__
+* `NNI practice on Recommenders <CommunitySharings/RecommendersSvd.rst>`__
+
+Release 0.7 - 4/29/2018
+-----------------------
+
+Major Features
+^^^^^^^^^^^^^^
+
+
+* `Support NNI on Windows <Tutorial/InstallationWin.rst>`__
+
+  * NNI running on windows for local mode
+
+* `New advisor: BOHB <Tuner/BohbAdvisor.rst>`__
+
+  * Support a new advisor BOHB, which is a robust and efficient hyperparameter tuning algorithm, combines the advantages of Bayesian optimization and Hyperband
+
+* `Support import and export experiment data through nnictl <Tutorial/Nnictl.rst>`__
+
+  * Generate analysis results report after the experiment execution
+  * Support import data to tuner and advisor for tuning
+
+* `Designated gpu devices for NNI trial jobs <Tutorial/ExperimentConfig.rst#localConfig>`__
+
+  * Specify GPU devices for NNI trial jobs by gpuIndices configuration, if gpuIndices is set in experiment configuration file, only the specified GPU devices are used for NNI trial jobs.
+
+* Web Portal enhancement
+
+  * Decimal format of metrics other than default on the Web UI
+  * Hints in WebUI about Multi-phase
+  * Enable copy/paste for hyperparameters as python dict
+  * Enable early stopped trials data for tuners.
+
+* NNICTL provide better error message
+
+  * nnictl provide more meaningful error message for YAML file format error
+
+Bug fix
+^^^^^^^
+
+
+* Unable to kill all python threads after nnictl stop in async dispatcher mode
+* nnictl --version does not work with make dev-install
+* All trail jobs status stays on 'waiting' for long time on OpenPAI platform
+
+Release 0.6 - 4/2/2019
+----------------------
+
+Major Features
+^^^^^^^^^^^^^^
+
+
+* `Version checking <TrainingService/PaiMode.rst>`__
+
+  * check whether the version is consistent between nniManager and trialKeeper
+
+* `Report final metrics for early stop job <https://github.com/microsoft/nni/issues/776>`__
+
+  * If includeIntermediateResults is true, the last intermediate result of the trial that is early stopped by assessor is sent to tuner as final result. The default value of includeIntermediateResults is false.
+
+* `Separate Tuner/Assessor <https://github.com/microsoft/nni/issues/841>`__
+
+  * Adds two pipes to separate message receiving channels for tuner and assessor.
+
+* Make log collection feature configurable
+* Add intermediate result graph for all trials
+
+Bug fix
+^^^^^^^
+
+
+* `Add shmMB config key for OpenPAI <https://github.com/microsoft/nni/issues/842>`__
+* Fix the bug that doesn't show any result if metrics is dict
+* Fix the number calculation issue for float types in hyperband
+* Fix a bug in the search space conversion in SMAC tuner
+* Fix the WebUI issue when parsing experiment.json with illegal format
+* Fix cold start issue in Metis Tuner
+
+Release 0.5.2 - 3/4/2019
+------------------------
+
+Improvements
+^^^^^^^^^^^^
+
+
+* Curve fitting assessor performance improvement.
+
+Documentation
+^^^^^^^^^^^^^
+
+
+* Chinese version document: https://nni.readthedocs.io/zh/latest/
+* Debuggability/serviceability document: https://nni.readthedocs.io/en/latest/Tutorial/HowToDebug.html
+* Tuner assessor reference: https://nni.readthedocs.io/en/latest/sdk_reference.html
+
+Bug Fixes and Other Changes
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+
+* Fix a race condition bug that does not store trial job cancel status correctly.
+* Fix search space parsing error when using SMAC tuner.
+* Fix cifar10 example broken pipe issue.
+* Add unit test cases for nnimanager and local training service.
+* Add integration test azure pipelines for remote machine, OpenPAI and kubeflow training services.
+* Support Pylon in OpenPAI webhdfs client.
+
+Release 0.5.1 - 1/31/2018
+-------------------------
+
+Improvements
+^^^^^^^^^^^^
+
+
+* Making `log directory <https://github.com/microsoft/nni/blob/v0.5.1/docs/ExperimentConfig.rst>`__ configurable
+* Support `different levels of logs <https://github.com/microsoft/nni/blob/v0.5.1/docs/ExperimentConfig.rst>`__\ , making it easier for debugging
+
+Documentation
+^^^^^^^^^^^^^
+
+
+* Reorganized documentation & New Homepage Released: https://nni.readthedocs.io/en/latest/
+
+Bug Fixes and Other Changes
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+
+* Fix the bug of installation in python virtualenv, and refactor the installation logic
+* Fix the bug of HDFS access failure on OpenPAI mode after OpenPAI is upgraded.
+* Fix the bug that sometimes in-place flushed stdout makes experiment crash
+
+Release 0.5.0 - 01/14/2019
+--------------------------
+
+Major Features
+^^^^^^^^^^^^^^
+
+New tuner and assessor supports
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+
+* Support `Metis tuner <Tuner/MetisTuner.rst>`__ as a new NNI tuner. Metis algorithm has been proofed to be well performed for **online** hyper-parameter tuning.
+* Support `ENAS customized tuner <https://github.com/countif/enas_nni>`__\ , a tuner contributed by github community user, is an algorithm for neural network search, it could learn neural network architecture via reinforcement learning and serve a better performance than NAS.
+* Support `Curve fitting assessor <Assessor/CurvefittingAssessor.rst>`__ for early stop policy using learning curve extrapolation.
+* Advanced Support of `Weight Sharing <https://github.com/microsoft/nni/blob/v0.5/docs/AdvancedNAS.rst>`__\ : Enable weight sharing for NAS tuners, currently through NFS.
+
+Training Service Enhancement
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+
+* `FrameworkController Training service <TrainingService/FrameworkControllerMode.rst>`__\ : Support run experiments using frameworkcontroller on kubernetes
+
+  * FrameworkController is a Controller on kubernetes that is general enough to run (distributed) jobs with various machine learning frameworks, such as tensorflow, pytorch, MXNet.
+  * NNI provides unified and simple specification for job definition.
+  * MNIST example for how to use FrameworkController.
+
+User Experience improvements
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+
+* A better trial logging support for NNI experiments in OpenPAI, Kubeflow and FrameworkController mode:
+
+  * An improved logging architecture to send stdout/stderr of trials to NNI manager via Http post. NNI manager will store trial's stdout/stderr messages in local log file.
+  * Show the link for trial log file on WebUI.
+
+* Support to show final result's all key-value pairs.
+
+Release 0.4.1 - 12/14/2018
+--------------------------
+
+Major Features
+^^^^^^^^^^^^^^
+
+New tuner supports
+^^^^^^^^^^^^^^^^^^
+
+
+* Support `network morphism <Tuner/NetworkmorphismTuner.rst>`__ as a new tuner
+
+Training Service improvements
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+
+* Migrate `Kubeflow training service <TrainingService/KubeflowMode.rst>`__\ 's dependency from kubectl CLI to `Kubernetes API <https://kubernetes.io/docs/concepts/overview/kubernetes-api/>`__ client
+* `Pytorch-operator <https://github.com/kubeflow/pytorch-operator>`__ support for Kubeflow training service
+* Improvement on local code files uploading to OpenPAI HDFS
+* Fixed OpenPAI integration WebUI bug: WebUI doesn't show latest trial job status, which is caused by OpenPAI token expiration
+
+NNICTL improvements
+^^^^^^^^^^^^^^^^^^^
+
+
+* Show version information both in nnictl and WebUI. You can run **nnictl -v** to show your current installed NNI version
+
+WebUI improvements
+^^^^^^^^^^^^^^^^^^
+
+
+* Enable modify concurrency number during experiment
+* Add feedback link to NNI github 'create issue' page
+* Enable customize top 10 trials regarding to metric numbers (largest or smallest)
+* Enable download logs for dispatcher & nnimanager
+* Enable automatic scaling of axes for metric number
+* Update annotation to support displaying real choice in searchspace
+
+New examples
+^^^^^^^^^^^^
+
+
+* `FashionMnist <https://github.com/microsoft/nni/tree/v1.9/examples/trials/network_morphism>`__\ , work together with network morphism tuner
+* `Distributed MNIST example <https://github.com/microsoft/nni/tree/v1.9/examples/trials/mnist-distributed-pytorch>`__ written in PyTorch
+
+Release 0.4 - 12/6/2018
+-----------------------
+
+Major Features
+^^^^^^^^^^^^^^
+
+
+* `Kubeflow Training service <TrainingService/KubeflowMode.rst>`__
+
+  * Support tf-operator
+  * `Distributed trial example <https://github.com/microsoft/nni/tree/v1.9/examples/trials/mnist-distributed/dist_mnist.py>`__ on Kubeflow
+
+* `Grid search tuner <Tuner/GridsearchTuner.rst>`__
+* `Hyperband tuner <Tuner/HyperbandAdvisor.rst>`__
+* Support launch NNI experiment on MAC
+* WebUI
+
+  * UI support for hyperband tuner
+  * Remove tensorboard button
+  * Show experiment error message
+  * Show line numbers in search space and trial profile
+  * Support search a specific trial by trial number
+  * Show trial's hdfsLogPath
+  * Download experiment parameters
+
+Others
+^^^^^^
+
+
+* Asynchronous dispatcher
+* Docker file update, add pytorch library
+* Refactor 'nnictl stop' process, send SIGTERM to nni manager process, rather than calling stop Rest API.
+* OpenPAI training service bug fix
+
+  * Support NNI Manager IP configuration(nniManagerIp) in OpenPAI cluster config file, to fix the issue that user’s machine has no eth0 device
+  * File number in codeDir is capped to 1000 now, to avoid user mistakenly fill root dir for codeDir
+  * Don’t print useless ‘metrics is empty’ log in OpenPAI job’s stdout. Only print useful message once new metrics are recorded, to reduce confusion when user checks OpenPAI trial’s output for debugging purpose
+  * Add timestamp at the beginning of each log entry in trial keeper.
+
+Release 0.3.0 - 11/2/2018
+-------------------------
+
+NNICTL new features and updates
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+
+* 
+  Support running multiple experiments simultaneously.
+
+  Before v0.3, NNI only supports running single experiment once a time. After this release, users are able to run multiple experiments simultaneously. Each experiment will require a unique port, the 1st experiment will be set to the default port as previous versions. You can specify a unique port for the rest experiments as below:
+
+  .. code-block:: bash
+
+     nnictl create --port 8081 --config <config file path>
+
+* 
+  Support updating max trial number.
+  use ``nnictl update --help`` to learn more. Or refer to `NNICTL Spec <Tutorial/Nnictl.rst>`__ for the fully usage of NNICTL.
+
+API new features and updates
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+
+* 
+  :raw-html:`<span style="color:red">**breaking change**</span>`\ : nn.get_parameters() is refactored to nni.get_next_parameter. All examples of prior releases can not run on v0.3, please clone nni repo to get new examples. If you had applied NNI to your own codes, please update the API accordingly.
+
+* 
+  New API **nni.get_sequence_id()**.
+  Each trial job is allocated a unique sequence number, which can be retrieved by nni.get_sequence_id() API.
+
+  .. code-block:: bash
+
+     git clone -b v0.3 https://github.com/microsoft/nni.git
+
+* 
+  **nni.report_final_result(result)** API supports more data types for result parameter.
+
+  It can be of following types:
+
+
+  * int
+  * float
+  * A python dict containing 'default' key, the value of 'default' key should be of type int or float. The dict can contain any other key value pairs.
+
+New tuner support
+^^^^^^^^^^^^^^^^^
+
+
+* **Batch Tuner** which iterates all parameter combination, can be used to submit batch trial jobs.
+
+New examples
+^^^^^^^^^^^^
+
+
+* 
+  A NNI Docker image for public usage:
+
+  .. code-block:: bash
+
+     docker pull msranni/nni:latest
+
+* 
+  New trial example: `NNI Sklearn Example <https://github.com/microsoft/nni/tree/v1.9/examples/trials/sklearn>`__
+
+* New competition example: `Kaggle Competition TGS Salt Example <https://github.com/microsoft/nni/tree/v1.9/examples/trials/kaggle-tgs-salt>`__
+
+Others
+^^^^^^
+
+
+* UI refactoring, refer to `WebUI doc <Tutorial/WebUI.rst>`__ for how to work with the new UI.
+* Continuous Integration: NNI had switched to Azure pipelines
+
+Release 0.2.0 - 9/29/2018
+-------------------------
+
+Major Features
+^^^^^^^^^^^^^^
+
+
+* Support `OpenPAI <https://github.com/microsoft/pai>`__ Training Platform (See `here <TrainingService/PaiMode.rst>`__ for instructions about how to submit NNI job in pai mode)
+
+  * Support training services on pai mode. NNI trials will be scheduled to run on OpenPAI cluster
+  * NNI trial's output (including logs and model file) will be copied to OpenPAI HDFS for further debugging and checking
+
+* Support `SMAC <https://www.cs.ubc.ca/~hutter/papers/10-TR-SMAC.pdf>`__ tuner (See `here <Tuner/SmacTuner.rst>`__ for instructions about how to use SMAC tuner)
+
+  * `SMAC <https://www.cs.ubc.ca/~hutter/papers/10-TR-SMAC.pdf>`__ is based on Sequential Model-Based Optimization (SMBO). It adapts the most prominent previously used model class (Gaussian stochastic process models) and introduces the model class of random forests to SMBO to handle categorical parameters. The SMAC supported by NNI is a wrapper on `SMAC3 <https://github.com/automl/SMAC3>`__
+
+* Support NNI installation on `conda <https://conda.io/docs/index.html>`__ and python virtual environment
+* Others
+
+  * Update ga squad example and related documentation
+  * WebUI UX small enhancement and bug fix
+
+Release 0.1.0 - 9/10/2018 (initial release)
+-------------------------------------------
+
+Initial release of Neural Network Intelligence (NNI).
+
+Major Features
+^^^^^^^^^^^^^^
+
+
+* Installation and Deployment
+
+  * Support pip install and source codes install
+  * Support training services on local mode(including Multi-GPU mode) as well as multi-machines mode
+
+* Tuners, Assessors and Trial
+
+  * Support AutoML algorithms including:  hyperopt_tpe, hyperopt_annealing, hyperopt_random, and evolution_tuner
+  * Support assessor(early stop) algorithms including: medianstop algorithm
+  * Provide Python API for user defined tuners and assessors
+  * Provide Python API for user to wrap trial code as NNI deployable codes
+
+* Experiments
+
+  * Provide a command line toolkit 'nnictl' for experiments management
+  * Provide a WebUI for viewing experiments details and managing experiments
+
+* Continuous Integration
+
+  * Support CI by providing out-of-box integration with `travis-ci <https://github.com/travis-ci>`__ on ubuntu
+
+* Others
+
+  * Support simple GPU job scheduling