Resolve conflicts for #4760 (#4762)

docs/source/TrialExample/KDExample.rst → docs/source/sharings/kd_example.rst

@@ -35,12 +35,12 @@ PyTorch code
          loss.backward()
 
 
-The complete code for fine-tuning the pruned model can be found :githublink:`here <examples/model_compress/pruning/finetune_kd_torch.py>`
+The complete code for fine-tuning the pruned model can be found :githublink:`here <examples/model_compress/pruning/legacy/finetune_kd_torch.py>`
 
-.. code-block:: python
+.. code-block:: bash
 
-      python finetune_kd_torch.py --model [model name] --teacher-model-dir [pretrained checkpoint path]  --student-model-dir [pruned checkpoint path] --mask-path [mask file path]
+   python finetune_kd_torch.py --model [model name] --teacher-model-dir [pretrained checkpoint path]  --student-model-dir [pruned checkpoint path] --mask-path [mask file path]
 
-Note that: for fine-tuning a pruned model, run :githublink:`basic_pruners_torch.py <examples/model_compress/pruning/basic_pruners_torch.py>` first to get the mask file, then pass the mask path as argument to the script.
+Note that: for fine-tuning a pruned model, run :githublink:`basic_pruners_torch.py <examples/model_compress/pruning/legacy/basic_pruners_torch.py>` first to get the mask file, then pass the mask path as argument to the script.
 
 

docs/source/CommunitySharings/ModelCompressionComparison.rst → docs/source/sharings/model_compress_comp.rst

@@ -13,7 +13,7 @@ The experiments are performed with the following pruners/datasets/models:
 
 
 * 
-  Models: :githublink:`VGG16, ResNet18, ResNet50 <examples/model_compress/pruning/models/cifar10>`
+  Models: :githublink:`VGG16, ResNet18, ResNet50 <examples/model_compress/models/cifar10>`
 
 * 
   Datasets: CIFAR-10
@@ -35,7 +35,7 @@ The experiments are performed with the following pruners/datasets/models:
     For the pruners with scheduling, ``L1Filter Pruner`` is used as the base algorithm. That is to say, after the sparsities distribution is decided by the scheduling algorithm, ``L1Filter Pruner`` is used to performn real pruning.
 
   * 
-    All the pruners listed above are implemented in :githublink:`nni <docs/en_US/Compression/Overview.rst>`.
+    All the pruners listed above are implemented in :doc:`nni </compression/overview>`.
 
 Experiment Result
 -----------------
@@ -50,24 +50,24 @@ The experiment result are shown in the following figures:
 CIFAR-10, VGG16:
 
 
-.. image:: ../../../examples/model_compress/pruning/comparison_of_pruners/img/performance_comparison_vgg16.png
-   :target: ../../../examples/model_compress/pruning/comparison_of_pruners/img/performance_comparison_vgg16.png
+.. image:: ../../../examples/model_compress/pruning/legacy/comparison_of_pruners/img/performance_comparison_vgg16.png
+   :target: ../../../examples/model_compress/pruning/legacy/comparison_of_pruners/img/performance_comparison_vgg16.png
    :alt: 
 
 
 CIFAR-10, ResNet18:
 
 
-.. image:: ../../../examples/model_compress/pruning/comparison_of_pruners/img/performance_comparison_resnet18.png
-   :target: ../../../examples/model_compress/pruning/comparison_of_pruners/img/performance_comparison_resnet18.png
+.. image:: ../../../examples/model_compress/pruning/legacy/comparison_of_pruners/img/performance_comparison_resnet18.png
+   :target: ../../../examples/model_compress/pruning/legacy/comparison_of_pruners/img/performance_comparison_resnet18.png
    :alt: 
 
 
 CIFAR-10, ResNet50:
 
 
-.. image:: ../../../examples/model_compress/pruning/comparison_of_pruners/img/performance_comparison_resnet50.png
-   :target: ../../../examples/model_compress/pruning/comparison_of_pruners/img/performance_comparison_resnet50.png
+.. image:: ../../../examples/model_compress/pruning/legacy/comparison_of_pruners/img/performance_comparison_resnet50.png
+   :target: ../../../examples/model_compress/pruning/legacy/comparison_of_pruners/img/performance_comparison_resnet50.png
    :alt: 
 
 
@@ -88,22 +88,19 @@ Implementation Details
 ^^^^^^^^^^^^^^^^^^^^^^
 
 
-* 
-  The experiment results are all collected with the default configuration of the pruners in nni, which means that when we call a pruner class in nni, we don't change any default class arguments.
+* The experiment results are all collected with the default configuration of the pruners in nni, which means that when we call a pruner class in nni, we don't change any default class arguments.
 
-* 
-  Both FLOPs and the number of parameters are counted with :githublink:`Model FLOPs/Parameters Counter <docs/en_US/Compression/CompressionUtils.md#model-flopsparameters-counter>` after :githublink:`model speed up <docs/en_US/Compression/ModelSpeedup.rst>`.
+* Both FLOPs and the number of parameters are counted with :ref:`Model FLOPs/Parameters Counter <flops-counter>` after :doc:`model speedup </tutorials/pruning_speedup>`.
   This avoids potential issues of counting them of masked models.
 
-* 
-  The experiment code can be found :githublink:`here <examples/model_compress/pruning/auto_pruners_torch.py>`.
+* The experiment code can be found :githublink:`here <examples/model_compress/pruning/legacy/auto_pruners_torch.py>`.
 
 Experiment Result Rendering
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 
 * 
-  If you follow the practice in the :githublink:`example <examples/model_compress/pruning/auto_pruners_torch.py>`\ , for every single pruning experiment, the experiment result will be saved in JSON format as follows:
+  If you follow the practice in the :githublink:`example <examples/model_compress/pruning/legacy/auto_pruners_torch.py>`\ , for every single pruning experiment, the experiment result will be saved in JSON format as follows:
 
   .. code-block:: json
 
@@ -114,8 +111,8 @@ Experiment Result Rendering
        }
 
 * 
-  The experiment results are saved :githublink:`here <examples/model_compress/pruning/comparison_of_pruners>`. 
-  You can refer to :githublink:`analyze <examples/model_compress/pruning/comparison_of_pruners/analyze.py>` to plot new performance comparison figures.
+  The experiment results are saved :githublink:`here <examples/model_compress/pruning/legacy/comparison_of_pruners>`. 
+  You can refer to :githublink:`analyze <examples/model_compress/pruning/legacy/comparison_of_pruners/analyze.py>` to plot new performance comparison figures.
 
 Contribution
 ------------

docs/source/sharings/model_compression_toctree.rst

+Model Compression
+=================
+
+..  toctree::
+    :maxdepth: 1
+
+    Knowledge distillation with NNI model compression <kd_example>
\ No newline at end of file

docs/source/sharings/nn_spider.rst

+nnSpider Emoticons
+==================
+
+* Comfort
+
+  .. image:: images/nn_spider/comfort.png
+     :width: 400
+
+* Crying
+
+  .. image:: images/nn_spider/crying.png
+     :width: 400
+
+* Cut
+
+  .. image:: images/nn_spider/cut.png
+     :width: 400
+
+* Error
+
+  .. image:: images/nn_spider/error.png
+     :width: 400
+
+* Holiday
+
+  .. image:: images/nn_spider/holiday.png
+     :width: 400
+
+* No bug
+
+  .. image:: images/nn_spider/nobug.png
+     :width: 400
+
+* Sign
+
+  .. image:: images/nn_spider/sign.png
+     :width: 400
+
+* Sweat
+
+  .. image:: images/nn_spider/sweat.png
+     :width: 400
+
+* Weaving
+
+  .. image:: images/nn_spider/weaving.png
+     :width: 400
+
+* Working
+
+  .. image:: images/nn_spider/working.png
+     :width: 400

docs/source/CommunitySharings/NNI_colab_support.rst → docs/source/sharings/nni_colab_support.rst

@@ -6,40 +6,42 @@ NNI can easily run on Google Colab platform. However, Colab doesn't expose its p
 How to Open NNI's Web UI on Google Colab
 ----------------------------------------
 
-
 #. Install required packages and softwares.
 
-.. code-block:: bash
+   .. code-block:: bash
 
-   ! pip install nni # install nni
-   ! wget https://bin.equinox.io/c/4VmDzA7iaHb/ngrok-stable-linux-amd64.zip # download ngrok and unzip it
-   ! unzip ngrok-stable-linux-amd64.zip
-   ! mkdir -p nni_repo
-   ! git clone https://github.com/microsoft/nni.git nni_repo/nni # clone NNI's offical repo to get examples
+      ! pip install nni # install nni
+      ! wget https://bin.equinox.io/c/4VmDzA7iaHb/ngrok-stable-linux-amd64.zip # download ngrok and unzip it
+      ! unzip ngrok-stable-linux-amd64.zip
+      ! mkdir -p nni_repo
+      ! git clone https://github.com/microsoft/nni.git nni_repo/nni # clone NNI's offical repo to get examples
 
 
-#. Register a ngrok account `here <https://ngrok.com/>`__\ , then connect to your account using your authtoken.
+#. Register a ngrok account `here <https://ngrok.com/>`__, then connect to your account using your authtoken.
 
-.. code-block:: bash
+   .. code-block:: bash
 
-   ! ./ngrok authtoken <your-authtoken>
+      ! ./ngrok authtoken YOUR_AUTH_TOKEN
 
 
 #. Start an NNI example on a port bigger than 1024, then start ngrok with the same port. If you want to use gpu, make sure gpuNum >= 1 in config.yml. Use ``get_ipython()`` to start ngrok since it will be stuck if you use ``! ngrok http 5000 &``.
 
-.. code-block:: bash
+   .. code-block:: bash
+
+      ! nnictl create --config nni_repo/nni/examples/trials/mnist-pytorch/config.yml --port 5000 &
+
+   .. code-block:: python
 
-   ! nnictl create --config nni_repo/nni/examples/trials/mnist-pytorch/config.yml --port 5000 &
-   get_ipython().system_raw('./ngrok http 5000 &')
+      get_ipython().system_raw('./ngrok http 5000 &')
 
 
 #. Check the public url.
 
-.. code-block:: bash
+   .. code-block:: bash
 
-   ! curl -s http://localhost:4040/api/tunnels # don't change the port number 4040
+      ! curl -s http://localhost:4040/api/tunnels # don't change the port number 4040
 
-You will see an url like http://xxxx.ngrok.io after step 4, open this url and you will find NNI's Web UI. Have fun :)
+   You will see an url like ``http://xxxx.ngrok.io`` after step 4, open this url and you will find NNI's Web UI. Have fun :)
 
 Access Web UI with frp
 ----------------------

docs/source/TrialExample/OpEvoExamples.rst → docs/source/sharings/op_evo_examples.rst

@@ -118,7 +118,7 @@ Citing OpEvo
 
 If you feel OpEvo is helpful, please consider citing the paper as follows:
 
-.. code-block:: bash
+.. code-block:: bib
 
    @misc{gao2020opevo,
        title={OpEvo: An Evolutionary Method for Tensor Operator Optimization},

docs/source/sharings/overview.rst

+Use Cases and Solutions
+=======================
+
+Different from the tutorials and examples in the rest of the document which show the usage of a feature, this part mainly introduces end-to-end scenarios and use cases to help users further understand how NNI can help them. NNI can be widely adopted in various scenarios. We also encourage community contributors to share their AutoML practices especially the NNI usage practices from their experience.
+
+Automatic Model Tuning
+----------------------
+
+NNI can be applied on various model tuning tasks. Some state-of-the-art model search algorithms, such as EfficientNet, can be easily built on NNI. Popular models, e.g., recommendation models, can be tuned with NNI. The following are some use cases to illustrate how to leverage NNI in your model tuning tasks and how to build your own pipeline with NNI.
+
+* :doc:`Tuning SVD automatically <recommenders_svd>`
+* :doc:`EfficientNet on NNI <efficientnet>`
+* :doc:`Automatic Model Architecture Search for Reading Comprehension <squad_evolution_examples>`
+* :doc:`Parallelizing Optimization for TPE <parallelizing_tpe_search>`
+
+Automatic System Tuning
+-----------------------
+
+The performance of systems, such as database, tensor operator implementaion, often need to be tuned to adapt to specific hardware configuration, targeted workload, etc. Manually tuning a system is complicated and often requires detailed understanding of hardware and workload. NNI can make such tasks much easier and help system owners find the best configuration to the system automatically. The detailed design philosophy of automatic system tuning can be found in this `paper <https://dl.acm.org/doi/10.1145/3352020.3352031>`__ . The following are some typical cases that NNI can help.
+
+* :doc:`Tuning SPTAG (Space Partition Tree And Graph) automatically <sptag_auto_tune>`
+* :doc:`Tuning the performance of RocksDB <rocksdb_examples>`
+* :doc:`Tuning Tensor Operators automatically <op_evo_examples>`
+
+Model Compression
+-----------------
+
+The following one shows how to apply knowledge distillation on NNI model compression. More use cases and solutions will be added in the future.
+
+* :doc:`Knowledge distillation with NNI model compression <kd_example>`
+
+Feature Engineering
+-------------------
+
+The following is an article about how NNI helps in auto feature engineering shared by a community contributor. More use cases and solutions will be added in the future.
+
+* :doc:`NNI review article from Zhihu: - By Garvin Li <nni_autofeatureeng>`
+
+Performance Measurement, Comparison and Analysis
+------------------------------------------------
+
+Performance comparison and analysis can help users decide a proper algorithm (e.g., tuner, NAS algorithm) for their scenario. The following are some measurement and comparison data for users' reference.
+
+* :doc:`Neural Architecture Search Comparison <nas_comparison>`
+* :doc:`Hyper-parameter Tuning Algorithm Comparsion <hpo_comparison>`
+* :doc:`Model Compression Algorithm Comparsion <model_compress_comp>`

docs/source/sharings/perf_compare_toctree.rst

+Performance Measurement, Comparison and Analysis
+================================================
+
+..  toctree::
+    :maxdepth: 1
+
+    Neural Architecture Search Comparison <nas_comparison>
+    Hyper-parameter Tuning Algorithm Comparsion <hpo_comparison>
+    Model Compression Algorithm Comparsion <model_compress_comp>
\ No newline at end of file

docs/source/TrialExample/RocksdbExamples.rst → docs/source/sharings/rocksdb_examples.rst

@@ -8,7 +8,7 @@ Overview
 
 The performance of RocksDB is highly contingent on its tuning. However, because of the complexity of its underlying technology and a large number of configurable parameters, a good configuration is sometimes hard to obtain. NNI can help to address this issue. NNI supports many kinds of tuning algorithms to search the best configuration of RocksDB, and support many kinds of environments like local machine, remote servers and cloud. 
 
-This example illustrates how to use NNI to search the best configuration of RocksDB for a ``fillrandom`` benchmark supported by a benchmark tool ``db_bench``\ , which is an official benchmark tool provided by RocksDB itself. Therefore, before running this example, please make sure NNI is installed and `db_bench <https://github.com/facebook/rocksdb/wiki/Benchmarking-tools>`__ is in your ``PATH``. Please refer to `here <../Tutorial/QuickStart.rst>`__ for detailed information about installation and preparing of NNI environment, and `here <https://github.com/facebook/rocksdb/blob/master/INSTALL.md>`__ for compiling RocksDB as well as ``db_bench``.
+This example illustrates how to use NNI to search the best configuration of RocksDB for a ``fillrandom`` benchmark supported by a benchmark tool ``db_bench``\ , which is an official benchmark tool provided by RocksDB itself. Therefore, before running this example, please make sure NNI is installed and `db_bench <https://github.com/facebook/rocksdb/wiki/Benchmarking-tools>`__ is in your ``PATH``. Please refer to :doc:`here </installation>` for detailed information about installation and preparing of NNI environment, and `here <https://github.com/facebook/rocksdb/blob/master/INSTALL.md>`__ for compiling RocksDB as well as ``db_bench``.
 
 We also provide a simple script :githublink:`db_bench_installation.sh <examples/trials/systems_auto_tuning/rocksdb-fillrandom/db_bench_installation.sh>` helping to compile and install ``db_bench`` as well as its dependencies on Ubuntu. Installing RocksDB on other systems can follow the same procedure.
 
@@ -24,7 +24,7 @@ Search Space
 
 For simplicity, this example tunes three parameters, ``write_buffer_size``\ , ``min_write_buffer_num`` and ``level0_file_num_compaction_trigger``\ , for writing 16M keys with 20 Bytes of key size and 100 Bytes of value size randomly, based on writing operations per second (OPS). ``write_buffer_size`` sets the size of a single memtable. Once memtable exceeds this size, it is marked immutable and a new one is created. ``min_write_buffer_num`` is the minimum number of memtables to be merged before flushing to storage. Once the number of files in level 0 reaches ``level0_file_num_compaction_trigger``\ , level 0 to level 1 compaction is triggered.
 
-In this example, the search space is specified by a ``search_space.json`` file as shown below. Detailed explanation of search space could be found `here <../Tutorial/SearchSpaceSpec.rst>`__.
+In this example, the search space is specified by a ``search_space.json`` file as shown below. Detailed explanation of search space could be found :doc:`here </hpo/search_space>`.
 
 .. code-block:: json
 
@@ -48,8 +48,7 @@ In this example, the search space is specified by a ``search_space.json`` file a
 Benchmark code
 ^^^^^^^^^^^^^^
 
-Benchmark code should receive a configuration from NNI manager, and report the corresponding benchmark result back. Following NNI APIs are designed for this purpose. In this example, writing operations per second (OPS) is used as a performance metric. Please refer to `here <Trials.rst>`__ for detailed information.
-
+Benchmark code should receive a configuration from NNI manager, and report the corresponding benchmark result back. Following NNI APIs are designed for this purpose. In this example, writing operations per second (OPS) is used as a performance metric.
 
 * Use ``nni.get_next_parameter()`` to get next system configuration.
 * Use ``nni.report_final_result(metric)`` to report the benchmark result.
@@ -59,7 +58,7 @@ Benchmark code should receive a configuration from NNI manager, and report the c
 Config file
 ^^^^^^^^^^^
 
-One could start a NNI experiment with a config file. A config file for NNI is a ``yaml`` file usually including experiment settings (\ ``trialConcurrency``\ , ``trialGpuNumber``\ , etc.), platform settings (\ ``trainingService``\ ), path settings (\ ``searchSpaceFile``\ , ``trialCodeDirectory``\ , etc.) and tuner settings (\ ``tuner``\ , ``tuner optimize_mode``\ , etc.). Please refer to `here <../Tutorial/QuickStart.rst>`__ for more information.
+One could start a NNI experiment with a config file. A config file for NNI is a ``yaml`` file usually including experiment settings (\ ``trialConcurrency``\ , ``trialGpuNumber``\ , etc.), platform settings (\ ``trainingService``\ ), path settings (\ ``searchSpaceFile``\ , ``trialCodeDirectory``\ , etc.) and tuner settings (\ ``tuner``\ , ``tuner optimize_mode``\ , etc.). Please refer to :doc:`/reference/experiment_config`.
 
 Here is an example of tuning RocksDB with SMAC algorithm:
 
@@ -69,7 +68,7 @@ Here is an example of tuning RocksDB with TPE algorithm:
 
 :githublink:`code directory <examples/trials/systems_auto_tuning/rocksdb-fillrandom/config_tpe.yml>`
 
-Other tuners can be easily adopted in the same way. Please refer to `here <../Tuner/BuiltinTuner.rst>`__ for more information.
+Other tuners can be easily adopted in the same way. Please refer to :doc:`here </hpo/tuners>` for more information.
 
 Finally, we could enter the example folder and start the experiment using following commands:
 

docs/source/TrialExample/SquadEvolutionExamples.rst → docs/source/sharings/squad_evolution_examples.rst

@@ -146,9 +146,9 @@ Among those files, ``trial.py`` and ``graph_to_tf.py`` are special.
            if topo_i == '|':
                continue
            if graph.layers[topo_i].graph_type == LayerType.input.value:
-               # ......
+               ...
            elif graph.layers[topo_i].graph_type == LayerType.attention.value:
-               # ......
+               ...
            # More layers to handle
 
 As we can see, this function is actually a compiler, that converts the internal model DAG configuration (which will be introduced in the ``Model configuration format`` section) ``graph``\ , to a Tensorflow computation graph.
@@ -162,6 +162,7 @@ performs topological sorting on the internal graph representation, and the code
 .. code-block:: python
 
    for _, topo_i in enumerate(topology):
+       ...
 
 performs actually conversion that maps each layer to a part in Tensorflow computation graph.
 

docs/source/training_services_zh.rst

-.. 4e054d96c7d211dc514c99d673415d8e
-
-NNI 支持的训练平台介绍
-=====================================
-
-..  toctree::
-    Overview <./TrainingService/Overview>
-    本机<./TrainingService/LocalMode>
-    远程<./TrainingService/RemoteMachineMode>
-    OpenPAI<./TrainingService/PaiMode>
-    Kubeflow<./TrainingService/KubeflowMode>
-    AdaptDL<./TrainingService/AdaptDLMode>
-    FrameworkController<./TrainingService/FrameworkControllerMode>
-    DLTS<./TrainingService/DLTSMode>
-    AML<./TrainingService/AMLMode>
-    PAI-DLC<./TrainingService/DLCMode>
-    混合模式 <./TrainingService/HybridMode>

docs/source/tutorials.rst

-Tutorials
-=========
-
-.. TOC
-
-.. toctree::
-   :maxdepth: 2
-   :hidden:
-
-   tutorials/nni_experiment
-   tutorials/nas_quick_start_mnist
-
-.. ----------------------
-
-.. cardlinkitem::
-   :header: Start and Manage a New Experiment
-   :description: Familiarize yourself with Pythonic API to manage a hyper-parameter tuning experiment
-   :link: tutorials/nni_experiment.html
-   :image: ../img/thumbnails/overview-31.png
-   :tags: Experiment/HPO
-
-.. cardlinkitem::
-   :header: Get started with NAS on MNIST
-   :description: bla bla bla bla
-   :link: tutorials/nas_quick_start_mnist.html
-   :image: ../img/thumbnails/overview-30.png
-   :background: cyan
-   :tags: NAS

docs/source/tutorials/hello_nas.ipynb

+{
+  "cells": [
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "collapsed": false
+      },
+      "outputs": [],
+      "source": [
+        "%matplotlib inline"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "\n# Hello, NAS!\n\nThis is the 101 tutorial of Neural Architecture Search (NAS) on NNI.\nIn this tutorial, we will search for a neural architecture on MNIST dataset with the help of NAS framework of NNI, i.e., *Retiarii*.\nWe use multi-trial NAS as an example to show how to construct and explore a model space.\n\nThere are mainly three crucial components for a neural architecture search task, namely,\n\n* Model search space that defines a set of models to explore.\n* A proper strategy as the method to explore this model space.\n* A model evaluator that reports the performance of every model in the space.\n\nCurrently, PyTorch is the only supported framework by Retiarii, and we have only tested **PyTorch 1.7 to 1.10**.\nThis tutorial assumes PyTorch context but it should also apply to other frameworks, which is in our future plan.\n\n## Define your Model Space\n\nModel space is defined by users to express a set of models that users want to explore, which contains potentially good-performing models.\nIn this framework, a model space is defined with two parts: a base model and possible mutations on the base model.\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "### Define Base Model\n\nDefining a base model is almost the same as defining a PyTorch (or TensorFlow) model.\nUsually, you only need to replace the code ``import torch.nn as nn`` with\n``import nni.retiarii.nn.pytorch as nn`` to use our wrapped PyTorch modules.\n\nBelow is a very simple example of defining a base model.\n\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "collapsed": false
+      },
+      "outputs": [],
+      "source": [
+        "import torch\nimport torch.nn.functional as F\nimport nni.retiarii.nn.pytorch as nn\nfrom nni.retiarii import model_wrapper\n\n\n@model_wrapper      # this decorator should be put on the out most\nclass Net(nn.Module):\n    def __init__(self):\n        super().__init__()\n        self.conv1 = nn.Conv2d(1, 32, 3, 1)\n        self.conv2 = nn.Conv2d(32, 64, 3, 1)\n        self.dropout1 = nn.Dropout(0.25)\n        self.dropout2 = nn.Dropout(0.5)\n        self.fc1 = nn.Linear(9216, 128)\n        self.fc2 = nn.Linear(128, 10)\n\n    def forward(self, x):\n        x = F.relu(self.conv1(x))\n        x = F.max_pool2d(self.conv2(x), 2)\n        x = torch.flatten(self.dropout1(x), 1)\n        x = self.fc2(self.dropout2(F.relu(self.fc1(x))))\n        output = F.log_softmax(x, dim=1)\n        return output"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        ".. tip:: Always keep in mind that you should use ``import nni.retiarii.nn.pytorch as nn`` and :meth:`nni.retiarii.model_wrapper`.\n         Many mistakes are a result of forgetting one of those.\n         Also, please use ``torch.nn`` for submodules of ``nn.init``, e.g., ``torch.nn.init`` instead of ``nn.init``.\n\n### Define Model Mutations\n\nA base model is only one concrete model not a model space. We provide :doc:`API and Primitives </nas/construct_space>`\nfor users to express how the base model can be mutated. That is, to build a model space which includes many models.\n\nBased on the above base model, we can define a model space as below.\n\n.. code-block:: diff\n\n  @model_wrapper\n  class Net(nn.Module):\n    def __init__(self):\n      super().__init__()\n      self.conv1 = nn.Conv2d(1, 32, 3, 1)\n  -   self.conv2 = nn.Conv2d(32, 64, 3, 1)\n  +   self.conv2 = nn.LayerChoice([\n  +       nn.Conv2d(32, 64, 3, 1),\n  +       DepthwiseSeparableConv(32, 64)\n  +   ])\n  -   self.dropout1 = nn.Dropout(0.25)\n  +   self.dropout1 = nn.Dropout(nn.ValueChoice([0.25, 0.5, 0.75]))\n      self.dropout2 = nn.Dropout(0.5)\n  -   self.fc1 = nn.Linear(9216, 128)\n  -   self.fc2 = nn.Linear(128, 10)\n  +   feature = nn.ValueChoice([64, 128, 256])\n  +   self.fc1 = nn.Linear(9216, feature)\n  +   self.fc2 = nn.Linear(feature, 10)\n\n    def forward(self, x):\n      x = F.relu(self.conv1(x))\n      x = F.max_pool2d(self.conv2(x), 2)\n      x = torch.flatten(self.dropout1(x), 1)\n      x = self.fc2(self.dropout2(F.relu(self.fc1(x))))\n      output = F.log_softmax(x, dim=1)\n      return output\n\nThis results in the following code:\n\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "collapsed": false
+      },
+      "outputs": [],
+      "source": [
+        "class DepthwiseSeparableConv(nn.Module):\n    def __init__(self, in_ch, out_ch):\n        super().__init__()\n        self.depthwise = nn.Conv2d(in_ch, in_ch, kernel_size=3, groups=in_ch)\n        self.pointwise = nn.Conv2d(in_ch, out_ch, kernel_size=1)\n\n    def forward(self, x):\n        return self.pointwise(self.depthwise(x))\n\n\n@model_wrapper\nclass ModelSpace(nn.Module):\n    def __init__(self):\n        super().__init__()\n        self.conv1 = nn.Conv2d(1, 32, 3, 1)\n        # LayerChoice is used to select a layer between Conv2d and DwConv.\n        self.conv2 = nn.LayerChoice([\n            nn.Conv2d(32, 64, 3, 1),\n            DepthwiseSeparableConv(32, 64)\n        ])\n        # ValueChoice is used to select a dropout rate.\n        # ValueChoice can be used as parameter of modules wrapped in `nni.retiarii.nn.pytorch`\n        # or customized modules wrapped with `@basic_unit`.\n        self.dropout1 = nn.Dropout(nn.ValueChoice([0.25, 0.5, 0.75]))  # choose dropout rate from 0.25, 0.5 and 0.75\n        self.dropout2 = nn.Dropout(0.5)\n        feature = nn.ValueChoice([64, 128, 256])\n        self.fc1 = nn.Linear(9216, feature)\n        self.fc2 = nn.Linear(feature, 10)\n\n    def forward(self, x):\n        x = F.relu(self.conv1(x))\n        x = F.max_pool2d(self.conv2(x), 2)\n        x = torch.flatten(self.dropout1(x), 1)\n        x = self.fc2(self.dropout2(F.relu(self.fc1(x))))\n        output = F.log_softmax(x, dim=1)\n        return output\n\n\nmodel_space = ModelSpace()\nmodel_space"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "This example uses two mutation APIs,\n:class:`nn.LayerChoice <nni.retiarii.nn.pytorch.LayerChoice>` and\n:class:`nn.InputChoice <nni.retiarii.nn.pytorch.ValueChoice>`.\n:class:`nn.LayerChoice <nni.retiarii.nn.pytorch.LayerChoice>`\ntakes a list of candidate modules (two in this example), one will be chosen for each sampled model.\nIt can be used like normal PyTorch module.\n:class:`nn.InputChoice <nni.retiarii.nn.pytorch.ValueChoice>` takes a list of candidate values,\none will be chosen to take effect for each sampled model.\n\nMore detailed API description and usage can be found :doc:`here </nas/construct_space>`.\n\n<div class=\"alert alert-info\"><h4>Note</h4><p>We are actively enriching the mutation APIs, to facilitate easy construction of model space.\n    If the currently supported mutation APIs cannot express your model space,\n    please refer to :doc:`this doc </nas/mutator>` for customizing mutators.</p></div>\n\n## Explore the Defined Model Space\n\nThere are basically two exploration approaches: (1) search by evaluating each sampled model independently,\nwhich is the search approach in `multi-trial NAS <multi-trial-nas>`\nand (2) one-shot weight-sharing based search, which is used in one-shot NAS.\nWe demonstrate the first approach in this tutorial. Users can refer to `here <one-shot-nas>` for the second approach.\n\nFirst, users need to pick a proper exploration strategy to explore the defined model space.\nSecond, users need to pick or customize a model evaluator to evaluate the performance of each explored model.\n\n### Pick an exploration strategy\n\nRetiarii supports many :doc:`exploration strategies </nas/exploration_strategy>`.\n\nSimply choosing (i.e., instantiate) an exploration strategy as below.\n\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "collapsed": false
+      },
+      "outputs": [],
+      "source": [
+        "import nni.retiarii.strategy as strategy\nsearch_strategy = strategy.Random(dedup=True)  # dedup=False if deduplication is not wanted"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "### Pick or customize a model evaluator\n\nIn the exploration process, the exploration strategy repeatedly generates new models. A model evaluator is for training\nand validating each generated model to obtain the model's performance.\nThe performance is sent to the exploration strategy for the strategy to generate better models.\n\nRetiarii has provided :doc:`built-in model evaluators </nas/evaluator>`, but to start with,\nit is recommended to use :class:`FunctionalEvaluator <nni.retiarii.evaluator.FunctionalEvaluator>`,\nthat is, to wrap your own training and evaluation code with one single function.\nThis function should receive one single model class and uses :func:`nni.report_final_result` to report the final score of this model.\n\nAn example here creates a simple evaluator that runs on MNIST dataset, trains for 2 epochs, and reports its validation accuracy.\n\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "collapsed": false
+      },
+      "outputs": [],
+      "source": [
+        "import nni\n\nfrom torchvision import transforms\nfrom torchvision.datasets import MNIST\nfrom torch.utils.data import DataLoader\n\n\ndef train_epoch(model, device, train_loader, optimizer, epoch):\n    loss_fn = torch.nn.CrossEntropyLoss()\n    model.train()\n    for batch_idx, (data, target) in enumerate(train_loader):\n        data, target = data.to(device), target.to(device)\n        optimizer.zero_grad()\n        output = model(data)\n        loss = loss_fn(output, target)\n        loss.backward()\n        optimizer.step()\n        if batch_idx % 10 == 0:\n            print('Train Epoch: {} [{}/{} ({:.0f}%)]\\tLoss: {:.6f}'.format(\n                epoch, batch_idx * len(data), len(train_loader.dataset),\n                100. * batch_idx / len(train_loader), loss.item()))\n\n\ndef test_epoch(model, device, test_loader):\n    model.eval()\n    test_loss = 0\n    correct = 0\n    with torch.no_grad():\n        for data, target in test_loader:\n            data, target = data.to(device), target.to(device)\n            output = model(data)\n            pred = output.argmax(dim=1, keepdim=True)\n            correct += pred.eq(target.view_as(pred)).sum().item()\n\n    test_loss /= len(test_loader.dataset)\n    accuracy = 100. * correct / len(test_loader.dataset)\n\n    print('\\nTest set: Accuracy: {}/{} ({:.0f}%)\\n'.format(\n          correct, len(test_loader.dataset), accuracy))\n\n    return accuracy\n\n\ndef evaluate_model(model_cls):\n    # \"model_cls\" is a class, need to instantiate\n    model = model_cls()\n\n    device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')\n    model.to(device)\n\n    optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)\n    transf = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.1307,), (0.3081,))])\n    train_loader = DataLoader(MNIST('data/mnist', download=True, transform=transf), batch_size=64, shuffle=True)\n    test_loader = DataLoader(MNIST('data/mnist', download=True, train=False, transform=transf), batch_size=64)\n\n    for epoch in range(3):\n        # train the model for one epoch\n        train_epoch(model, device, train_loader, optimizer, epoch)\n        # test the model for one epoch\n        accuracy = test_epoch(model, device, test_loader)\n        # call report intermediate result. Result can be float or dict\n        nni.report_intermediate_result(accuracy)\n\n    # report final test result\n    nni.report_final_result(accuracy)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Create the evaluator\n\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "collapsed": false
+      },
+      "outputs": [],
+      "source": [
+        "from nni.retiarii.evaluator import FunctionalEvaluator\nevaluator = FunctionalEvaluator(evaluate_model)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "The ``train_epoch`` and ``test_epoch`` here can be any customized function,\nwhere users can write their own training recipe.\n\nIt is recommended that the ``evaluate_model`` here accepts no additional arguments other than ``model_cls``.\nHowever, in the :doc:`advanced tutorial </nas/evaluator>`, we will show how to use additional arguments in case you actually need those.\nIn future, we will support mutation on the arguments of evaluators, which is commonly called \"Hyper-parmeter tuning\".\n\n## Launch an Experiment\n\nAfter all the above are prepared, it is time to start an experiment to do the model search. An example is shown below.\n\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "collapsed": false
+      },
+      "outputs": [],
+      "source": [
+        "from nni.retiarii.experiment.pytorch import RetiariiExperiment, RetiariiExeConfig\nexp = RetiariiExperiment(model_space, evaluator, [], search_strategy)\nexp_config = RetiariiExeConfig('local')\nexp_config.experiment_name = 'mnist_search'"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "The following configurations are useful to control how many trials to run at most / at the same time.\n\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "collapsed": false
+      },
+      "outputs": [],
+      "source": [
+        "exp_config.max_trial_number = 4   # spawn 4 trials at most\nexp_config.trial_concurrency = 2  # will run two trials concurrently"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Remember to set the following config if you want to GPU.\n``use_active_gpu`` should be set true if you wish to use an occupied GPU (possibly running a GUI).\n\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "collapsed": false
+      },
+      "outputs": [],
+      "source": [
+        "exp_config.trial_gpu_number = 1\nexp_config.training_service.use_active_gpu = True"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Launch the experiment. The experiment should take several minutes to finish on a workstation with 2 GPUs.\n\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "collapsed": false
+      },
+      "outputs": [],
+      "source": [
+        "exp.run(exp_config, 8081)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Users can also run Retiarii Experiment with :doc:`different training services </experiment/training_service/overview>`\nbesides ``local`` training service.\n\n## Visualize the Experiment\n\nUsers can visualize their experiment in the same way as visualizing a normal hyper-parameter tuning experiment.\nFor example, open ``localhost:8081`` in your browser, 8081 is the port that you set in ``exp.run``.\nPlease refer to :doc:`here </experiment/web_portal/web_portal>` for details.\n\nWe support visualizing models with 3rd-party visualization engines (like `Netron <https://netron.app/>`__).\nThis can be used by clicking ``Visualization`` in detail panel for each trial.\nNote that current visualization is based on `onnx <https://onnx.ai/>`__ ,\nthus visualization is not feasible if the model cannot be exported into onnx.\n\nBuilt-in evaluators (e.g., Classification) will automatically export the model into a file.\nFor your own evaluator, you need to save your file into ``$NNI_OUTPUT_DIR/model.onnx`` to make this work.\nFor instance,\n\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "collapsed": false
+      },
+      "outputs": [],
+      "source": [
+        "import os\nfrom pathlib import Path\n\n\ndef evaluate_model_with_visualization(model_cls):\n    model = model_cls()\n    # dump the model into an onnx\n    if 'NNI_OUTPUT_DIR' in os.environ:\n        dummy_input = torch.zeros(1, 3, 32, 32)\n        torch.onnx.export(model, (dummy_input, ),\n                          Path(os.environ['NNI_OUTPUT_DIR']) / 'model.onnx')\n    evaluate_model(model_cls)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Relaunch the experiment, and a button is shown on Web portal.\n\n<img src=\"file://../../img/netron_entrance_webui.png\">\n\n## Export Top Models\n\nUsers can export top models after the exploration is done using ``export_top_models``.\n\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "collapsed": false
+      },
+      "outputs": [],
+      "source": [
+        "for model_dict in exp.export_top_models(formatter='dict'):\n    print(model_dict)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "The output is ``json`` object which records the mutation actions of the top model.\nIf users want to output source code of the top model,\nthey can use `graph-based execution engine <graph-based-execution-engine>` for the experiment,\nby simply adding the following two lines.\n\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "collapsed": false
+      },
+      "outputs": [],
+      "source": [
+        "exp_config.execution_engine = 'base'\nexport_formatter = 'code'"
+      ]
+    }
+  ],
+  "metadata": {
+    "kernelspec": {
+      "display_name": "Python 3",
+      "language": "python",
+      "name": "python3"
+    },
+    "language_info": {
+      "codemirror_mode": {
+        "name": "ipython",
+        "version": 3
+      },
+      "file_extension": ".py",
+      "mimetype": "text/x-python",
+      "name": "python",
+      "nbconvert_exporter": "python",
+      "pygments_lexer": "ipython3",
+      "version": "3.8.8"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
+}
\ No newline at end of file

docs/source/tutorials/hello_nas.py

+"""
+Hello, NAS!
+===========
+
+This is the 101 tutorial of Neural Architecture Search (NAS) on NNI.
+In this tutorial, we will search for a neural architecture on MNIST dataset with the help of NAS framework of NNI, i.e., *Retiarii*.
+We use multi-trial NAS as an example to show how to construct and explore a model space.
+
+There are mainly three crucial components for a neural architecture search task, namely,
+
+* Model search space that defines a set of models to explore.
+* A proper strategy as the method to explore this model space.
+* A model evaluator that reports the performance of every model in the space.
+
+Currently, PyTorch is the only supported framework by Retiarii, and we have only tested **PyTorch 1.7 to 1.10**.
+This tutorial assumes PyTorch context but it should also apply to other frameworks, which is in our future plan.
+
+Define your Model Space
+-----------------------
+
+Model space is defined by users to express a set of models that users want to explore, which contains potentially good-performing models.
+In this framework, a model space is defined with two parts: a base model and possible mutations on the base model.
+"""
+
+# %%
+#
+# Define Base Model
+# ^^^^^^^^^^^^^^^^^
+#
+# Defining a base model is almost the same as defining a PyTorch (or TensorFlow) model.
+# Usually, you only need to replace the code ``import torch.nn as nn`` with
+# ``import nni.retiarii.nn.pytorch as nn`` to use our wrapped PyTorch modules.
+#
+# Below is a very simple example of defining a base model.
+
+import torch
+import torch.nn.functional as F
+import nni.retiarii.nn.pytorch as nn
+from nni.retiarii import model_wrapper
+
+
+@model_wrapper      # this decorator should be put on the out most
+class Net(nn.Module):
+    def __init__(self):
+        super().__init__()
+        self.conv1 = nn.Conv2d(1, 32, 3, 1)
+        self.conv2 = nn.Conv2d(32, 64, 3, 1)
+        self.dropout1 = nn.Dropout(0.25)
+        self.dropout2 = nn.Dropout(0.5)
+        self.fc1 = nn.Linear(9216, 128)
+        self.fc2 = nn.Linear(128, 10)
+
+    def forward(self, x):
+        x = F.relu(self.conv1(x))
+        x = F.max_pool2d(self.conv2(x), 2)
+        x = torch.flatten(self.dropout1(x), 1)
+        x = self.fc2(self.dropout2(F.relu(self.fc1(x))))
+        output = F.log_softmax(x, dim=1)
+        return output
+
+# %%
+# .. tip:: Always keep in mind that you should use ``import nni.retiarii.nn.pytorch as nn`` and :meth:`nni.retiarii.model_wrapper`.
+#          Many mistakes are a result of forgetting one of those.
+#          Also, please use ``torch.nn`` for submodules of ``nn.init``, e.g., ``torch.nn.init`` instead of ``nn.init``.
+#
+# Define Model Mutations
+# ^^^^^^^^^^^^^^^^^^^^^^
+#
+# A base model is only one concrete model not a model space. We provide :doc:`API and Primitives </nas/construct_space>`
+# for users to express how the base model can be mutated. That is, to build a model space which includes many models.
+#
+# Based on the above base model, we can define a model space as below.
+#
+# .. code-block:: diff
+#
+#   @model_wrapper
+#   class Net(nn.Module):
+#     def __init__(self):
+#       super().__init__()
+#       self.conv1 = nn.Conv2d(1, 32, 3, 1)
+#   -   self.conv2 = nn.Conv2d(32, 64, 3, 1)
+#   +   self.conv2 = nn.LayerChoice([
+#   +       nn.Conv2d(32, 64, 3, 1),
+#   +       DepthwiseSeparableConv(32, 64)
+#   +   ])
+#   -   self.dropout1 = nn.Dropout(0.25)
+#   +   self.dropout1 = nn.Dropout(nn.ValueChoice([0.25, 0.5, 0.75]))
+#       self.dropout2 = nn.Dropout(0.5)
+#   -   self.fc1 = nn.Linear(9216, 128)
+#   -   self.fc2 = nn.Linear(128, 10)
+#   +   feature = nn.ValueChoice([64, 128, 256])
+#   +   self.fc1 = nn.Linear(9216, feature)
+#   +   self.fc2 = nn.Linear(feature, 10)
+#
+#     def forward(self, x):
+#       x = F.relu(self.conv1(x))
+#       x = F.max_pool2d(self.conv2(x), 2)
+#       x = torch.flatten(self.dropout1(x), 1)
+#       x = self.fc2(self.dropout2(F.relu(self.fc1(x))))
+#       output = F.log_softmax(x, dim=1)
+#       return output
+#
+# This results in the following code:
+
+
+class DepthwiseSeparableConv(nn.Module):
+    def __init__(self, in_ch, out_ch):
+        super().__init__()
+        self.depthwise = nn.Conv2d(in_ch, in_ch, kernel_size=3, groups=in_ch)
+        self.pointwise = nn.Conv2d(in_ch, out_ch, kernel_size=1)
+
+    def forward(self, x):
+        return self.pointwise(self.depthwise(x))
+
+
+@model_wrapper
+class ModelSpace(nn.Module):
+    def __init__(self):
+        super().__init__()
+        self.conv1 = nn.Conv2d(1, 32, 3, 1)
+        # LayerChoice is used to select a layer between Conv2d and DwConv.
+        self.conv2 = nn.LayerChoice([
+            nn.Conv2d(32, 64, 3, 1),
+            DepthwiseSeparableConv(32, 64)
+        ])
+        # ValueChoice is used to select a dropout rate.
+        # ValueChoice can be used as parameter of modules wrapped in `nni.retiarii.nn.pytorch`
+        # or customized modules wrapped with `@basic_unit`.
+        self.dropout1 = nn.Dropout(nn.ValueChoice([0.25, 0.5, 0.75]))  # choose dropout rate from 0.25, 0.5 and 0.75
+        self.dropout2 = nn.Dropout(0.5)
+        feature = nn.ValueChoice([64, 128, 256])
+        self.fc1 = nn.Linear(9216, feature)
+        self.fc2 = nn.Linear(feature, 10)
+
+    def forward(self, x):
+        x = F.relu(self.conv1(x))
+        x = F.max_pool2d(self.conv2(x), 2)
+        x = torch.flatten(self.dropout1(x), 1)
+        x = self.fc2(self.dropout2(F.relu(self.fc1(x))))
+        output = F.log_softmax(x, dim=1)
+        return output
+
+
+model_space = ModelSpace()
+model_space
+
+# %%
+# This example uses two mutation APIs,
+# :class:`nn.LayerChoice <nni.retiarii.nn.pytorch.LayerChoice>` and
+# :class:`nn.InputChoice <nni.retiarii.nn.pytorch.ValueChoice>`.
+# :class:`nn.LayerChoice <nni.retiarii.nn.pytorch.LayerChoice>`
+# takes a list of candidate modules (two in this example), one will be chosen for each sampled model.
+# It can be used like normal PyTorch module.
+# :class:`nn.InputChoice <nni.retiarii.nn.pytorch.ValueChoice>` takes a list of candidate values,
+# one will be chosen to take effect for each sampled model.
+#
+# More detailed API description and usage can be found :doc:`here </nas/construct_space>`.
+#
+# .. note::
+#
+#     We are actively enriching the mutation APIs, to facilitate easy construction of model space.
+#     If the currently supported mutation APIs cannot express your model space,
+#     please refer to :doc:`this doc </nas/mutator>` for customizing mutators.
+#
+# Explore the Defined Model Space
+# -------------------------------
+#
+# There are basically two exploration approaches: (1) search by evaluating each sampled model independently,
+# which is the search approach in :ref:`multi-trial NAS <multi-trial-nas>`
+# and (2) one-shot weight-sharing based search, which is used in one-shot NAS.
+# We demonstrate the first approach in this tutorial. Users can refer to :ref:`here <one-shot-nas>` for the second approach.
+#
+# First, users need to pick a proper exploration strategy to explore the defined model space.
+# Second, users need to pick or customize a model evaluator to evaluate the performance of each explored model.
+#
+# Pick an exploration strategy
+# ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+#
+# Retiarii supports many :doc:`exploration strategies </nas/exploration_strategy>`.
+#
+# Simply choosing (i.e., instantiate) an exploration strategy as below.
+
+import nni.retiarii.strategy as strategy
+search_strategy = strategy.Random(dedup=True)  # dedup=False if deduplication is not wanted
+
+# %%
+# Pick or customize a model evaluator
+# ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+#
+# In the exploration process, the exploration strategy repeatedly generates new models. A model evaluator is for training
+# and validating each generated model to obtain the model's performance.
+# The performance is sent to the exploration strategy for the strategy to generate better models.
+#
+# Retiarii has provided :doc:`built-in model evaluators </nas/evaluator>`, but to start with,
+# it is recommended to use :class:`FunctionalEvaluator <nni.retiarii.evaluator.FunctionalEvaluator>`,
+# that is, to wrap your own training and evaluation code with one single function.
+# This function should receive one single model class and uses :func:`nni.report_final_result` to report the final score of this model.
+#
+# An example here creates a simple evaluator that runs on MNIST dataset, trains for 2 epochs, and reports its validation accuracy.
+
+import nni
+
+from torchvision import transforms
+from torchvision.datasets import MNIST
+from torch.utils.data import DataLoader
+
+
+def train_epoch(model, device, train_loader, optimizer, epoch):
+    loss_fn = torch.nn.CrossEntropyLoss()
+    model.train()
+    for batch_idx, (data, target) in enumerate(train_loader):
+        data, target = data.to(device), target.to(device)
+        optimizer.zero_grad()
+        output = model(data)
+        loss = loss_fn(output, target)
+        loss.backward()
+        optimizer.step()
+        if batch_idx % 10 == 0:
+            print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
+                epoch, batch_idx * len(data), len(train_loader.dataset),
+                100. * batch_idx / len(train_loader), loss.item()))
+
+
+def test_epoch(model, device, test_loader):
+    model.eval()
+    test_loss = 0
+    correct = 0
+    with torch.no_grad():
+        for data, target in test_loader:
+            data, target = data.to(device), target.to(device)
+            output = model(data)
+            pred = output.argmax(dim=1, keepdim=True)
+            correct += pred.eq(target.view_as(pred)).sum().item()
+
+    test_loss /= len(test_loader.dataset)
+    accuracy = 100. * correct / len(test_loader.dataset)
+
+    print('\nTest set: Accuracy: {}/{} ({:.0f}%)\n'.format(
+          correct, len(test_loader.dataset), accuracy))
+
+    return accuracy
+
+
+def evaluate_model(model_cls):
+    # "model_cls" is a class, need to instantiate
+    model = model_cls()
+
+    device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')
+    model.to(device)
+
+    optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
+    transf = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.1307,), (0.3081,))])
+    train_loader = DataLoader(MNIST('data/mnist', download=True, transform=transf), batch_size=64, shuffle=True)
+    test_loader = DataLoader(MNIST('data/mnist', download=True, train=False, transform=transf), batch_size=64)
+
+    for epoch in range(3):
+        # train the model for one epoch
+        train_epoch(model, device, train_loader, optimizer, epoch)
+        # test the model for one epoch
+        accuracy = test_epoch(model, device, test_loader)
+        # call report intermediate result. Result can be float or dict
+        nni.report_intermediate_result(accuracy)
+
+    # report final test result
+    nni.report_final_result(accuracy)
+
+
+# %%
+# Create the evaluator
+
+from nni.retiarii.evaluator import FunctionalEvaluator
+evaluator = FunctionalEvaluator(evaluate_model)
+
+# %%
+#
+# The ``train_epoch`` and ``test_epoch`` here can be any customized function,
+# where users can write their own training recipe.
+#
+# It is recommended that the ``evaluate_model`` here accepts no additional arguments other than ``model_cls``.
+# However, in the :doc:`advanced tutorial </nas/evaluator>`, we will show how to use additional arguments in case you actually need those.
+# In future, we will support mutation on the arguments of evaluators, which is commonly called "Hyper-parmeter tuning".
+#
+# Launch an Experiment
+# --------------------
+#
+# After all the above are prepared, it is time to start an experiment to do the model search. An example is shown below.
+
+from nni.retiarii.experiment.pytorch import RetiariiExperiment, RetiariiExeConfig
+exp = RetiariiExperiment(model_space, evaluator, [], search_strategy)
+exp_config = RetiariiExeConfig('local')
+exp_config.experiment_name = 'mnist_search'
+
+# %%
+# The following configurations are useful to control how many trials to run at most / at the same time.
+
+exp_config.max_trial_number = 4   # spawn 4 trials at most
+exp_config.trial_concurrency = 2  # will run two trials concurrently
+
+# %%
+# Remember to set the following config if you want to GPU.
+# ``use_active_gpu`` should be set true if you wish to use an occupied GPU (possibly running a GUI).
+
+exp_config.trial_gpu_number = 1
+exp_config.training_service.use_active_gpu = True
+
+# %%
+# Launch the experiment. The experiment should take several minutes to finish on a workstation with 2 GPUs.
+
+exp.run(exp_config, 8081)
+
+# %%
+# Users can also run Retiarii Experiment with :doc:`different training services </experiment/training_service/overview>`
+# besides ``local`` training service.
+#
+# Visualize the Experiment
+# ------------------------
+#
+# Users can visualize their experiment in the same way as visualizing a normal hyper-parameter tuning experiment.
+# For example, open ``localhost:8081`` in your browser, 8081 is the port that you set in ``exp.run``.
+# Please refer to :doc:`here </experiment/web_portal/web_portal>` for details.
+#
+# We support visualizing models with 3rd-party visualization engines (like `Netron <https://netron.app/>`__).
+# This can be used by clicking ``Visualization`` in detail panel for each trial.
+# Note that current visualization is based on `onnx <https://onnx.ai/>`__ ,
+# thus visualization is not feasible if the model cannot be exported into onnx.
+#
+# Built-in evaluators (e.g., Classification) will automatically export the model into a file.
+# For your own evaluator, you need to save your file into ``$NNI_OUTPUT_DIR/model.onnx`` to make this work.
+# For instance,
+
+import os
+from pathlib import Path
+
+
+def evaluate_model_with_visualization(model_cls):
+    model = model_cls()
+    # dump the model into an onnx
+    if 'NNI_OUTPUT_DIR' in os.environ:
+        dummy_input = torch.zeros(1, 3, 32, 32)
+        torch.onnx.export(model, (dummy_input, ),
+                          Path(os.environ['NNI_OUTPUT_DIR']) / 'model.onnx')
+    evaluate_model(model_cls)
+
+# %%
+# Relaunch the experiment, and a button is shown on Web portal.
+#
+# .. image:: ../../img/netron_entrance_webui.png
+#
+# Export Top Models
+# -----------------
+#
+# Users can export top models after the exploration is done using ``export_top_models``.
+
+for model_dict in exp.export_top_models(formatter='dict'):
+    print(model_dict)
+
+# %%
+# The output is ``json`` object which records the mutation actions of the top model.
+# If users want to output source code of the top model,
+# they can use :ref:`graph-based execution engine <graph-based-execution-engine>` for the experiment,
+# by simply adding the following two lines.
+
+exp_config.execution_engine = 'base'
+export_formatter = 'code'