create branch for v2.9

docs/source/locales/zh/LC_MESSAGES/hpo.po

+# SOME DESCRIPTIVE TITLE.
+# Copyright (C) 2022, Microsoft
+# This file is distributed under the same license as the NNI package.
+# FIRST AUTHOR <EMAIL@ADDRESS>, 2022.
+#
+#, fuzzy
+msgid ""
+msgstr ""
+"Project-Id-Version: NNI \n"
+"Report-Msgid-Bugs-To: \n"
+"POT-Creation-Date: 2022-04-20 05:50+0000\n"
+"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
+"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
+"Language-Team: LANGUAGE <LL@li.org>\n"
+"MIME-Version: 1.0\n"
+"Content-Type: text/plain; charset=utf-8\n"
+"Content-Transfer-Encoding: 8bit\n"
+"Generated-By: Babel 2.9.1\n"
+
+#: ../../source/hpo/overview.rst:2
+msgid "Hyperparameter Optimization Overview"
+msgstr ""
+
+#: ../../source/hpo/overview.rst:4
+msgid ""
+"Auto hyperparameter optimization (HPO), or auto tuning, is one of the key"
+" features of NNI."
+msgstr ""
+
+#: ../../source/hpo/overview.rst:7
+msgid "Introduction to HPO"
+msgstr ""
+
+#: ../../source/hpo/overview.rst:9
+msgid ""
+"In machine learning, a hyperparameter is a parameter whose value is used "
+"to control learning process, and HPO is the problem of choosing a set of "
+"optimal hyperparameters for a learning algorithm. (`From "
+"<https://en.wikipedia.org/wiki/Hyperparameter_(machine_learning)>`__ "
+"`Wikipedia "
+"<https://en.wikipedia.org/wiki/Hyperparameter_optimization>`__)"
+msgstr ""
+
+#: ../../source/hpo/overview.rst:14
+msgid "Following code snippet demonstrates a naive HPO process:"
+msgstr ""
+
+#: ../../source/hpo/overview.rst:34
+msgid ""
+"You may have noticed, the example will train 4×10×3=120 models in total. "
+"Since it consumes so much computing resources, you may want to:"
+msgstr ""
+
+#: ../../source/hpo/overview.rst:37
+msgid ""
+":ref:`Find the best hyperparameter set with less iterations. <hpo-"
+"overview-tuners>`"
+msgstr ""
+
+#: ../../source/hpo/overview.rst:38
+msgid ":ref:`Train the models on distributed platforms. <hpo-overview-platforms>`"
+msgstr ""
+
+#: ../../source/hpo/overview.rst:39
+msgid ""
+":ref:`Have a portal to monitor and control the process. <hpo-overview-"
+"portal>`"
+msgstr ""
+
+#: ../../source/hpo/overview.rst:41
+msgid "NNI will do them for you."
+msgstr ""
+
+#: ../../source/hpo/overview.rst:44
+msgid "Key Features of NNI HPO"
+msgstr ""
+
+#: ../../source/hpo/overview.rst:49
+msgid "Tuning Algorithms"
+msgstr ""
+
+#: ../../source/hpo/overview.rst:51
+msgid ""
+"NNI provides *tuners* to speed up the process of finding best "
+"hyperparameter set."
+msgstr ""
+
+#: ../../source/hpo/overview.rst:53
+msgid ""
+"A tuner, or a tuning algorithm, decides the order in which hyperparameter"
+" sets are evaluated. Based on the results of historical hyperparameter "
+"sets, an efficient tuner can predict where the best hyperparameters "
+"locates around, and finds them in much fewer attempts."
+msgstr ""
+
+#: ../../source/hpo/overview.rst:57
+msgid ""
+"The naive example above evaluates all possible hyperparameter sets in "
+"constant order, ignoring the historical results. This is the brute-force "
+"tuning algorithm called *grid search*."
+msgstr ""
+
+#: ../../source/hpo/overview.rst:60
+msgid ""
+"NNI has out-of-the-box support for a variety of popular tuners. It "
+"includes naive algorithms like random search and grid search, Bayesian-"
+"based algorithms like TPE and SMAC, RL based algorithms like PPO, and "
+"much more."
+msgstr ""
+
+#: ../../source/hpo/overview.rst:64
+msgid "Main article: :doc:`tuners`"
+msgstr ""
+
+#: ../../source/hpo/overview.rst:69
+msgid "Training Platforms"
+msgstr ""
+
+#: ../../source/hpo/overview.rst:71
+msgid ""
+"If you are not interested in distributed platforms, you can simply run "
+"NNI HPO with current computer, just like any ordinary Python library."
+msgstr ""
+
+#: ../../source/hpo/overview.rst:74
+msgid ""
+"And when you want to leverage more computing resources, NNI provides "
+"built-in integration for training platforms from simple on-premise "
+"servers to scalable commercial clouds."
+msgstr ""
+
+#: ../../source/hpo/overview.rst:77
+msgid ""
+"With NNI you can write one piece of model code, and concurrently evaluate"
+" hyperparameter sets on local machine, SSH servers, Kubernetes-based "
+"clusters, AzureML service, and much more."
+msgstr ""
+
+#: ../../source/hpo/overview.rst:80
+msgid "Main article: :doc:`/experiment/training_service/overview`"
+msgstr ""
+
+#: ../../source/hpo/overview.rst:85
+msgid "Web Portal"
+msgstr ""
+
+#: ../../source/hpo/overview.rst:87
+msgid ""
+"NNI provides a web portal to monitor training progress, to visualize "
+"hyperparameter performance, to manually customize hyperparameters, and to"
+" manage multiple HPO experiments."
+msgstr ""
+
+#: ../../source/hpo/overview.rst:90
+msgid "Main article: :doc:`/experiment/web_portal/web_portal`"
+msgstr ""
+
+#: ../../source/hpo/overview.rst:96
+msgid "Tutorials"
+msgstr ""
+
+#: ../../source/hpo/overview.rst:98
+msgid ""
+"To start using NNI HPO, choose the quickstart tutorial of your favorite "
+"framework:"
+msgstr ""
+
+#: ../../source/hpo/overview.rst:100
+msgid ":doc:`PyTorch tutorial </tutorials/hpo_quickstart_pytorch/main>`"
+msgstr ""
+
+#: ../../source/hpo/overview.rst:101
+msgid ":doc:`TensorFlow tutorial </tutorials/hpo_quickstart_tensorflow/main>`"
+msgstr ""
+
+#: ../../source/hpo/overview.rst:104
+msgid "Extra Features"
+msgstr ""
+
+#: ../../source/hpo/overview.rst:106
+msgid ""
+"After you are familiar with basic usage, you can explore more HPO "
+"features:"
+msgstr ""
+
+#: ../../source/hpo/overview.rst:108
+msgid ""
+":doc:`Use command line tool to create and manage experiments (nnictl) "
+"</reference/nnictl>`"
+msgstr ""
+
+#: ../../source/hpo/overview.rst:110
+msgid ":doc:`nnictl example </tutorials/hpo_nnictl/nnictl>`"
+msgstr ""
+
+#: ../../source/hpo/overview.rst:112
+msgid ":doc:`Early stop non-optimal models (assessor) <assessors>`"
+msgstr ""
+
+#: ../../source/hpo/overview.rst:113
+msgid ":doc:`TensorBoard integration </experiment/web_portal/tensorboard>`"
+msgstr ""
+
+#: ../../source/hpo/overview.rst:114
+msgid ":doc:`Implement your own algorithm <custom_algorithm>`"
+msgstr ""
+
+#: ../../source/hpo/overview.rst:115
+msgid ":doc:`Benchmark tuners <hpo_benchmark>`"
+msgstr ""
+

docs/source/locales/zh/LC_MESSAGES/index.po

+# SOME DESCRIPTIVE TITLE.
+# Copyright (C) 2022, Microsoft
+# This file is distributed under the same license as the NNI package.
+# FIRST AUTHOR <EMAIL@ADDRESS>, 2022.
+#
+#, fuzzy
+msgid ""
+msgstr ""
+"Project-Id-Version: NNI \n"
+"Report-Msgid-Bugs-To: \n"
+"POT-Creation-Date: 2022-04-20 05:50+0000\n"
+"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
+"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
+"Language-Team: LANGUAGE <LL@li.org>\n"
+"MIME-Version: 1.0\n"
+"Content-Type: text/plain; charset=utf-8\n"
+"Content-Transfer-Encoding: 8bit\n"
+"Generated-By: Babel 2.9.1\n"
+
+#: ../../source/index.rst:4 ../../source/index.rst:52
+msgid "Get Started"
+msgstr ""
+
+#: ../../source/index.rst:12
+msgid "Model Compression"
+msgstr ""
+
+#: ../../source/index.rst:12
+msgid "User Guide"
+msgstr ""
+
+#: ../../source/index.rst:23
+msgid "Python API"
+msgstr ""
+
+#: ../../source/index.rst:23
+msgid "References"
+msgstr ""
+
+#: ../../source/index.rst:32
+msgid "Misc"
+msgstr ""
+
+#: ../../source/index.rst:2
+msgid "NNI Documentation"
+msgstr ""
+
+#: ../../source/index.rst:44
+msgid ""
+"**NNI (Neural Network Intelligence)** is a lightweight but powerful "
+"toolkit to help users **automate**:"
+msgstr ""
+
+#: ../../source/index.rst:46
+msgid ":doc:`Hyperparameter Optimization </hpo/overview>`"
+msgstr ""
+
+#: ../../source/index.rst:47
+msgid ":doc:`Neural Architecture Search </nas/overview>`"
+msgstr ""
+
+#: ../../source/index.rst:48
+msgid ":doc:`Model Compression </compression/overview>`"
+msgstr ""
+
+#: ../../source/index.rst:49
+msgid ":doc:`Feature Engineering </feature_engineering/overview>`"
+msgstr ""
+
+#: ../../source/index.rst:54
+msgid "To install the current release:"
+msgstr ""
+
+#: ../../source/index.rst:60
+msgid ""
+"See the :doc:`installation guide </installation>` if you need additional "
+"help on installation."
+msgstr ""
+
+#: ../../source/index.rst:63
+msgid "Try your first NNI experiment"
+msgstr ""
+
+#: ../../source/index.rst:69
+msgid ""
+"You need to have `PyTorch <https://pytorch.org/>`_ (as well as "
+"`torchvision <https://pytorch.org/vision/stable/index.html>`_) installed "
+"to run this experiment."
+msgstr ""
+
+#: ../../source/index.rst:71
+msgid ""
+"To start your journey now, please follow the :doc:`absolute quickstart of"
+" NNI <quickstart>`!"
+msgstr ""
+
+#: ../../source/index.rst:74
+msgid "Why choose NNI?"
+msgstr ""
+
+#: ../../source/index.rst:77
+msgid "NNI makes AutoML techniques plug-and-play"
+msgstr ""
+
+#: ../../source/index.rst:221
+msgid "NNI eases the effort to scale and manage AutoML experiments"
+msgstr ""
+
+#: ../../source/index.rst:229
+msgid ""
+"An AutoML experiment requires many trials to explore feasible and "
+"potentially good-performing models. **Training service** aims to make the"
+" tuning process easily scalable in a distributed platforms. It provides a"
+" unified user experience for diverse computation resources (e.g., local "
+"machine, remote servers, AKS). Currently, NNI supports **more than 9** "
+"kinds of training services."
+msgstr ""
+
+#: ../../source/index.rst:240
+msgid ""
+"Web portal visualizes the tuning process, exposing the ability to "
+"inspect, monitor and control the experiment."
+msgstr ""
+
+#: ../../source/index.rst:251
+msgid ""
+"The DNN model tuning often requires more than one experiment. Users might"
+" try different tuning algorithms, fine-tune their search space, or switch"
+" to another training service. **Experiment management** provides the "
+"power to aggregate and compare tuning results from multiple experiments, "
+"so that the tuning workflow becomes clean and organized."
+msgstr ""
+
+#: ../../source/index.rst:257
+msgid "Get Support and Contribute Back"
+msgstr ""
+
+#: ../../source/index.rst:259
+msgid ""
+"NNI is maintained on the `NNI GitHub repository "
+"<https://github.com/microsoft/nni>`_. We collect feedbacks and new "
+"proposals/ideas on GitHub. You can:"
+msgstr ""
+
+#: ../../source/index.rst:261
+msgid ""
+"Open a `GitHub issue <https://github.com/microsoft/nni/issues>`_ for bugs"
+" and feature requests."
+msgstr ""
+
+#: ../../source/index.rst:262
+msgid ""
+"Open a `pull request <https://github.com/microsoft/nni/pulls>`_ to "
+"contribute code (make sure to read the :doc:`contribution guide "
+"<notes/contributing>` before doing this)."
+msgstr ""
+
+#: ../../source/index.rst:263
+msgid ""
+"Participate in `NNI Discussion "
+"<https://github.com/microsoft/nni/discussions>`_ for general questions "
+"and new ideas."
+msgstr ""
+
+#: ../../source/index.rst:264
+msgid "Join the following IM groups."
+msgstr ""
+
+#: ../../source/index.rst:270
+msgid "Gitter"
+msgstr ""
+
+#: ../../source/index.rst:271
+msgid "WeChat"
+msgstr ""
+
+#: ../../source/index.rst:278
+msgid "Citing NNI"
+msgstr ""
+
+#: ../../source/index.rst:280
+msgid ""
+"If you use NNI in a scientific publication, please consider citing NNI in"
+" your references."
+msgstr ""
+
+#: ../../source/index.rst:282
+msgid ""
+"Microsoft. Neural Network Intelligence (version |release|). "
+"https://github.com/microsoft/nni"
+msgstr ""
+
+#: ../../source/index.rst:284
+msgid ""
+"Bibtex entry (please replace the version with the particular version you "
+"are using): ::"
+msgstr ""
+
+#~ msgid "Hyperparameter Optimization"
+#~ msgstr ""
+
+#~ msgid "To run your first NNI experiment:"
+#~ msgstr ""
+
+#~ msgid ""
+#~ "you need to have `PyTorch "
+#~ "<https://pytorch.org/>`_ (as well as "
+#~ "`torchvision <https://pytorch.org/vision/stable/index.html>`_)"
+#~ " installed to run this experiment."
+#~ msgstr ""
+
+#~ msgid ""
+#~ "Open a `pull request "
+#~ "<https://github.com/microsoft/nni/pulls>`_ to contribute"
+#~ " code (make sure to read the "
+#~ "`contribution guide </contribution>` before "
+#~ "doing this)."
+#~ msgstr ""
+

docs/source/locales/zh/LC_MESSAGES/installation.po

+# SOME DESCRIPTIVE TITLE.
+# Copyright (C) 2022, Microsoft
+# This file is distributed under the same license as the NNI package.
+# FIRST AUTHOR <EMAIL@ADDRESS>, 2022.
+#
+#, fuzzy
+msgid ""
+msgstr ""
+"Project-Id-Version: NNI \n"
+"Report-Msgid-Bugs-To: \n"
+"POT-Creation-Date: 2022-04-13 03:14+0000\n"
+"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
+"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
+"Language-Team: LANGUAGE <LL@li.org>\n"
+"MIME-Version: 1.0\n"
+"Content-Type: text/plain; charset=utf-8\n"
+"Content-Transfer-Encoding: 8bit\n"
+"Generated-By: Babel 2.9.1\n"
+
+#: ../../source/installation.rst:2
+msgid "Install NNI"
+msgstr ""
+
+#: ../../source/installation.rst:4
+msgid ""
+"NNI requires Python >= 3.7. It is tested and supported on Ubuntu >= "
+"18.04, Windows 10 >= 21H2, and macOS >= 11."
+msgstr ""
+
+#: ../../source/installation.rst:8
+msgid "There are 3 ways to install NNI:"
+msgstr ""
+
+#: ../../source/installation.rst:10
+msgid ":ref:`Using pip <installation-pip>`"
+msgstr ""
+
+#: ../../source/installation.rst:11
+msgid ":ref:`Build source code <installation-source>`"
+msgstr ""
+
+#: ../../source/installation.rst:12
+msgid ":ref:`Using Docker <installation-docker>`"
+msgstr ""
+
+#: ../../source/installation.rst:17
+msgid "Using pip"
+msgstr ""
+
+#: ../../source/installation.rst:19
+msgid ""
+"NNI provides official packages for x86-64 CPUs. They can be installed "
+"with pip:"
+msgstr ""
+
+#: ../../source/installation.rst:25
+msgid "Or to upgrade to latest version:"
+msgstr ""
+
+#: ../../source/installation.rst:31
+msgid "You can check installation with:"
+msgstr ""
+
+#: ../../source/installation.rst:37
+msgid ""
+"On Linux systems without Conda, you may encounter ``bash: nnictl: command"
+" not found`` error. In this case you need to add pip script directory to "
+"``PATH``:"
+msgstr ""
+
+#: ../../source/installation.rst:48
+msgid "Installing from Source Code"
+msgstr ""
+
+#: ../../source/installation.rst:50
+msgid "NNI hosts source code on `GitHub <https://github.com/microsoft/nni>`__."
+msgstr ""
+
+#: ../../source/installation.rst:52
+msgid ""
+"NNI has experimental support for ARM64 CPUs, including Apple M1. It "
+"requires to install from source code."
+msgstr ""
+
+#: ../../source/installation.rst:55
+msgid "See :doc:`/notes/build_from_source`."
+msgstr ""
+
+#: ../../source/installation.rst:60
+msgid "Using Docker"
+msgstr ""
+
+#: ../../source/installation.rst:62
+msgid ""
+"NNI provides official Docker image on `Docker Hub "
+"<https://hub.docker.com/r/msranni/nni>`__."
+msgstr ""
+
+#: ../../source/installation.rst:69
+msgid "Installing Extra Dependencies"
+msgstr ""
+
+#: ../../source/installation.rst:71
+msgid ""
+"Some built-in algorithms of NNI requires extra packages. Use ``nni"
+"[<algorithm-name>]`` to install their dependencies."
+msgstr ""
+
+#: ../../source/installation.rst:74
+msgid ""
+"For example, to install dependencies of :class:`DNGO "
+"tuner<nni.algorithms.hpo.dngo_tuner.DNGOTuner>` :"
+msgstr ""
+
+#: ../../source/installation.rst:80
+msgid ""
+"This command will not reinstall NNI itself, even if it was installed in "
+"development mode."
+msgstr ""
+
+#: ../../source/installation.rst:82
+msgid "Alternatively, you may install all extra dependencies at once:"
+msgstr ""
+
+#: ../../source/installation.rst:88
+msgid ""
+"**NOTE**: SMAC tuner depends on swig3, which requires a manual downgrade "
+"on Ubuntu:"
+msgstr ""
+

docs/source/locales/zh/LC_MESSAGES/nas.po

+# SOME DESCRIPTIVE TITLE.
+# Copyright (C) 2022, Microsoft
+# This file is distributed under the same license as the NNI package.
+# FIRST AUTHOR <EMAIL@ADDRESS>, 2022.
+#
+#, fuzzy
+msgid ""
+msgstr ""
+"Project-Id-Version: NNI \n"
+"Report-Msgid-Bugs-To: \n"
+"POT-Creation-Date: 2022-04-20 05:50+0000\n"
+"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
+"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
+"Language-Team: LANGUAGE <LL@li.org>\n"
+"MIME-Version: 1.0\n"
+"Content-Type: text/plain; charset=utf-8\n"
+"Content-Transfer-Encoding: 8bit\n"
+"Generated-By: Babel 2.9.1\n"
+
+#: ../../source/nas/overview.rst:2
+msgid "Overview"
+msgstr ""
+
+#: ../../source/nas/overview.rst:4
+msgid ""
+"NNI's latest NAS supports are all based on Retiarii Framework, users who "
+"are still on `early version using NNI NAS v1.0 "
+"<https://nni.readthedocs.io/en/v2.2/nas.html>`__ shall migrate your work "
+"to Retiarii as soon as possible. We plan to remove the legacy NAS "
+"framework in the next few releases."
+msgstr ""
+
+#: ../../source/nas/overview.rst:6
+msgid ""
+"PyTorch is the **only supported framework on Retiarii**. Inquiries of NAS"
+" support on Tensorflow is in `this discussion "
+"<https://github.com/microsoft/nni/discussions/4605>`__. If you intend to "
+"run NAS with DL frameworks other than PyTorch and Tensorflow, please "
+"`open new issues <https://github.com/microsoft/nni/issues>`__ to let us "
+"know."
+msgstr ""
+
+#: ../../source/nas/overview.rst:9
+msgid "Basics"
+msgstr ""
+
+#: ../../source/nas/overview.rst:11
+msgid ""
+"Automatic neural architecture search is playing an increasingly important"
+" role in finding better models. Recent research has proven the "
+"feasibility of automatic NAS and has led to models that beat many "
+"manually designed and tuned models. Representative works include `NASNet "
+"<https://arxiv.org/abs/1707.07012>`__, `ENAS "
+"<https://arxiv.org/abs/1802.03268>`__, `DARTS "
+"<https://arxiv.org/abs/1806.09055>`__, `Network Morphism "
+"<https://arxiv.org/abs/1806.10282>`__, and `Evolution "
+"<https://arxiv.org/abs/1703.01041>`__. In addition, new innovations "
+"continue to emerge."
+msgstr ""
+
+#: ../../source/nas/overview.rst:13
+msgid ""
+"High-level speaking, aiming to solve any particular task with neural "
+"architecture search typically requires: search space design, search "
+"strategy selection, and performance evaluation. The three components work"
+" together with the following loop (from the famous `NAS survey "
+"<https://arxiv.org/abs/1808.05377>`__):"
+msgstr ""
+
+#: ../../source/nas/overview.rst:19
+msgid "In this figure:"
+msgstr ""
+
+#: ../../source/nas/overview.rst:21
+msgid ""
+"*Model search space*  means a set of models from which the best model is "
+"explored/searched. Sometimes we use *search space* or *model space* in "
+"short."
+msgstr ""
+
+#: ../../source/nas/overview.rst:22
+msgid ""
+"*Exploration strategy* is the algorithm that is used to explore a model "
+"search space. Sometimes we also call it *search strategy*."
+msgstr ""
+
+#: ../../source/nas/overview.rst:23
+msgid ""
+"*Model evaluator* is responsible for training a model and evaluating its "
+"performance."
+msgstr ""
+
+#: ../../source/nas/overview.rst:25
+msgid ""
+"The process is similar to :doc:`Hyperparameter Optimization "
+"</hpo/overview>`, except that the target is the best architecture rather "
+"than hyperparameter. Concretely, an exploration strategy selects an "
+"architecture from a predefined search space. The architecture is passed "
+"to a performance evaluation to get a score, which represents how well "
+"this architecture performs on a particular task. This process is repeated"
+" until the search process is able to find the best architecture."
+msgstr ""
+
+#: ../../source/nas/overview.rst:28
+msgid "Key Features"
+msgstr ""
+
+#: ../../source/nas/overview.rst:30
+msgid ""
+"The current NAS framework in NNI is powered by the research of `Retiarii:"
+" A Deep Learning Exploratory-Training Framework "
+"<https://www.usenix.org/system/files/osdi20-zhang_quanlu.pdf>`__, where "
+"we highlight the following features:"
+msgstr ""
+
+#: ../../source/nas/overview.rst:32
+msgid ":doc:`Simple APIs to construct search space easily <construct_space>`"
+msgstr ""
+
+#: ../../source/nas/overview.rst:33
+msgid ":doc:`SOTA NAS algorithms to explore search space <exploration_strategy>`"
+msgstr ""
+
+#: ../../source/nas/overview.rst:34
+msgid ""
+":doc:`Experiment backend support to scale up experiments on large-scale "
+"AI platforms </experiment/overview>`"
+msgstr ""
+
+#: ../../source/nas/overview.rst:37
+msgid "Why NAS with NNI"
+msgstr ""
+
+#: ../../source/nas/overview.rst:39
+msgid ""
+"We list out the three perspectives where NAS can be particularly "
+"challegning without NNI. NNI provides solutions to relieve users' "
+"engineering effort when they want to try NAS techniques in their own "
+"scenario."
+msgstr ""
+
+#: ../../source/nas/overview.rst:42
+msgid "Search Space Design"
+msgstr ""
+
+#: ../../source/nas/overview.rst:44
+msgid ""
+"The search space defines which architectures can be represented in "
+"principle. Incorporating prior knowledge about typical properties of "
+"architectures well-suited for a task can reduce the size of the search "
+"space and simplify the search. However, this also introduces a human "
+"bias, which may prevent finding novel architectural building blocks that "
+"go beyond the current human knowledge. Search space design can be very "
+"challenging for beginners, who might not possess the experience to "
+"balance the richness and simplicity."
+msgstr ""
+
+#: ../../source/nas/overview.rst:46
+msgid ""
+"In NNI, we provide a wide range of APIs to build the search space. There "
+"are :doc:`high-level APIs <construct_space>`, that enables the "
+"possibility to incorporate human knowledge about what makes a good "
+"architecture or search space. There are also :doc:`low-level APIs "
+"<mutator>`, that is a list of primitives to construct a network from "
+"operation to operation."
+msgstr ""
+
+#: ../../source/nas/overview.rst:49
+msgid "Exploration strategy"
+msgstr ""
+
+#: ../../source/nas/overview.rst:51
+msgid ""
+"The exploration strategy details how to explore the search space (which "
+"is often exponentially large). It encompasses the classical exploration-"
+"exploitation trade-off since, on the one hand, it is desirable to find "
+"well-performing architectures quickly, while on the other hand, premature"
+" convergence to a region of suboptimal architectures should be avoided. "
+"The \"best\" exploration strategy for a particular scenario is usually "
+"found via trial-and-error. As many state-of-the-art strategies are "
+"implemented with their own code-base, it becomes very troublesome to "
+"switch from one to another."
+msgstr ""
+
+#: ../../source/nas/overview.rst:53
+msgid ""
+"In NNI, we have also provided :doc:`a list of strategies "
+"<exploration_strategy>`. Some of them are powerful yet time consuming, "
+"while others might be suboptimal but really efficient. Given that all "
+"strategies are implemented with a unified interface, users can always "
+"find one that matches their need."
+msgstr ""
+
+#: ../../source/nas/overview.rst:56
+msgid "Performance estimation"
+msgstr ""
+
+#: ../../source/nas/overview.rst:58
+msgid ""
+"The objective of NAS is typically to find architectures that achieve high"
+" predictive performance on unseen data. Performance estimation refers to "
+"the process of estimating this performance. The problem with performance "
+"estimation is mostly its scalability, i.e., how can I run and manage "
+"multiple trials simultaneously."
+msgstr ""
+
+#: ../../source/nas/overview.rst:60
+msgid ""
+"In NNI, we standardize this process is implemented with :doc:`evaluator "
+"<evaluator>`, which is responsible of estimating a model's performance. "
+"NNI has quite a few built-in supports of evaluators, ranging from the "
+"simplest option, e.g., to perform a standard training and validation of "
+"the architecture on data, to complex configurations and implementations. "
+"Evaluators are run in *trials*, where trials can be spawn onto "
+"distributed platforms with our powerful :doc:`training service "
+"</experiment/training_service/overview>`."
+msgstr ""
+
+#: ../../source/nas/overview.rst:63
+msgid "Tutorials"
+msgstr ""
+
+#: ../../source/nas/overview.rst:65
+msgid ""
+"To start using NNI NAS framework, we recommend at least going through the"
+" following tutorials:"
+msgstr ""
+
+#: ../../source/nas/overview.rst:67
+msgid ":doc:`Quickstart </tutorials/hello_nas>`"
+msgstr ""
+
+#: ../../source/nas/overview.rst:68
+msgid ":doc:`construct_space`"
+msgstr ""
+
+#: ../../source/nas/overview.rst:69
+msgid ":doc:`exploration_strategy`"
+msgstr ""
+
+#: ../../source/nas/overview.rst:70
+msgid ":doc:`evaluator`"
+msgstr ""
+
+#: ../../source/nas/overview.rst:73
+msgid "Resources"
+msgstr ""
+
+#: ../../source/nas/overview.rst:75
+msgid ""
+"The following articles will help with a better understanding of the "
+"current arts of NAS:"
+msgstr ""
+
+#: ../../source/nas/overview.rst:77
+msgid ""
+"`Neural Architecture Search: A Survey "
+"<https://arxiv.org/abs/1808.05377>`__"
+msgstr ""
+
+#: ../../source/nas/overview.rst:78
+msgid ""
+"`A Comprehensive Survey of Neural Architecture Search: Challenges and "
+"Solutions <https://arxiv.org/abs/2006.02903>`__"
+msgstr ""
+
+#~ msgid "Basics"
+#~ msgstr ""
+
+#~ msgid "Basic Concepts"
+#~ msgstr ""
+
+#~ msgid ""
+#~ "The process is similar to "
+#~ ":doc:`Hyperparameter Optimization </hpo/index>`, "
+#~ "except that the target is the best"
+#~ " architecture rather than hyperparameter. "
+#~ "Concretely, an exploration strategy selects"
+#~ " an architecture from a predefined "
+#~ "search space. The architecture is passed"
+#~ " to a performance evaluation to get"
+#~ " a score, which represents how well"
+#~ " this architecture performs on a "
+#~ "particular task. This process is "
+#~ "repeated until the search process is "
+#~ "able to find the best architecture."
+#~ msgstr ""
+
+#~ msgid ""
+#~ "In NNI, we provide a wide range"
+#~ " of APIs to build the search "
+#~ "space. There are :doc:`high-level APIs"
+#~ " <construct_space>`, that enables incorporating"
+#~ " human knowledge about what makes a"
+#~ " good architecture or search space. "
+#~ "There are also :doc:`low-level APIs "
+#~ "<mutator>`, that is a list of "
+#~ "primitives to construct a network from"
+#~ " operator to operator."
+#~ msgstr ""
+
+#~ msgid ""
+#~ "In NNI, we standardize this process "
+#~ "is implemented with :doc:`evaluator "
+#~ "<evaluator>`, which is responsible of "
+#~ "estimating a model's performance. The "
+#~ "choices of evaluators also range from"
+#~ " the simplest option, e.g., to "
+#~ "perform a standard training and "
+#~ "validation of the architecture on data,"
+#~ " to complex configurations and "
+#~ "implementations. Evaluators are run in "
+#~ "*trials*, where trials can be spawn "
+#~ "onto distributed platforms with our "
+#~ "powerful :doc:`training service "
+#~ "</experiment/training_service/overview>`."
+#~ msgstr ""
+

docs/source/locales/zh/LC_MESSAGES/quickstart.po

+# SOME DESCRIPTIVE TITLE.
+# Copyright (C) 2022, Microsoft
+# This file is distributed under the same license as the NNI package.
+# FIRST AUTHOR <EMAIL@ADDRESS>, 2022.
+#
+#, fuzzy
+msgid ""
+msgstr ""
+"Project-Id-Version: NNI \n"
+"Report-Msgid-Bugs-To: \n"
+"POT-Creation-Date: 2022-04-13 03:14+0000\n"
+"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
+"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
+"Language-Team: LANGUAGE <LL@li.org>\n"
+"MIME-Version: 1.0\n"
+"Content-Type: text/plain; charset=utf-8\n"
+"Content-Transfer-Encoding: 8bit\n"
+"Generated-By: Babel 2.9.1\n"
+
+#: ../../source/quickstart.rst:2
+msgid "Quickstart"
+msgstr ""
+

docs/source/locales/zh/LC_MESSAGES/sphinx.po

+# SOME DESCRIPTIVE TITLE.
+# Copyright (C) 2022, Microsoft
+# This file is distributed under the same license as the NNI package.
+# FIRST AUTHOR <EMAIL@ADDRESS>, 2022.
+#
+#, fuzzy
+msgid ""
+msgstr ""
+"Project-Id-Version: NNI \n"
+"Report-Msgid-Bugs-To: \n"
+"POT-Creation-Date: 2022-04-12 17:35+0000\n"
+"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
+"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
+"Language-Team: LANGUAGE <LL@li.org>\n"
+"MIME-Version: 1.0\n"
+"Content-Type: text/plain; charset=utf-8\n"
+"Content-Transfer-Encoding: 8bit\n"
+"Generated-By: Babel 2.9.1\n"
+
+#: ../../templates/globaltoc.html:6
+msgid "Overview"
+msgstr ""
+

docs/source/locales/zh/LC_MESSAGES/tutorials.po

+# SOME DESCRIPTIVE TITLE.
+# Copyright (C) 2022, Microsoft
+# This file is distributed under the same license as the NNI package.
+# FIRST AUTHOR <EMAIL@ADDRESS>, 2022.
+#
+#, fuzzy
+msgid ""
+msgstr ""
+"Project-Id-Version: NNI \n"
+"Report-Msgid-Bugs-To: \n"
+"POT-Creation-Date: 2022-05-27 16:52+0800\n"
+"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
+"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
+"Language-Team: LANGUAGE <LL@li.org>\n"
+"MIME-Version: 1.0\n"
+"Content-Type: text/plain; charset=utf-8\n"
+"Content-Transfer-Encoding: 8bit\n"
+"Generated-By: Babel 2.9.1\n"
+
+#: ../../source/tutorials/hello_nas.rst:13
+msgid ""
+"Click :ref:`here <sphx_glr_download_tutorials_hello_nas.py>` to download "
+"the full example code"
+msgstr ""
+
+#: ../../source/tutorials/hello_nas.rst:22
+msgid "Hello, NAS!"
+msgstr ""
+
+#: ../../source/tutorials/hello_nas.rst:24
+msgid ""
+"This is the 101 tutorial of Neural Architecture Search (NAS) on NNI. In "
+"this tutorial, we will search for a neural architecture on MNIST dataset "
+"with the help of NAS framework of NNI, i.e., *Retiarii*. We use multi-"
+"trial NAS as an example to show how to construct and explore a model "
+"space."
+msgstr ""
+
+#: ../../source/tutorials/hello_nas.rst:28
+msgid ""
+"There are mainly three crucial components for a neural architecture "
+"search task, namely,"
+msgstr ""
+
+#: ../../source/tutorials/hello_nas.rst:30
+msgid "Model search space that defines a set of models to explore."
+msgstr ""
+
+#: ../../source/tutorials/hello_nas.rst:31
+msgid "A proper strategy as the method to explore this model space."
+msgstr ""
+
+#: ../../source/tutorials/hello_nas.rst:32
+msgid ""
+"A model evaluator that reports the performance of every model in the "
+"space."
+msgstr ""
+
+#: ../../source/tutorials/hello_nas.rst:34
+msgid ""
+"Currently, PyTorch is the only supported framework by Retiarii, and we "
+"have only tested **PyTorch 1.7 to 1.10**. This tutorial assumes PyTorch "
+"context but it should also apply to other frameworks, which is in our "
+"future plan."
+msgstr ""
+
+#: ../../source/tutorials/hello_nas.rst:38
+msgid "Define your Model Space"
+msgstr ""
+
+#: ../../source/tutorials/hello_nas.rst:40
+msgid ""
+"Model space is defined by users to express a set of models that users "
+"want to explore, which contains potentially good-performing models. In "
+"this framework, a model space is defined with two parts: a base model and"
+" possible mutations on the base model."
+msgstr ""
+
+#: ../../source/tutorials/hello_nas.rst:46
+msgid "Define Base Model"
+msgstr ""
+
+#: ../../source/tutorials/hello_nas.rst:48
+msgid ""
+"Defining a base model is almost the same as defining a PyTorch (or "
+"TensorFlow) model. Usually, you only need to replace the code ``import "
+"torch.nn as nn`` with ``import nni.retiarii.nn.pytorch as nn`` to use our"
+" wrapped PyTorch modules."
+msgstr ""
+
+#: ../../source/tutorials/hello_nas.rst:52
+msgid "Below is a very simple example of defining a base model."
+msgstr ""
+
+#: ../../source/tutorials/hello_nas.rst:93
+msgid ""
+"Always keep in mind that you should use ``import nni.retiarii.nn.pytorch "
+"as nn`` and :meth:`nni.retiarii.model_wrapper`. Many mistakes are a "
+"result of forgetting one of those. Also, please use ``torch.nn`` for "
+"submodules of ``nn.init``, e.g., ``torch.nn.init`` instead of "
+"``nn.init``."
+msgstr ""
+
+#: ../../source/tutorials/hello_nas.rst:98
+msgid "Define Model Mutations"
+msgstr ""
+
+#: ../../source/tutorials/hello_nas.rst:100
+msgid ""
+"A base model is only one concrete model not a model space. We provide "
+":doc:`API and Primitives </nas/construct_space>` for users to express how"
+" the base model can be mutated. That is, to build a model space which "
+"includes many models."
+msgstr ""
+
+#: ../../source/tutorials/hello_nas.rst:103
+msgid "Based on the above base model, we can define a model space as below."
+msgstr ""
+
+#: ../../source/tutorials/hello_nas.rst:134
+msgid "This results in the following code:"
+msgstr ""
+
+#: ../../source/tutorials/hello_nas.rst:189
+#: ../../source/tutorials/hello_nas.rst:259
+#: ../../source/tutorials/hello_nas.rst:471
+#: ../../source/tutorials/hello_nas.rst:564
+#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:244
+#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:281
+#: ../../source/tutorials/pruning_quick_start_mnist.rst:70
+#: ../../source/tutorials/pruning_quick_start_mnist.rst:112
+#: ../../source/tutorials/pruning_quick_start_mnist.rst:177
+#: ../../source/tutorials/pruning_quick_start_mnist.rst:223
+#: ../../source/tutorials/pruning_quick_start_mnist.rst:260
+#: ../../source/tutorials/pruning_quick_start_mnist.rst:288
+msgid "Out:"
+msgstr ""
+
+#: ../../source/tutorials/hello_nas.rst:210
+msgid ""
+"This example uses two mutation APIs, :class:`nn.LayerChoice "
+"<nni.retiarii.nn.pytorch.LayerChoice>` and :class:`nn.InputChoice "
+"<nni.retiarii.nn.pytorch.ValueChoice>`. :class:`nn.LayerChoice "
+"<nni.retiarii.nn.pytorch.LayerChoice>` takes a list of candidate modules "
+"(two in this example), one will be chosen for each sampled model. It can "
+"be used like normal PyTorch module. :class:`nn.InputChoice "
+"<nni.retiarii.nn.pytorch.ValueChoice>` takes a list of candidate values, "
+"one will be chosen to take effect for each sampled model."
+msgstr ""
+
+#: ../../source/tutorials/hello_nas.rst:219
+msgid ""
+"More detailed API description and usage can be found :doc:`here "
+"</nas/construct_space>`."
+msgstr ""
+
+#: ../../source/tutorials/hello_nas.rst:223
+msgid ""
+"We are actively enriching the mutation APIs, to facilitate easy "
+"construction of model space. If the currently supported mutation APIs "
+"cannot express your model space, please refer to :doc:`this doc "
+"</nas/mutator>` for customizing mutators."
+msgstr ""
+
+#: ../../source/tutorials/hello_nas.rst:228
+msgid "Explore the Defined Model Space"
+msgstr ""
+
+#: ../../source/tutorials/hello_nas.rst:230
+msgid ""
+"There are basically two exploration approaches: (1) search by evaluating "
+"each sampled model independently, which is the search approach in :ref"
+":`multi-trial NAS <multi-trial-nas>` and (2) one-shot weight-sharing "
+"based search, which is used in one-shot NAS. We demonstrate the first "
+"approach in this tutorial. Users can refer to :ref:`here <one-shot-nas>` "
+"for the second approach."
+msgstr ""
+
+#: ../../source/tutorials/hello_nas.rst:235
+msgid ""
+"First, users need to pick a proper exploration strategy to explore the "
+"defined model space. Second, users need to pick or customize a model "
+"evaluator to evaluate the performance of each explored model."
+msgstr ""
+
+#: ../../source/tutorials/hello_nas.rst:239
+msgid "Pick an exploration strategy"
+msgstr ""
+
+#: ../../source/tutorials/hello_nas.rst:241
+msgid ""
+"Retiarii supports many :doc:`exploration strategies "
+"</nas/exploration_strategy>`."
+msgstr ""
+
+#: ../../source/tutorials/hello_nas.rst:243
+msgid "Simply choosing (i.e., instantiate) an exploration strategy as below."
+msgstr ""
+
+#: ../../source/tutorials/hello_nas.rst:273
+msgid "Pick or customize a model evaluator"
+msgstr ""
+
+#: ../../source/tutorials/hello_nas.rst:275
+msgid ""
+"In the exploration process, the exploration strategy repeatedly generates"
+" new models. A model evaluator is for training and validating each "
+"generated model to obtain the model's performance. The performance is "
+"sent to the exploration strategy for the strategy to generate better "
+"models."
+msgstr ""
+
+#: ../../source/tutorials/hello_nas.rst:279
+msgid ""
+"Retiarii has provided :doc:`built-in model evaluators </nas/evaluator>`, "
+"but to start with, it is recommended to use :class:`FunctionalEvaluator "
+"<nni.retiarii.evaluator.FunctionalEvaluator>`, that is, to wrap your own "
+"training and evaluation code with one single function. This function "
+"should receive one single model class and uses "
+":func:`nni.report_final_result` to report the final score of this model."
+msgstr ""
+
+#: ../../source/tutorials/hello_nas.rst:284
+msgid ""
+"An example here creates a simple evaluator that runs on MNIST dataset, "
+"trains for 2 epochs, and reports its validation accuracy."
+msgstr ""
+
+#: ../../source/tutorials/hello_nas.rst:367
+msgid "Create the evaluator"
+msgstr ""
+
+#: ../../source/tutorials/hello_nas.rst:386
+msgid ""
+"The ``train_epoch`` and ``test_epoch`` here can be any customized "
+"function, where users can write their own training recipe."
+msgstr ""
+
+#: ../../source/tutorials/hello_nas.rst:389
+msgid ""
+"It is recommended that the ``evaluate_model`` here accepts no additional "
+"arguments other than ``model_cls``. However, in the :doc:`advanced "
+"tutorial </nas/evaluator>`, we will show how to use additional arguments "
+"in case you actually need those. In future, we will support mutation on "
+"the arguments of evaluators, which is commonly called \"Hyper-parmeter "
+"tuning\"."
+msgstr ""
+
+#: ../../source/tutorials/hello_nas.rst:394
+msgid "Launch an Experiment"
+msgstr ""
+
+#: ../../source/tutorials/hello_nas.rst:396
+msgid ""
+"After all the above are prepared, it is time to start an experiment to do"
+" the model search. An example is shown below."
+msgstr ""
+
+#: ../../source/tutorials/hello_nas.rst:417
+msgid ""
+"The following configurations are useful to control how many trials to run"
+" at most / at the same time."
+msgstr ""
+
+#: ../../source/tutorials/hello_nas.rst:436
+msgid ""
+"Remember to set the following config if you want to GPU. "
+"``use_active_gpu`` should be set true if you wish to use an occupied GPU "
+"(possibly running a GUI)."
+msgstr ""
+
+#: ../../source/tutorials/hello_nas.rst:456
+msgid ""
+"Launch the experiment. The experiment should take several minutes to "
+"finish on a workstation with 2 GPUs."
+msgstr ""
+
+#: ../../source/tutorials/hello_nas.rst:495
+msgid ""
+"Users can also run Retiarii Experiment with :doc:`different training "
+"services </experiment/training_service/overview>` besides ``local`` "
+"training service."
+msgstr ""
+
+#: ../../source/tutorials/hello_nas.rst:499
+msgid "Visualize the Experiment"
+msgstr ""
+
+#: ../../source/tutorials/hello_nas.rst:501
+msgid ""
+"Users can visualize their experiment in the same way as visualizing a "
+"normal hyper-parameter tuning experiment. For example, open "
+"``localhost:8081`` in your browser, 8081 is the port that you set in "
+"``exp.run``. Please refer to :doc:`here "
+"</experiment/web_portal/web_portal>` for details."
+msgstr ""
+
+#: ../../source/tutorials/hello_nas.rst:505
+msgid ""
+"We support visualizing models with 3rd-party visualization engines (like "
+"`Netron <https://netron.app/>`__). This can be used by clicking "
+"``Visualization`` in detail panel for each trial. Note that current "
+"visualization is based on `onnx <https://onnx.ai/>`__ , thus "
+"visualization is not feasible if the model cannot be exported into onnx."
+msgstr ""
+
+#: ../../source/tutorials/hello_nas.rst:510
+msgid ""
+"Built-in evaluators (e.g., Classification) will automatically export the "
+"model into a file. For your own evaluator, you need to save your file "
+"into ``$NNI_OUTPUT_DIR/model.onnx`` to make this work. For instance,"
+msgstr ""
+
+#: ../../source/tutorials/hello_nas.rst:541
+msgid "Relaunch the experiment, and a button is shown on Web portal."
+msgstr ""
+
+#: ../../source/tutorials/hello_nas.rst:546
+msgid "Export Top Models"
+msgstr ""
+
+#: ../../source/tutorials/hello_nas.rst:548
+msgid ""
+"Users can export top models after the exploration is done using "
+"``export_top_models``."
+msgstr ""
+
+#: ../../source/tutorials/hello_nas.rst:575
+msgid ""
+"The output is ``json`` object which records the mutation actions of the "
+"top model. If users want to output source code of the top model, they can"
+" use :ref:`graph-based execution engine <graph-based-execution-engine>` "
+"for the experiment, by simply adding the following two lines."
+msgstr ""
+
+#: ../../source/tutorials/hello_nas.rst:597
+msgid "**Total running time of the script:** ( 2 minutes  4.499 seconds)"
+msgstr ""
+
+#: ../../source/tutorials/hello_nas.rst:612
+msgid ":download:`Download Python source code: hello_nas.py <hello_nas.py>`"
+msgstr ""
+
+#: ../../source/tutorials/hello_nas.rst:618
+msgid ":download:`Download Jupyter notebook: hello_nas.ipynb <hello_nas.ipynb>`"
+msgstr ""
+
+#: ../../source/tutorials/hello_nas.rst:625
+#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:335
+#: ../../source/tutorials/pruning_quick_start_mnist.rst:362
+msgid "`Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_"
+msgstr ""
+
+#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:13
+msgid ""
+"Click :ref:`here "
+"<sphx_glr_download_tutorials_hpo_quickstart_pytorch_main.py>` to download"
+" the full example code"
+msgstr ""
+
+#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:22
+msgid "HPO Quickstart with PyTorch"
+msgstr ""
+
+#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:23
+msgid ""
+"This tutorial optimizes the model in `official PyTorch quickstart`_ with "
+"auto-tuning."
+msgstr ""
+
+#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:25
+msgid "The tutorial consists of 4 steps:"
+msgstr ""
+
+#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:27
+msgid "Modify the model for auto-tuning."
+msgstr ""
+
+#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:28
+msgid "Define hyperparameters' search space."
+msgstr ""
+
+#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:29
+msgid "Configure the experiment."
+msgstr ""
+
+#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:30
+msgid "Run the experiment."
+msgstr ""
+
+#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:37
+msgid "Step 1: Prepare the model"
+msgstr ""
+
+#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:38
+msgid "In first step, we need to prepare the model to be tuned."
+msgstr ""
+
+#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:40
+msgid ""
+"The model should be put in a separate script. It will be evaluated many "
+"times concurrently, and possibly will be trained on distributed "
+"platforms."
+msgstr ""
+
+#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:44
+msgid "In this tutorial, the model is defined in :doc:`model.py <model>`."
+msgstr ""
+
+#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:46
+msgid "In short, it is a PyTorch model with 3 additional API calls:"
+msgstr ""
+
+#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:48
+msgid ""
+"Use :func:`nni.get_next_parameter` to fetch the hyperparameters to be "
+"evalutated."
+msgstr ""
+
+#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:49
+msgid ""
+"Use :func:`nni.report_intermediate_result` to report per-epoch accuracy "
+"metrics."
+msgstr ""
+
+#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:50
+msgid "Use :func:`nni.report_final_result` to report final accuracy."
+msgstr ""
+
+#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:52
+msgid "Please understand the model code before continue to next step."
+msgstr ""
+
+#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:57
+msgid "Step 2: Define search space"
+msgstr ""
+
+#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:58
+msgid ""
+"In model code, we have prepared 3 hyperparameters to be tuned: "
+"*features*, *lr*, and *momentum*."
+msgstr ""
+
+#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:61
+msgid ""
+"Here we need to define their *search space* so the tuning algorithm can "
+"sample them in desired range."
+msgstr ""
+
+#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:63
+msgid "Assuming we have following prior knowledge for these hyperparameters:"
+msgstr ""
+
+#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:65
+msgid "*features* should be one of 128, 256, 512, 1024."
+msgstr ""
+
+#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:66
+msgid ""
+"*lr* should be a float between 0.0001 and 0.1, and it follows exponential"
+" distribution."
+msgstr ""
+
+#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:67
+msgid "*momentum* should be a float between 0 and 1."
+msgstr ""
+
+#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:69
+msgid ""
+"In NNI, the space of *features* is called ``choice``; the space of *lr* "
+"is called ``loguniform``; and the space of *momentum* is called "
+"``uniform``. You may have noticed, these names are derived from "
+"``numpy.random``."
+msgstr ""
+
+#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:74
+msgid ""
+"For full specification of search space, check :doc:`the reference "
+"</hpo/search_space>`."
+msgstr ""
+
+#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:76
+msgid "Now we can define the search space as follow:"
+msgstr ""
+
+#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:99
+msgid "Step 3: Configure the experiment"
+msgstr ""
+
+#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:100
+msgid ""
+"NNI uses an *experiment* to manage the HPO process. The *experiment "
+"config* defines how to train the models and how to explore the search "
+"space."
+msgstr ""
+
+#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:103
+msgid ""
+"In this tutorial we use a *local* mode experiment, which means models "
+"will be trained on local machine, without using any special training "
+"platform."
+msgstr ""
+
+#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:122
+msgid "Now we start to configure the experiment."
+msgstr ""
+
+#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:125
+msgid "Configure trial code"
+msgstr ""
+
+#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:126
+msgid ""
+"In NNI evaluation of each hyperparameter set is called a *trial*. So the "
+"model script is called *trial code*."
+msgstr ""
+
+#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:144
+msgid ""
+"When ``trial_code_directory`` is a relative path, it relates to current "
+"working directory. To run ``main.py`` in a different path, you can set "
+"trial code directory to ``Path(__file__).parent``. (`__file__ "
+"<https://docs.python.org/3.10/reference/datamodel.html#index-43>`__ is "
+"only available in standard Python, not in Jupyter Notebook.)"
+msgstr ""
+
+#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:151
+msgid ""
+"If you are using Linux system without Conda, you may need to change "
+"``\"python model.py\"`` to ``\"python3 model.py\"``."
+msgstr ""
+
+#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:157
+msgid "Configure search space"
+msgstr ""
+
+#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:175
+msgid "Configure tuning algorithm"
+msgstr ""
+
+#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:176
+msgid "Here we use :doc:`TPE tuner </hpo/tuners>`."
+msgstr ""
+
+#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:195
+msgid "Configure how many trials to run"
+msgstr ""
+
+#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:196
+msgid ""
+"Here we evaluate 10 sets of hyperparameters in total, and concurrently "
+"evaluate 2 sets at a time."
+msgstr ""
+
+#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:213
+msgid "You may also set ``max_experiment_duration = '1h'`` to limit running time."
+msgstr ""
+
+#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:215
+msgid ""
+"If neither ``max_trial_number`` nor ``max_experiment_duration`` are set, "
+"the experiment will run forever until you press Ctrl-C."
+msgstr ""
+
+#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:220
+msgid ""
+"``max_trial_number`` is set to 10 here for a fast example. In real world "
+"it should be set to a larger number. With default config TPE tuner "
+"requires 20 trials to warm up."
+msgstr ""
+
+#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:227
+msgid "Step 4: Run the experiment"
+msgstr ""
+
+#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:228
+msgid ""
+"Now the experiment is ready. Choose a port and launch it. (Here we use "
+"port 8080.)"
+msgstr ""
+
+#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:230
+msgid ""
+"You can use the web portal to view experiment status: "
+"http://localhost:8080."
+msgstr ""
+
+#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:260
+msgid "After the experiment is done"
+msgstr ""
+
+#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:261
+msgid "Everything is done and it is safe to exit now. The following are optional."
+msgstr ""
+
+#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:263
+msgid ""
+"If you are using standard Python instead of Jupyter Notebook, you can add"
+" ``input()`` or ``signal.pause()`` to prevent Python from exiting, "
+"allowing you to view the web portal after the experiment is done."
+msgstr ""
+
+#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:293
+msgid ""
+":meth:`nni.experiment.Experiment.stop` is automatically invoked when "
+"Python exits, so it can be omitted in your code."
+msgstr ""
+
+#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:296
+msgid ""
+"After the experiment is stopped, you can run "
+":meth:`nni.experiment.Experiment.view` to restart web portal."
+msgstr ""
+
+#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:300
+msgid ""
+"This example uses :doc:`Python API </reference/experiment>` to create "
+"experiment."
+msgstr ""
+
+#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:302
+msgid ""
+"You can also create and manage experiments with :doc:`command line tool "
+"<../hpo_nnictl/nnictl>`."
+msgstr ""
+
+#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:307
+msgid "**Total running time of the script:** ( 1 minutes  24.367 seconds)"
+msgstr ""
+
+#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:322
+msgid ":download:`Download Python source code: main.py <main.py>`"
+msgstr ""
+
+#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:328
+msgid ":download:`Download Jupyter notebook: main.ipynb <main.ipynb>`"
+msgstr ""
+
+#: ../../source/tutorials/pruning_quick_start_mnist.rst:13
+msgid ""
+"Click :ref:`here "
+"<sphx_glr_download_tutorials_pruning_quick_start_mnist.py>` to download "
+"the full example code"
+msgstr ""
+
+#: ../../source/tutorials/pruning_quick_start_mnist.rst:22
+msgid "Pruning Quickstart"
+msgstr ""
+
+#: ../../source/tutorials/pruning_quick_start_mnist.rst:24
+msgid "Here is a three-minute video to get you started with model pruning."
+msgstr ""
+
+#: ../../source/tutorials/pruning_quick_start_mnist.rst:29
+msgid ""
+"Model pruning is a technique to reduce the model size and computation by "
+"reducing model weight size or intermediate state size. There are three "
+"common practices for pruning a DNN model:"
+msgstr ""
+
+#: ../../source/tutorials/pruning_quick_start_mnist.rst:32
+msgid "Pre-training a model -> Pruning the model -> Fine-tuning the pruned model"
+msgstr ""
+
+#: ../../source/tutorials/pruning_quick_start_mnist.rst:33
+msgid ""
+"Pruning a model during training (i.e., pruning aware training) -> Fine-"
+"tuning the pruned model"
+msgstr ""
+
+#: ../../source/tutorials/pruning_quick_start_mnist.rst:34
+msgid "Pruning a model -> Training the pruned model from scratch"
+msgstr ""
+
+#: ../../source/tutorials/pruning_quick_start_mnist.rst:36
+msgid ""
+"NNI supports all of the above pruning practices by working on the key "
+"pruning stage. Following this tutorial for a quick look at how to use NNI"
+" to prune a model in a common practice."
+msgstr ""
+
+#: ../../source/tutorials/pruning_quick_start_mnist.rst:42
+msgid "Preparation"
+msgstr ""
+
+#: ../../source/tutorials/pruning_quick_start_mnist.rst:44
+msgid ""
+"In this tutorial, we use a simple model and pre-trained on MNIST dataset."
+" If you are familiar with defining a model and training in pytorch, you "
+"can skip directly to `Pruning Model`_."
+msgstr ""
+
+#: ../../source/tutorials/pruning_quick_start_mnist.rst:126
+msgid "Pruning Model"
+msgstr ""
+
+#: ../../source/tutorials/pruning_quick_start_mnist.rst:128
+msgid ""
+"Using L1NormPruner to prune the model and generate the masks. Usually, a "
+"pruner requires original model and ``config_list`` as its inputs. "
+"Detailed about how to write ``config_list`` please refer "
+":doc:`compression config specification "
+"<../compression/compression_config_list>`."
+msgstr ""
+
+#: ../../source/tutorials/pruning_quick_start_mnist.rst:132
+msgid ""
+"The following `config_list` means all layers whose type is `Linear` or "
+"`Conv2d` will be pruned, except the layer named `fc3`, because `fc3` is "
+"`exclude`. The final sparsity ratio for each layer is 50%. The layer "
+"named `fc3` will not be pruned."
+msgstr ""
+
+#: ../../source/tutorials/pruning_quick_start_mnist.rst:158
+msgid "Pruners usually require `model` and `config_list` as input arguments."
+msgstr ""
+
+#: ../../source/tutorials/pruning_quick_start_mnist.rst:237
+msgid ""
+"Speedup the original model with masks, note that `ModelSpeedup` requires "
+"an unwrapped model. The model becomes smaller after speedup, and reaches "
+"a higher sparsity ratio because `ModelSpeedup` will propagate the masks "
+"across layers."
+msgstr ""
+
+#: ../../source/tutorials/pruning_quick_start_mnist.rst:274
+msgid "the model will become real smaller after speedup"
+msgstr ""
+
+#: ../../source/tutorials/pruning_quick_start_mnist.rst:312
+msgid "Fine-tuning Compacted Model"
+msgstr ""
+
+#: ../../source/tutorials/pruning_quick_start_mnist.rst:313
+msgid ""
+"Note that if the model has been sped up, you need to re-initialize a new "
+"optimizer for fine-tuning. Because speedup will replace the masked big "
+"layers with dense small ones."
+msgstr ""
+
+#: ../../source/tutorials/pruning_quick_start_mnist.rst:334
+msgid "**Total running time of the script:** ( 1 minutes  30.730 seconds)"
+msgstr ""
+
+#: ../../source/tutorials/pruning_quick_start_mnist.rst:349
+msgid ""
+":download:`Download Python source code: pruning_quick_start_mnist.py "
+"<pruning_quick_start_mnist.py>`"
+msgstr ""
+
+#: ../../source/tutorials/pruning_quick_start_mnist.rst:355
+msgid ""
+":download:`Download Jupyter notebook: pruning_quick_start_mnist.ipynb "
+"<pruning_quick_start_mnist.ipynb>`"
+msgstr ""
+
+#~ msgid "**Total running time of the script:** ( 2 minutes  15.810 seconds)"
+#~ msgstr ""
+
+#~ msgid "NNI HPO Quickstart with PyTorch"
+#~ msgstr ""
+
+#~ msgid ""
+#~ "There is also a :doc:`TensorFlow "
+#~ "version<../hpo_quickstart_tensorflow/main>` if you "
+#~ "prefer it."
+#~ msgstr ""
+
+#~ msgid ""
+#~ "You can also create and manage "
+#~ "experiments with :doc:`command line tool "
+#~ "</reference/nnictl>`."
+#~ msgstr ""
+
+#~ msgid "**Total running time of the script:** ( 1 minutes  24.393 seconds)"
+#~ msgstr ""
+
+#~ msgid "**Total running time of the script:** ( 0 minutes  58.337 seconds)"
+#~ msgstr ""
+

docs/source/nas/advanced_usage.rst

+Advanced Usage
+==============
+
+..  toctree::
+    :maxdepth: 2
+
+    execution_engine
+    hardware_aware_nas
+    mutator
+    customize_strategy
+    serialization
+    benchmarks_toctree

docs/source/nas/benchmarks.rst

+NAS Benchmark
+=============
+
+.. note:: :doc:`Example usage of NAS benchmarks </tutorials/nasbench_as_dataset>`.
+
+To improve the reproducibility of NAS algorithms as well as reducing computing resource requirements, researchers proposed a series of NAS benchmarks such as `NAS-Bench-101 <https://arxiv.org/abs/1902.09635>`__, `NAS-Bench-201 <https://arxiv.org/abs/2001.00326>`__, `NDS <https://arxiv.org/abs/1905.13214>`__, etc. NNI provides a query interface for users to acquire these benchmarks. Within just a few lines of code, researcher are able to evaluate their NAS algorithms easily and fairly by utilizing these benchmarks.
+
+Prerequisites
+-------------
+
+* Please prepare a folder to household all the benchmark databases. By default, it can be found at ``${HOME}/.cache/nni/nasbenchmark``. Or you can place it anywhere you like, and specify it in ``NASBENCHMARK_DIR`` via ``export NASBENCHMARK_DIR=/path/to/your/nasbenchmark`` before importing NNI.
+* Please install ``peewee`` via ``pip3 install peewee``, which NNI uses to connect to database.
+
+Data Preparation
+----------------
+
+Option 1 (Recommended)
+^^^^^^^^^^^^^^^^^^^^^^
+
+You can download the preprocessed benchmark files via ``python -m nni.nas.benchmarks.download <benchmark_name>``, where ``<benchmark_name>`` can be ``nasbench101``, ``nasbench201``, and etc. Add ``--help`` to the command for supported command line arguments.
+
+Option 2
+^^^^^^^^
+
+.. note:: If you have files that are processed before v2.5, it is recommended that you delete them and try option 1.
+
+#. Clone NNI to your machine and enter ``examples/nas/benchmarks`` directory.
+
+   .. code-block:: bash
+
+      git clone -b ${NNI_VERSION} https://github.com/microsoft/nni
+      cd nni/examples/nas/benchmarks
+
+   Replace ``${NNI_VERSION}`` with a released version name or branch name, e.g., ``v2.4``.
+
+#. Install dependencies via ``pip3 install -r xxx.requirements.txt``. ``xxx`` can be ``nasbench101``, ``nasbench201`` or ``nds``.
+
+#. Generate the database via ``./xxx.sh``. The directory that stores the benchmark file can be configured with ``NASBENCHMARK_DIR`` environment variable, which defaults to ``~/.nni/nasbenchmark``. Note that the NAS-Bench-201 dataset will be downloaded from a google drive.
+
+Please make sure there is at least 10GB free disk space and note that the conversion process can take up to hours to complete.
+
+Example Usages
+--------------
+
+Please refer to :doc:`examples usages of Benchmarks API </tutorials/nasbench_as_dataset>`.
+
+NAS-Bench-101
+-------------
+
+* `Paper link <https://arxiv.org/abs/1902.09635>`__ 
+* `Open-source <https://github.com/google-research/nasbench>`__
+
+NAS-Bench-101 contains 423,624 unique neural networks, combined with 4 variations in number of epochs (4, 12, 36, 108), each of which is trained 3 times. It is a cell-wise search space, which constructs and stacks a cell by enumerating DAGs with at most 7 operators, and no more than 9 connections. All operators can be chosen from ``CONV3X3_BN_RELU``, ``CONV1X1_BN_RELU`` and ``MAXPOOL3X3``, except the first operator (always ``INPUT``\ ) and last operator (always ``OUTPUT``\ ).
+
+Notably, NAS-Bench-101 eliminates invalid cells (e.g., there is no path from input to output, or there is redundant computation). Furthermore, isomorphic cells are de-duplicated, i.e., all the remaining cells are computationally unique.
+
+See :doc:`example usages </tutorials/nasbench_as_dataset>` and :ref:`API references <nas-bench-101-reference>`.
+
+NAS-Bench-201
+-------------
+
+* `Paper link <https://arxiv.org/abs/2001.00326>`__ 
+* `Open-source API <https://github.com/D-X-Y/NAS-Bench-201>`__ 
+* `Implementations <https://github.com/D-X-Y/AutoDL-Projects>`__
+
+NAS-Bench-201 is a cell-wise search space that views nodes as tensors and edges as operators. The search space contains all possible densely-connected DAGs with 4 nodes, resulting in 15,625 candidates in total. Each operator (i.e., edge) is selected from a pre-defined operator set (\ ``NONE``, ``SKIP_CONNECT``, ``CONV_1X1``, ``CONV_3X3`` and ``AVG_POOL_3X3``\ ). Training appraoches vary in the dataset used (CIFAR-10, CIFAR-100, ImageNet) and number of epochs scheduled (12 and 200). Each combination of architecture and training approach is repeated 1 - 3 times with different random seeds.
+
+See :doc:`example usages </tutorials/nasbench_as_dataset>` and :ref:`API references <nas-bench-201-reference>`.
+
+NDS
+---
+
+* `Paper link <https://arxiv.org/abs/1905.13214>`__ 
+* `Open-source <https://github.com/facebookresearch/nds>`__
+
+*On Network Design Spaces for Visual Recognition* released trial statistics of over 100,000 configurations (models + hyper-parameters) sampled from multiple model families, including vanilla (feedforward network loosely inspired by VGG), ResNet and ResNeXt (residual basic block and residual bottleneck block) and NAS cells (following popular design from NASNet, Ameoba, PNAS, ENAS and DARTS). Most configurations are trained only once with a fixed seed, except a few that are trained twice or three times.
+
+Instead of storing results obtained with different configurations in separate files, we dump them into one single database to enable comparison in multiple dimensions. Specifically, we use ``model_family`` to distinguish model types, ``model_spec`` for all hyper-parameters needed to build this model, ``cell_spec`` for detailed information on operators and connections if it is a NAS cell, ``generator`` to denote the sampling policy through which this configuration is generated. Refer to API documentation for details.
+
+Here is a list of available operators used in NDS.
+
+.. automodule:: nni.nas.benchmarks.nds.constants
+   :noindex:
+
+See :doc:`example usages </tutorials/nasbench_as_dataset>` and :ref:`API references <nds-reference>`.

docs/source/nas/benchmarks_toctree.rst

+NAS Benchmark
+=============
+
+.. toctree::
+   :hidden:
+
+   Overview <benchmarks>
+   Examples </tutorials/nasbench_as_dataset>

docs/source/nas/construct_space.rst

+Construct Model Space
+=====================
+
+NNI provides powerful (and multi-level) APIs for users to easily express model space (or search space).
+
+* *Mutation Primitives*: high-level APIs (e.g., ValueChoice, LayerChoice) that are utilities to build blocks in search space. In most cases, mutation pritimives should be straightforward yet expressive enough. **We strongly recommend users to try them first,** and report issues if those APIs are not satisfying.
+* *Hyper-module Library*: plug-and-play modules that are proved useful. They are usually well studied in research, and comes with pre-searched results. (For example, the optimal activation function in `AutoActivation <https://arxiv.org/abs/1710.05941>`__ is reported to be `Swish <https://pytorch.org/docs/stable/generated/torch.nn.SiLU.html>`__).
+* *Mutator*: for advanced users only. NNI provides interface to customize new mutators for expressing more complicated model spaces.
+
+The following table summarizes all the APIs we have provided for constructing search space.
+
+.. list-table::
+   :header-rows: 1
+   :widths: auto
+
+   * - Name
+     - Category
+     - Brief Description
+   * - :class:`LayerChoice <nni.retiarii.nn.pytorch.LayerChoice>`
+     - :ref:`Mutation Primitives <mutation-primitives>`
+     - Select from some PyTorch modules
+   * - :class:`InputChoice <nni.retiarii.nn.pytorch.InputChoice>`
+     - :ref:`Mutation Primitives <mutation-primitives>`
+     - Select from some inputs (tensors)
+   * - :class:`ValueChoice <nni.retiarii.nn.pytorch.ValueChoice>`
+     - :ref:`Mutation Primitives <mutation-primitives>`
+     - Select from some candidate values
+   * - :class:`Repeat <nni.retiarii.nn.pytorch.Repeat>`
+     - :ref:`Mutation Primitives <mutation-primitives>`
+     - Repeat a block by a variable number of times
+   * - :class:`Cell <nni.retiarii.nn.pytorch.Cell>`
+     - :ref:`Mutation Primitives <mutation-primitives>`
+     - Cell structure popularly used in literature
+   * - :class:`NasBench101Cell <nni.retiarii.nn.pytorch.NasBench101Cell>`
+     - :ref:`Mutation Primitives <mutation-primitives>`
+     - Cell structure (variant) proposed by NAS-Bench-101
+   * - :class:`NasBench201Cell <nni.retiarii.nn.pytorch.NasBench201Cell>`
+     - :ref:`Mutation Primitives <mutation-primitives>`
+     - Cell structure (variant) proposed by NAS-Bench-201
+   * - :class:`AutoActivation <nni.retiarii.nn.pytorch.AutoActivation>`
+     - :ref:`Hyper-modules Library <hyper-modules>`
+     - Searching for activation functions
+   * - :class:`Mutator <nni.retiarii.Mutator>`
+     - :doc:`Mutator <mutator>`
+     - Flexible mutations on graphs. :doc:`See tutorial here <mutator>`

docs/source/nas/customize_strategy.rst

+Customize Exploration Strategy
+==============================
+
+Customize Multi-trial Strategy
+------------------------------
+
+If users want to innovate a new exploration strategy, they can easily customize a new one following the interface provided by NNI. Specifically, users should inherit the base strategy class :class:`nni.retiarii.strategy.BaseStrategy`, then implement the member function ``run``. This member function takes ``base_model`` and ``applied_mutators`` as its input arguments. It can simply apply the user specified mutators in ``applied_mutators`` onto ``base_model`` to generate a new model. When a mutator is applied, it should be bound with a sampler (e.g., ``RandomSampler``). Every sampler implements the ``choice`` function which chooses value(s) from candidate values. The ``choice`` functions invoked in mutators are executed with the sampler.
+
+Below is a very simple random strategy, which makes the choices completely random.
+
+.. code-block:: python
+
+    from nni.retiarii import Sampler
+
+    class RandomSampler(Sampler):
+        def choice(self, candidates, mutator, model, index):
+            return random.choice(candidates)
+
+    class RandomStrategy(BaseStrategy):
+        def __init__(self):
+            self.random_sampler = RandomSampler()
+
+        def run(self, base_model, applied_mutators):
+            _logger.info('stargety start...')
+            while True:
+                avail_resource = query_available_resources()
+                if avail_resource > 0:
+                    model = base_model
+                    _logger.info('apply mutators...')
+                    _logger.info('mutators: %s', str(applied_mutators))
+                    for mutator in applied_mutators:
+                        mutator.bind_sampler(self.random_sampler)
+                        model = mutator.apply(model)
+                    # run models
+                    submit_models(model)
+                else:
+                    time.sleep(2)
+
+You can find that this strategy does not know the search space beforehand, it passively makes decisions every time ``choice`` is invoked from mutators. If a strategy wants to know the whole search space before making any decision (e.g., TPE, SMAC), it can use ``dry_run`` function provided by ``Mutator`` to obtain the space. An example strategy can be found :githublink:`here <nni/retiarii/strategy/tpe_strategy.py>`.
+
+After generating a new model, the strategy can use our provided APIs (e.g., :func:`nni.retiarii.execution.submit_models`, :func:`nni.retiarii.execution.is_stopped_exec`) to submit the model and get its reported results.
+
+Customize a New One-shot Trainer (legacy)
+-----------------------------------------
+
+One-shot trainers should inherit :class:`nni.retiarii.oneshot.BaseOneShotTrainer`, and need to implement ``fit()`` (used to conduct the fitting and searching process) and ``export()`` method (used to return the searched best architecture).
+
+Writing a one-shot trainer is very different to single-arch evaluator. First of all, there are no more restrictions on init method arguments, any Python arguments are acceptable. Secondly, the model fed into one-shot trainers might be a model with Retiarii-specific modules, such as LayerChoice and InputChoice. Such model cannot directly forward-propagate and trainers need to decide how to handle those modules.
+
+A typical example is DartsTrainer, where learnable-parameters are used to combine multiple choices in LayerChoice. Retiarii provides ease-to-use utility functions for module-replace purposes, namely :meth:`nni.retiarii.oneshot.pytorch.utils.replace_layer_choice`, :meth:`nni.retiarii.oneshot.pytorch.utils.replace_input_choice`. A simplified example is as follows: 
+
+.. code-block:: python
+
+    from nni.retiarii.oneshot import BaseOneShotTrainer
+    from nni.retiarii.oneshot.pytorch.utils import replace_layer_choice, replace_input_choice
+
+
+    class DartsLayerChoice(nn.Module):
+        def __init__(self, layer_choice):
+            super(DartsLayerChoice, self).__init__()
+            self.name = layer_choice.label
+            self.op_choices = nn.ModuleDict(layer_choice.named_children())
+            self.alpha = nn.Parameter(torch.randn(len(self.op_choices)) * 1e-3)
+
+        def forward(self, *args, **kwargs):
+            op_results = torch.stack([op(*args, **kwargs) for op in self.op_choices.values()])
+            alpha_shape = [-1] + [1] * (len(op_results.size()) - 1)
+            return torch.sum(op_results * F.softmax(self.alpha, -1).view(*alpha_shape), 0)
+
+
+    class DartsTrainer(BaseOneShotTrainer):
+
+        def __init__(self, model, loss, metrics, optimizer):
+            self.model = model
+            self.loss = loss
+            self.metrics = metrics
+            self.num_epochs = 10
+
+            self.nas_modules = []
+            replace_layer_choice(self.model, DartsLayerChoice, self.nas_modules)
+
+            ... # init dataloaders and optimizers
+
+        def fit(self):
+            for i in range(self.num_epochs):
+                for (trn_X, trn_y), (val_X, val_y) in zip(self.train_loader, self.valid_loader):
+                    self.train_architecture(val_X, val_y)
+                    self.train_model_weight(trn_X, trn_y)
+
+        @torch.no_grad()
+        def export(self):
+            result = dict()
+            for name, module in self.nas_modules:
+                if name not in result:
+                    result[name] = select_best_of_module(module)
+            return result
+
+The full code of DartsTrainer is available to Retiarii source code. Please have a check at :githublink:`DartsTrainer <nni/retiarii/oneshot/pytorch/darts.py>`.

docs/source/nas/evaluator.rst

+Model Evaluator
+===============
+
+A model evaluator is for training and validating each generated model. They are necessary to evaluate the performance of new explored models.
+
+.. _functional-evaluator:
+
+Customize Evaluator with Any Function
+-------------------------------------
+
+The simplest way to customize a new evaluator is with :class:`~nni.retiarii.evaluator.FunctionalEvaluator`, which is very easy when training code is already available. Users only need to write a fit function that wraps everything, which usually includes training, validating and testing of a single model. This function takes one positional arguments (``model_cls``) and possible keyword arguments. The keyword arguments (other than ``model_cls``) are fed to :class:`~nni.retiarii.evaluator.FunctionalEvaluator` as its initialization parameters (note that they will be :doc:`serialized <./serialization>`). In this way, users get everything under their control, but expose less information to the framework and as a result, further optimizations like :ref:`CGO <cgo-execution-engine>` might be not feasible. An example is as belows:
+
+.. code-block:: python
+
+    from nni.retiarii.evaluator import FunctionalEvaluator
+    from nni.retiarii.experiment.pytorch import RetiariiExperiment
+
+    def fit(model_cls, dataloader):
+        model = model_cls()
+        train(model, dataloader)
+        acc = test(model, dataloader)
+        nni.report_final_result(acc)
+
+    # The dataloader will be serialized, thus ``nni.trace`` is needed here.
+    # See serialization tutorial for more details.
+    evaluator = FunctionalEvaluator(fit, dataloader=nni.trace(DataLoader)(foo, bar))
+    experiment = RetiariiExperiment(base_model, evaluator, mutators, strategy)
+
+.. tip::
+
+    When using customized evaluators, if you want to visualize models, you need to export your model and save it into ``$NNI_OUTPUT_DIR/model.onnx`` in your evaluator. An example here:
+
+    .. code-block:: python
+
+        def fit(model_cls):
+            model = model_cls()
+            onnx_path = Path(os.environ.get('NNI_OUTPUT_DIR', '.')) / 'model.onnx'
+            onnx_path.parent.mkdir(exist_ok=True)
+            dummy_input = torch.randn(10, 3, 224, 224)
+            torch.onnx.export(model, dummy_input, onnx_path)
+            # the rest of training code here
+
+    If the conversion is successful, the model will be able to be visualized with powerful tools `Netron <https://netron.app/>`__.
+
+Use Evaluators to Train and Evaluate Models
+-------------------------------------------
+
+Users can use evaluators to train or evaluate a single, concrete architecture. This is very useful when:
+
+* Debugging your evaluator against a baseline model.
+* Fully train, validate and test your model after the search process is complete.
+
+The usage is shown below:
+
+.. code-block:: python
+
+   # Class definition of single model, for example, ResNet.
+   class SingleModel(nn.Module):
+       def __init__():  # Can't have init parameters here.
+           ...
+
+   # Use a callable returning a model
+   evaluator.evaluate(SingleModel)
+   # Or initialize the model beforehand
+   evaluator.evaluate(SingleModel())
+
+The underlying implementation of :meth:`~nni.retiarii.Evaluator.evaluate` depends on concrete evaluator that you used.
+For example, if :class:`~nni.retiarii.evaluator.FunctionalEvaluator` is used, it will run your customized fit function.
+If lightning evaluators like :class:`nni.retiarii.evaluator.pytorch.Classification` are used, it will invoke the ``trainer.fit()`` of Lightning.
+
+To evaluate an architecture that is exported from experiment (i.e., from :meth:`~nni.retiarii.experiment.pytorch.RetiariiExperiment.export_top_models`), use :func:`nni.retiarii.fixed_arch` to instantiate the exported model::
+
+    with fixed_arch(exported_model):
+        model = ModelSpace()
+    # Then use evaluator.evaluate
+    evaluator.evaluate(model)
+
+.. tip:: There is a way to port the trained checkpoint of super-net produced by one-shot strategies, to the concrete chosen architecture, thanks to :func:`nni.retiarii.utils.original_state_dict_hooks`. This is helpful in implementing recent multi-stage NAS algorithms like `SPOS <https://arxiv.org/abs/1904.00420>`__.
+
+.. _lightning-evaluator:
+
+Evaluators with PyTorch-Lightning
+---------------------------------
+
+Use Built-in Evaluators
+^^^^^^^^^^^^^^^^^^^^^^^
+
+NNI provides some commonly used model evaluators for users' convenience. These evaluators are built upon the awesome library PyTorch-Lightning. Read the :doc:`reference </reference/nas/evaluator>` for their detailed usages.
+
+* :class:`nni.retiarii.evaluator.pytorch.Classification`: for classification tasks.
+* :class:`nni.retiarii.evaluator.pytorch.Regression`: for regression tasks.
+
+We recommend to read the :doc:`serialization tutorial <serialization>` before using these evaluators. A few notes to summarize the tutorial:
+
+1. :class:`nni.retiarii.evaluator.pytorch.DataLoader` should be used in place of ``torch.utils.data.DataLoader``.
+2. The datasets used in data-loader should be decorated with :meth:`nni.trace` recursively.
+
+For example,
+
+.. code-block:: python
+
+  import nni.retiarii.evaluator.pytorch.lightning as pl
+  from torchvision import transforms
+
+  transform = nni.trace(transforms.Compose, [nni.trace(transforms.ToTensor()), nni.trace(transforms.Normalize, (0.1307,), (0.3081,))])
+  train_dataset = nni.trace(MNIST, root='data/mnist', train=True, download=True, transform=transform)
+  test_dataset = nni.trace(MNIST, root='data/mnist', train=False, download=True, transform=transform)
+
+  # pl.DataLoader and pl.Classification is already traced and supports serialization.
+  evaluator = pl.Classification(train_dataloaders=pl.DataLoader(train_dataset, batch_size=100),
+                                val_dataloaders=pl.DataLoader(test_dataset, batch_size=100),
+                                max_epochs=10)
+
+Customize Evaluator with PyTorch-Lightning
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Another approach is to write training code in PyTorch-Lightning style, that is, to write a LightningModule that defines all elements needed for training (e.g., loss function, optimizer) and to define a trainer that takes (optional) dataloaders to execute the training. Before that, please read the `document of PyTorch-lightning <https://pytorch-lightning.readthedocs.io/>`__ to learn the basic concepts and components provided by PyTorch-lightning.
+
+In practice, writing a new training module in Retiarii should inherit :class:`nni.retiarii.evaluator.pytorch.LightningModule`, which has a ``set_model`` that will be called after ``__init__`` to save the candidate model (generated by strategy) as ``self.model``. The rest of the process (like ``training_step``) should be the same as writing any other lightning module. Evaluators should also communicate with strategies via two API calls (:meth:`nni.report_intermediate_result` for periodical metrics and :meth:`nni.report_final_result` for final metrics), added in ``on_validation_epoch_end`` and ``teardown`` respectively. 
+
+An example is as follows:
+
+.. code-block:: python
+
+    from nni.retiarii.evaluator.pytorch.lightning import LightningModule  # please import this one
+
+    @nni.trace
+    class AutoEncoder(LightningModule):
+        def __init__(self):
+            super().__init__()
+            self.decoder = nn.Sequential(
+                nn.Linear(3, 64),
+                nn.ReLU(),
+                nn.Linear(64, 28*28)
+            )
+
+        def forward(self, x):
+            embedding = self.model(x)  # let's search for encoder
+            return embedding
+
+        def training_step(self, batch, batch_idx):
+            # training_step defined the train loop.
+            # It is independent of forward
+            x, y = batch
+            x = x.view(x.size(0), -1)
+            z = self.model(x)  # model is the one that is searched for
+            x_hat = self.decoder(z)
+            loss = F.mse_loss(x_hat, x)
+            # Logging to TensorBoard by default
+            self.log('train_loss', loss)
+            return loss
+
+        def validation_step(self, batch, batch_idx):
+            x, y = batch
+            x = x.view(x.size(0), -1)
+            z = self.model(x)
+            x_hat = self.decoder(z)
+            loss = F.mse_loss(x_hat, x)
+            self.log('val_loss', loss)
+
+        def configure_optimizers(self):
+            optimizer = torch.optim.Adam(self.parameters(), lr=1e-3)
+            return optimizer
+
+        def on_validation_epoch_end(self):
+            nni.report_intermediate_result(self.trainer.callback_metrics['val_loss'].item())
+
+        def teardown(self, stage):
+            if stage == 'fit':
+                nni.report_final_result(self.trainer.callback_metrics['val_loss'].item())
+
+.. note::
+
+   If you are trying to use your customized evaluator with one-shot strategy, bear in mind that your defined methods will be reassembled into another LightningModule, which might result in extra constraints when writing the LightningModule. For example, your validation step could appear else where (e.g., in ``training_step``). This prohibits you from returning arbitrary object in ``validation_step``.
+
+Then, users need to wrap everything (including LightningModule, trainer and dataloaders) into a :class:`nni.retiarii.evaluator.pytorch.Lightning` object, and pass this object into a Retiarii experiment.
+
+.. code-block:: python
+
+    import nni.retiarii.evaluator.pytorch.lightning as pl
+    from nni.retiarii.experiment.pytorch import RetiariiExperiment
+
+    lightning = pl.Lightning(AutoEncoder(),
+                             pl.Trainer(max_epochs=10),
+                             train_dataloaders=pl.DataLoader(train_dataset, batch_size=100),
+                             val_dataloaders=pl.DataLoader(test_dataset, batch_size=100))
+    experiment = RetiariiExperiment(base_model, lightning, mutators, strategy)

docs/source/nas/execution_engine.rst

+Execution Engines
+=================
+
+Execution engine is for running Retiarii Experiment. NNI supports three execution engines, users can choose a specific engine according to the type of their model mutation definition and their requirements for cross-model optimizations. 
+
+* **Pure-python execution engine** is the default engine, it supports the model space expressed by :doc:`mutation primitives <construct_space>`.
+
+* **Graph-based execution engine** supports the use of :doc:`mutation primitives <construct_space>` and model spaces represented by :doc:`mutators <mutator>`. It requires the user's model to be parsed by `TorchScript <https://pytorch.org/docs/stable/jit.html>`__.
+
+* **CGO execution engine** has the same requirements and capabilities as the **Graph-based execution engine**. But further enables cross-model optimizations, which makes model space exploration faster.
+
+.. _pure-python-execution-engine:
+
+Pure-python Execution Engine
+----------------------------
+
+Pure-python Execution Engine is the default engine, we recommend users to keep using this execution engine, if they are new to NNI NAS. Pure-python execution engine plays magic within the scope of inline mutation APIs, while does not touch the rest of user model. Thus, it has minimal requirement on user model. 
+
+Rememeber to add :meth:`nni.retiarii.model_wrapper` decorator outside the whole PyTorch model before using this engine.
+
+.. note:: You should always use ``super().__init__()`` instead of ``super(MyNetwork, self).__init__()`` in the PyTorch model, because the latter one has issues with model wrapper.
+
+.. _graph-based-execution-engine:
+
+Graph-based Execution Engine
+----------------------------
+
+For graph-based execution engine, it converts user-defined model to a graph representation (called graph IR) using `TorchScript <https://pytorch.org/docs/stable/jit.html>`__, each instantiated module in the model is converted to a subgraph. Then mutations are applied to the graph to generate new graphs. Each new graph is then converted back to PyTorch code and executed on the user specified training service.
+
+Users may find ``@basic_unit`` helpful in some cases. :meth:`nni.retiarii.basic_unit` here means the module will not be converted to a subgraph, instead, it is converted to a single graph node as a basic unit.
+
+``@basic_unit`` is usually used in the following cases:
+
+* When users want to tune initialization parameters of a module using :class:`nni.retiarii.nn.pytorch.ValueChoice`, then decorate the module with ``@basic_unit``. For example, ``self.conv = MyConv(kernel_size=nn.ValueChoice([1, 3, 5]))``, here ``MyConv`` should be decorated.
+
+* When a module cannot be successfully parsed to a subgraph, decorate the module with ``@basic_unit``. The parse failure could be due to complex control flow. Currently Retiarii does not support adhoc loop, if there is adhoc loop in a module's forward, this class should be decorated as serializable module. For example, the following ``MyModule`` should be decorated.
+
+  .. code-block:: python
+
+    @basic_unit
+    class MyModule(nn.Module):
+      def __init__(self):
+        ...
+      def forward(self, x):
+        for i in range(10): # <- adhoc loop
+          ...
+
+* Some inline mutation APIs require their handled module to be decorated with ``@basic_unit``. For example, user-defined module that is provided to :class:`nni.retiarii.nn.pytorch.LayerChoice` as a candidate op should be decorated.
+
+Three steps are need to use graph-based execution engine.
+
+1. Remove ``@nni.retiarii.model_wrapper`` if there is any in your model.
+2. Add ``config.execution_engine = 'base'`` to :class:`nni.retiarii.experiment.pytorch.RetiariiExeConfig`. The default value of ``execution_engine`` is 'py', which means pure-python execution engine.
+3. Add ``@basic_unit`` when necessary following the above guidelines.
+
+For exporting top models, graph-based execution engine supports exporting source code for top models by running ``exp.export_top_models(formatter='code')``.
+
+.. _cgo-execution-engine:
+
+CGO Execution Engine (experimental)
+-----------------------------------
+
+CGO (Cross-Graph Optimization) execution engine does cross-model optimizations based on the graph-based execution engine. In CGO execution engine, multiple models could be merged and trained together in one trial.
+Currently, it only supports ``DedupInputOptimizer`` that can merge graphs sharing the same dataset to only loading and pre-processing each batch of data once, which can avoid bottleneck on data loading. 
+
+.. note :: To use CGO engine, PyTorch Lightning >= 1.6.1 is required.
+
+To enable CGO execution engine, you need to follow these steps:
+
+1. Create RetiariiExeConfig with remote training service. CGO execution engine currently only supports remote training service.
+2. Add configurations for remote training service
+3. Add configurations for CGO engine
+
+.. code-block:: python
+  
+    exp = RetiariiExperiment(base_model, trainer, mutators, strategy)
+    config = RetiariiExeConfig('remote')
+    
+    # ...
+    # other configurations of RetiariiExeConfig
+
+    config.execution_engine = 'cgo' # set execution engine to CGO
+    config.max_concurrency_cgo = 3 # the maximum number of concurrent models to merge
+    config.batch_waiting_time = 10  # how many seconds CGO execution engine should wait before optimizing a new batch of models
+
+    rm_conf = RemoteMachineConfig()
+
+    # ...
+    # server configuration in rm_conf
+    rm_conf.gpu_indices = [0, 1, 2, 3] # gpu_indices must be set in RemoteMachineConfig for CGO execution engine
+
+    config.training_service.machine_list = [rm_conf]
+    exp.run(config, 8099)
+
+CGO Execution Engine only supports pytorch-lightning trainer that inherits :class:`nni.retiarii.evaluator.pytorch.cgo.evaluator.MultiModelSupervisedLearningModule`.
+For a trial running multiple models, the trainers inheriting :class:`nni.retiarii.evaluator.pytorch.cgo.evaluator.MultiModelSupervisedLearningModule` can handle the multiple outputs from the merged model for training, test and validation.
+We have already implemented two trainers: :class:`nni.retiarii.evaluator.pytorch.cgo.evaluator.Classification` and :class:`nni.retiarii.evaluator.pytorch.cgo.evaluator.Regression`.
+
+.. code-block:: python
+
+  from nni.retiarii.evaluator.pytorch.cgo.evaluator import Classification
+
+  trainer = Classification(train_dataloaders=pl.DataLoader(train_dataset, batch_size=100),
+                                val_dataloaders=pl.DataLoader(test_dataset, batch_size=100),
+                                max_epochs=1, limit_train_batches=0.2)
+
+Advanced users can also implement their own trainers by inheriting ``MultiModelSupervisedLearningModule``.
+
+Sometimes, a mutated model cannot be executed (e.g., due to shape mismatch). When a trial running multiple models contains 
+a bad model, CGO execution engine will re-run each model independently in separate trials without cross-model optimizations.

docs/source/nas/exploration_strategy.rst

+Exploration Strategy
+====================
+
+There are two types of model space exploration approach: **Multi-trial strategy** and **One-shot strategy**. When the model space has been constructed, users can use either exploration approach to explore the model space. 
+
+* :ref:`Mutli-trial strategy <multi-trial-nas>` trains each sampled model in the model space independently.
+* :ref:`One-shot strategy <one-shot-nas>` samples the model from a super model.
+
+Here is the list of exploration strategies that NNI has supported.
+
+.. list-table::
+   :header-rows: 1
+   :widths: auto
+
+   * - Name
+     - Category
+     - Brief Description
+   * - :class:`Random <nni.retiarii.strategy.Random>`
+     - :ref:`Multi-trial <multi-trial-nas>`
+     - Randomly sample an architecture each time
+   * - :class:`GridSearch <nni.retiarii.strategy.GridSearch>`
+     - :ref:`Multi-trial <multi-trial-nas>`
+     - Traverse the search space and try all possibilities
+   * - :class:`RegularizedEvolution <nni.retiarii.strategy.RegularizedEvolution>`
+     - :ref:`Multi-trial <multi-trial-nas>`
+     - Evolution algorithm for NAS. `Reference <https://arxiv.org/abs/1802.01548>`__
+   * - :class:`TPE <nni.retiarii.strategy.TPE>`
+     - :ref:`Multi-trial <multi-trial-nas>`
+     - Tree-structured Parzen Estimator (TPE). `Reference <https://papers.nips.cc/paper/4443-algorithms-for-hyper-parameter-optimization.pdf>`__
+   * - :class:`PolicyBasedRL <nni.retiarii.strategy.PolicyBasedRL>`
+     - :ref:`Multi-trial <multi-trial-nas>`
+     - Policy-based reinforcement learning, based on implementation of tianshou. `Reference <https://arxiv.org/abs/1611.01578>`__
+   * - :class:`DARTS <nni.retiarii.strategy.DARTS>`
+     - :ref:`One-shot <one-shot-nas>`
+     - Continuous relaxation of the architecture representation, allowing efficient search of the architecture using gradient descent. `Reference <https://arxiv.org/abs/1806.09055>`__
+   * - :class:`ENAS <nni.retiarii.strategy.ENAS>`
+     - :ref:`One-shot <one-shot-nas>`
+     - RL controller learns to generate the best network on a super-net. `Reference <https://arxiv.org/abs/1802.03268>`__
+   * - :class:`GumbelDARTS <nni.retiarii.strategy.GumbelDARTS>`
+     - :ref:`One-shot <one-shot-nas>`
+     - Choose the best block by using Gumbel Softmax random sampling and differentiable training. `Reference <https://arxiv.org/abs/1812.03443>`__
+   * - :class:`RandomOneShot <nni.retiarii.strategy.RandomOneShot>`
+     - :ref:`One-shot <one-shot-nas>`
+     - Train a super-net with uniform path sampling. `Reference <https://arxiv.org/abs/1904.00420>`__
+   * - :class:`Proxyless <nni.retiarii.strategy.Proxyless>`
+     - :ref:`One-shot <one-shot-nas>`
+     - A low-memory-consuming optimized version of differentiable architecture search. `Reference <https://arxiv.org/abs/1812.00332>`__
+
+.. _multi-trial-nas:
+
+Multi-trial strategy
+--------------------
+
+Multi-trial NAS means each sampled model from model space is trained independently. A typical multi-trial NAS is `NASNet <https://arxiv.org/abs/1707.07012>`__. In multi-trial NAS, users need model evaluator to evaluate the performance of each sampled model, and need an exploration strategy to sample models from a defined model space. Here, users could use NNI provided model evaluators or write their own model evalutor. They can simply choose a exploration strategy. Advanced users can also customize new exploration strategy.
+
+To use an exploration strategy, users simply instantiate an exploration strategy and pass the instantiated object to :class:`~nni.retiarii.experiment.pytorch.RetiariiExperiment`. Below is a simple example.
+
+.. code-block:: python
+
+   import nni.retiarii.strategy as strategy
+   exploration_strategy = strategy.Random(dedup=True)
+
+Rather than using :class:`strategy.Random <nni.retiarii.strategy.Random>`, users can choose one of the strategies from the table above.
+
+.. _one-shot-nas:
+
+One-shot strategy
+-----------------
+
+One-shot NAS algorithms leverage weight sharing among models in neural architecture search space to train a supernet, and use this supernet to guide the selection of better models. This type of algorihtms greatly reduces computational resource compared to independently training each model from scratch (which we call "Multi-trial NAS").
+
+Starting from v2.8, the usage of one-shot strategies are much alike to multi-trial strategies. Users simply need to create a strategy and run :class:`~nni.retiarii.experiment.pytorch.RetiariiExperiment`. Since one-shot strategies will manipulate the training recipe, to use a one-shot strategy, the evaluator needs to be one of the :ref:`PyTorch-Lightning evaluators <lightning-evaluator>`, either built-in or customized. Last but not least, don't forget to set execution engine to ``oneshot``. Example follows:
+
+.. code-block:: python
+
+   import nni.retiarii.strategy as strategy
+   import nni.retiarii.evaluator.pytorch.lightning as pl
+   evaluator = pl.Classification(
+     # Need to use `pl.DataLoader` instead of `torch.utils.data.DataLoader` here,
+     # or use `nni.trace` to wrap `torch.utils.data.DataLoader`.
+     train_dataloaders=pl.DataLoader(train_dataset, batch_size=100),
+     val_dataloaders=pl.DataLoader(test_dataset, batch_size=100),
+     # Other keyword arguments passed to pytorch_lightning.Trainer.
+     max_epochs=10,
+     gpus=1,
+   )
+   exploration_strategy = strategy.DARTS()
+
+   exp_config.execution_engine = 'oneshot'
+
+One-shot strategies only support a limited set of :ref:`mutation-primitives`, and does not support :doc:`customizing mutators <mutator>` at all. See the :ref:`reference <one-shot-strategy-reference>` for the detailed support list of each algorithm.
+
+.. versionadded:: 2.8
+
+   One-shot strategy is now compatible with `Lightning accelerators <https://pytorch-lightning.readthedocs.io/en/stable/accelerators/gpu.html>`__. It means that, you can accelerate one-shot strategies on hardwares like multiple GPUs. To enable this feature, you only need to pass the keyword arguments which used to be set in ``pytorch_lightning.Trainer``, to your evaluator. See :doc:`this reference </reference/nas/evaluator>` for more details.
+
+One-shot strategy (legacy)
+--------------------------
+
+.. warning::
+
+   .. deprecated:: 2.8
+
+      The following usages are deprecated and will be removed in future releases. If you intend to use them, the references can be found :doc:`here </deprecated/oneshot_legacy>`.
+
+The usage of one-shot NAS strategy is a little different from multi-trial strategy. One-shot strategy is implemented with a special type of objects named *Trainer*. Following the common practice of one-shot NAS, *Trainer* trains the super-net and searches for the optimal architecture in a single run. For example,
+
+.. code-block:: python
+
+   from nni.retiarii.oneshot.pytorch import DartsTrainer
+
+   trainer = DartsTrainer(
+      model=model,
+      loss=criterion,
+      metrics=lambda output, target: accuracy(output, target, topk=(1,)),
+      optimizer=optim,
+      dataset=dataset_train,
+      batch_size=32,
+      log_frequency=50
+   )
+   trainer.fit()
+
+One-shot strategy can be used without :class:`~nni.retiarii.experiment.pytorch.RetiariiExperiment`. Thus, the ``trainer.fit()`` here runs the experiment locally.
+
+After ``trainer.fit()`` completes, we can use ``trainer.export()`` to export the searched architecture (a dict of choices) to a file.
+
+.. code-block:: python
+
+   final_architecture = trainer.export()
+   print('Final architecture:', trainer.export())
+   json.dump(trainer.export(), open('checkpoint.json', 'w'))
+
+.. tip:: The trained super-net (neither the weights or exported JSON) can't be used directly. It's only an intermediate result used for deriving the final architecture. The exported architecture (can be retrieved with :meth:`nni.retiarii.fixed_arch`) needs to be *retrained* with a standard training recipe to get the final model.

docs/source/nas/hardware_aware_nas.rst

+Hardware-aware NAS
+==================
+
+.. This file should be rewritten as a tutorial
+
+End-to-end Multi-trial SPOS Demo
+--------------------------------
+
+To empower affordable DNN on the edge and mobile devices, hardware-aware NAS searches both high accuracy and low latency models. In particular, the search algorithm only considers the models within the target latency constraints during the search process.
+
+To run this demo, first install nn-Meter by running:
+
+.. code-block:: bash
+
+  pip install nn-meter
+
+Then run multi-trail SPOS demo:
+
+.. code-block:: bash
+
+  cd ${NNI_ROOT}/examples/nas/oneshot/spos/
+  python search.py --latency-filter cortexA76cpu_tflite21
+
+
+How the demo works
+^^^^^^^^^^^^^^^^^^
+
+To support hardware-aware NAS, you first need a ``Strategy`` that supports filtering the models by latency. We provide such a filter named ``LatencyFilter`` in NNI and initialize a ``RegularizedEvolution`` strategy with the filter:
+
+.. code-block:: python
+
+  evolution_strategy = strategy.RegularizedEvolution(
+        model_filter=latency_filter,
+        sample_size=args.evolution_sample_size, population_size=args.evolution_population_size, cycles=args.evolution_cycles
+        )
+
+``LatencyFilter`` will predict the models\' latency by using nn-Meter and filter out the models whose latency are larger than the threshold (i.e., ``100`` in this example).
+You can also build your own strategies and filters to support more flexible NAS such as sorting the models according to latency.
+
+Then, pass this strategy to ``RetiariiExperiment``:
+
+.. code-block:: python
+
+  exp = RetiariiExperiment(base_model, evaluator, strategy=evolution_strategy)
+
+  exp_config = RetiariiExeConfig('local')
+  ...
+  exp_config.dummy_input = [1, 3, 224, 224]
+
+  exp.run(exp_config, args.port)
+
+In ``exp_config``, ``dummy_input`` is required for tracing shape info in latency predictor.
+
+
+End-to-end ProxylessNAS with Latency Constraints
+------------------------------------------------
+
+`ProxylessNAS <https://arxiv.org/abs/1812.00332>`__ is a hardware-aware one-shot NAS algorithm. ProxylessNAS applies the expected latency of the model to build a differentiable metric and design efficient neural network architectures for hardware. The latency loss is added as a regularization term for architecture parameter optimization. In this example, nn-Meter provides a latency estimator to predict expected latency for the mixed operation on other types of mobile and edge hardware. 
+
+To run the one-shot ProxylessNAS demo, first install nn-Meter by running:
+
+.. code-block:: bash
+
+  pip install nn-meter
+
+Then run one-shot ProxylessNAS demo:
+
+.. code-block:: bash
+
+   python ${NNI_ROOT}/examples/nas/oneshot/proxylessnas/main.py --applied_hardware HARDWARE --reference_latency REFERENCE_LATENCY_MS
+
+How the demo works
+^^^^^^^^^^^^^^^^^^
+
+In the implementation of ProxylessNAS ``trainer``, we provide a ``HardwareLatencyEstimator`` which currently builds a lookup table, that stores the measured latency of each candidate building block in the search space. The latency sum of all building blocks in a candidate model will be treated as the model inference latency. The latency prediction is obtained by ``nn-Meter``. ``HardwareLatencyEstimator`` predicts expected latency for the mixed operation based on the path weight of ``ProxylessLayerChoice``. With leveraging ``nn-Meter`` in NNI, users can apply ProxylessNAS to search efficient DNN models on more types of edge devices. 
+
+Despite of ``applied_hardware`` and ``reference_latency``, There are some other parameters related to hardware-aware ProxylessNAS training in this :githublink:`example <examples/nas/oneshot/proxylessnas/main.py>`:
+
+* ``grad_reg_loss_type``: Regularization type to add hardware related loss. Allowed types include ``"mul#log"`` and ``"add#linear"``. Type of ``mul#log`` is calculate by ``(torch.log(expected_latency) / math.log(reference_latency)) ** beta``. Type of ``"add#linear"`` is calculate by ``reg_lambda * (expected_latency - reference_latency) / reference_latency``. 
+* ``grad_reg_loss_lambda``: Regularization params, is set to ``0.1`` by default.
+* ``grad_reg_loss_alpha``: Regularization params, is set to ``0.2`` by default.
+* ``grad_reg_loss_beta``: Regularization params, is set to ``0.3`` by default.
+* ``dummy_input``: The dummy input shape when applied to the target hardware. This parameter is set as (1, 3, 224, 224) by default.

docs/source/nas/mutator.rst

+Construct Space with Mutator
+============================
+
+Besides the mutation primitives demonstrated in the :doc:`basic tutorial <construct_space>`, NNI provides a more general approach to express a model space, i.e., *Mutator*, to cover more complex model spaces. The high-level APIs are also implemented with mutator in the underlying system, which can be seen as a special case of model mutation.
+
+.. warning:: Mutator and inline mutation APIs can NOT be used together.
+
+A mutator is a piece of logic to express how to mutate a given model. Users are free to write their own mutators. Then a model space is expressed with a base model and a list of mutators. A model in the model space is sampled by applying the mutators on the base model one after another. An example is shown below.
+
+.. code-block:: python
+
+  applied_mutators = []
+  applied_mutators.append(BlockMutator('mutable_0'))
+  applied_mutators.append(BlockMutator('mutable_1'))
+
+``BlockMutator`` is defined by users to express how to mutate the base model. 
+
+Write a mutator
+---------------
+
+User-defined mutator should inherit :class:`nni.retiarii.Mutator` class, and implement mutation logic in the member function :meth:`nni.retiarii.Mutator.mutate`.
+
+.. code-block:: python
+
+  from nni.retiarii import Mutator
+  class BlockMutator(Mutator):
+    def __init__(self, target: str, candidates: List):
+        super(BlockMutator, self).__init__()
+        self.target = target
+        self.candidate_op_list = candidates
+
+    def mutate(self, model):
+      nodes = model.get_nodes_by_label(self.target)
+      for node in nodes:
+        chosen_op = self.choice(self.candidate_op_list)
+        node.update_operation(chosen_op.type, chosen_op.params)
+
+The input of :meth:`nni.retiarii.Mutator.mutate` is graph IR (Intermediate Representation) of the base model, users can mutate the graph using the graph's member functions (e.g., :meth:`nni.retiarii.Model.get_nodes_by_label`). The mutation operations can be combined with the API ``self.choice``, in order to express a set of possible mutations. In the above example, the node's operation can be changed to any operation from ``candidate_op_list``.
+
+Use placeholder to make mutation easier: :class:`nni.retiarii.nn.pytorch.Placeholder`. If you want to mutate a subgraph or node of your model, you can define a placeholder in this model to represent the subgraph or node. Then, use mutator to mutate this placeholder to make it real modules.
+
+.. code-block:: python
+
+  ph = nn.Placeholder(
+    label='mutable_0',
+    kernel_size_options=[1, 3, 5],
+    n_layer_options=[1, 2, 3, 4],
+    exp_ratio=exp_ratio,
+    stride=stride
+  )
+
+``label`` is used by mutator to identify this placeholder. The other parameters are the information that is required by mutator. They can be accessed from ``node.operation.parameters`` as a dict, it could include any information that users want to put to pass it to user defined mutator. The complete example code can be found in :githublink:`Mnasnet base model <examples/nas/multi-trial/mnasnet/base_mnasnet.py>`.
+
+Starting an experiment is almost the same as using inline mutation APIs. The only difference is that the applied mutators should be passed to :class:`nni.retiarii.experiment.pytorch.RetiariiExperiment`. Below is a simple example.
+
+.. code-block:: python
+
+  exp = RetiariiExperiment(base_model, trainer, applied_mutators, simple_strategy)
+  exp_config = RetiariiExeConfig('local')
+  exp_config.experiment_name = 'mnasnet_search'
+  exp_config.trial_concurrency = 2
+  exp_config.max_trial_number = 10
+  exp_config.training_service.use_active_gpu = False
+  exp.run(exp_config, 8081)

docs/source/nas/overview.rst

+Overview
+========
+
+.. attention:: NNI's latest NAS supports are all based on Retiarii Framework, users who are still on `early version using NNI NAS v1.0 <https://nni.readthedocs.io/en/v2.2/nas.html>`__ shall migrate your work to Retiarii as soon as possible. We plan to remove the legacy NAS framework in the next few releases.
+
+.. note:: PyTorch is the **only supported framework on Retiarii**. Inquiries of NAS support on Tensorflow is in `this discussion <https://github.com/microsoft/nni/discussions/4605>`__. If you intend to run NAS with DL frameworks other than PyTorch and Tensorflow, please `open new issues <https://github.com/microsoft/nni/issues>`__ to let us know.
+
+Basics
+------
+
+Automatic neural architecture search is playing an increasingly important role in finding better models. Recent research has proven the feasibility of automatic NAS and has led to models that beat many manually designed and tuned models. Representative works include `NASNet <https://arxiv.org/abs/1707.07012>`__, `ENAS <https://arxiv.org/abs/1802.03268>`__, `DARTS <https://arxiv.org/abs/1806.09055>`__, `Network Morphism <https://arxiv.org/abs/1806.10282>`__, and `Evolution <https://arxiv.org/abs/1703.01041>`__. In addition, new innovations continue to emerge.
+
+High-level speaking, aiming to solve any particular task with neural architecture search typically requires: search space design, search strategy selection, and performance evaluation. The three components work together with the following loop (from the famous `NAS survey <https://arxiv.org/abs/1808.05377>`__):
+
+.. image:: ../../img/nas_abstract_illustration.png
+   :align: center
+   :width: 700
+
+In this figure:
+
+* *Model search space*  means a set of models from which the best model is explored/searched. Sometimes we use *search space* or *model space* in short.
+* *Exploration strategy* is the algorithm that is used to explore a model search space. Sometimes we also call it *search strategy*.
+* *Model evaluator* is responsible for training a model and evaluating its performance.
+
+The process is similar to :doc:`Hyperparameter Optimization </hpo/overview>`, except that the target is the best architecture rather than hyperparameter. Concretely, an exploration strategy selects an architecture from a predefined search space. The architecture is passed to a performance evaluation to get a score, which represents how well this architecture performs on a particular task. This process is repeated until the search process is able to find the best architecture.
+
+Key Features
+------------
+
+The current NAS framework in NNI is powered by the research of `Retiarii: A Deep Learning Exploratory-Training Framework <https://www.usenix.org/system/files/osdi20-zhang_quanlu.pdf>`__, where we highlight the following features:
+
+* :doc:`Simple APIs to construct search space easily <construct_space>`
+* :doc:`SOTA NAS algorithms to explore search space <exploration_strategy>`
+* :doc:`Experiment backend support to scale up experiments on large-scale AI platforms </experiment/overview>`
+
+Why NAS with NNI
+----------------
+
+We list out the three perspectives where NAS can be particularly challegning without NNI. NNI provides solutions to relieve users' engineering effort when they want to try NAS techniques in their own scenario.
+
+Search Space Design
+^^^^^^^^^^^^^^^^^^^
+
+The search space defines which architectures can be represented in principle. Incorporating prior knowledge about typical properties of architectures well-suited for a task can reduce the size of the search space and simplify the search. However, this also introduces a human bias, which may prevent finding novel architectural building blocks that go beyond the current human knowledge. Search space design can be very challenging for beginners, who might not possess the experience to balance the richness and simplicity.
+
+In NNI, we provide a wide range of APIs to build the search space. There are :doc:`high-level APIs <construct_space>`, that enables the possibility to incorporate human knowledge about what makes a good architecture or search space. There are also :doc:`low-level APIs <mutator>`, that is a list of primitives to construct a network from operation to operation.
+
+Exploration strategy
+^^^^^^^^^^^^^^^^^^^^
+
+The exploration strategy details how to explore the search space (which is often exponentially large). It encompasses the classical exploration-exploitation trade-off since, on the one hand, it is desirable to find well-performing architectures quickly, while on the other hand, premature convergence to a region of suboptimal architectures should be avoided. The "best" exploration strategy for a particular scenario is usually found via trial-and-error. As many state-of-the-art strategies are implemented with their own code-base, it becomes very troublesome to switch from one to another.
+
+In NNI, we have also provided :doc:`a list of strategies <exploration_strategy>`. Some of them are powerful yet time consuming, while others might be suboptimal but really efficient. Given that all strategies are implemented with a unified interface, users can always find one that matches their need.
+
+Performance estimation
+^^^^^^^^^^^^^^^^^^^^^^
+
+The objective of NAS is typically to find architectures that achieve high predictive performance on unseen data. Performance estimation refers to the process of estimating this performance. The problem with performance estimation is mostly its scalability, i.e., how can I run and manage multiple trials simultaneously.
+
+In NNI, we standardize this process is implemented with :doc:`evaluator <evaluator>`, which is responsible of estimating a model's performance. NNI has quite a few built-in supports of evaluators, ranging from the simplest option, e.g., to perform a standard training and validation of the architecture on data, to complex configurations and implementations. Evaluators are run in *trials*, where trials can be spawn onto distributed platforms with our powerful :doc:`training service </experiment/training_service/overview>`.
+
+Tutorials
+---------
+
+To start using NNI NAS framework, we recommend at least going through the following tutorials:
+
+* :doc:`Quickstart </tutorials/hello_nas>`
+* :doc:`construct_space`
+* :doc:`exploration_strategy`
+* :doc:`evaluator`
+
+Resources
+---------
+
+The following articles will help with a better understanding of the current arts of NAS:
+
+* `Neural Architecture Search: A Survey <https://arxiv.org/abs/1808.05377>`__
+* `A Comprehensive Survey of Neural Architecture Search: Challenges and Solutions <https://arxiv.org/abs/2006.02903>`__

docs/source/nas/overview_zh.rst

+.. 48c39585a539a877461aadef63078c48
+
+神经架构搜索
+===========================
+
+.. toctree::
+   :hidden:
+
+   快速入门 </tutorials/hello_nas>
+   构建搜索空间 <construct_space>
+   探索策略 <exploration_strategy>
+   评估器 <evaluator>
+   高级用法 <advanced_usage>
+
+.. attention:: NNI 最新的架构搜索支持都是基于 Retiarii 框架，还在使用 `NNI 架构搜索的早期版本 <https://nni.readthedocs.io/en/v2.2/nas.html>`__ 的用户应尽快将您的工作迁移到 Retiarii。我们计划在接下来的几个版本中删除旧的架构搜索框架。
+
+.. attention:: PyTorch 是 **Retiarii 唯一支持的框架**。有关 Tensorflow 上架构搜索支持的需求在 `此讨论 <https://github.com/microsoft/nni/discussions/4605>`__ 中。另外，如果您打算使用 PyTorch 和 Tensorflow 以外的 DL 框架运行 NAS，请 `创建新 issue <https://github.com/microsoft/nni/issues>`__ 让我们知道。
+
+概述
+------
+
+自动神经架构搜索 (Neural Architecture Search, NAS）在寻找更好的模型方面发挥着越来越重要的作用。最近的研究证明了自动架构搜索的可行性，并导致模型击败了许多手动设计和调整的模型。其中具有代表性的有 `NASNet <https://arxiv.org/abs/1707.07012>`__、 `ENAS <https://arxiv.org/abs/1802.03268>`__、 `DARTS <https://arxiv.org/ abs/1806.09055>`__、 `Network Morphism <https://arxiv.org/abs/1806.10282>`__ 和 `进化算法 <https://arxiv.org/abs/1703.01041>`__。此外，新的创新正不断涌现。
+
+总的来说，使用神经架构搜索解决任何特定任务通常需要：搜索空间设计、搜索策略选择和性能评估。这三个组件形成如下的循环（图来自于 `架构搜索综述 <https://arxiv.org/abs/1808.05377>`__）：
+
+.. image:: ../../img/nas_abstract_illustration.png
+   :align: center
+   :width: 700
+
+在这个图中：
+
+* *模型搜索空间* 是指一组模型，从中探索/搜索最佳模型，简称为 *搜索空间* 或 *模型空间*。
+* *探索策略* 是用于探索模型搜索空间的算法。有时我们也称它为 *搜索策略*。
+* *模型评估者* 负责训练模型并评估其性能。
+
+该过程类似于 :doc:`超参数优化 </hpo/overview>`，只不过目标是最佳网络结构而不是最优超参数。具体来说，探索策略从预定义的搜索空间中选择架构。该架构被传递给性能评估以获得评分，该评分表示这个网络结构在特定任务上的表现。重复此过程，直到搜索过程能够找到最优的网络结构。
+
+主要特点
+------------
+
+NNI 中当前的架构搜索框架由 `Retiarii: A Deep Learning Exploratory-Training Framework <https://www.usenix.org/system/files/osdi20-zhang_quanlu.pdf>`__ 的研究支撑，具有以下特点：
+
+* :doc:`简单的 API，让您轻松构建搜索空间 <construct_space>`
+* :doc:`SOTA 架构搜索算法，以高效探索搜索空间 <exploration_strategy>`
+* :doc:`后端支持，在大规模 AI 平台上运行实验 </experiment/overview>`
+
+为什么使用 NNI 的架构搜索
+-------------------------------
+
+若没有 NNI，实现架构搜索将极具挑战性，主要包含以下三个方面。当用户想在自己的场景中尝试架构搜索技术时，NNI 提供的解决方案可以极大程度上减轻用户的工作量。
+
+搜索空间设计
+^^^^^^^^^^^^^^^^^^^
+
+搜索空间定义了架构的可行域集合。为了简化搜索，我们通常需要结合任务相关的先验知识，减小搜索空间的规模。然而，这也引入了人类的偏见，在某种程度上可能会丧失突破人类认知的可能性。无论如何，对于初学者来说，搜索空间设计是一个极具挑战性的任务，因为他们可能无法在简单的空间和丰富的想象力之间取得平衡。
+
+在 NNI 中，我们提供了不同层级的 API 来构建搜索空间。有 :doc:`高层 API <construct_space>`，引入大量先验，帮助用户迅速了解什么是好的架构或搜索空间；也有 :doc:`底层 API <mutator>`，提供了最底层的算子和图变换原语。
+
+探索策略
+^^^^^^^^^^^^^^^^^^^^
+
+探索策略定义了如何探索搜索空间（通常是指数级规模的）。它包含经典的探索-利用权衡。一方面，我们希望快速找到性能良好的架构；而另一方面，我们也应避免过早收敛到次优架构的区域。我们往往需要通常通过反复试验找到特定场景的“最佳”探索策略。由于许多近期发表的探索策略都是使用自己的代码库实现的，因此从一个切换到另一个变得非常麻烦。
+
+在 NNI 中，我们还提供了 :doc:`一系列的探索策略 <exploration_strategy>`。其中一些功能强大但耗时，而另一些可能不能找到最优架构但非常高效。鉴于所有策略都使用统一的用户接口实现，用户可以轻松找到符合他们需求的策略。
+
+性能评估
+^^^^^^^^^^^^^^^^^^^^^^
+
+架构搜索的目标通常是找到能够在测试数据集表现理想的网络结构。性能评估的作用便是量化每个网络的好坏。其主要难点在于可扩展性，即如何在大规模训练平台上同时运行和管理多个试验。
+
+在 NNI 中，我们使用 :doc:`evaluator <evaluator>` 来标准化性能评估流程。它负责估计模型的性能。NNI 内建了不少性能评估器，从最简单的交叉验证，到复杂的自定义配置。评估器在 *试验 (trials)* 中运行，可以通过我们强大的 :doc:`训练平台 </experiment/training_service/overview>` 将试验分发到大规模训练平台上。
+
+教程
+---------
+
+要开始使用 NNI 架构搜索框架，我们建议至少阅读以下教程：
+
+* :doc:`快速入门 </tutorials/hello_nas>`
+* :doc:`构建搜索空间 <construct_space>`
+* :doc:`探索策略 <exploration_strategy>`
+* :doc:`评估器 <evaluator>`
+
+资源
+---------
+
+以下文章将有助于更好地了解 NAS 的最新发展：
+
+* `神经架构搜索：综述 <https://arxiv.org/abs/1808.05377>`__
+* `神经架构搜索的综述：挑战和解决方案 <https://arxiv.org/abs/2006.02903>`__

docs/source/nas/serialization.rst

+Serialization
+=============
+
+In multi-trial NAS, a sampled model should be able to be executed on a remote machine or a training platform (e.g., AzureML, OpenPAI). "Serialization" enables re-instantiation of model evaluator in another process or machine, such that, both the model and its model evaluator should be correctly serialized. To make NNI correctly serialize model evaluator, users should apply :func:`nni.trace <nni.common.serializer.trace>` on some of their functions and objects. API references can be found in :func:`nni.trace <nni.common.serializer.trace>`.
+
+Serialization is implemented as a combination of `json-tricks <https://json-tricks.readthedocs.io/en/latest/>`_ and `cloudpickle <https://github.com/cloudpipe/cloudpickle>`_. Essentially, it is json-tricks, that is a enhanced version of Python JSON, enabling handling of serialization of numpy arrays, date/times, decimal, fraction and etc. The difference lies in the handling of class instances. Json-tricks deals with class instances with ``__dict__`` and ``__class__``, which in most of our cases are not reliable (e.g., datasets, dataloaders). Rather, our serialization deals with class instances with two methods:
+
+1. If the class / factory that creates the object is decorated with :func:`nni.trace <nni.common.serializer.trace>`, we can serialize the class / factory function, along with the parameters, such that the instance can be re-instantiated.
+2. Otherwise, cloudpickle is used to serialize the object into a binary.
+
+The recommendation is, unless you are absolutely certain that there is no problem and extra burden to serialize the object into binary, always add :func:`nni.trace <nni.common.serializer.trace>`. In most cases, it will be more clean and neat, and enables possibilities such as mutation of parameters (will be supported in future).
+
+.. warning::
+
+    **What will happen if I forget to "trace" my objects?**
+
+    It is likely that the program can still run. NNI will try to serialize the untraced object into a binary. It might fail in complex cases. For example, when the object is too large. Even if it succeeds, the result might be a substantially large object. For example, if you forgot to add :func:`nni.trace <nni.common.serializer.trace>` on ``MNIST``, the MNIST dataset object wil be serialized into binary, which will be dozens of megabytes because the object has the whole 60k images stored inside. You might see warnings and even errors when running experiments. To avoid such issues, the easiest way is to always remember to add :func:`nni.trace <nni.common.serializer.trace>` to non-primitive objects.
+
+.. note:: In Retiarii, serializer will throw exception when one of an single object in the recursive serialization is larger than 64 KB when binary serialized. This indicates that such object needs to be wrapped by :func:`nni.trace <nni.common.serializer.trace>`. In rare cases, if you insist on pickling large data, the limit can be overridden by setting an environment variable ``PICKLE_SIZE_LIMIT``, whose unit is byte. Please note that even if the experiment might be able to run, this can still cause performance issues and even the crash of NNI experiment.
+
+To trace a function or class, users can use decorator like,
+
+.. code-block:: python
+
+    @nni.trace
+    class MyClass:
+        ...
+
+Inline trace that traces instantly on the object instantiation or function invoke is also acceptable:
+
+.. code-block:: python
+
+   nni.trace(MyClass)(parameters)
+
+Assuming a class ``cls`` is already traced, when it is serialized, its class type along with initialization parameters will be dumped. As the parameters are possibly class instances (if not primitive types like ``int`` and ``str``), their serialization will be a similar problem. We recommend decorate them with :func:`nni.trace <nni.common.serializer.trace>` as well. In other words, :func:`nni.trace <nni.common.serializer.trace>` should be applied recursively if necessary.
+
+Below is an example, ``transforms.Compose``, ``transforms.Normalize``, and ``MNIST`` are serialized manually using :func:`nni.trace <nni.common.serializer.trace>`. :func:`nni.trace <nni.common.serializer.trace>` takes a class / function as its argument, and returns a wrapped class and function that has the same behavior with the original class / function. The usage of the wrapped class / function is also identical to the original one, except that the arguments are recorded. No need to apply :func:`nni.trace <nni.common.serializer.trace>` to :class:`pl.Classification <nni.retiarii.evaluator.pytorch.Classification>` and :class:`pl.DataLoader <nni.retiarii.evaluator.pytorch.DataLoader>` because they are already traced.
+
+.. code-block:: python
+
+  import nni
+  import nni.retiarii.evaluator.pytorch.lightning as pl
+  from torchvision import transforms
+
+  def create_mnist_dataset(root, transform):
+    return MNIST(root='data/mnist', train=False, download=True, transform=transform)
+
+  transform = nni.trace(transforms.Compose)([nni.trace(transforms.ToTensor)(), nni.trace(transforms.Normalize)((0.1307,), (0.3081,))])
+
+  # If you write like following, the whole transform will be serialized into a pickle.
+  # This actually works fine, but we do NOT recommend such practice.
+  # transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.1307,), (0.3081,))])
+
+  train_dataset = nni.trace(MNIST)(root='data/mnist', train=True, download=True, transform=transform)
+  test_dataset = nni.trace(create_mnist_dataset)('data/mnist', transform=transform)  # factory is also acceptable
+  evaluator = pl.Classification(train_dataloaders=pl.DataLoader(train_dataset, batch_size=100),
+                                val_dataloaders=pl.DataLoader(test_dataset, batch_size=100),
+                                max_epochs=10)
+
+.. note::
+
+    **What's the relationship between model_wrapper, basic_unit and nni.trace?**
+
+    They are fundamentally different. :func:`model_wrapper <nni.retiarii.model_wrapper>` is used to wrap a base model (search space), :func:`basic_unit <nni.retiarii.basic_unit>` to annotate a module as primitive. :func:`nni.trace <nni.common.serializer.trace>` is to enable serialization of general objects. Though they share similar underlying implementations, but do keep in mind that you will experience errors if you mix them up.
+
+    Please refer to API reference of :meth:`nni.retiarii.model_wrapper`, :meth:`nni.retiarii.basic_unit`, and :func:`nni.trace <nni.common.serializer.trace>`.