Skip to content

GitLab

"vscode:/vscode.git/clone" did not exist on "f774299be269a19217d8ad9d80f4ca2eb61de1d4"
Unverified Commit a911b856 authored by Yuge Zhang's avatar Yuge Zhang Committed by GitHub
Browse files

Resolve conflicts for #4760 (#4762)

parent 14d2966b
Hide whitespace changes
Inline Side-by-side
Showing with 1838 additions and 154 deletions
docs/source/locales/zh/LC_MESSAGES/installation.po 0 → 100644
View file @ a911b856
# SOME DESCRIPTIVE TITLE.
# Copyright (C) 2022, Microsoft
# This file is distributed under the same license as the NNI package.
# FIRST AUTHOR <EMAIL@ADDRESS>, 2022.
#
#, fuzzy
msgid ""
msgstr ""
"Project-Id-Version: NNI \n"
"Report-Msgid-Bugs-To: \n"
"POT-Creation-Date: 2022-04-13 03:14+0000\n"
"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
"Language-Team: LANGUAGE <LL@li.org>\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=utf-8\n"
"Content-Transfer-Encoding: 8bit\n"
"Generated-By: Babel 2.9.1\n"
#: ../../source/installation.rst:2
msgid "Install NNI"
msgstr ""
#: ../../source/installation.rst:4
msgid ""
"NNI requires Python >= 3.7. It is tested and supported on Ubuntu >= "
"18.04, Windows 10 >= 21H2, and macOS >= 11."
msgstr ""
#: ../../source/installation.rst:8
msgid "There are 3 ways to install NNI:"
msgstr ""
#: ../../source/installation.rst:10
msgid ":ref:`Using pip <installation-pip>`"
msgstr ""
#: ../../source/installation.rst:11
msgid ":ref:`Build source code <installation-source>`"
msgstr ""
#: ../../source/installation.rst:12
msgid ":ref:`Using Docker <installation-docker>`"
msgstr ""
#: ../../source/installation.rst:17
msgid "Using pip"
msgstr ""
#: ../../source/installation.rst:19
msgid ""
"NNI provides official packages for x86-64 CPUs. They can be installed "
"with pip:"
msgstr ""
#: ../../source/installation.rst:25
msgid "Or to upgrade to latest version:"
msgstr ""
#: ../../source/installation.rst:31
msgid "You can check installation with:"
msgstr ""
#: ../../source/installation.rst:37
msgid ""
"On Linux systems without Conda, you may encounter ``bash: nnictl: command"
" not found`` error. In this case you need to add pip script directory to "
"``PATH``:"
msgstr ""
#: ../../source/installation.rst:48
msgid "Installing from Source Code"
msgstr ""
#: ../../source/installation.rst:50
msgid "NNI hosts source code on `GitHub <https://github.com/microsoft/nni>`__."
msgstr ""
#: ../../source/installation.rst:52
msgid ""
"NNI has experimental support for ARM64 CPUs, including Apple M1. It "
"requires to install from source code."
msgstr ""
#: ../../source/installation.rst:55
msgid "See :doc:`/notes/build_from_source`."
msgstr ""
#: ../../source/installation.rst:60
msgid "Using Docker"
msgstr ""
#: ../../source/installation.rst:62
msgid ""
"NNI provides official Docker image on `Docker Hub "
"<https://hub.docker.com/r/msranni/nni>`__."
msgstr ""
#: ../../source/installation.rst:69
msgid "Installing Extra Dependencies"
msgstr ""
#: ../../source/installation.rst:71
msgid ""
"Some built-in algorithms of NNI requires extra packages. Use ``nni"
"[<algorithm-name>]`` to install their dependencies."
msgstr ""
#: ../../source/installation.rst:74
msgid ""
"For example, to install dependencies of :class:`DNGO "
"tuner<nni.algorithms.hpo.dngo_tuner.DNGOTuner>` :"
msgstr ""
#: ../../source/installation.rst:80
msgid ""
"This command will not reinstall NNI itself, even if it was installed in "
"development mode."
msgstr ""
#: ../../source/installation.rst:82
msgid "Alternatively, you may install all extra dependencies at once:"
msgstr ""
#: ../../source/installation.rst:88
msgid ""
"**NOTE**: SMAC tuner depends on swig3, which requires a manual downgrade "
"on Ubuntu:"
msgstr ""
docs/source/locales/zh/LC_MESSAGES/nas.po 0 → 100644
View file @ a911b856
# SOME DESCRIPTIVE TITLE.
# Copyright (C) 2022, Microsoft
# This file is distributed under the same license as the NNI package.
# FIRST AUTHOR <EMAIL@ADDRESS>, 2022.
#
#, fuzzy
msgid ""
msgstr ""
"Project-Id-Version: NNI \n"
"Report-Msgid-Bugs-To: \n"
"POT-Creation-Date: 2022-04-20 05:50+0000\n"
"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
"Language-Team: LANGUAGE <LL@li.org>\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=utf-8\n"
"Content-Transfer-Encoding: 8bit\n"
"Generated-By: Babel 2.9.1\n"
#: ../../source/nas/overview.rst:2
msgid "Overview"
msgstr ""
#: ../../source/nas/overview.rst:4
msgid ""
"NNI's latest NAS supports are all based on Retiarii Framework, users who "
"are still on `early version using NNI NAS v1.0 "
"<https://nni.readthedocs.io/en/v2.2/nas.html>`__ shall migrate your work "
"to Retiarii as soon as possible. We plan to remove the legacy NAS "
"framework in the next few releases."
msgstr ""
#: ../../source/nas/overview.rst:6
msgid ""
"PyTorch is the **only supported framework on Retiarii**. Inquiries of NAS"
" support on Tensorflow is in `this discussion "
"<https://github.com/microsoft/nni/discussions/4605>`__. If you intend to "
"run NAS with DL frameworks other than PyTorch and Tensorflow, please "
"`open new issues <https://github.com/microsoft/nni/issues>`__ to let us "
"know."
msgstr ""
#: ../../source/nas/overview.rst:9
msgid "Basics"
msgstr ""
#: ../../source/nas/overview.rst:11
msgid ""
"Automatic neural architecture search is playing an increasingly important"
" role in finding better models. Recent research has proven the "
"feasibility of automatic NAS and has led to models that beat many "
"manually designed and tuned models. Representative works include `NASNet "
"<https://arxiv.org/abs/1707.07012>`__, `ENAS "
"<https://arxiv.org/abs/1802.03268>`__, `DARTS "
"<https://arxiv.org/abs/1806.09055>`__, `Network Morphism "
"<https://arxiv.org/abs/1806.10282>`__, and `Evolution "
"<https://arxiv.org/abs/1703.01041>`__. In addition, new innovations "
"continue to emerge."
msgstr ""
#: ../../source/nas/overview.rst:13
msgid ""
"High-level speaking, aiming to solve any particular task with neural "
"architecture search typically requires: search space design, search "
"strategy selection, and performance evaluation. The three components work"
" together with the following loop (from the famous `NAS survey "
"<https://arxiv.org/abs/1808.05377>`__):"
msgstr ""
#: ../../source/nas/overview.rst:19
msgid "In this figure:"
msgstr ""
#: ../../source/nas/overview.rst:21
msgid ""
"*Model search space* means a set of models from which the best model is "
"explored/searched. Sometimes we use *search space* or *model space* in "
"short."
msgstr ""
#: ../../source/nas/overview.rst:22
msgid ""
"*Exploration strategy* is the algorithm that is used to explore a model "
"search space. Sometimes we also call it *search strategy*."
msgstr ""
#: ../../source/nas/overview.rst:23
msgid ""
"*Model evaluator* is responsible for training a model and evaluating its "
"performance."
msgstr ""
#: ../../source/nas/overview.rst:25
msgid ""
"The process is similar to :doc:`Hyperparameter Optimization "
"</hpo/overview>`, except that the target is the best architecture rather "
"than hyperparameter. Concretely, an exploration strategy selects an "
"architecture from a predefined search space. The architecture is passed "
"to a performance evaluation to get a score, which represents how well "
"this architecture performs on a particular task. This process is repeated"
" until the search process is able to find the best architecture."
msgstr ""
#: ../../source/nas/overview.rst:28
msgid "Key Features"
msgstr ""
#: ../../source/nas/overview.rst:30
msgid ""
"The current NAS framework in NNI is powered by the research of `Retiarii:"
" A Deep Learning Exploratory-Training Framework "
"<https://www.usenix.org/system/files/osdi20-zhang_quanlu.pdf>`__, where "
"we highlight the following features:"
msgstr ""
#: ../../source/nas/overview.rst:32
msgid ":doc:`Simple APIs to construct search space easily <construct_space>`"
msgstr ""
#: ../../source/nas/overview.rst:33
msgid ":doc:`SOTA NAS algorithms to explore search space <exploration_strategy>`"
msgstr ""
#: ../../source/nas/overview.rst:34
msgid ""
":doc:`Experiment backend support to scale up experiments on large-scale "
"AI platforms </experiment/overview>`"
msgstr ""
#: ../../source/nas/overview.rst:37
msgid "Why NAS with NNI"
msgstr ""
#: ../../source/nas/overview.rst:39
msgid ""
"We list out the three perspectives where NAS can be particularly "
"challegning without NNI. NNI provides solutions to relieve users' "
"engineering effort when they want to try NAS techniques in their own "
"scenario."
msgstr ""
#: ../../source/nas/overview.rst:42
msgid "Search Space Design"
msgstr ""
#: ../../source/nas/overview.rst:44
msgid ""
"The search space defines which architectures can be represented in "
"principle. Incorporating prior knowledge about typical properties of "
"architectures well-suited for a task can reduce the size of the search "
"space and simplify the search. However, this also introduces a human "
"bias, which may prevent finding novel architectural building blocks that "
"go beyond the current human knowledge. Search space design can be very "
"challenging for beginners, who might not possess the experience to "
"balance the richness and simplicity."
msgstr ""
#: ../../source/nas/overview.rst:46
msgid ""
"In NNI, we provide a wide range of APIs to build the search space. There "
"are :doc:`high-level APIs <construct_space>`, that enables the "
"possibility to incorporate human knowledge about what makes a good "
"architecture or search space. There are also :doc:`low-level APIs "
"<mutator>`, that is a list of primitives to construct a network from "
"operation to operation."
msgstr ""
#: ../../source/nas/overview.rst:49
msgid "Exploration strategy"
msgstr ""
#: ../../source/nas/overview.rst:51
msgid ""
"The exploration strategy details how to explore the search space (which "
"is often exponentially large). It encompasses the classical exploration-"
"exploitation trade-off since, on the one hand, it is desirable to find "
"well-performing architectures quickly, while on the other hand, premature"
" convergence to a region of suboptimal architectures should be avoided. "
"The \"best\" exploration strategy for a particular scenario is usually "
"found via trial-and-error. As many state-of-the-art strategies are "
"implemented with their own code-base, it becomes very troublesome to "
"switch from one to another."
msgstr ""
#: ../../source/nas/overview.rst:53
msgid ""
"In NNI, we have also provided :doc:`a list of strategies "
"<exploration_strategy>`. Some of them are powerful yet time consuming, "
"while others might be suboptimal but really efficient. Given that all "
"strategies are implemented with a unified interface, users can always "
"find one that matches their need."
msgstr ""
#: ../../source/nas/overview.rst:56
msgid "Performance estimation"
msgstr ""
#: ../../source/nas/overview.rst:58
msgid ""
"The objective of NAS is typically to find architectures that achieve high"
" predictive performance on unseen data. Performance estimation refers to "
"the process of estimating this performance. The problem with performance "
"estimation is mostly its scalability, i.e., how can I run and manage "
"multiple trials simultaneously."
msgstr ""
#: ../../source/nas/overview.rst:60
msgid ""
"In NNI, we standardize this process is implemented with :doc:`evaluator "
"<evaluator>`, which is responsible of estimating a model's performance. "
"NNI has quite a few built-in supports of evaluators, ranging from the "
"simplest option, e.g., to perform a standard training and validation of "
"the architecture on data, to complex configurations and implementations. "
"Evaluators are run in *trials*, where trials can be spawn onto "
"distributed platforms with our powerful :doc:`training service "
"</experiment/training_service/overview>`."
msgstr ""
#: ../../source/nas/overview.rst:63
msgid "Tutorials"
msgstr ""
#: ../../source/nas/overview.rst:65
msgid ""
"To start using NNI NAS framework, we recommend at least going through the"
" following tutorials:"
msgstr ""
#: ../../source/nas/overview.rst:67
msgid ":doc:`Quickstart </tutorials/hello_nas>`"
msgstr ""
#: ../../source/nas/overview.rst:68
msgid ":doc:`construct_space`"
msgstr ""
#: ../../source/nas/overview.rst:69
msgid ":doc:`exploration_strategy`"
msgstr ""
#: ../../source/nas/overview.rst:70
msgid ":doc:`evaluator`"
msgstr ""
#: ../../source/nas/overview.rst:73
msgid "Resources"
msgstr ""
#: ../../source/nas/overview.rst:75
msgid ""
"The following articles will help with a better understanding of the "
"current arts of NAS:"
msgstr ""
#: ../../source/nas/overview.rst:77
msgid ""
"`Neural Architecture Search: A Survey "
"<https://arxiv.org/abs/1808.05377>`__"
msgstr ""
#: ../../source/nas/overview.rst:78
msgid ""
"`A Comprehensive Survey of Neural Architecture Search: Challenges and "
"Solutions <https://arxiv.org/abs/2006.02903>`__"
msgstr ""
#~ msgid "Basics"
#~ msgstr ""
#~ msgid "Basic Concepts"
#~ msgstr ""
#~ msgid ""
#~ "The process is similar to "
#~ ":doc:`Hyperparameter Optimization </hpo/index>`, "
#~ "except that the target is the best"
#~ " architecture rather than hyperparameter. "
#~ "Concretely, an exploration strategy selects"
#~ " an architecture from a predefined "
#~ "search space. The architecture is passed"
#~ " to a performance evaluation to get"
#~ " a score, which represents how well"
#~ " this architecture performs on a "
#~ "particular task. This process is "
#~ "repeated until the search process is "
#~ "able to find the best architecture."
#~ msgstr ""
#~ msgid ""
#~ "In NNI, we provide a wide range"
#~ " of APIs to build the search "
#~ "space. There are :doc:`high-level APIs"
#~ " <construct_space>`, that enables incorporating"
#~ " human knowledge about what makes a"
#~ " good architecture or search space. "
#~ "There are also :doc:`low-level APIs "
#~ "<mutator>`, that is a list of "
#~ "primitives to construct a network from"
#~ " operator to operator."
#~ msgstr ""
#~ msgid ""
#~ "In NNI, we standardize this process "
#~ "is implemented with :doc:`evaluator "
#~ "<evaluator>`, which is responsible of "
#~ "estimating a model's performance. The "
#~ "choices of evaluators also range from"
#~ " the simplest option, e.g., to "
#~ "perform a standard training and "
#~ "validation of the architecture on data,"
#~ " to complex configurations and "
#~ "implementations. Evaluators are run in "
#~ "*trials*, where trials can be spawn "
#~ "onto distributed platforms with our "
#~ "powerful :doc:`training service "
#~ "</experiment/training_service/overview>`."
#~ msgstr ""
docs/source/locales/zh/LC_MESSAGES/quickstart.po 0 → 100644
View file @ a911b856
# SOME DESCRIPTIVE TITLE.
# Copyright (C) 2022, Microsoft
# This file is distributed under the same license as the NNI package.
# FIRST AUTHOR <EMAIL@ADDRESS>, 2022.
#
#, fuzzy
msgid ""
msgstr ""
"Project-Id-Version: NNI \n"
"Report-Msgid-Bugs-To: \n"
"POT-Creation-Date: 2022-04-13 03:14+0000\n"
"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
"Language-Team: LANGUAGE <LL@li.org>\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=utf-8\n"
"Content-Transfer-Encoding: 8bit\n"
"Generated-By: Babel 2.9.1\n"
#: ../../source/quickstart.rst:2
msgid "Quickstart"
msgstr ""
docs/source/locales/zh/LC_MESSAGES/sphinx.po 0 → 100644
View file @ a911b856
# SOME DESCRIPTIVE TITLE.
# Copyright (C) 2022, Microsoft
# This file is distributed under the same license as the NNI package.
# FIRST AUTHOR <EMAIL@ADDRESS>, 2022.
#
#, fuzzy
msgid ""
msgstr ""
"Project-Id-Version: NNI \n"
"Report-Msgid-Bugs-To: \n"
"POT-Creation-Date: 2022-04-12 17:35+0000\n"
"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
"Language-Team: LANGUAGE <LL@li.org>\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=utf-8\n"
"Content-Transfer-Encoding: 8bit\n"
"Generated-By: Babel 2.9.1\n"
#: ../../templates/globaltoc.html:6
msgid "Overview"
msgstr ""
docs/source/locales/zh/LC_MESSAGES/tutorials.po 0 → 100644
View file @ a911b856
# SOME DESCRIPTIVE TITLE.
# Copyright (C) 2022, Microsoft
# This file is distributed under the same license as the NNI package.
# FIRST AUTHOR <EMAIL@ADDRESS>, 2022.
#
#, fuzzy
msgid ""
msgstr ""
"Project-Id-Version: NNI \n"
"Report-Msgid-Bugs-To: \n"
"POT-Creation-Date: 2022-04-20 05:50+0000\n"
"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
"Language-Team: LANGUAGE <LL@li.org>\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=utf-8\n"
"Content-Transfer-Encoding: 8bit\n"
"Generated-By: Babel 2.9.1\n"
#: ../../source/tutorials/hello_nas.rst:13
msgid ""
"Click :ref:`here <sphx_glr_download_tutorials_hello_nas.py>` to download "
"the full example code"
msgstr ""
#: ../../source/tutorials/hello_nas.rst:22
msgid "Hello, NAS!"
msgstr ""
#: ../../source/tutorials/hello_nas.rst:24
msgid ""
"This is the 101 tutorial of Neural Architecture Search (NAS) on NNI. In "
"this tutorial, we will search for a neural architecture on MNIST dataset "
"with the help of NAS framework of NNI, i.e., *Retiarii*. We use multi-"
"trial NAS as an example to show how to construct and explore a model "
"space."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:28
msgid ""
"There are mainly three crucial components for a neural architecture "
"search task, namely,"
msgstr ""
#: ../../source/tutorials/hello_nas.rst:30
msgid "Model search space that defines a set of models to explore."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:31
msgid "A proper strategy as the method to explore this model space."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:32
msgid ""
"A model evaluator that reports the performance of every model in the "
"space."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:34
msgid ""
"Currently, PyTorch is the only supported framework by Retiarii, and we "
"have only tested **PyTorch 1.7 to 1.10**. This tutorial assumes PyTorch "
"context but it should also apply to other frameworks, which is in our "
"future plan."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:38
msgid "Define your Model Space"
msgstr ""
#: ../../source/tutorials/hello_nas.rst:40
msgid ""
"Model space is defined by users to express a set of models that users "
"want to explore, which contains potentially good-performing models. In "
"this framework, a model space is defined with two parts: a base model and"
" possible mutations on the base model."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:46
msgid "Define Base Model"
msgstr ""
#: ../../source/tutorials/hello_nas.rst:48
msgid ""
"Defining a base model is almost the same as defining a PyTorch (or "
"TensorFlow) model. Usually, you only need to replace the code ``import "
"torch.nn as nn`` with ``import nni.retiarii.nn.pytorch as nn`` to use our"
" wrapped PyTorch modules."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:52
msgid "Below is a very simple example of defining a base model."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:93
msgid ""
"Always keep in mind that you should use ``import nni.retiarii.nn.pytorch "
"as nn`` and :meth:`nni.retiarii.model_wrapper`. Many mistakes are a "
"result of forgetting one of those. Also, please use ``torch.nn`` for "
"submodules of ``nn.init``, e.g., ``torch.nn.init`` instead of "
"``nn.init``."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:98
msgid "Define Model Mutations"
msgstr ""
#: ../../source/tutorials/hello_nas.rst:100
msgid ""
"A base model is only one concrete model not a model space. We provide "
":doc:`API and Primitives </nas/construct_space>` for users to express how"
" the base model can be mutated. That is, to build a model space which "
"includes many models."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:103
msgid "Based on the above base model, we can define a model space as below."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:134
msgid "This results in the following code:"
msgstr ""
#: ../../source/tutorials/hello_nas.rst:189
#: ../../source/tutorials/hello_nas.rst:259
#: ../../source/tutorials/hello_nas.rst:471
#: ../../source/tutorials/hello_nas.rst:564
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:244
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:281
#: ../../source/tutorials/pruning_quick_start_mnist.rst:65
#: ../../source/tutorials/pruning_quick_start_mnist.rst:107
#: ../../source/tutorials/pruning_quick_start_mnist.rst:172
#: ../../source/tutorials/pruning_quick_start_mnist.rst:218
#: ../../source/tutorials/pruning_quick_start_mnist.rst:255
#: ../../source/tutorials/pruning_quick_start_mnist.rst:283
msgid "Out:"
msgstr ""
#: ../../source/tutorials/hello_nas.rst:210
msgid ""
"This example uses two mutation APIs, :class:`nn.LayerChoice "
"<nni.retiarii.nn.pytorch.LayerChoice>` and :class:`nn.InputChoice "
"<nni.retiarii.nn.pytorch.ValueChoice>`. :class:`nn.LayerChoice "
"<nni.retiarii.nn.pytorch.LayerChoice>` takes a list of candidate modules "
"(two in this example), one will be chosen for each sampled model. It can "
"be used like normal PyTorch module. :class:`nn.InputChoice "
"<nni.retiarii.nn.pytorch.ValueChoice>` takes a list of candidate values, "
"one will be chosen to take effect for each sampled model."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:219
msgid ""
"More detailed API description and usage can be found :doc:`here "
"</nas/construct_space>`."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:223
msgid ""
"We are actively enriching the mutation APIs, to facilitate easy "
"construction of model space. If the currently supported mutation APIs "
"cannot express your model space, please refer to :doc:`this doc "
"</nas/mutator>` for customizing mutators."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:228
msgid "Explore the Defined Model Space"
msgstr ""
#: ../../source/tutorials/hello_nas.rst:230
msgid ""
"There are basically two exploration approaches: (1) search by evaluating "
"each sampled model independently, which is the search approach in :ref"
":`multi-trial NAS <multi-trial-nas>` and (2) one-shot weight-sharing "
"based search, which is used in one-shot NAS. We demonstrate the first "
"approach in this tutorial. Users can refer to :ref:`here <one-shot-nas>` "
"for the second approach."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:235
msgid ""
"First, users need to pick a proper exploration strategy to explore the "
"defined model space. Second, users need to pick or customize a model "
"evaluator to evaluate the performance of each explored model."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:239
msgid "Pick an exploration strategy"
msgstr ""
#: ../../source/tutorials/hello_nas.rst:241
msgid ""
"Retiarii supports many :doc:`exploration strategies "
"</nas/exploration_strategy>`."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:243
msgid "Simply choosing (i.e., instantiate) an exploration strategy as below."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:273
msgid "Pick or customize a model evaluator"
msgstr ""
#: ../../source/tutorials/hello_nas.rst:275
msgid ""
"In the exploration process, the exploration strategy repeatedly generates"
" new models. A model evaluator is for training and validating each "
"generated model to obtain the model's performance. The performance is "
"sent to the exploration strategy for the strategy to generate better "
"models."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:279
msgid ""
"Retiarii has provided :doc:`built-in model evaluators </nas/evaluator>`, "
"but to start with, it is recommended to use :class:`FunctionalEvaluator "
"<nni.retiarii.evaluator.FunctionalEvaluator>`, that is, to wrap your own "
"training and evaluation code with one single function. This function "
"should receive one single model class and uses "
":func:`nni.report_final_result` to report the final score of this model."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:284
msgid ""
"An example here creates a simple evaluator that runs on MNIST dataset, "
"trains for 2 epochs, and reports its validation accuracy."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:367
msgid "Create the evaluator"
msgstr ""
#: ../../source/tutorials/hello_nas.rst:386
msgid ""
"The ``train_epoch`` and ``test_epoch`` here can be any customized "
"function, where users can write their own training recipe."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:389
msgid ""
"It is recommended that the ``evaluate_model`` here accepts no additional "
"arguments other than ``model_cls``. However, in the :doc:`advanced "
"tutorial </nas/evaluator>`, we will show how to use additional arguments "
"in case you actually need those. In future, we will support mutation on "
"the arguments of evaluators, which is commonly called \"Hyper-parmeter "
"tuning\"."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:394
msgid "Launch an Experiment"
msgstr ""
#: ../../source/tutorials/hello_nas.rst:396
msgid ""
"After all the above are prepared, it is time to start an experiment to do"
" the model search. An example is shown below."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:417
msgid ""
"The following configurations are useful to control how many trials to run"
" at most / at the same time."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:436
msgid ""
"Remember to set the following config if you want to GPU. "
"``use_active_gpu`` should be set true if you wish to use an occupied GPU "
"(possibly running a GUI)."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:456
msgid ""
"Launch the experiment. The experiment should take several minutes to "
"finish on a workstation with 2 GPUs."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:495
msgid ""
"Users can also run Retiarii Experiment with :doc:`different training "
"services </experiment/training_service/overview>` besides ``local`` "
"training service."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:499
msgid "Visualize the Experiment"
msgstr ""
#: ../../source/tutorials/hello_nas.rst:501
msgid ""
"Users can visualize their experiment in the same way as visualizing a "
"normal hyper-parameter tuning experiment. For example, open "
"``localhost:8081`` in your browser, 8081 is the port that you set in "
"``exp.run``. Please refer to :doc:`here "
"</experiment/web_portal/web_portal>` for details."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:505
msgid ""
"We support visualizing models with 3rd-party visualization engines (like "
"`Netron <https://netron.app/>`__). This can be used by clicking "
"``Visualization`` in detail panel for each trial. Note that current "
"visualization is based on `onnx <https://onnx.ai/>`__ , thus "
"visualization is not feasible if the model cannot be exported into onnx."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:510
msgid ""
"Built-in evaluators (e.g., Classification) will automatically export the "
"model into a file. For your own evaluator, you need to save your file "
"into ``$NNI_OUTPUT_DIR/model.onnx`` to make this work. For instance,"
msgstr ""
#: ../../source/tutorials/hello_nas.rst:541
msgid "Relaunch the experiment, and a button is shown on Web portal."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:546
msgid "Export Top Models"
msgstr ""
#: ../../source/tutorials/hello_nas.rst:548
msgid ""
"Users can export top models after the exploration is done using "
"``export_top_models``."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:575
msgid ""
"The output is ``json`` object which records the mutation actions of the "
"top model. If users want to output source code of the top model, they can"
" use :ref:`graph-based execution engine <graph-based-execution-engine>` "
"for the experiment, by simply adding the following two lines."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:597
msgid "**Total running time of the script:** ( 2 minutes 4.499 seconds)"
msgstr ""
#: ../../source/tutorials/hello_nas.rst:612
msgid ":download:`Download Python source code: hello_nas.py <hello_nas.py>`"
msgstr ""
#: ../../source/tutorials/hello_nas.rst:618
msgid ":download:`Download Jupyter notebook: hello_nas.ipynb <hello_nas.ipynb>`"
msgstr ""
#: ../../source/tutorials/hello_nas.rst:625
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:335
#: ../../source/tutorials/pruning_quick_start_mnist.rst:357
msgid "`Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_"
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:13
msgid ""
"Click :ref:`here "
"<sphx_glr_download_tutorials_hpo_quickstart_pytorch_main.py>` to download"
" the full example code"
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:22
msgid "HPO Quickstart with PyTorch"
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:23
msgid ""
"This tutorial optimizes the model in `official PyTorch quickstart`_ with "
"auto-tuning."
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:25
msgid "The tutorial consists of 4 steps:"
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:27
msgid "Modify the model for auto-tuning."
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:28
msgid "Define hyperparameters' search space."
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:29
msgid "Configure the experiment."
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:30
msgid "Run the experiment."
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:37
msgid "Step 1: Prepare the model"
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:38
msgid "In first step, we need to prepare the model to be tuned."
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:40
msgid ""
"The model should be put in a separate script. It will be evaluated many "
"times concurrently, and possibly will be trained on distributed "
"platforms."
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:44
msgid "In this tutorial, the model is defined in :doc:`model.py <model>`."
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:46
msgid "In short, it is a PyTorch model with 3 additional API calls:"
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:48
msgid ""
"Use :func:`nni.get_next_parameter` to fetch the hyperparameters to be "
"evalutated."
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:49
msgid ""
"Use :func:`nni.report_intermediate_result` to report per-epoch accuracy "
"metrics."
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:50
msgid "Use :func:`nni.report_final_result` to report final accuracy."
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:52
msgid "Please understand the model code before continue to next step."
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:57
msgid "Step 2: Define search space"
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:58
msgid ""
"In model code, we have prepared 3 hyperparameters to be tuned: "
"*features*, *lr*, and *momentum*."
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:61
msgid ""
"Here we need to define their *search space* so the tuning algorithm can "
"sample them in desired range."
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:63
msgid "Assuming we have following prior knowledge for these hyperparameters:"
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:65
msgid "*features* should be one of 128, 256, 512, 1024."
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:66
msgid ""
"*lr* should be a float between 0.0001 and 0.1, and it follows exponential"
" distribution."
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:67
msgid "*momentum* should be a float between 0 and 1."
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:69
msgid ""
"In NNI, the space of *features* is called ``choice``; the space of *lr* "
"is called ``loguniform``; and the space of *momentum* is called "
"``uniform``. You may have noticed, these names are derived from "
"``numpy.random``."
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:74
msgid ""
"For full specification of search space, check :doc:`the reference "
"</hpo/search_space>`."
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:76
msgid "Now we can define the search space as follow:"
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:99
msgid "Step 3: Configure the experiment"
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:100
msgid ""
"NNI uses an *experiment* to manage the HPO process. The *experiment "
"config* defines how to train the models and how to explore the search "
"space."
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:103
msgid ""
"In this tutorial we use a *local* mode experiment, which means models "
"will be trained on local machine, without using any special training "
"platform."
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:122
msgid "Now we start to configure the experiment."
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:125
msgid "Configure trial code"
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:126
msgid ""
"In NNI evaluation of each hyperparameter set is called a *trial*. So the "
"model script is called *trial code*."
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:144
msgid ""
"When ``trial_code_directory`` is a relative path, it relates to current "
"working directory. To run ``main.py`` in a different path, you can set "
"trial code directory to ``Path(__file__).parent``. (`__file__ "
"<https://docs.python.org/3.10/reference/datamodel.html#index-43>`__ is "
"only available in standard Python, not in Jupyter Notebook.)"
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:151
msgid ""
"If you are using Linux system without Conda, you may need to change "
"``\"python model.py\"`` to ``\"python3 model.py\"``."
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:157
msgid "Configure search space"
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:175
msgid "Configure tuning algorithm"
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:176
msgid "Here we use :doc:`TPE tuner </hpo/tuners>`."
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:195
msgid "Configure how many trials to run"
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:196
msgid ""
"Here we evaluate 10 sets of hyperparameters in total, and concurrently "
"evaluate 2 sets at a time."
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:213
msgid "You may also set ``max_experiment_duration = '1h'`` to limit running time."
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:215
msgid ""
"If neither ``max_trial_number`` nor ``max_experiment_duration`` are set, "
"the experiment will run forever until you press Ctrl-C."
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:220
msgid ""
"``max_trial_number`` is set to 10 here for a fast example. In real world "
"it should be set to a larger number. With default config TPE tuner "
"requires 20 trials to warm up."
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:227
msgid "Step 4: Run the experiment"
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:228
msgid ""
"Now the experiment is ready. Choose a port and launch it. (Here we use "
"port 8080.)"
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:230
msgid ""
"You can use the web portal to view experiment status: "
"http://localhost:8080."
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:260
msgid "After the experiment is done"
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:261
msgid "Everything is done and it is safe to exit now. The following are optional."
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:263
msgid ""
"If you are using standard Python instead of Jupyter Notebook, you can add"
" ``input()`` or ``signal.pause()`` to prevent Python from exiting, "
"allowing you to view the web portal after the experiment is done."
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:293
msgid ""
":meth:`nni.experiment.Experiment.stop` is automatically invoked when "
"Python exits, so it can be omitted in your code."
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:296
msgid ""
"After the experiment is stopped, you can run "
":meth:`nni.experiment.Experiment.view` to restart web portal."
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:300
msgid ""
"This example uses :doc:`Python API </reference/experiment>` to create "
"experiment."
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:302
msgid ""
"You can also create and manage experiments with :doc:`command line tool "
"<../hpo_nnictl/nnictl>`."
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:307
msgid "**Total running time of the script:** ( 1 minutes 24.367 seconds)"
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:322
msgid ":download:`Download Python source code: main.py <main.py>`"
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:328
msgid ":download:`Download Jupyter notebook: main.ipynb <main.ipynb>`"
msgstr ""
#: ../../source/tutorials/pruning_quick_start_mnist.rst:13
msgid ""
"Click :ref:`here "
"<sphx_glr_download_tutorials_pruning_quick_start_mnist.py>` to download "
"the full example code"
msgstr ""
#: ../../source/tutorials/pruning_quick_start_mnist.rst:22
msgid "Pruning Quickstart"
msgstr ""
#: ../../source/tutorials/pruning_quick_start_mnist.rst:24
msgid ""
"Model pruning is a technique to reduce the model size and computation by "
"reducing model weight size or intermediate state size. There are three "
"common practices for pruning a DNN model:"
msgstr ""
#: ../../source/tutorials/pruning_quick_start_mnist.rst:27
msgid "Pre-training a model -> Pruning the model -> Fine-tuning the pruned model"
msgstr ""
#: ../../source/tutorials/pruning_quick_start_mnist.rst:28
msgid ""
"Pruning a model during training (i.e., pruning aware training) -> Fine-"
"tuning the pruned model"
msgstr ""
#: ../../source/tutorials/pruning_quick_start_mnist.rst:29
msgid "Pruning a model -> Training the pruned model from scratch"
msgstr ""
#: ../../source/tutorials/pruning_quick_start_mnist.rst:31
msgid ""
"NNI supports all of the above pruning practices by working on the key "
"pruning stage. Following this tutorial for a quick look at how to use NNI"
" to prune a model in a common practice."
msgstr ""
#: ../../source/tutorials/pruning_quick_start_mnist.rst:37
msgid "Preparation"
msgstr ""
#: ../../source/tutorials/pruning_quick_start_mnist.rst:39
msgid ""
"In this tutorial, we use a simple model and pre-trained on MNIST dataset."
" If you are familiar with defining a model and training in pytorch, you "
"can skip directly to `Pruning Model`_."
msgstr ""
#: ../../source/tutorials/pruning_quick_start_mnist.rst:121
msgid "Pruning Model"
msgstr ""
#: ../../source/tutorials/pruning_quick_start_mnist.rst:123
msgid ""
"Using L1NormPruner to prune the model and generate the masks. Usually, a "
"pruner requires original model and ``config_list`` as its inputs. "
"Detailed about how to write ``config_list`` please refer "
":doc:`compression config specification "
"<../compression/compression_config_list>`."
msgstr ""
#: ../../source/tutorials/pruning_quick_start_mnist.rst:127
msgid ""
"The following `config_list` means all layers whose type is `Linear` or "
"`Conv2d` will be pruned, except the layer named `fc3`, because `fc3` is "
"`exclude`. The final sparsity ratio for each layer is 50%. The layer "
"named `fc3` will not be pruned."
msgstr ""
#: ../../source/tutorials/pruning_quick_start_mnist.rst:153
msgid "Pruners usually require `model` and `config_list` as input arguments."
msgstr ""
#: ../../source/tutorials/pruning_quick_start_mnist.rst:232
msgid ""
"Speedup the original model with masks, note that `ModelSpeedup` requires "
"an unwrapped model. The model becomes smaller after speedup, and reaches "
"a higher sparsity ratio because `ModelSpeedup` will propagate the masks "
"across layers."
msgstr ""
#: ../../source/tutorials/pruning_quick_start_mnist.rst:269
msgid "the model will become real smaller after speedup"
msgstr ""
#: ../../source/tutorials/pruning_quick_start_mnist.rst:307
msgid "Fine-tuning Compacted Model"
msgstr ""
#: ../../source/tutorials/pruning_quick_start_mnist.rst:308
msgid ""
"Note that if the model has been sped up, you need to re-initialize a new "
"optimizer for fine-tuning. Because speedup will replace the masked big "
"layers with dense small ones."
msgstr ""
#: ../../source/tutorials/pruning_quick_start_mnist.rst:329
msgid "**Total running time of the script:** ( 0 minutes 58.337 seconds)"
msgstr ""
#: ../../source/tutorials/pruning_quick_start_mnist.rst:344
msgid ""
":download:`Download Python source code: pruning_quick_start_mnist.py "
"<pruning_quick_start_mnist.py>`"
msgstr ""
#: ../../source/tutorials/pruning_quick_start_mnist.rst:350
msgid ""
":download:`Download Jupyter notebook: pruning_quick_start_mnist.ipynb "
"<pruning_quick_start_mnist.ipynb>`"
msgstr ""
#~ msgid "**Total running time of the script:** ( 2 minutes 15.810 seconds)"
#~ msgstr ""
#~ msgid "NNI HPO Quickstart with PyTorch"
#~ msgstr ""
#~ msgid ""
#~ "There is also a :doc:`TensorFlow "
#~ "version<../hpo_quickstart_tensorflow/main>` if you "
#~ "prefer it."
#~ msgstr ""
#~ msgid ""
#~ "You can also create and manage "
#~ "experiments with :doc:`command line tool "
#~ "</reference/nnictl>`."
#~ msgstr ""
#~ msgid "**Total running time of the script:** ( 1 minutes 24.393 seconds)"
#~ msgstr ""
docs/source/model_compression.rst deleted 100644 → 0
View file @ 14d2966b
#################
Model Compression
#################
Deep neural networks (DNNs) have achieved great success in many tasks. However, typical neural networks are both
computationally expensive and energy intensive, can be difficult to be deployed on devices with low computation
resources or with strict latency requirements. Therefore, a natural thought is to perform model compression to
reduce model size and accelerate model training/inference without losing performance significantly. Model compression
techniques can be divided into two categories: pruning and quantization. The pruning methods explore the redundancy
in the model weights and try to remove/prune the redundant and uncritical weights. Quantization refers to compressing
models by reducing the number of bits required to represent weights or activations.
NNI provides an easy-to-use toolkit to help user design and use model pruning and quantization algorithms.
It supports Tensorflow and PyTorch with unified interface.
For users to compress their models, they only need to add several lines in their code.
There are some popular model compression algorithms built-in in NNI.
Users could further use NNI's auto tuning power to find the best compressed model,
which is detailed in Auto Model Compression.
On the other hand, users could easily customize their new compression algorithms using NNI's interface.
For details, please refer to the following tutorials:
.. toctree::
:maxdepth: 2
Overview <Compression/Overview>
Quick Start <Compression/QuickStart>
Tutorial <Compression/Tutorial>
Pruning <Compression/pruning>
Pruning V2 <Compression/v2_pruning>
Quantization <Compression/quantization>
Utilities <Compression/CompressionUtils>
Advanced Usage <Compression/advanced>
API Reference <Compression/CompressionReference>
docs/source/model_compression_zh.rst deleted 100644 → 0
View file @ 14d2966b
.. da97b4cdd507bd8fad43d640f3d2bfef
#################
模型压缩
#################
深度神经网络（DNNs）在许多领域都取得了巨大的成功。 然而，典型的神经网络是
计算和能源密集型的，很难将其部署在计算资源匮乏
或具有严格延迟要求的设备上。 因此，一个自然的想法就是对模型进行压缩
以减小模型大小并加速模型训练/推断，同时不会显着降低模型性能。 模型压缩
技术可以分为两类：剪枝和量化。 剪枝方法探索模型权重中的冗余，
并尝试删除/修剪冗余和非关键的权重。 量化是指通过减少
权重表示或激活所需的比特数来压缩模型。
NNI 提供了易于使用的工具包来帮助用户设计并使用剪枝和量化算法。
其使用了统一的接口来支持 TensorFlow 和 PyTorch。
对用户来说， 只需要添加几行代码即可压缩模型。
NNI 中也内置了一些主流的模型压缩算法。
用户可以进一步利用 NNI 的自动调优功能找到最佳的压缩模型，
该功能在自动模型压缩部分有详细介绍。
另一方面，用户可以使用 NNI 的接口自定义新的压缩算法。
详细信息，参考以下教程：
.. toctree::
:maxdepth: 2
概述 <Compression/Overview>
快速入门 <Compression/QuickStart>
教程 <Compression/Tutorial>
剪枝 <Compression/pruning>
剪枝（V2版本） <Compression/v2_pruning>
量化 <Compression/quantization>
工具 <Compression/CompressionUtils>
高级用法 <Compression/advanced>
API 参考 <Compression/CompressionReference>
docs/source/nas.rst deleted 100644 → 0
View file @ 14d2966b
#############################################
Retiarii for Neural Architecture Search (NAS)
#############################################
Automatic neural architecture search is taking an increasingly important role on finding better models.
Recent research works have proved the feasibility of automatic NAS, and also found some models that could beat manually tuned models.
Some of representative works are NASNet, ENAS, DARTS, Network Morphism, and Evolution. Moreover, new innovations keep emerging.
However, it takes great efforts to implement NAS algorithms, and it is hard to reuse code base of existing algorithms in a new one.
To facilitate NAS innovations (e.g., design and implement new NAS models, compare different NAS models side-by-side),
an easy-to-use and flexible programming interface is crucial.
Thus, we design `Retiarii <https://www.usenix.org/system/files/osdi20-zhang_quanlu.pdf>`__. It is a deep learning framework that supports the exploratory training on a neural network model space, rather than on a single neural network model.
Exploratory training with Retiarii allows user to express various search spaces for *Neural Architecture Search* and *Hyper-Parameter Tuning* with high flexibility.
Some frequently used terminologies in this document:
* *Model search space*: it means a set of models from which the best model is explored/searched. Sometimes we use *search space* or *model space* in short.
* *Exploration strategy*: the algorithm that is used to explore a model search space.
* *Model evaluator*: it is used to train a model and evaluate the model's performance.
Follow the instructions below to start your journey with Retiarii.
.. toctree::
:maxdepth: 2
Overview <NAS/Overview>
Quick Start <NAS/QuickStart>
Construct Model Space <NAS/construct_space>
Multi-trial NAS <NAS/multi_trial_nas>
One-shot NAS <NAS/one_shot_nas>
Hardware-aware NAS <NAS/HardwareAwareNAS>
NAS Benchmarks <NAS/Benchmarks>
NAS API References <NAS/ApiReference>
docs/source/Compression/advanced.rst → docs/source/nas/advanced_usage.rst
View file @ a911b856
......@@ -4,6 +4,9 @@ Advanced Usage
.. toctree::
:maxdepth: 2
Framework <./Framework>
Customize a new algorithm <./CustomizeCompressor>
Automatic Model Compression (Beta) <./AutoCompression>
execution_engine
hardware_aware_nas
mutator
customize_strategy
serialization
benchmarks_toctree
docs/source/NAS/Benchmarks.rst → docs/source/nas/benchmarks.rst
View file @ a911b856
NAS Benchmarks
==============
NAS Benchmark
=============
.. code-block::
.. note:: :doc:`Example usage of NAS benchmarks </tutorials/nasbench_as_dataset>`.
.. toctree::
:hidden:
Example Usages <BenchmarksExample>
Introduction
------------
To improve the reproducibility of NAS algorithms as well as reducing computing resource requirements, researchers proposed a series of NAS benchmarks such as `NAS-Bench-101 <https://arxiv.org/abs/1902.09635>`__\ , `NAS-Bench-201 <https://arxiv.org/abs/2001.00326>`__\ , `NDS <https://arxiv.org/abs/1905.13214>`__\ , etc. NNI provides a query interface for users to acquire these benchmarks. Within just a few lines of code, researcher are able to evaluate their NAS algorithms easily and fairly by utilizing these benchmarks.
To improve the reproducibility of NAS algorithms as well as reducing computing resource requirements, researchers proposed a series of NAS benchmarks such as `NAS-Bench-101 <https://arxiv.org/abs/1902.09635>`__, `NAS-Bench-201 <https://arxiv.org/abs/2001.00326>`__, `NDS <https://arxiv.org/abs/1905.13214>`__, etc. NNI provides a query interface for users to acquire these benchmarks. Within just a few lines of code, researcher are able to evaluate their NAS algorithms easily and fairly by utilizing these benchmarks.
Prerequisites
-------------
* Please prepare a folder to household all the benchmark databases. By default, it can be found at ``${HOME}/.cache/nni/nasbenchmark``. Or you can place it anywhere you like, and specify it in ``NASBENCHMARK_DIR`` via ``export NASBENCHMARK_DIR=/path/to/your/nasbenchmark`` before importing NNI.
* Please install ``peewee`` via ``pip3 install peewee``\ , which NNI uses to connect to database.
* Please install ``peewee`` via ``pip3 install peewee``, which NNI uses to connect to database.
Data Preparation
----------------
......@@ -33,8 +24,7 @@ Option 2
.. note:: If you have files that are processed before v2.5, it is recommended that you delete them and try option 1.
#.
Clone NNI to your machine and enter ``examples/nas/benchmarks`` directory.
#. Clone NNI to your machine and enter ``examples/nas/benchmarks`` directory.
.. code-block:: bash
......@@ -43,8 +33,7 @@ Option 2
Replace ``${NNI_VERSION}`` with a released version name or branch name, e.g., ``v2.4``.
#.
Install dependencies via ``pip3 install -r xxx.requirements.txt``. ``xxx`` can be ``nasbench101``\ , ``nasbench201`` or ``nds``.
#. Install dependencies via ``pip3 install -r xxx.requirements.txt``. ``xxx`` can be ``nasbench101``, ``nasbench201`` or ``nds``.
#. Generate the database via ``./xxx.sh``. The directory that stores the benchmark file can be configured with ``NASBENCHMARK_DIR`` environment variable, which defaults to ``~/.nni/nasbenchmark``. Note that the NAS-Bench-201 dataset will be downloaded from a google drive.
......@@ -53,7 +42,7 @@ Please make sure there is at least 10GB free disk space and note that the conver
Example Usages
--------------
Please refer to `examples usages of Benchmarks API <./BenchmarksExample.rst>`__.
Please refer to :doc:`examples usages of Benchmarks API </tutorials/nasbench_as_dataset>`.
NAS-Bench-101
-------------
......@@ -61,36 +50,11 @@ NAS-Bench-101
* `Paper link <https://arxiv.org/abs/1902.09635>`__
* `Open-source <https://github.com/google-research/nasbench>`__
NAS-Bench-101 contains 423,624 unique neural networks, combined with 4 variations in number of epochs (4, 12, 36, 108), each of which is trained 3 times. It is a cell-wise search space, which constructs and stacks a cell by enumerating DAGs with at most 7 operators, and no more than 9 connections. All operators can be chosen from ``CONV3X3_BN_RELU``\ , ``CONV1X1_BN_RELU`` and ``MAXPOOL3X3``\ , except the first operator (always ``INPUT``\ ) and last operator (always ``OUTPUT``\ ).
NAS-Bench-101 contains 423,624 unique neural networks, combined with 4 variations in number of epochs (4, 12, 36, 108), each of which is trained 3 times. It is a cell-wise search space, which constructs and stacks a cell by enumerating DAGs with at most 7 operators, and no more than 9 connections. All operators can be chosen from ``CONV3X3_BN_RELU``, ``CONV1X1_BN_RELU`` and ``MAXPOOL3X3``, except the first operator (always ``INPUT``\ ) and last operator (always ``OUTPUT``\ ).
Notably, NAS-Bench-101 eliminates invalid cells (e.g., there is no path from input to output, or there is redundant computation). Furthermore, isomorphic cells are de-duplicated, i.e., all the remaining cells are computationally unique.
API Documentation
^^^^^^^^^^^^^^^^^
.. autofunction:: nni.nas.benchmarks.nasbench101.query_nb101_trial_stats
.. autoattribute:: nni.nas.benchmarks.nasbench101.INPUT
.. autoattribute:: nni.nas.benchmarks.nasbench101.OUTPUT
.. autoattribute:: nni.nas.benchmarks.nasbench101.CONV3X3_BN_RELU
.. autoattribute:: nni.nas.benchmarks.nasbench101.CONV1X1_BN_RELU
.. autoattribute:: nni.nas.benchmarks.nasbench101.MAXPOOL3X3
.. autoclass:: nni.nas.benchmarks.nasbench101.Nb101TrialConfig
.. autoclass:: nni.nas.benchmarks.nasbench101.Nb101TrialStats
.. autoclass:: nni.nas.benchmarks.nasbench101.Nb101IntermediateStats
.. autofunction:: nni.nas.benchmarks.nasbench101.graph_util.nasbench_format_to_architecture_repr
.. autofunction:: nni.nas.benchmarks.nasbench101.graph_util.infer_num_vertices
.. autofunction:: nni.nas.benchmarks.nasbench101.graph_util.hash_module
See :doc:`example usages </tutorials/nasbench_as_dataset>` and :ref:`API references <nas-bench-101-reference>`.
NAS-Bench-201
-------------
......@@ -99,28 +63,9 @@ NAS-Bench-201
* `Open-source API <https://github.com/D-X-Y/NAS-Bench-201>`__
* `Implementations <https://github.com/D-X-Y/AutoDL-Projects>`__
NAS-Bench-201 is a cell-wise search space that views nodes as tensors and edges as operators. The search space contains all possible densely-connected DAGs with 4 nodes, resulting in 15,625 candidates in total. Each operator (i.e., edge) is selected from a pre-defined operator set (\ ``NONE``\ , ``SKIP_CONNECT``\ , ``CONV_1X1``\ , ``CONV_3X3`` and ``AVG_POOL_3X3``\ ). Training appraoches vary in the dataset used (CIFAR-10, CIFAR-100, ImageNet) and number of epochs scheduled (12 and 200). Each combination of architecture and training approach is repeated 1 - 3 times with different random seeds.
API Documentation
^^^^^^^^^^^^^^^^^
.. autofunction:: nni.nas.benchmarks.nasbench201.query_nb201_trial_stats
.. autoattribute:: nni.nas.benchmarks.nasbench201.NONE
.. autoattribute:: nni.nas.benchmarks.nasbench201.SKIP_CONNECT
.. autoattribute:: nni.nas.benchmarks.nasbench201.CONV_1X1
.. autoattribute:: nni.nas.benchmarks.nasbench201.CONV_3X3
.. autoattribute:: nni.nas.benchmarks.nasbench201.AVG_POOL_3X3
NAS-Bench-201 is a cell-wise search space that views nodes as tensors and edges as operators. The search space contains all possible densely-connected DAGs with 4 nodes, resulting in 15,625 candidates in total. Each operator (i.e., edge) is selected from a pre-defined operator set (\ ``NONE``, ``SKIP_CONNECT``, ``CONV_1X1``, ``CONV_3X3`` and ``AVG_POOL_3X3``\ ). Training appraoches vary in the dataset used (CIFAR-10, CIFAR-100, ImageNet) and number of epochs scheduled (12 and 200). Each combination of architecture and training approach is repeated 1 - 3 times with different random seeds.
.. autoclass:: nni.nas.benchmarks.nasbench201.Nb201TrialConfig
.. autoclass:: nni.nas.benchmarks.nasbench201.Nb201TrialStats
.. autoclass:: nni.nas.benchmarks.nasbench201.Nb201IntermediateStats
See :doc:`example usages </tutorials/nasbench_as_dataset>` and :ref:`API references <nas-bench-201-reference>`.
NDS
---
......@@ -132,50 +77,9 @@ NDS
Instead of storing results obtained with different configurations in separate files, we dump them into one single database to enable comparison in multiple dimensions. Specifically, we use ``model_family`` to distinguish model types, ``model_spec`` for all hyper-parameters needed to build this model, ``cell_spec`` for detailed information on operators and connections if it is a NAS cell, ``generator`` to denote the sampling policy through which this configuration is generated. Refer to API documentation for details.
Available Operators
-------------------
Here is a list of available operators used in NDS.
.. autoattribute:: nni.nas.benchmarks.nds.constants.NONE
.. autoattribute:: nni.nas.benchmarks.nds.constants.SKIP_CONNECT
.. autoattribute:: nni.nas.benchmarks.nds.constants.AVG_POOL_3X3
.. autoattribute:: nni.nas.benchmarks.nds.constants.MAX_POOL_3X3
.. autoattribute:: nni.nas.benchmarks.nds.constants.MAX_POOL_5X5
.. autoattribute:: nni.nas.benchmarks.nds.constants.MAX_POOL_7X7
.. autoattribute:: nni.nas.benchmarks.nds.constants.CONV_1X1
.. autoattribute:: nni.nas.benchmarks.nds.constants.CONV_3X3
.. autoattribute:: nni.nas.benchmarks.nds.constants.CONV_3X1_1X3
.. autoattribute:: nni.nas.benchmarks.nds.constants.CONV_7X1_1X7
.. autoattribute:: nni.nas.benchmarks.nds.constants.DIL_CONV_3X3
.. autoattribute:: nni.nas.benchmarks.nds.constants.DIL_CONV_5X5
.. autoattribute:: nni.nas.benchmarks.nds.constants.SEP_CONV_3X3
.. autoattribute:: nni.nas.benchmarks.nds.constants.SEP_CONV_5X5
.. autoattribute:: nni.nas.benchmarks.nds.constants.SEP_CONV_7X7
.. autoattribute:: nni.nas.benchmarks.nds.constants.DIL_SEP_CONV_3X3
API Documentation
^^^^^^^^^^^^^^^^^
.. autofunction:: nni.nas.benchmarks.nds.query_nds_trial_stats
.. autoclass:: nni.nas.benchmarks.nds.NdsTrialConfig
.. autoclass:: nni.nas.benchmarks.nds.NdsTrialStats
.. automodule:: nni.nas.benchmarks.nds.constants
:noindex:
.. autoclass:: nni.nas.benchmarks.nds.NdsIntermediateStats
See :doc:`example usages </tutorials/nasbench_as_dataset>` and :ref:`API references <nds-reference>`.
docs/source/nas/benchmarks_toctree.rst 0 → 100644
View file @ a911b856
NAS Benchmark
=============
.. toctree::
:hidden:
Overview <benchmarks>
Examples </tutorials/nasbench_as_dataset>
docs/source/nas/construct_space.rst 0 → 100644
View file @ a911b856
Construct Model Space
=====================
NNI provides powerful (and multi-level) APIs for users to easily express model space (or search space).
* *Mutation Primitives*: high-level APIs (e.g., ValueChoice, LayerChoice) that are utilities to build blocks in search space. In most cases, mutation pritimives should be straightforward yet expressive enough. **We strongly recommend users to try them first,** and report issues if those APIs are not satisfying.
* *Hyper-module Library*: plug-and-play modules that are proved useful. They are usually well studied in research, and comes with pre-searched results. (For example, the optimal activation function in `AutoActivation <https://arxiv.org/abs/1710.05941>`__ is reported to be `Swish <https://pytorch.org/docs/stable/generated/torch.nn.SiLU.html>`__).
* *Mutator*: for advanced users only. NNI provides interface to customize new mutators for expressing more complicated model spaces.
The following table summarizes all the APIs we have provided for constructing search space.
.. list-table::
:header-rows: 1
:widths: auto
* - Name
- Category
- Brief Description
* - :class:`LayerChoice <nni.retiarii.nn.pytorch.LayerChoice>`
- :ref:`Mutation Primitives <mutation-primitives>`
- Select from some PyTorch modules
* - :class:`InputChoice <nni.retiarii.nn.pytorch.InputChoice>`
- :ref:`Mutation Primitives <mutation-primitives>`
- Select from some inputs (tensors)
* - :class:`ValueChoice <nni.retiarii.nn.pytorch.ValueChoice>`
- :ref:`Mutation Primitives <mutation-primitives>`
- Select from some candidate values
* - :class:`Repeat <nni.retiarii.nn.pytorch.Repeat>`
- :ref:`Mutation Primitives <mutation-primitives>`
- Repeat a block by a variable number of times
* - :class:`Cell <nni.retiarii.nn.pytorch.Cell>`
- :ref:`Mutation Primitives <mutation-primitives>`
- Cell structure popularly used in literature
* - :class:`NasBench101Cell <nni.retiarii.nn.pytorch.NasBench101Cell>`
- :ref:`Mutation Primitives <mutation-primitives>`
- Cell structure (variant) proposed by NAS-Bench-101
* - :class:`NasBench201Cell <nni.retiarii.nn.pytorch.NasBench201Cell>`
- :ref:`Mutation Primitives <mutation-primitives>`
- Cell structure (variant) proposed by NAS-Bench-201
* - :class:`AutoActivation <nni.retiarii.nn.pytorch.AutoActivation>`
- :ref:`Hyper-modules Library <hyper-modules>`
- Searching for activation functions
* - :class:`Mutator <nni.retiarii.Mutator>`
- :doc:`Mutator <mutator>`
- Flexible mutations on graphs. :doc:`See tutorial here <mutator>`
docs/source/NAS/WriteOneshot.rst → docs/source/nas/customize_strategy.rst
View file @ a911b856
Customize a New One-shot Trainer
================================
Customize Exploration Strategy
==============================
One-shot trainers should inherit ``nni.retiarii.oneshot.BaseOneShotTrainer``, and need to implement ``fit()`` (used to conduct the fitting and searching process) and ``export()`` method (used to return the searched best architecture).
Customize Multi-trial Strategy
------------------------------
If users want to innovate a new exploration strategy, they can easily customize a new one following the interface provided by NNI. Specifically, users should inherit the base strategy class :class:`nni.retiarii.strategy.BaseStrategy`, then implement the member function ``run``. This member function takes ``base_model`` and ``applied_mutators`` as its input arguments. It can simply apply the user specified mutators in ``applied_mutators`` onto ``base_model`` to generate a new model. When a mutator is applied, it should be bound with a sampler (e.g., ``RandomSampler``). Every sampler implements the ``choice`` function which chooses value(s) from candidate values. The ``choice`` functions invoked in mutators are executed with the sampler.
Below is a very simple random strategy, which makes the choices completely random.
.. code-block:: python
from nni.retiarii import Sampler
class RandomSampler(Sampler):
def choice(self, candidates, mutator, model, index):
return random.choice(candidates)
class RandomStrategy(BaseStrategy):
def __init__(self):
self.random_sampler = RandomSampler()
def run(self, base_model, applied_mutators):
_logger.info('stargety start...')
while True:
avail_resource = query_available_resources()
if avail_resource > 0:
model = base_model
_logger.info('apply mutators...')
_logger.info('mutators: %s', str(applied_mutators))
for mutator in applied_mutators:
mutator.bind_sampler(self.random_sampler)
model = mutator.apply(model)
# run models
submit_models(model)
else:
time.sleep(2)
You can find that this strategy does not know the search space beforehand, it passively makes decisions every time ``choice`` is invoked from mutators. If a strategy wants to know the whole search space before making any decision (e.g., TPE, SMAC), it can use ``dry_run`` function provided by ``Mutator`` to obtain the space. An example strategy can be found :githublink:`here <nni/retiarii/strategy/tpe_strategy.py>`.
After generating a new model, the strategy can use our provided APIs (e.g., :func:`nni.retiarii.execution.submit_models`, :func:`nni.retiarii.execution.is_stopped_exec`) to submit the model and get its reported results.
Customize a New One-shot Trainer (legacy)
-----------------------------------------
One-shot trainers should inherit :class:`nni.retiarii.oneshot.BaseOneShotTrainer`, and need to implement ``fit()`` (used to conduct the fitting and searching process) and ``export()`` method (used to return the searched best architecture).
Writing a one-shot trainer is very different to single-arch evaluator. First of all, there are no more restrictions on init method arguments, any Python arguments are acceptable. Secondly, the model fed into one-shot trainers might be a model with Retiarii-specific modules, such as LayerChoice and InputChoice. Such model cannot directly forward-propagate and trainers need to decide how to handle those modules.
A typical example is DartsTrainer, where learnable-parameters are used to combine multiple choices in LayerChoice. Retiarii provides ease-to-use utility functions for module-replace purposes, namely ``replace_layer_choice``, ``replace_input_choice``. A simplified example is as follows:
A typical example is DartsTrainer, where learnable-parameters are used to combine multiple choices in LayerChoice. Retiarii provides ease-to-use utility functions for module-replace purposes, namely :meth:`nni.retiarii.oneshot.pytorch.utils.replace_layer_choice`, :meth:`nni.retiarii.oneshot.pytorch.utils.replace_input_choice`. A simplified example is as follows:
.. code-block:: python
from nni.retiarii.oneshot import BaseOneShotTrainer
from nni.retiarii.oneshot.pytorch import replace_layer_choice, replace_input_choice
from nni.retiarii.oneshot.pytorch.utils import replace_layer_choice, replace_input_choice
class DartsLayerChoice(nn.Module):
......
docs/source/NAS/ModelEvaluators.rst → docs/source/nas/evaluator.rst
View file @ a911b856
Model Evaluators
================
Model Evaluator
===============
A model evaluator is for training and validating each generated model. They are necessary to evaluate the performance of new explored models.
.. _functional-evaluator:
Customize Evaluator with Any Function
-------------------------------------
The simplest way to customize a new evaluator is with functional APIs, which is very easy when training code is already available. Users only need to write a fit function that wraps everything, which usually includes training, validating and testing of a single model. This function takes one positional arguments (``model_cls``) and possible keyword arguments. The keyword arguments (other than ``model_cls``) are fed to FunctionEvaluator as its initialization parameters (note that they will be `serialized <./Serialization.rst>`__). In this way, users get everything under their control, but expose less information to the framework and as a result, further optimizations like `CGO <./ExecutionEngines.rst#cgo-execution-engine-experimental>`__ might be not feasible. An example is as belows:
The simplest way to customize a new evaluator is with :class:`FunctionalEvaluator <nni.retiarii.evaluator.FunctionalEvaluator>`, which is very easy when training code is already available. Users only need to write a fit function that wraps everything, which usually includes training, validating and testing of a single model. This function takes one positional arguments (``model_cls``) and possible keyword arguments. The keyword arguments (other than ``model_cls``) are fed to :class:`FunctionalEvaluator <nni.retiarii.evaluator.FunctionalEvaluator>` as its initialization parameters (note that they will be :doc:`serialized <./serialization>`). In this way, users get everything under their control, but expose less information to the framework and as a result, further optimizations like :ref:`CGO <cgo-execution-engine>` might be not feasible. An example is as belows:
.. code-block:: python
......@@ -46,12 +48,15 @@ Evaluators with PyTorch-Lightning
Use Built-in Evaluators
^^^^^^^^^^^^^^^^^^^^^^^
NNI provides some commonly used model evaluators for users' convenience. These evaluators are built upon the awesome library PyTorch-Lightning.
NNI provides some commonly used model evaluators for users' convenience. These evaluators are built upon the awesome library PyTorch-Lightning. Read the :doc:`reference </reference/nas/evaluator>` for their detailed usages.
* :class:`nni.retiarii.evaluator.pytorch.Classification`: for classification tasks.
* :class:`nni.retiarii.evaluator.pytorch.Regression`: for regression tasks.
We recommend to read the `serialization tutorial <./Serialization.rst>`__ before using these evaluators. A few notes to summarize the tutorial:
We recommend to read the :doc:`serialization tutorial <serialization>` before using these evaluators. A few notes to summarize the tutorial:
1. ``pl.DataLoader`` should be used in place of ``torch.utils.data.DataLoader``.
2. The datasets used in data-loader should be decorated with ``nni.trace`` recursively.
1. :class:`nni.retiarii.evaluator.pytorch.DataLoader` should be used in place of ``torch.utils.data.DataLoader``.
2. The datasets used in data-loader should be decorated with :meth:`nni.trace` recursively.
For example,
......@@ -69,18 +74,12 @@ For example,
val_dataloaders=pl.DataLoader(test_dataset, batch_size=100),
max_epochs=10)
.. autoclass:: nni.retiarii.evaluator.pytorch.lightning.Classification
:noindex:
.. autoclass:: nni.retiarii.evaluator.pytorch.lightning.Regression
:noindex:
Customize Evaluator with PyTorch-Lightning
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Another approach is to write training code in PyTorch-Lightning style, that is, to write a LightningModule that defines all elements needed for training (e.g., loss function, optimizer) and to define a trainer that takes (optional) dataloaders to execute the training. Before that, please read the `document of PyTorch-lightning <https://pytorch-lightning.readthedocs.io/>`__ to learn the basic concepts and components provided by PyTorch-lightning.
In practice, writing a new training module in Retiarii should inherit ``nni.retiarii.evaluator.pytorch.lightning.LightningModule``, which has a ``set_model`` that will be called after ``__init__`` to save the candidate model (generated by strategy) as ``self.model``. The rest of the process (like ``training_step``) should be the same as writing any other lightning module. Evaluators should also communicate with strategies via two API calls (``nni.report_intermediate_result`` for periodical metrics and ``nni.report_final_result`` for final metrics), added in ``on_validation_epoch_end`` and ``teardown`` respectively.
In practice, writing a new training module in Retiarii should inherit :class:`nni.retiarii.evaluator.pytorch.LightningModule`, which has a ``set_model`` that will be called after ``__init__`` to save the candidate model (generated by strategy) as ``self.model``. The rest of the process (like ``training_step``) should be the same as writing any other lightning module. Evaluators should also communicate with strategies via two API calls (:meth:`nni.report_intermediate_result` for periodical metrics and :meth:`nni.report_final_result` for final metrics), added in ``on_validation_epoch_end`` and ``teardown`` respectively.
An example is as follows:
......@@ -133,7 +132,7 @@ An example is as follows:
if stage == 'fit':
nni.report_final_result(self.trainer.callback_metrics['val_loss'].item())
Then, users need to wrap everything (including LightningModule, trainer and dataloaders) into a ``Lightning`` object, and pass this object into a Retiarii experiment.
Then, users need to wrap everything (including LightningModule, trainer and dataloaders) into a :class:`nni.retiarii.evaluator.pytorch.Lightning` object, and pass this object into a Retiarii experiment.
.. code-block:: python
......
docs/source/NAS/ExecutionEngines.rst → docs/source/nas/execution_engine.rst
View file @ a911b856
Execution Engines
=================
Execution engine is for running Retiarii Experiment. NNI supports three execution engines, users can choose a speicific engine according to the type of their model mutation definition and their requirements for cross-model optimizations.
Execution engine is for running Retiarii Experiment. NNI supports three execution engines, users can choose a specific engine according to the type of their model mutation definition and their requirements for cross-model optimizations.
* **Pure-python execution engine** is the default engine, it supports the model space expressed by `inline mutation API <./MutationPrimitives.rst>`__.
* **Pure-python execution engine** is the default engine, it supports the model space expressed by :doc:`mutation primitives <construct_space>`.
* **Graph-based execution engine** supports the use of `inline mutation APIs <./MutationPrimitives.rst>`__ and model spaces represented by `mutators <./Mutators.rst>`__. It requires the user's model to be parsed by `TorchScript <https://pytorch.org/docs/stable/jit.html>`__.
* **Graph-based execution engine** supports the use of :doc:`mutation primitives <construct_space>` and model spaces represented by :doc:`mutators <mutator>`. It requires the user's model to be parsed by `TorchScript <https://pytorch.org/docs/stable/jit.html>`__.
* **CGO execution engine** has the same requirements and capabilities as the **Graph-based execution engine**. But further enables cross-model optimizations, which makes model space exploration faster.
.. _pure-python-execution-engine:
Pure-python Execution Engine
----------------------------
Pure-python Execution Engine is the default engine, we recommend users to keep using this execution engine, if they are new to NNI NAS. Pure-python execution engine plays magic within the scope of inline mutation APIs, while does not touch the rest of user model. Thus, it has minimal requirement on user model.
One steps are needed to use this engine now.
1. Add ``@nni.retiarii.model_wrapper`` decorator outside the whole PyTorch model.
Rememeber to add :meth:`nni.retiarii.model_wrapper` decorator outside the whole PyTorch model before using this engine.
.. note:: You should always use ``super().__init__()`` instead of ``super(MyNetwork, self).__init__()`` in the PyTorch model, because the latter one has issues with model wrapper.
.. _graph-based-execution-engine:
Graph-based Execution Engine
----------------------------
For graph-based execution engine, it converts user-defined model to a graph representation (called graph IR) using `TorchScript <https://pytorch.org/docs/stable/jit.html>`__, each instantiated module in the model is converted to a subgraph. Then mutations are applied to the graph to generate new graphs. Each new graph is then converted back to PyTorch code and executed on the user specified training service.
Users may find ``@basic_unit`` helpful in some cases. ``@basic_unit`` here means the module will not be converted to a subgraph, instead, it is converted to a single graph node as a basic unit.
Users may find ``@basic_unit`` helpful in some cases. :meth:`nni.retiarii.basic_unit` here means the module will not be converted to a subgraph, instead, it is converted to a single graph node as a basic unit.
``@basic_unit`` is usually used in the following cases:
* When users want to tune initialization parameters of a module using ``ValueChoice``, then decorate the module with ``@basic_unit``. For example, ``self.conv = MyConv(kernel_size=nn.ValueChoice([1, 3, 5]))``, here ``MyConv`` should be decorated.
* When users want to tune initialization parameters of a module using :class:`nni.retiarii.nn.pytorch.ValueChoice`, then decorate the module with ``@basic_unit``. For example, ``self.conv = MyConv(kernel_size=nn.ValueChoice([1, 3, 5]))``, here ``MyConv`` should be decorated.
* When a module cannot be successfully parsed to a subgraph, decorate the module with ``@basic_unit``. The parse failure could be due to complex control flow. Currently Retiarii does not support adhoc loop, if there is adhoc loop in a module's forward, this class should be decorated as serializable module. For example, the following ``MyModule`` should be decorated.
......@@ -43,23 +45,25 @@ Users may find ``@basic_unit`` helpful in some cases. ``@basic_unit`` here means
for i in range(10): # <- adhoc loop
...
* Some inline mutation APIs require their handled module to be decorated with ``@basic_unit``. For example, user-defined module that is provided to ``LayerChoice`` as a candidate op should be decorated.
* Some inline mutation APIs require their handled module to be decorated with ``@basic_unit``. For example, user-defined module that is provided to :class:`nni.retiarii.nn.pytorch.LayerChoice` as a candidate op should be decorated.
Three steps are need to use graph-based execution engine.
1. Remove ``@nni.retiarii.model_wrapper`` if there is any in your model.
2. Add ``config.execution_engine = 'base'`` to ``RetiariiExeConfig``. The default value of ``execution_engine`` is 'py', which means pure-python execution engine.
2. Add ``config.execution_engine = 'base'`` to :class:`nni.retiarii.experiment.pytorch.RetiariiExeConfig`. The default value of ``execution_engine`` is 'py', which means pure-python execution engine.
3. Add ``@basic_unit`` when necessary following the above guidelines.
For exporting top models, graph-based execution engine supports exporting source code for top models by running ``exp.export_top_models(formatter='code')``.
.. _cgo-execution-engine:
CGO Execution Engine (experimental)
-----------------------------------
CGO（Cross-Graph Optimization) execution engine does cross-model optimizations based on the graph-based execution engine. In CGO execution engine, multiple models could be merged and trained together in one trial.
CGO (Cross-Graph Optimization) execution engine does cross-model optimizations based on the graph-based execution engine. In CGO execution engine, multiple models could be merged and trained together in one trial.
Currently, it only supports ``DedupInputOptimizer`` that can merge graphs sharing the same dataset to only loading and pre-processing each batch of data once, which can avoid bottleneck on data loading.
.. note :: To use CGO engine, PyTorch-lightning above version 1.4.2 is required.
.. note :: To use CGO engine, PyTorch Lightning of 1.5.x is required.
To enable CGO execution engine, you need to follow these steps:
......@@ -67,7 +71,7 @@ To enable CGO execution engine, you need to follow these steps:
2. Add configurations for remote training service
3. Add configurations for CGO engine
.. code-block:: python
.. code-block:: python
exp = RetiariiExperiment(base_model, trainer, mutators, strategy)
config = RetiariiExeConfig('remote')
......@@ -103,4 +107,4 @@ We have already implemented two trainers: :class:`nni.retiarii.evaluator.pytorch
Advanced users can also implement their own trainers by inheriting ``MultiModelSupervisedLearningModule``.
Sometimes, a mutated model cannot be executed (e.g., due to shape mismatch). When a trial running multiple models contains
a bad model, CGO execution engine will re-run each model independently in seperate trials without cross-model optimizations.
a bad model, CGO execution engine will re-run each model independently in separate trials without cross-model optimizations.
docs/source/nas/exploration_strategy.rst 0 → 100644
View file @ a911b856
Exploration Strategy
====================
There are two types of model space exploration approach: **Multi-trial strategy** and **One-shot strategy**. When the model space has been constructed, users can use either exploration approach to explore the model space.
* :ref:`Mutli-trial strategy <multi-trial-nas>` trains each sampled model in the model space independently.
* :ref:`One-shot strategy <one-shot-nas>` samples the model from a super model.
Here is the list of exploration strategies that NNI has supported.
.. list-table::
:header-rows: 1
:widths: auto
* - Name
- Category
- Brief Description
* - :class:`Random <nni.retiarii.strategy.Random>`
- :ref:`Multi-trial <multi-trial-nas>`
- Randomly sample an architecture each time
* - :class:`GridSearch <nni.retiarii.strategy.GridSearch>`
- :ref:`Multi-trial <multi-trial-nas>`
- Traverse the search space and try all possibilities
* - :class:`RegularizedEvolution <nni.retiarii.strategy.RegularizedEvolution>`
- :ref:`Multi-trial <multi-trial-nas>`
- Evolution algorithm for NAS. `Reference <https://arxiv.org/abs/1802.01548>`__
* - :class:`TPE <nni.retiarii.strategy.TPE>`
- :ref:`Multi-trial <multi-trial-nas>`
- Tree-structured Parzen Estimator (TPE). `Reference <https://papers.nips.cc/paper/4443-algorithms-for-hyper-parameter-optimization.pdf>`__
* - :class:`PolicyBasedRL <nni.retiarii.strategy.PolicyBasedRL>`
- :ref:`Multi-trial <multi-trial-nas>`
- Policy-based reinforcement learning, based on implementation of tianshou. `Reference <https://arxiv.org/abs/1611.01578>`__
* - :ref:`darts-strategy`
- :ref:`One-shot <one-shot-nas>`
- Continuous relaxation of the architecture representation, allowing efficient search of the architecture using gradient descent. `Reference <https://arxiv.org/abs/1806.09055>`__
* - :ref:`enas-strategy`
- :ref:`One-shot <one-shot-nas>`
- RL controller learns to generate the best network on a super-net. `Reference <https://arxiv.org/abs/1802.03268>`__
* - :ref:`fbnet-strategy`
- :ref:`One-shot <one-shot-nas>`
- Choose the best block by using Gumbel Softmax random sampling and differentiable training. `Reference <https://arxiv.org/abs/1812.03443>`__
* - :ref:`spos-strategy`
- :ref:`One-shot <one-shot-nas>`
- Train a super-net with uniform path sampling. `Reference <https://arxiv.org/abs/1904.00420>`__
* - :ref:`proxylessnas-strategy`
- :ref:`One-shot <one-shot-nas>`
- A low-memory-consuming optimized version of differentiable architecture search. `Reference <https://arxiv.org/abs/1812.00332>`__
.. _multi-trial-nas:
Multi-trial strategy
--------------------
Multi-trial NAS means each sampled model from model space is trained independently. A typical multi-trial NAS is `NASNet <https://arxiv.org/abs/1707.07012>`__. In multi-trial NAS, users need model evaluator to evaluate the performance of each sampled model, and need an exploration strategy to sample models from a defined model space. Here, users could use NNI provided model evaluators or write their own model evalutor. They can simply choose a exploration strategy. Advanced users can also customize new exploration strategy.
To use an exploration strategy, users simply instantiate an exploration strategy and pass the instantiated object to :class:`RetiariiExperiment <nni.retiarii.experiment.pytorch.RetiariiExperiment>`. Below is a simple example.
.. code-block:: python
import nni.retiarii.strategy as strategy
exploration_strategy = strategy.Random(dedup=True)
Rather than using :class:`strategy.Random <nni.retiarii.strategy.Random>`, users can choose one of the strategies from the table above.
.. _one-shot-nas:
One-shot strategy
-----------------
One-shot NAS algorithms leverage weight sharing among models in neural architecture search space to train a supernet, and use this supernet to guide the selection of better models. This type of algorihtms greatly reduces computational resource compared to independently training each model from scratch (which we call "Multi-trial NAS").
Currently, the usage of one-shot NAS strategy is a little different from multi-trial strategy. One-shot strategy is implemented with a special type of objects named *Trainer*. Following the common practice of one-shot NAS, *Trainer* trains the super-net and searches for the optimal architecture in a single run. For example,
.. code-block:: python
from nni.retiarii.oneshot.pytorch import DartsTrainer
trainer = DartsTrainer(
model=model,
loss=criterion,
metrics=lambda output, target: accuracy(output, target, topk=(1,)),
optimizer=optim,
dataset=dataset_train,
batch_size=32,
log_frequency=50
)
trainer.fit()
One-shot strategy can be used without :class:`RetiariiExperiment <nni.retiarii.experiment.pytorch.RetiariiExperiment>`. Thus, the ``trainer.fit()`` here runs the experiment locally.
After ``trainer.fit()`` completes, we can use ``trainer.export()`` to export the searched architecture (a dict of choices) to a file.
.. code-block:: python
final_architecture = trainer.export()
print('Final architecture:', trainer.export())
json.dump(trainer.export(), open('checkpoint.json', 'w'))
.. tip:: The trained super-net (neither the weights or exported JSON) can't be used directly. It's only an intermediate result used for deriving the final architecture. The exported architecture (can be retrieved with :meth:`nni.retiarii.fixed_arch`) needs to be *retrained* with a standard training recipe to get the final model.
docs/source/NAS/HardwareAwareNAS.rst → docs/source/nas/hardware_aware_nas.rst
View file @ a911b856
Hardware-aware NAS
==================
.. contents::
.. This file should be rewritten as a tutorial
End-to-end Multi-trial SPOS Demo
--------------------------------
......@@ -18,17 +18,21 @@ Then run multi-trail SPOS demo:
.. code-block:: bash
python ${NNI_ROOT}/examples/nas/oneshot/spos/multi_trial.py
cd ${NNI_ROOT}/examples/nas/oneshot/spos/
python search.py --latency-filter cortexA76cpu_tflite21
How the demo works
^^^^^^^^^^^^^^^^^^
To support hardware-aware NAS, you first need a ``Strategy`` that supports filtering the models by latency. We provide such a filter named ``LatencyFilter`` in NNI and initialize a ``Random`` strategy with the filter:
To support hardware-aware NAS, you first need a ``Strategy`` that supports filtering the models by latency. We provide such a filter named ``LatencyFilter`` in NNI and initialize a ``RegularizedEvolution`` strategy with the filter:
.. code-block:: python
simple_strategy = strategy.Random(model_filter=LatencyFilter(threshold=100, predictor=base_predictor))
evolution_strategy = strategy.RegularizedEvolution(
model_filter=latency_filter,
sample_size=args.evolution_sample_size, population_size=args.evolution_population_size, cycles=args.evolution_cycles
)
``LatencyFilter`` will predict the models\' latency by using nn-Meter and filter out the models whose latency are larger than the threshold (i.e., ``100`` in this example).
You can also build your own strategies and filters to support more flexible NAS such as sorting the models according to latency.
......@@ -37,21 +41,21 @@ Then, pass this strategy to ``RetiariiExperiment``:
.. code-block:: python
exp = RetiariiExperiment(base_model, trainer, strategy=simple_strategy)
exp = RetiariiExperiment(base_model, evaluator, strategy=evolution_strategy)
exp_config = RetiariiExeConfig('local')
...
exp_config.dummy_input = [1, 3, 32, 32]
exp_config.dummy_input = [1, 3, 224, 224]
exp.run(exp_config, port)
exp.run(exp_config, args.port)
In ``exp_config``, ``dummy_input`` is required for tracing shape info.
In ``exp_config``, ``dummy_input`` is required for tracing shape info in latency predictor.
End-to-end ProxylessNAS with Latency Constraints
------------------------------------------------
`ProxylessNAS <https://arxiv.org/pdf/1812.00332.pdf>`__ is a hardware-aware one-shot NAS algorithm. ProxylessNAS applies the expected latency of the model to build a differentiable metric and design efficient neural network architectures for hardware. The latency loss is added as a regularization term for architecture parameter optimization. In this example, nn-Meter provides a latency estimator to predict expected latency for the mixed operation on other types of mobile and edge hardware.
`ProxylessNAS <https://arxiv.org/abs/1812.00332>`__ is a hardware-aware one-shot NAS algorithm. ProxylessNAS applies the expected latency of the model to build a differentiable metric and design efficient neural network architectures for hardware. The latency loss is added as a regularization term for architecture parameter optimization. In this example, nn-Meter provides a latency estimator to predict expected latency for the mixed operation on other types of mobile and edge hardware.
To run the one-shot ProxylessNAS demo, first install nn-Meter by running:
......@@ -61,14 +65,14 @@ To run the one-shot ProxylessNAS demo, first install nn-Meter by running:
Then run one-shot ProxylessNAS demo:
```bash
python ${NNI_ROOT}/examples/nas/oneshot/proxylessnas/main.py --applied_hardware <hardware> --reference_latency <reference latency (ms)>
```
.. code-block:: bash
python ${NNI_ROOT}/examples/nas/oneshot/proxylessnas/main.py --applied_hardware HARDWARE --reference_latency REFERENCE_LATENCY_MS
How the demo works
^^^^^^^^^^^^^^^^^^
In the implementation of ProxylessNAS ``trainer``, we provide a ``HardwareLatencyEstimator`` which currently builds a lookup table, that stores the measured latency of each candidate building block in the search space. The latency sum of all building blocks in a candidate model will be treated as the model inference latency. The latency prediction is obtained by ``nn-Meter``. ``HardwareLatencyEstimator`` predicts expected latency for the mixed operation based on the path weight of `ProxylessLayerChoice`. With leveraging ``nn-Meter`` in NNI, users can apply ProxylessNAS to search efficient DNN models on more types of edge devices.
In the implementation of ProxylessNAS ``trainer``, we provide a ``HardwareLatencyEstimator`` which currently builds a lookup table, that stores the measured latency of each candidate building block in the search space. The latency sum of all building blocks in a candidate model will be treated as the model inference latency. The latency prediction is obtained by ``nn-Meter``. ``HardwareLatencyEstimator`` predicts expected latency for the mixed operation based on the path weight of ``ProxylessLayerChoice``. With leveraging ``nn-Meter`` in NNI, users can apply ProxylessNAS to search efficient DNN models on more types of edge devices.
Despite of ``applied_hardware`` and ``reference_latency``, There are some other parameters related to hardware-aware ProxylessNAS training in this :githublink:`example <examples/nas/oneshot/proxylessnas/main.py>`:
......
docs/source/NAS/Mutators.rst → docs/source/nas/mutator.rst
View file @ a911b856
Express Mutations with Mutators
===============================
Construct Space with Mutator
============================
Besides the inline mutation APIs demonstrated `here <./MutationPrimitives.rst>`__, NNI provides a more general approach to express a model space, i.e., *Mutator*, to cover more complex model spaces. Those inline mutation APIs are also implemented with mutator in the underlying system, which can be seen as a special case of model mutation.
Besides the mutation primitives demonstrated in the :doc:`basic tutorial <construct_space>`, NNI provides a more general approach to express a model space, i.e., *Mutator*, to cover more complex model spaces. The high-level APIs are also implemented with mutator in the underlying system, which can be seen as a special case of model mutation.
.. note:: Mutator and inline mutation APIs cannot be used together.
.. warning:: Mutator and inline mutation APIs can NOT be used together.
A mutator is a piece of logic to express how to mutate a given model. Users are free to write their own mutators. Then a model space is expressed with a base model and a list of mutators. A model in the model space is sampled by applying the mutators on the base model one after another. An example is shown below.
......@@ -18,7 +18,7 @@ A mutator is a piece of logic to express how to mutate a given model. Users are
Write a mutator
---------------
User-defined mutator should inherit ``Mutator`` class, and implement mutation logic in the member function ``mutate``.
User-defined mutator should inherit :class:`nni.retiarii.Mutator` class, and implement mutation logic in the member function :meth:`nni.retiarii.Mutator.mutate`.
.. code-block:: python
......@@ -35,9 +35,9 @@ User-defined mutator should inherit ``Mutator`` class, and implement mutation lo
chosen_op = self.choice(self.candidate_op_list)
node.update_operation(chosen_op.type, chosen_op.params)
The input of ``mutate`` is graph IR (Intermediate Representation) of the base model (please refer to `here <./ApiReference.rst>`__ for the format and APIs of the IR), users can mutate the graph using the graph's member functions (e.g., ``get_nodes_by_label``, ``update_operation``). The mutation operations can be combined with the API ``self.choice``, in order to express a set of possible mutations. In the above example, the node's operation can be changed to any operation from ``candidate_op_list``.
The input of :meth:`nni.retiarii.Mutator.mutate` is graph IR (Intermediate Representation) of the base model, users can mutate the graph using the graph's member functions (e.g., :meth:`nni.retiarii.Model.get_nodes_by_label`). The mutation operations can be combined with the API ``self.choice``, in order to express a set of possible mutations. In the above example, the node's operation can be changed to any operation from ``candidate_op_list``.
Use placehoder to make mutation easier: ``nn.Placeholder``. If you want to mutate a subgraph or node of your model, you can define a placeholder in this model to represent the subgraph or node. Then, use mutator to mutate this placeholder to make it real modules.
Use placeholder to make mutation easier: :class:`nni.retiarii.nn.pytorch.Placeholder`. If you want to mutate a subgraph or node of your model, you can define a placeholder in this model to represent the subgraph or node. Then, use mutator to mutate this placeholder to make it real modules.
.. code-block:: python
......@@ -51,7 +51,7 @@ Use placehoder to make mutation easier: ``nn.Placeholder``. If you want to mutat
``label`` is used by mutator to identify this placeholder. The other parameters are the information that is required by mutator. They can be accessed from ``node.operation.parameters`` as a dict, it could include any information that users want to put to pass it to user defined mutator. The complete example code can be found in :githublink:`Mnasnet base model <examples/nas/multi-trial/mnasnet/base_mnasnet.py>`.
Starting an experiment is almost the same as using inline mutation APIs. The only difference is that the applied mutators should be passed to ``RetiariiExperiment``. Below is a simple example.
Starting an experiment is almost the same as using inline mutation APIs. The only difference is that the applied mutators should be passed to :class:`nni.retiarii.experiment.pytorch.RetiariiExperiment`. Below is a simple example.
.. code-block:: python
......
docs/source/nas/overview.rst 0 → 100644
View file @ a911b856
Overview
========
.. attention:: NNI's latest NAS supports are all based on Retiarii Framework, users who are still on `early version using NNI NAS v1.0 <https://nni.readthedocs.io/en/v2.2/nas.html>`__ shall migrate your work to Retiarii as soon as possible. We plan to remove the legacy NAS framework in the next few releases.
.. note:: PyTorch is the **only supported framework on Retiarii**. Inquiries of NAS support on Tensorflow is in `this discussion <https://github.com/microsoft/nni/discussions/4605>`__. If you intend to run NAS with DL frameworks other than PyTorch and Tensorflow, please `open new issues <https://github.com/microsoft/nni/issues>`__ to let us know.
Basics
------
Automatic neural architecture search is playing an increasingly important role in finding better models. Recent research has proven the feasibility of automatic NAS and has led to models that beat many manually designed and tuned models. Representative works include `NASNet <https://arxiv.org/abs/1707.07012>`__, `ENAS <https://arxiv.org/abs/1802.03268>`__, `DARTS <https://arxiv.org/abs/1806.09055>`__, `Network Morphism <https://arxiv.org/abs/1806.10282>`__, and `Evolution <https://arxiv.org/abs/1703.01041>`__. In addition, new innovations continue to emerge.
High-level speaking, aiming to solve any particular task with neural architecture search typically requires: search space design, search strategy selection, and performance evaluation. The three components work together with the following loop (from the famous `NAS survey <https://arxiv.org/abs/1808.05377>`__):
.. image:: ../../img/nas_abstract_illustration.png
:align: center
:width: 700
In this figure:
* *Model search space* means a set of models from which the best model is explored/searched. Sometimes we use *search space* or *model space* in short.
* *Exploration strategy* is the algorithm that is used to explore a model search space. Sometimes we also call it *search strategy*.
* *Model evaluator* is responsible for training a model and evaluating its performance.
The process is similar to :doc:`Hyperparameter Optimization </hpo/overview>`, except that the target is the best architecture rather than hyperparameter. Concretely, an exploration strategy selects an architecture from a predefined search space. The architecture is passed to a performance evaluation to get a score, which represents how well this architecture performs on a particular task. This process is repeated until the search process is able to find the best architecture.
Key Features
------------
The current NAS framework in NNI is powered by the research of `Retiarii: A Deep Learning Exploratory-Training Framework <https://www.usenix.org/system/files/osdi20-zhang_quanlu.pdf>`__, where we highlight the following features:
* :doc:`Simple APIs to construct search space easily <construct_space>`
* :doc:`SOTA NAS algorithms to explore search space <exploration_strategy>`
* :doc:`Experiment backend support to scale up experiments on large-scale AI platforms </experiment/overview>`
Why NAS with NNI
----------------
We list out the three perspectives where NAS can be particularly challegning without NNI. NNI provides solutions to relieve users' engineering effort when they want to try NAS techniques in their own scenario.
Search Space Design
^^^^^^^^^^^^^^^^^^^
The search space defines which architectures can be represented in principle. Incorporating prior knowledge about typical properties of architectures well-suited for a task can reduce the size of the search space and simplify the search. However, this also introduces a human bias, which may prevent finding novel architectural building blocks that go beyond the current human knowledge. Search space design can be very challenging for beginners, who might not possess the experience to balance the richness and simplicity.
In NNI, we provide a wide range of APIs to build the search space. There are :doc:`high-level APIs <construct_space>`, that enables the possibility to incorporate human knowledge about what makes a good architecture or search space. There are also :doc:`low-level APIs <mutator>`, that is a list of primitives to construct a network from operation to operation.
Exploration strategy
^^^^^^^^^^^^^^^^^^^^
The exploration strategy details how to explore the search space (which is often exponentially large). It encompasses the classical exploration-exploitation trade-off since, on the one hand, it is desirable to find well-performing architectures quickly, while on the other hand, premature convergence to a region of suboptimal architectures should be avoided. The "best" exploration strategy for a particular scenario is usually found via trial-and-error. As many state-of-the-art strategies are implemented with their own code-base, it becomes very troublesome to switch from one to another.
In NNI, we have also provided :doc:`a list of strategies <exploration_strategy>`. Some of them are powerful yet time consuming, while others might be suboptimal but really efficient. Given that all strategies are implemented with a unified interface, users can always find one that matches their need.
Performance estimation
^^^^^^^^^^^^^^^^^^^^^^
The objective of NAS is typically to find architectures that achieve high predictive performance on unseen data. Performance estimation refers to the process of estimating this performance. The problem with performance estimation is mostly its scalability, i.e., how can I run and manage multiple trials simultaneously.
In NNI, we standardize this process is implemented with :doc:`evaluator <evaluator>`, which is responsible of estimating a model's performance. NNI has quite a few built-in supports of evaluators, ranging from the simplest option, e.g., to perform a standard training and validation of the architecture on data, to complex configurations and implementations. Evaluators are run in *trials*, where trials can be spawn onto distributed platforms with our powerful :doc:`training service </experiment/training_service/overview>`.
Tutorials
---------
To start using NNI NAS framework, we recommend at least going through the following tutorials:
* :doc:`Quickstart </tutorials/hello_nas>`
* :doc:`construct_space`
* :doc:`exploration_strategy`
* :doc:`evaluator`
Resources
---------
The following articles will help with a better understanding of the current arts of NAS:
* `Neural Architecture Search: A Survey <https://arxiv.org/abs/1808.05377>`__
* `A Comprehensive Survey of Neural Architecture Search: Challenges and Solutions <https://arxiv.org/abs/2006.02903>`__
docs/source/nas/overview_zh.rst 0 → 100644
View file @ a911b856
.. 48c39585a539a877461aadef63078c48
神经架构搜索
===========================
.. toctree::
:hidden:
快速入门 </tutorials/hello_nas>
构建搜索空间 <construct_space>
探索策略 <exploration_strategy>
评估器 <evaluator>
高级用法 <advanced_usage>
.. attention:: NNI 最新的架构搜索支持都是基于 Retiarii 框架，还在使用 `NNI 架构搜索的早期版本 <https://nni.readthedocs.io/en/v2.2/nas.html>`__ 的用户应尽快将您的工作迁移到 Retiarii。我们计划在接下来的几个版本中删除旧的架构搜索框架。
.. attention:: PyTorch 是 **Retiarii 唯一支持的框架**。有关 Tensorflow 上架构搜索支持的需求在 `此讨论 <https://github.com/microsoft/nni/discussions/4605>`__ 中。另外，如果您打算使用 PyTorch 和 Tensorflow 以外的 DL 框架运行 NAS，请 `创建新 issue <https://github.com/microsoft/nni/issues>`__ 让我们知道。
概述
------
自动神经架构搜索 (Neural Architecture Search, NAS）在寻找更好的模型方面发挥着越来越重要的作用。最近的研究证明了自动架构搜索的可行性，并导致模型击败了许多手动设计和调整的模型。其中具有代表性的有 `NASNet <https://arxiv.org/abs/1707.07012>`__、 `ENAS <https://arxiv.org/abs/1802.03268>`__、 `DARTS <https://arxiv.org/ abs/1806.09055>`__、 `Network Morphism <https://arxiv.org/abs/1806.10282>`__ 和 `进化算法 <https://arxiv.org/abs/1703.01041>`__。此外，新的创新正不断涌现。
总的来说，使用神经架构搜索解决任何特定任务通常需要：搜索空间设计、搜索策略选择和性能评估。这三个组件形成如下的循环（图来自于 `架构搜索综述 <https://arxiv.org/abs/1808.05377>`__）：
.. image:: ../../img/nas_abstract_illustration.png
:align: center
:width: 700
在这个图中：
* *模型搜索空间* 是指一组模型，从中探索/搜索最佳模型，简称为 *搜索空间* 或 *模型空间*。
* *探索策略* 是用于探索模型搜索空间的算法。有时我们也称它为 *搜索策略*。
* *模型评估者* 负责训练模型并评估其性能。
该过程类似于 :doc:`超参数优化 </hpo/overview>`，只不过目标是最佳网络结构而不是最优超参数。具体来说，探索策略从预定义的搜索空间中选择架构。该架构被传递给性能评估以获得评分，该评分表示这个网络结构在特定任务上的表现。重复此过程，直到搜索过程能够找到最优的网络结构。
主要特点
------------
NNI 中当前的架构搜索框架由 `Retiarii: A Deep Learning Exploratory-Training Framework <https://www.usenix.org/system/files/osdi20-zhang_quanlu.pdf>`__ 的研究支撑，具有以下特点：
* :doc:`简单的 API，让您轻松构建搜索空间 <construct_space>`
* :doc:`SOTA 架构搜索算法，以高效探索搜索空间 <exploration_strategy>`
* :doc:`后端支持，在大规模 AI 平台上运行实验 </experiment/overview>`
为什么使用 NNI 的架构搜索
-------------------------------
若没有 NNI，实现架构搜索将极具挑战性，主要包含以下三个方面。当用户想在自己的场景中尝试架构搜索技术时，NNI 提供的解决方案可以极大程度上减轻用户的工作量。
搜索空间设计
^^^^^^^^^^^^^^^^^^^
搜索空间定义了架构的可行域集合。为了简化搜索，我们通常需要结合任务相关的先验知识，减小搜索空间的规模。然而，这也引入了人类的偏见，在某种程度上可能会丧失突破人类认知的可能性。无论如何，对于初学者来说，搜索空间设计是一个极具挑战性的任务，因为他们可能无法在简单的空间和丰富的想象力之间取得平衡。
在 NNI 中，我们提供了不同层级的 API 来构建搜索空间。有 :doc:`高层 API <construct_space>`，引入大量先验，帮助用户迅速了解什么是好的架构或搜索空间；也有 :doc:`底层 API <mutator>`，提供了最底层的算子和图变换原语。
探索策略
^^^^^^^^^^^^^^^^^^^^
探索策略定义了如何探索搜索空间（通常是指数级规模的）。它包含经典的探索-利用权衡。一方面，我们希望快速找到性能良好的架构；而另一方面，我们也应避免过早收敛到次优架构的区域。我们往往需要通常通过反复试验找到特定场景的“最佳”探索策略。由于许多近期发表的探索策略都是使用自己的代码库实现的，因此从一个切换到另一个变得非常麻烦。
在 NNI 中，我们还提供了 :doc:`一系列的探索策略 <exploration_strategy>`。其中一些功能强大但耗时，而另一些可能不能找到最优架构但非常高效。鉴于所有策略都使用统一的用户接口实现，用户可以轻松找到符合他们需求的策略。
性能评估
^^^^^^^^^^^^^^^^^^^^^^
架构搜索的目标通常是找到能够在测试数据集表现理想的网络结构。性能评估的作用便是量化每个网络的好坏。其主要难点在于可扩展性，即如何在大规模训练平台上同时运行和管理多个试验。
在 NNI 中，我们使用 :doc:`evaluator <evaluator>` 来标准化性能评估流程。它负责估计模型的性能。NNI 内建了不少性能评估器，从最简单的交叉验证，到复杂的自定义配置。评估器在 *试验 (trials)* 中运行，可以通过我们强大的 :doc:`训练平台 </experiment/training_service/overview>` 将试验分发到大规模训练平台上。
教程
---------
要开始使用 NNI 架构搜索框架，我们建议至少阅读以下教程：
* :doc:`快速入门 </tutorials/hello_nas>`
* :doc:`构建搜索空间 <construct_space>`
* :doc:`探索策略 <exploration_strategy>`
* :doc:`评估器 <evaluator>`
资源
---------
以下文章将有助于更好地了解 NAS 的最新发展：
* `神经架构搜索：综述 <https://arxiv.org/abs/1808.05377>`__
* `神经架构搜索的综述：挑战和解决方案 <https://arxiv.org/abs/2006.02903>`__
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment