Skip to content

GitLab

"docs/primer.md" did not exist on "2134e3fd857d952e03ce76064fad5ac6e9036104"
Commit 9f73153f authored by zhanggzh's avatar zhanggzh
Browse files

add dtk24.04 code

parent eb77376e
Show whitespace changes
Inline Side-by-side
Showing with 2156 additions and 0 deletions
docs/source/locales/zh/LC_MESSAGES/quickstart.po 0 → 100644
View file @ 9f73153f
# SOME DESCRIPTIVE TITLE.
# Copyright (C) 2022, Microsoft
# This file is distributed under the same license as the NNI package.
# FIRST AUTHOR <EMAIL@ADDRESS>, 2022.
#
#, fuzzy
msgid ""
msgstr ""
"Project-Id-Version: NNI \n"
"Report-Msgid-Bugs-To: \n"
"POT-Creation-Date: 2022-04-13 03:14+0000\n"
"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
"Language-Team: LANGUAGE <LL@li.org>\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=utf-8\n"
"Content-Transfer-Encoding: 8bit\n"
"Generated-By: Babel 2.9.1\n"
#: ../../source/quickstart.rst:2
msgid "Quickstart"
msgstr ""
docs/source/locales/zh/LC_MESSAGES/sphinx.po 0 → 100644
View file @ 9f73153f
# SOME DESCRIPTIVE TITLE.
# Copyright (C) 2022, Microsoft
# This file is distributed under the same license as the NNI package.
# FIRST AUTHOR <EMAIL@ADDRESS>, 2022.
#
#, fuzzy
msgid ""
msgstr ""
"Project-Id-Version: NNI \n"
"Report-Msgid-Bugs-To: \n"
"POT-Creation-Date: 2022-04-12 17:35+0000\n"
"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
"Language-Team: LANGUAGE <LL@li.org>\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=utf-8\n"
"Content-Transfer-Encoding: 8bit\n"
"Generated-By: Babel 2.9.1\n"
#: ../../templates/globaltoc.html:6
msgid "Overview"
msgstr ""
docs/source/locales/zh/LC_MESSAGES/tutorials.po 0 → 100644
View file @ 9f73153f
# SOME DESCRIPTIVE TITLE.
# Copyright (C) 2022, Microsoft
# This file is distributed under the same license as the NNI package.
# FIRST AUTHOR <EMAIL@ADDRESS>, 2022.
#
#, fuzzy
msgid ""
msgstr ""
"Project-Id-Version: NNI \n"
"Report-Msgid-Bugs-To: \n"
"POT-Creation-Date: 2022-10-18 19:27+0800\n"
"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
"Language-Team: LANGUAGE <LL@li.org>\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=utf-8\n"
"Content-Transfer-Encoding: 8bit\n"
"Generated-By: Babel 2.9.1\n"
#: ../../source/tutorials/hello_nas.rst:13
msgid ""
"Click :ref:`here <sphx_glr_download_tutorials_hello_nas.py>` to download "
"the full example code"
msgstr ""
#: ../../source/tutorials/hello_nas.rst:22
msgid "Hello, NAS!"
msgstr ""
#: ../../source/tutorials/hello_nas.rst:24
msgid ""
"This is the 101 tutorial of Neural Architecture Search (NAS) on NNI. In "
"this tutorial, we will search for a neural architecture on MNIST dataset "
"with the help of NAS framework of NNI, i.e., *Retiarii*. We use multi-"
"trial NAS as an example to show how to construct and explore a model "
"space."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:28
msgid ""
"There are mainly three crucial components for a neural architecture "
"search task, namely,"
msgstr ""
#: ../../source/tutorials/hello_nas.rst:30
msgid "Model search space that defines a set of models to explore."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:31
msgid "A proper strategy as the method to explore this model space."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:32
msgid ""
"A model evaluator that reports the performance of every model in the "
"space."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:34
msgid ""
"Currently, PyTorch is the only supported framework by Retiarii, and we "
"have only tested **PyTorch 1.7 to 1.10**. This tutorial assumes PyTorch "
"context but it should also apply to other frameworks, which is in our "
"future plan."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:38
msgid "Define your Model Space"
msgstr ""
#: ../../source/tutorials/hello_nas.rst:40
msgid ""
"Model space is defined by users to express a set of models that users "
"want to explore, which contains potentially good-performing models. In "
"this framework, a model space is defined with two parts: a base model and"
" possible mutations on the base model."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:46
msgid "Define Base Model"
msgstr ""
#: ../../source/tutorials/hello_nas.rst:48
msgid ""
"Defining a base model is almost the same as defining a PyTorch (or "
"TensorFlow) model. Usually, you only need to replace the code ``import "
"torch.nn as nn`` with ``import nni.retiarii.nn.pytorch as nn`` to use our"
" wrapped PyTorch modules."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:52
msgid "Below is a very simple example of defining a base model."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:93
msgid ""
"Always keep in mind that you should use ``import nni.retiarii.nn.pytorch "
"as nn`` and :meth:`nni.retiarii.model_wrapper`. Many mistakes are a "
"result of forgetting one of those. Also, please use ``torch.nn`` for "
"submodules of ``nn.init``, e.g., ``torch.nn.init`` instead of "
"``nn.init``."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:98
msgid "Define Model Mutations"
msgstr ""
#: ../../source/tutorials/hello_nas.rst:100
msgid ""
"A base model is only one concrete model not a model space. We provide "
":doc:`API and Primitives </nas/construct_space>` for users to express how"
" the base model can be mutated. That is, to build a model space which "
"includes many models."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:103
msgid "Based on the above base model, we can define a model space as below."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:134
msgid "This results in the following code:"
msgstr ""
#: ../../source/tutorials/hello_nas.rst:189
#: ../../source/tutorials/hello_nas.rst:259
#: ../../source/tutorials/hello_nas.rst:471
#: ../../source/tutorials/hello_nas.rst:564
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:244
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:281
msgid "Out:"
msgstr ""
#: ../../source/tutorials/hello_nas.rst:210
msgid ""
"This example uses two mutation APIs, :class:`nn.LayerChoice "
"<nni.retiarii.nn.pytorch.LayerChoice>` and :class:`nn.InputChoice "
"<nni.retiarii.nn.pytorch.ValueChoice>`. :class:`nn.LayerChoice "
"<nni.retiarii.nn.pytorch.LayerChoice>` takes a list of candidate modules "
"(two in this example), one will be chosen for each sampled model. It can "
"be used like normal PyTorch module. :class:`nn.InputChoice "
"<nni.retiarii.nn.pytorch.ValueChoice>` takes a list of candidate values, "
"one will be chosen to take effect for each sampled model."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:219
msgid ""
"More detailed API description and usage can be found :doc:`here "
"</nas/construct_space>`."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:223
msgid ""
"We are actively enriching the mutation APIs, to facilitate easy "
"construction of model space. If the currently supported mutation APIs "
"cannot express your model space, please refer to :doc:`this doc "
"</nas/mutator>` for customizing mutators."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:228
msgid "Explore the Defined Model Space"
msgstr ""
#: ../../source/tutorials/hello_nas.rst:230
msgid ""
"There are basically two exploration approaches: (1) search by evaluating "
"each sampled model independently, which is the search approach in :ref"
":`multi-trial NAS <multi-trial-nas>` and (2) one-shot weight-sharing "
"based search, which is used in one-shot NAS. We demonstrate the first "
"approach in this tutorial. Users can refer to :ref:`here <one-shot-nas>` "
"for the second approach."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:235
msgid ""
"First, users need to pick a proper exploration strategy to explore the "
"defined model space. Second, users need to pick or customize a model "
"evaluator to evaluate the performance of each explored model."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:239
msgid "Pick an exploration strategy"
msgstr ""
#: ../../source/tutorials/hello_nas.rst:241
msgid ""
"Retiarii supports many :doc:`exploration strategies "
"</nas/exploration_strategy>`."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:243
msgid "Simply choosing (i.e., instantiate) an exploration strategy as below."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:273
msgid "Pick or customize a model evaluator"
msgstr ""
#: ../../source/tutorials/hello_nas.rst:275
msgid ""
"In the exploration process, the exploration strategy repeatedly generates"
" new models. A model evaluator is for training and validating each "
"generated model to obtain the model's performance. The performance is "
"sent to the exploration strategy for the strategy to generate better "
"models."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:279
msgid ""
"Retiarii has provided :doc:`built-in model evaluators </nas/evaluator>`, "
"but to start with, it is recommended to use :class:`FunctionalEvaluator "
"<nni.retiarii.evaluator.FunctionalEvaluator>`, that is, to wrap your own "
"training and evaluation code with one single function. This function "
"should receive one single model class and uses "
":func:`nni.report_final_result` to report the final score of this model."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:284
msgid ""
"An example here creates a simple evaluator that runs on MNIST dataset, "
"trains for 2 epochs, and reports its validation accuracy."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:367
msgid "Create the evaluator"
msgstr ""
#: ../../source/tutorials/hello_nas.rst:386
msgid ""
"The ``train_epoch`` and ``test_epoch`` here can be any customized "
"function, where users can write their own training recipe."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:389
msgid ""
"It is recommended that the ``evaluate_model`` here accepts no additional "
"arguments other than ``model_cls``. However, in the :doc:`advanced "
"tutorial </nas/evaluator>`, we will show how to use additional arguments "
"in case you actually need those. In future, we will support mutation on "
"the arguments of evaluators, which is commonly called \"Hyper-parmeter "
"tuning\"."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:394
msgid "Launch an Experiment"
msgstr ""
#: ../../source/tutorials/hello_nas.rst:396
msgid ""
"After all the above are prepared, it is time to start an experiment to do"
" the model search. An example is shown below."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:417
msgid ""
"The following configurations are useful to control how many trials to run"
" at most / at the same time."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:436
msgid ""
"Remember to set the following config if you want to GPU. "
"``use_active_gpu`` should be set true if you wish to use an occupied GPU "
"(possibly running a GUI)."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:456
msgid ""
"Launch the experiment. The experiment should take several minutes to "
"finish on a workstation with 2 GPUs."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:495
msgid ""
"Users can also run Retiarii Experiment with :doc:`different training "
"services </experiment/training_service/overview>` besides ``local`` "
"training service."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:499
msgid "Visualize the Experiment"
msgstr ""
#: ../../source/tutorials/hello_nas.rst:501
msgid ""
"Users can visualize their experiment in the same way as visualizing a "
"normal hyper-parameter tuning experiment. For example, open "
"``localhost:8081`` in your browser, 8081 is the port that you set in "
"``exp.run``. Please refer to :doc:`here "
"</experiment/web_portal/web_portal>` for details."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:505
msgid ""
"We support visualizing models with 3rd-party visualization engines (like "
"`Netron <https://netron.app/>`__). This can be used by clicking "
"``Visualization`` in detail panel for each trial. Note that current "
"visualization is based on `onnx <https://onnx.ai/>`__ , thus "
"visualization is not feasible if the model cannot be exported into onnx."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:510
msgid ""
"Built-in evaluators (e.g., Classification) will automatically export the "
"model into a file. For your own evaluator, you need to save your file "
"into ``$NNI_OUTPUT_DIR/model.onnx`` to make this work. For instance,"
msgstr ""
#: ../../source/tutorials/hello_nas.rst:541
msgid "Relaunch the experiment, and a button is shown on Web portal."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:546
msgid "Export Top Models"
msgstr ""
#: ../../source/tutorials/hello_nas.rst:548
msgid ""
"Users can export top models after the exploration is done using "
"``export_top_models``."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:575
msgid ""
"The output is ``json`` object which records the mutation actions of the "
"top model. If users want to output source code of the top model, they can"
" use :ref:`graph-based execution engine <graph-based-execution-engine>` "
"for the experiment, by simply adding the following two lines."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:597
msgid "**Total running time of the script:** ( 2 minutes 4.499 seconds)"
msgstr ""
#: ../../source/tutorials/hello_nas.rst:612
msgid ":download:`Download Python source code: hello_nas.py <hello_nas.py>`"
msgstr ""
#: ../../source/tutorials/hello_nas.rst:618
msgid ":download:`Download Jupyter notebook: hello_nas.ipynb <hello_nas.ipynb>`"
msgstr ""
#: ../../source/tutorials/hello_nas.rst:625
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:335
#: ../../source/tutorials/pruning_quick_start_mnist.rst:343
msgid "`Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_"
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:13
msgid ""
"Click :ref:`here "
"<sphx_glr_download_tutorials_hpo_quickstart_pytorch_main.py>` to download"
" the full example code"
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:22
msgid "HPO Quickstart with PyTorch"
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:23
msgid ""
"This tutorial optimizes the model in `official PyTorch quickstart`_ with "
"auto-tuning."
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:25
msgid "The tutorial consists of 4 steps:"
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:27
msgid "Modify the model for auto-tuning."
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:28
msgid "Define hyperparameters' search space."
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:29
msgid "Configure the experiment."
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:30
msgid "Run the experiment."
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:37
msgid "Step 1: Prepare the model"
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:38
msgid "In first step, we need to prepare the model to be tuned."
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:40
msgid ""
"The model should be put in a separate script. It will be evaluated many "
"times concurrently, and possibly will be trained on distributed "
"platforms."
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:44
msgid "In this tutorial, the model is defined in :doc:`model.py <model>`."
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:46
msgid "In short, it is a PyTorch model with 3 additional API calls:"
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:48
msgid ""
"Use :func:`nni.get_next_parameter` to fetch the hyperparameters to be "
"evalutated."
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:49
msgid ""
"Use :func:`nni.report_intermediate_result` to report per-epoch accuracy "
"metrics."
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:50
msgid "Use :func:`nni.report_final_result` to report final accuracy."
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:52
msgid "Please understand the model code before continue to next step."
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:57
msgid "Step 2: Define search space"
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:58
msgid ""
"In model code, we have prepared 3 hyperparameters to be tuned: "
"*features*, *lr*, and *momentum*."
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:61
msgid ""
"Here we need to define their *search space* so the tuning algorithm can "
"sample them in desired range."
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:63
msgid "Assuming we have following prior knowledge for these hyperparameters:"
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:65
msgid "*features* should be one of 128, 256, 512, 1024."
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:66
msgid ""
"*lr* should be a float between 0.0001 and 0.1, and it follows exponential"
" distribution."
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:67
msgid "*momentum* should be a float between 0 and 1."
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:69
msgid ""
"In NNI, the space of *features* is called ``choice``; the space of *lr* "
"is called ``loguniform``; and the space of *momentum* is called "
"``uniform``. You may have noticed, these names are derived from "
"``numpy.random``."
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:74
msgid ""
"For full specification of search space, check :doc:`the reference "
"</hpo/search_space>`."
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:76
msgid "Now we can define the search space as follow:"
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:99
msgid "Step 3: Configure the experiment"
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:100
msgid ""
"NNI uses an *experiment* to manage the HPO process. The *experiment "
"config* defines how to train the models and how to explore the search "
"space."
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:103
msgid ""
"In this tutorial we use a *local* mode experiment, which means models "
"will be trained on local machine, without using any special training "
"platform."
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:122
msgid "Now we start to configure the experiment."
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:125
msgid "Configure trial code"
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:126
msgid ""
"In NNI evaluation of each hyperparameter set is called a *trial*. So the "
"model script is called *trial code*."
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:144
msgid ""
"When ``trial_code_directory`` is a relative path, it relates to current "
"working directory. To run ``main.py`` in a different path, you can set "
"trial code directory to ``Path(__file__).parent``. (`__file__ "
"<https://docs.python.org/3.10/reference/datamodel.html#index-43>`__ is "
"only available in standard Python, not in Jupyter Notebook.)"
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:151
msgid ""
"If you are using Linux system without Conda, you may need to change "
"``\"python model.py\"`` to ``\"python3 model.py\"``."
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:157
msgid "Configure search space"
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:175
msgid "Configure tuning algorithm"
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:176
msgid "Here we use :doc:`TPE tuner </hpo/tuners>`."
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:195
msgid "Configure how many trials to run"
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:196
msgid ""
"Here we evaluate 10 sets of hyperparameters in total, and concurrently "
"evaluate 2 sets at a time."
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:213
msgid "You may also set ``max_experiment_duration = '1h'`` to limit running time."
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:215
msgid ""
"If neither ``max_trial_number`` nor ``max_experiment_duration`` are set, "
"the experiment will run forever until you press Ctrl-C."
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:220
msgid ""
"``max_trial_number`` is set to 10 here for a fast example. In real world "
"it should be set to a larger number. With default config TPE tuner "
"requires 20 trials to warm up."
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:227
msgid "Step 4: Run the experiment"
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:228
msgid ""
"Now the experiment is ready. Choose a port and launch it. (Here we use "
"port 8080.)"
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:230
msgid ""
"You can use the web portal to view experiment status: "
"http://localhost:8080."
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:260
msgid "After the experiment is done"
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:261
msgid "Everything is done and it is safe to exit now. The following are optional."
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:263
msgid ""
"If you are using standard Python instead of Jupyter Notebook, you can add"
" ``input()`` or ``signal.pause()`` to prevent Python from exiting, "
"allowing you to view the web portal after the experiment is done."
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:293
msgid ""
":meth:`nni.experiment.Experiment.stop` is automatically invoked when "
"Python exits, so it can be omitted in your code."
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:296
msgid ""
"After the experiment is stopped, you can run "
":meth:`nni.experiment.Experiment.view` to restart web portal."
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:300
msgid ""
"This example uses :doc:`Python API </reference/experiment>` to create "
"experiment."
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:302
msgid ""
"You can also create and manage experiments with :doc:`command line tool "
"<../hpo_nnictl/nnictl>`."
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:307
msgid "**Total running time of the script:** ( 1 minutes 24.367 seconds)"
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:322
msgid ":download:`Download Python source code: main.py <main.py>`"
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:328
msgid ":download:`Download Jupyter notebook: main.ipynb <main.ipynb>`"
msgstr ""
#: ../../source/tutorials/pruning_quick_start_mnist.rst:13
msgid ""
"Click :ref:`here "
"<sphx_glr_download_tutorials_pruning_quick_start_mnist.py>` to download "
"the full example code"
msgstr ""
#: ../../source/tutorials/pruning_quick_start_mnist.rst:22
msgid "Pruning Quickstart"
msgstr ""
#: ../../source/tutorials/pruning_quick_start_mnist.rst:24
msgid "Here is a three-minute video to get you started with model pruning."
msgstr ""
#: ../../source/tutorials/pruning_quick_start_mnist.rst:29
msgid ""
"Model pruning is a technique to reduce the model size and computation by "
"reducing model weight size or intermediate state size. There are three "
"common practices for pruning a DNN model:"
msgstr ""
#: ../../source/tutorials/pruning_quick_start_mnist.rst:32
msgid "Pre-training a model -> Pruning the model -> Fine-tuning the pruned model"
msgstr ""
#: ../../source/tutorials/pruning_quick_start_mnist.rst:33
msgid ""
"Pruning a model during training (i.e., pruning aware training) -> Fine-"
"tuning the pruned model"
msgstr ""
#: ../../source/tutorials/pruning_quick_start_mnist.rst:34
msgid "Pruning a model -> Training the pruned model from scratch"
msgstr ""
#: ../../source/tutorials/pruning_quick_start_mnist.rst:36
msgid ""
"NNI supports all of the above pruning practices by working on the key "
"pruning stage. Following this tutorial for a quick look at how to use NNI"
" to prune a model in a common practice."
msgstr ""
#: ../../source/tutorials/pruning_quick_start_mnist.rst:42
msgid "Preparation"
msgstr ""
#: ../../source/tutorials/pruning_quick_start_mnist.rst:44
msgid ""
"In this tutorial, we use a simple model and pre-trained on MNIST dataset."
" If you are familiar with defining a model and training in pytorch, you "
"can skip directly to `Pruning Model`_."
msgstr ""
#: ../../source/tutorials/pruning_quick_start_mnist.rst:122
msgid "Pruning Model"
msgstr ""
#: ../../source/tutorials/pruning_quick_start_mnist.rst:124
msgid ""
"Using L1NormPruner to prune the model and generate the masks. Usually, a "
"pruner requires original model and ``config_list`` as its inputs. "
"Detailed about how to write ``config_list`` please refer "
":doc:`compression config specification "
"<../compression/compression_config_list>`."
msgstr ""
#: ../../source/tutorials/pruning_quick_start_mnist.rst:128
msgid ""
"The following `config_list` means all layers whose type is `Linear` or "
"`Conv2d` will be pruned, except the layer named `fc3`, because `fc3` is "
"`exclude`. The final sparsity ratio for each layer is 50%. The layer "
"named `fc3` will not be pruned."
msgstr ""
#: ../../source/tutorials/pruning_quick_start_mnist.rst:154
msgid "Pruners usually require `model` and `config_list` as input arguments."
msgstr ""
#: ../../source/tutorials/pruning_quick_start_mnist.rst:229
msgid ""
"Speedup the original model with masks, note that `ModelSpeedup` requires "
"an unwrapped model. The model becomes smaller after speedup, and reaches "
"a higher sparsity ratio because `ModelSpeedup` will propagate the masks "
"across layers."
msgstr ""
#: ../../source/tutorials/pruning_quick_start_mnist.rst:262
msgid "the model will become real smaller after speedup"
msgstr ""
#: ../../source/tutorials/pruning_quick_start_mnist.rst:298
msgid "Fine-tuning Compacted Model"
msgstr ""
#: ../../source/tutorials/pruning_quick_start_mnist.rst:299
msgid ""
"Note that if the model has been sped up, you need to re-initialize a new "
"optimizer for fine-tuning. Because speedup will replace the masked big "
"layers with dense small ones."
msgstr ""
#: ../../source/tutorials/pruning_quick_start_mnist.rst:320
msgid "**Total running time of the script:** ( 1 minutes 0.810 seconds)"
msgstr ""
#: ../../source/tutorials/pruning_quick_start_mnist.rst:332
msgid ""
":download:`Download Python source code: pruning_quick_start_mnist.py "
"<pruning_quick_start_mnist.py>`"
msgstr ""
#: ../../source/tutorials/pruning_quick_start_mnist.rst:336
msgid ""
":download:`Download Jupyter notebook: pruning_quick_start_mnist.ipynb "
"<pruning_quick_start_mnist.ipynb>`"
msgstr ""
#~ msgid "**Total running time of the script:** ( 2 minutes 15.810 seconds)"
#~ msgstr ""
#~ msgid "NNI HPO Quickstart with PyTorch"
#~ msgstr ""
#~ msgid ""
#~ "There is also a :doc:`TensorFlow "
#~ "version<../hpo_quickstart_tensorflow/main>` if you "
#~ "prefer it."
#~ msgstr ""
#~ msgid ""
#~ "You can also create and manage "
#~ "experiments with :doc:`command line tool "
#~ "</reference/nnictl>`."
#~ msgstr ""
#~ msgid "**Total running time of the script:** ( 1 minutes 24.393 seconds)"
#~ msgstr ""
#~ msgid "**Total running time of the script:** ( 0 minutes 58.337 seconds)"
#~ msgstr ""
#~ msgid "**Total running time of the script:** ( 1 minutes 30.730 seconds)"
#~ msgstr ""
docs/source/nas/advanced_usage.rst 0 → 100644
View file @ 9f73153f
Advanced Usage
==============
.. toctree::
:maxdepth: 2
execution_engine
hardware_aware_nas
mutator
customize_strategy
serialization
benchmarks_toctree
docs/source/nas/benchmarks.rst 0 → 100644
View file @ 9f73153f
NAS Benchmark
=============
.. note:: :doc:`Example usage of NAS benchmarks </tutorials/nasbench_as_dataset>`.
To improve the reproducibility of NAS algorithms as well as reducing computing resource requirements, researchers proposed a series of NAS benchmarks such as `NAS-Bench-101 <https://arxiv.org/abs/1902.09635>`__, `NAS-Bench-201 <https://arxiv.org/abs/2001.00326>`__, `NDS <https://arxiv.org/abs/1905.13214>`__, etc. NNI provides a query interface for users to acquire these benchmarks. Within just a few lines of code, researcher are able to evaluate their NAS algorithms easily and fairly by utilizing these benchmarks.
Prerequisites
-------------
* Please prepare a folder to household all the benchmark databases. By default, it can be found at ``${HOME}/.cache/nni/nasbenchmark``. Or you can place it anywhere you like, and specify it in ``NASBENCHMARK_DIR`` via ``export NASBENCHMARK_DIR=/path/to/your/nasbenchmark`` before importing NNI.
* Please install ``peewee`` via ``pip3 install peewee``, which NNI uses to connect to database.
Data Preparation
----------------
Option 1 (Recommended)
^^^^^^^^^^^^^^^^^^^^^^
You can download the preprocessed benchmark files via ``python -m nni.nas.benchmarks.download <benchmark_name>``, where ``<benchmark_name>`` can be ``nasbench101``, ``nasbench201``, and etc. Add ``--help`` to the command for supported command line arguments.
Option 2
^^^^^^^^
.. note:: If you have files that are processed before v2.5, it is recommended that you delete them and try option 1.
#. Clone NNI to your machine and enter ``examples/nas/benchmarks`` directory.
.. code-block:: bash
git clone -b ${NNI_VERSION} https://github.com/microsoft/nni
cd nni/examples/nas/benchmarks
Replace ``${NNI_VERSION}`` with a released version name or branch name, e.g., ``v2.4``.
#. Install dependencies via ``pip3 install -r xxx.requirements.txt``. ``xxx`` can be ``nasbench101``, ``nasbench201`` or ``nds``.
#. Generate the database via ``./xxx.sh``. The directory that stores the benchmark file can be configured with ``NASBENCHMARK_DIR`` environment variable, which defaults to ``~/.nni/nasbenchmark``. Note that the NAS-Bench-201 dataset will be downloaded from a google drive.
Please make sure there is at least 10GB free disk space and note that the conversion process can take up to hours to complete.
Example Usages
--------------
Please refer to :doc:`examples usages of Benchmarks API </tutorials/nasbench_as_dataset>`.
NAS-Bench-101
-------------
* `Paper link <https://arxiv.org/abs/1902.09635>`__
* `Open-source <https://github.com/google-research/nasbench>`__
NAS-Bench-101 contains 423,624 unique neural networks, combined with 4 variations in number of epochs (4, 12, 36, 108), each of which is trained 3 times. It is a cell-wise search space, which constructs and stacks a cell by enumerating DAGs with at most 7 operators, and no more than 9 connections. All operators can be chosen from ``CONV3X3_BN_RELU``, ``CONV1X1_BN_RELU`` and ``MAXPOOL3X3``, except the first operator (always ``INPUT``\ ) and last operator (always ``OUTPUT``\ ).
Notably, NAS-Bench-101 eliminates invalid cells (e.g., there is no path from input to output, or there is redundant computation). Furthermore, isomorphic cells are de-duplicated, i.e., all the remaining cells are computationally unique.
See :doc:`example usages </tutorials/nasbench_as_dataset>` and :ref:`API references <nas-bench-101-reference>`.
NAS-Bench-201
-------------
* `Paper link <https://arxiv.org/abs/2001.00326>`__
* `Open-source API <https://github.com/D-X-Y/NAS-Bench-201>`__
* `Implementations <https://github.com/D-X-Y/AutoDL-Projects>`__
NAS-Bench-201 is a cell-wise search space that views nodes as tensors and edges as operators. The search space contains all possible densely-connected DAGs with 4 nodes, resulting in 15,625 candidates in total. Each operator (i.e., edge) is selected from a pre-defined operator set (\ ``NONE``, ``SKIP_CONNECT``, ``CONV_1X1``, ``CONV_3X3`` and ``AVG_POOL_3X3``\ ). Training appraoches vary in the dataset used (CIFAR-10, CIFAR-100, ImageNet) and number of epochs scheduled (12 and 200). Each combination of architecture and training approach is repeated 1 - 3 times with different random seeds.
See :doc:`example usages </tutorials/nasbench_as_dataset>` and :ref:`API references <nas-bench-201-reference>`.
NDS
---
* `Paper link <https://arxiv.org/abs/1905.13214>`__
* `Open-source <https://github.com/facebookresearch/nds>`__
*On Network Design Spaces for Visual Recognition* released trial statistics of over 100,000 configurations (models + hyper-parameters) sampled from multiple model families, including vanilla (feedforward network loosely inspired by VGG), ResNet and ResNeXt (residual basic block and residual bottleneck block) and NAS cells (following popular design from NASNet, Ameoba, PNAS, ENAS and DARTS). Most configurations are trained only once with a fixed seed, except a few that are trained twice or three times.
Instead of storing results obtained with different configurations in separate files, we dump them into one single database to enable comparison in multiple dimensions. Specifically, we use ``model_family`` to distinguish model types, ``model_spec`` for all hyper-parameters needed to build this model, ``cell_spec`` for detailed information on operators and connections if it is a NAS cell, ``generator`` to denote the sampling policy through which this configuration is generated. Refer to API documentation for details.
Here is a list of available operators used in NDS.
.. automodule:: nni.nas.benchmarks.nds.constants
:noindex:
See :doc:`example usages </tutorials/nasbench_as_dataset>` and :ref:`API references <nds-reference>`.
docs/source/nas/benchmarks_toctree.rst 0 → 100644
View file @ 9f73153f
NAS Benchmark
=============
.. toctree::
:hidden:
Overview <benchmarks>
Examples </tutorials/nasbench_as_dataset>
docs/source/nas/construct_space.rst 0 → 100644
View file @ 9f73153f
Construct Model Space
=====================
NNI provides powerful (and multi-level) APIs for users to easily express model space (or search space).
* *Mutation Primitives*: high-level APIs (e.g., ValueChoice, LayerChoice) that are utilities to build blocks in search space. In most cases, mutation pritimives should be straightforward yet expressive enough. **We strongly recommend users to try them first,** and report issues if those APIs are not satisfying.
* *Hyper-module Library*: plug-and-play modules that are proved useful. They are usually well studied in research, and comes with pre-searched results. (For example, the optimal activation function in `AutoActivation <https://arxiv.org/abs/1710.05941>`__ is reported to be `Swish <https://pytorch.org/docs/stable/generated/torch.nn.SiLU.html>`__).
* *Mutator*: for advanced users only. NNI provides interface to customize new mutators for expressing more complicated model spaces.
The following table summarizes all the APIs we have provided for constructing search space.
.. list-table::
:header-rows: 1
:widths: auto
* - Name
- Category
- Brief Description
* - :class:`LayerChoice <nni.retiarii.nn.pytorch.LayerChoice>`
- :ref:`Mutation Primitives <mutation-primitives>`
- Select from some PyTorch modules
* - :class:`InputChoice <nni.retiarii.nn.pytorch.InputChoice>`
- :ref:`Mutation Primitives <mutation-primitives>`
- Select from some inputs (tensors)
* - :class:`ValueChoice <nni.retiarii.nn.pytorch.ValueChoice>`
- :ref:`Mutation Primitives <mutation-primitives>`
- Select from some candidate values
* - :class:`Repeat <nni.retiarii.nn.pytorch.Repeat>`
- :ref:`Mutation Primitives <mutation-primitives>`
- Repeat a block by a variable number of times
* - :class:`Cell <nni.retiarii.nn.pytorch.Cell>`
- :ref:`Mutation Primitives <mutation-primitives>`
- Cell structure popularly used in literature
* - :class:`NasBench101Cell <nni.retiarii.nn.pytorch.NasBench101Cell>`
- :ref:`Mutation Primitives <mutation-primitives>`
- Cell structure (variant) proposed by NAS-Bench-101
* - :class:`NasBench201Cell <nni.retiarii.nn.pytorch.NasBench201Cell>`
- :ref:`Mutation Primitives <mutation-primitives>`
- Cell structure (variant) proposed by NAS-Bench-201
* - :class:`AutoActivation <nni.retiarii.nn.pytorch.AutoActivation>`
- :ref:`Hyper-modules Library <hyper-modules>`
- Searching for activation functions
* - :class:`Mutator <nni.retiarii.Mutator>`
- :doc:`Mutator <mutator>`
- Flexible mutations on graphs. :doc:`See tutorial here <mutator>`
docs/source/nas/customize_strategy.rst 0 → 100644
View file @ 9f73153f
Customize Exploration Strategy
==============================
Customize Multi-trial Strategy
------------------------------
If users want to innovate a new exploration strategy, they can easily customize a new one following the interface provided by NNI. Specifically, users should inherit the base strategy class :class:`nni.retiarii.strategy.BaseStrategy`, then implement the member function ``run``. This member function takes ``base_model`` and ``applied_mutators`` as its input arguments. It can simply apply the user specified mutators in ``applied_mutators`` onto ``base_model`` to generate a new model. When a mutator is applied, it should be bound with a sampler (e.g., ``RandomSampler``). Every sampler implements the ``choice`` function which chooses value(s) from candidate values. The ``choice`` functions invoked in mutators are executed with the sampler.
Below is a very simple random strategy, which makes the choices completely random.
.. code-block:: python
from nni.retiarii import Sampler
class RandomSampler(Sampler):
def choice(self, candidates, mutator, model, index):
return random.choice(candidates)
class RandomStrategy(BaseStrategy):
def __init__(self):
self.random_sampler = RandomSampler()
def run(self, base_model, applied_mutators):
_logger.info('stargety start...')
while True:
avail_resource = query_available_resources()
if avail_resource > 0:
model = base_model
_logger.info('apply mutators...')
_logger.info('mutators: %s', str(applied_mutators))
for mutator in applied_mutators:
mutator.bind_sampler(self.random_sampler)
model = mutator.apply(model)
# run models
submit_models(model)
else:
time.sleep(2)
You can find that this strategy does not know the search space beforehand, it passively makes decisions every time ``choice`` is invoked from mutators. If a strategy wants to know the whole search space before making any decision (e.g., TPE, SMAC), it can use ``dry_run`` function provided by ``Mutator`` to obtain the space. An example strategy can be found :githublink:`here <nni/retiarii/strategy/tpe_strategy.py>`.
After generating a new model, the strategy can use our provided APIs (e.g., :func:`nni.retiarii.execution.submit_models`, :func:`nni.retiarii.execution.is_stopped_exec`) to submit the model and get its reported results.
Customize a New One-shot Trainer (legacy)
-----------------------------------------
One-shot trainers should inherit :class:`nni.retiarii.oneshot.BaseOneShotTrainer`, and need to implement ``fit()`` (used to conduct the fitting and searching process) and ``export()`` method (used to return the searched best architecture).
Writing a one-shot trainer is very different to single-arch evaluator. First of all, there are no more restrictions on init method arguments, any Python arguments are acceptable. Secondly, the model fed into one-shot trainers might be a model with Retiarii-specific modules, such as LayerChoice and InputChoice. Such model cannot directly forward-propagate and trainers need to decide how to handle those modules.
A typical example is DartsTrainer, where learnable-parameters are used to combine multiple choices in LayerChoice. Retiarii provides ease-to-use utility functions for module-replace purposes, namely :meth:`nni.retiarii.oneshot.pytorch.utils.replace_layer_choice`, :meth:`nni.retiarii.oneshot.pytorch.utils.replace_input_choice`. A simplified example is as follows:
.. code-block:: python
from nni.retiarii.oneshot import BaseOneShotTrainer
from nni.retiarii.oneshot.pytorch.utils import replace_layer_choice, replace_input_choice
class DartsLayerChoice(nn.Module):
def __init__(self, layer_choice):
super(DartsLayerChoice, self).__init__()
self.name = layer_choice.label
self.op_choices = nn.ModuleDict(layer_choice.named_children())
self.alpha = nn.Parameter(torch.randn(len(self.op_choices)) * 1e-3)
def forward(self, *args, **kwargs):
op_results = torch.stack([op(*args, **kwargs) for op in self.op_choices.values()])
alpha_shape = [-1] + [1] * (len(op_results.size()) - 1)
return torch.sum(op_results * F.softmax(self.alpha, -1).view(*alpha_shape), 0)
class DartsTrainer(BaseOneShotTrainer):
def __init__(self, model, loss, metrics, optimizer):
self.model = model
self.loss = loss
self.metrics = metrics
self.num_epochs = 10
self.nas_modules = []
replace_layer_choice(self.model, DartsLayerChoice, self.nas_modules)
... # init dataloaders and optimizers
def fit(self):
for i in range(self.num_epochs):
for (trn_X, trn_y), (val_X, val_y) in zip(self.train_loader, self.valid_loader):
self.train_architecture(val_X, val_y)
self.train_model_weight(trn_X, trn_y)
@torch.no_grad()
def export(self):
result = dict()
for name, module in self.nas_modules:
if name not in result:
result[name] = select_best_of_module(module)
return result
The full code of DartsTrainer is available to Retiarii source code. Please have a check at :githublink:`DartsTrainer <nni/retiarii/oneshot/pytorch/darts.py>`.
docs/source/nas/evaluator.rst 0 → 100644
View file @ 9f73153f
Model Evaluator
===============
A model evaluator is for training and validating each generated model. They are necessary to evaluate the performance of new explored models.
.. _functional-evaluator:
Customize Evaluator with Any Function
-------------------------------------
The simplest way to customize a new evaluator is with :class:`~nni.retiarii.evaluator.FunctionalEvaluator`, which is very easy when training code is already available. Users only need to write a fit function that wraps everything, which usually includes training, validating and testing of a single model. This function takes one positional arguments (``model_cls``) and possible keyword arguments. The keyword arguments (other than ``model_cls``) are fed to :class:`~nni.retiarii.evaluator.FunctionalEvaluator` as its initialization parameters (note that they will be :doc:`serialized <./serialization>`). In this way, users get everything under their control, but expose less information to the framework and as a result, further optimizations like :ref:`CGO <cgo-execution-engine>` might be not feasible. An example is as belows:
.. code-block:: python
from nni.retiarii.evaluator import FunctionalEvaluator
from nni.retiarii.experiment.pytorch import RetiariiExperiment
def fit(model_cls, dataloader):
model = model_cls()
train(model, dataloader)
acc = test(model, dataloader)
nni.report_final_result(acc)
# The dataloader will be serialized, thus ``nni.trace`` is needed here.
# See serialization tutorial for more details.
evaluator = FunctionalEvaluator(fit, dataloader=nni.trace(DataLoader)(foo, bar))
experiment = RetiariiExperiment(base_model, evaluator, mutators, strategy)
.. tip::
When using customized evaluators, if you want to visualize models, you need to export your model and save it into ``$NNI_OUTPUT_DIR/model.onnx`` in your evaluator. An example here:
.. code-block:: python
def fit(model_cls):
model = model_cls()
onnx_path = Path(os.environ.get('NNI_OUTPUT_DIR', '.')) / 'model.onnx'
onnx_path.parent.mkdir(exist_ok=True)
dummy_input = torch.randn(10, 3, 224, 224)
torch.onnx.export(model, dummy_input, onnx_path)
# the rest of training code here
If the conversion is successful, the model will be able to be visualized with powerful tools `Netron <https://netron.app/>`__.
Use Evaluators to Train and Evaluate Models
-------------------------------------------
Users can use evaluators to train or evaluate a single, concrete architecture. This is very useful when:
* Debugging your evaluator against a baseline model.
* Fully train, validate and test your model after the search process is complete.
The usage is shown below:
.. code-block:: python
# Class definition of single model, for example, ResNet.
class SingleModel(nn.Module):
def __init__(): # Can't have init parameters here.
...
# Use a callable returning a model
evaluator.evaluate(SingleModel)
# Or initialize the model beforehand
evaluator.evaluate(SingleModel())
The underlying implementation of :meth:`~nni.retiarii.Evaluator.evaluate` depends on concrete evaluator that you used.
For example, if :class:`~nni.retiarii.evaluator.FunctionalEvaluator` is used, it will run your customized fit function.
If lightning evaluators like :class:`nni.retiarii.evaluator.pytorch.Classification` are used, it will invoke the ``trainer.fit()`` of Lightning.
To evaluate an architecture that is exported from experiment (i.e., from :meth:`~nni.retiarii.experiment.pytorch.RetiariiExperiment.export_top_models`), use :func:`nni.retiarii.fixed_arch` to instantiate the exported model::
with fixed_arch(exported_model):
model = ModelSpace()
# Then use evaluator.evaluate
evaluator.evaluate(model)
.. tip:: There is a way to port the trained checkpoint of super-net produced by one-shot strategies, to the concrete chosen architecture, thanks to :func:`nni.retiarii.utils.original_state_dict_hooks`. This is helpful in implementing recent multi-stage NAS algorithms like `SPOS <https://arxiv.org/abs/1904.00420>`__.
.. _lightning-evaluator:
Evaluators with PyTorch-Lightning
---------------------------------
Use Built-in Evaluators
^^^^^^^^^^^^^^^^^^^^^^^
NNI provides some commonly used model evaluators for users' convenience. These evaluators are built upon the awesome library PyTorch-Lightning. Read the :doc:`reference </reference/nas/evaluator>` for their detailed usages.
* :class:`nni.retiarii.evaluator.pytorch.Classification`: for classification tasks.
* :class:`nni.retiarii.evaluator.pytorch.Regression`: for regression tasks.
We recommend to read the :doc:`serialization tutorial <serialization>` before using these evaluators. A few notes to summarize the tutorial:
1. :class:`nni.retiarii.evaluator.pytorch.DataLoader` should be used in place of ``torch.utils.data.DataLoader``.
2. The datasets used in data-loader should be decorated with :meth:`nni.trace` recursively.
For example,
.. code-block:: python
import nni.retiarii.evaluator.pytorch.lightning as pl
from torchvision import transforms
transform = nni.trace(transforms.Compose, [nni.trace(transforms.ToTensor()), nni.trace(transforms.Normalize, (0.1307,), (0.3081,))])
train_dataset = nni.trace(MNIST, root='data/mnist', train=True, download=True, transform=transform)
test_dataset = nni.trace(MNIST, root='data/mnist', train=False, download=True, transform=transform)
# pl.DataLoader and pl.Classification is already traced and supports serialization.
evaluator = pl.Classification(train_dataloaders=pl.DataLoader(train_dataset, batch_size=100),
val_dataloaders=pl.DataLoader(test_dataset, batch_size=100),
max_epochs=10)
Customize Evaluator with PyTorch-Lightning
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Another approach is to write training code in PyTorch-Lightning style, that is, to write a LightningModule that defines all elements needed for training (e.g., loss function, optimizer) and to define a trainer that takes (optional) dataloaders to execute the training. Before that, please read the `document of PyTorch-lightning <https://pytorch-lightning.readthedocs.io/>`__ to learn the basic concepts and components provided by PyTorch-lightning.
In practice, writing a new training module in Retiarii should inherit :class:`nni.retiarii.evaluator.pytorch.LightningModule`, which has a ``set_model`` that will be called after ``__init__`` to save the candidate model (generated by strategy) as ``self.model``. The rest of the process (like ``training_step``) should be the same as writing any other lightning module. Evaluators should also communicate with strategies via two API calls (:meth:`nni.report_intermediate_result` for periodical metrics and :meth:`nni.report_final_result` for final metrics), added in ``on_validation_epoch_end`` and ``teardown`` respectively.
An example is as follows:
.. code-block:: python
from nni.retiarii.evaluator.pytorch.lightning import LightningModule # please import this one
@nni.trace
class AutoEncoder(LightningModule):
def __init__(self):
super().__init__()
self.decoder = nn.Sequential(
nn.Linear(3, 64),
nn.ReLU(),
nn.Linear(64, 28*28)
)
def forward(self, x):
embedding = self.model(x) # let's search for encoder
return embedding
def training_step(self, batch, batch_idx):
# training_step defined the train loop.
# It is independent of forward
x, y = batch
x = x.view(x.size(0), -1)
z = self.model(x) # model is the one that is searched for
x_hat = self.decoder(z)
loss = F.mse_loss(x_hat, x)
# Logging to TensorBoard by default
self.log('train_loss', loss)
return loss
def validation_step(self, batch, batch_idx):
x, y = batch
x = x.view(x.size(0), -1)
z = self.model(x)
x_hat = self.decoder(z)
loss = F.mse_loss(x_hat, x)
self.log('val_loss', loss)
def configure_optimizers(self):
optimizer = torch.optim.Adam(self.parameters(), lr=1e-3)
return optimizer
def on_validation_epoch_end(self):
nni.report_intermediate_result(self.trainer.callback_metrics['val_loss'].item())
def teardown(self, stage):
if stage == 'fit':
nni.report_final_result(self.trainer.callback_metrics['val_loss'].item())
.. note::
If you are trying to use your customized evaluator with one-shot strategy, bear in mind that your defined methods will be reassembled into another LightningModule, which might result in extra constraints when writing the LightningModule. For example, your validation step could appear else where (e.g., in ``training_step``). This prohibits you from returning arbitrary object in ``validation_step``.
Then, users need to wrap everything (including LightningModule, trainer and dataloaders) into a :class:`nni.retiarii.evaluator.pytorch.Lightning` object, and pass this object into a Retiarii experiment.
.. code-block:: python
import nni.retiarii.evaluator.pytorch.lightning as pl
from nni.retiarii.experiment.pytorch import RetiariiExperiment
lightning = pl.Lightning(AutoEncoder(),
pl.Trainer(max_epochs=10),
train_dataloaders=pl.DataLoader(train_dataset, batch_size=100),
val_dataloaders=pl.DataLoader(test_dataset, batch_size=100))
experiment = RetiariiExperiment(base_model, lightning, mutators, strategy)
docs/source/nas/execution_engine.rst 0 → 100644
View file @ 9f73153f
Execution Engines
=================
Execution engine is for running Retiarii Experiment. NNI supports three execution engines, users can choose a specific engine according to the type of their model mutation definition and their requirements for cross-model optimizations.
* **Pure-python execution engine** is the default engine, it supports the model space expressed by :doc:`mutation primitives <construct_space>`.
* **Graph-based execution engine** supports the use of :doc:`mutation primitives <construct_space>` and model spaces represented by :doc:`mutators <mutator>`. It requires the user's model to be parsed by `TorchScript <https://pytorch.org/docs/stable/jit.html>`__.
* **CGO execution engine** has the same requirements and capabilities as the **Graph-based execution engine**. But further enables cross-model optimizations, which makes model space exploration faster.
.. _pure-python-execution-engine:
Pure-python Execution Engine
----------------------------
Pure-python Execution Engine is the default engine, we recommend users to keep using this execution engine, if they are new to NNI NAS. Pure-python execution engine plays magic within the scope of inline mutation APIs, while does not touch the rest of user model. Thus, it has minimal requirement on user model.
Rememeber to add :meth:`nni.retiarii.model_wrapper` decorator outside the whole PyTorch model before using this engine.
.. note:: You should always use ``super().__init__()`` instead of ``super(MyNetwork, self).__init__()`` in the PyTorch model, because the latter one has issues with model wrapper.
.. _graph-based-execution-engine:
Graph-based Execution Engine
----------------------------
For graph-based execution engine, it converts user-defined model to a graph representation (called graph IR) using `TorchScript <https://pytorch.org/docs/stable/jit.html>`__, each instantiated module in the model is converted to a subgraph. Then mutations are applied to the graph to generate new graphs. Each new graph is then converted back to PyTorch code and executed on the user specified training service.
Users may find ``@basic_unit`` helpful in some cases. :meth:`nni.retiarii.basic_unit` here means the module will not be converted to a subgraph, instead, it is converted to a single graph node as a basic unit.
``@basic_unit`` is usually used in the following cases:
* When users want to tune initialization parameters of a module using :class:`nni.retiarii.nn.pytorch.ValueChoice`, then decorate the module with ``@basic_unit``. For example, ``self.conv = MyConv(kernel_size=nn.ValueChoice([1, 3, 5]))``, here ``MyConv`` should be decorated.
* When a module cannot be successfully parsed to a subgraph, decorate the module with ``@basic_unit``. The parse failure could be due to complex control flow. Currently Retiarii does not support adhoc loop, if there is adhoc loop in a module's forward, this class should be decorated as serializable module. For example, the following ``MyModule`` should be decorated.
.. code-block:: python
@basic_unit
class MyModule(nn.Module):
def __init__(self):
...
def forward(self, x):
for i in range(10): # <- adhoc loop
...
* Some inline mutation APIs require their handled module to be decorated with ``@basic_unit``. For example, user-defined module that is provided to :class:`nni.retiarii.nn.pytorch.LayerChoice` as a candidate op should be decorated.
Three steps are need to use graph-based execution engine.
1. Remove ``@nni.retiarii.model_wrapper`` if there is any in your model.
2. Add ``config.execution_engine = 'base'`` to :class:`nni.retiarii.experiment.pytorch.RetiariiExeConfig`. The default value of ``execution_engine`` is 'py', which means pure-python execution engine.
3. Add ``@basic_unit`` when necessary following the above guidelines.
For exporting top models, graph-based execution engine supports exporting source code for top models by running ``exp.export_top_models(formatter='code')``.
.. _cgo-execution-engine:
CGO Execution Engine (experimental)
-----------------------------------
CGO (Cross-Graph Optimization) execution engine does cross-model optimizations based on the graph-based execution engine. In CGO execution engine, multiple models could be merged and trained together in one trial.
Currently, it only supports ``DedupInputOptimizer`` that can merge graphs sharing the same dataset to only loading and pre-processing each batch of data once, which can avoid bottleneck on data loading.
.. note :: To use CGO engine, PyTorch Lightning >= 1.6.1 is required.
To enable CGO execution engine, you need to follow these steps:
1. Create RetiariiExeConfig with remote training service. CGO execution engine currently only supports remote training service.
2. Add configurations for remote training service
3. Add configurations for CGO engine
.. code-block:: python
exp = RetiariiExperiment(base_model, trainer, mutators, strategy)
config = RetiariiExeConfig('remote')
# ...
# other configurations of RetiariiExeConfig
config.execution_engine = 'cgo' # set execution engine to CGO
config.max_concurrency_cgo = 3 # the maximum number of concurrent models to merge
config.batch_waiting_time = 10 # how many seconds CGO execution engine should wait before optimizing a new batch of models
rm_conf = RemoteMachineConfig()
# ...
# server configuration in rm_conf
rm_conf.gpu_indices = [0, 1, 2, 3] # gpu_indices must be set in RemoteMachineConfig for CGO execution engine
config.training_service.machine_list = [rm_conf]
exp.run(config, 8099)
CGO Execution Engine only supports pytorch-lightning trainer that inherits :class:`nni.retiarii.evaluator.pytorch.cgo.evaluator.MultiModelSupervisedLearningModule`.
For a trial running multiple models, the trainers inheriting :class:`nni.retiarii.evaluator.pytorch.cgo.evaluator.MultiModelSupervisedLearningModule` can handle the multiple outputs from the merged model for training, test and validation.
We have already implemented two trainers: :class:`nni.retiarii.evaluator.pytorch.cgo.evaluator.Classification` and :class:`nni.retiarii.evaluator.pytorch.cgo.evaluator.Regression`.
.. code-block:: python
from nni.retiarii.evaluator.pytorch.cgo.evaluator import Classification
trainer = Classification(train_dataloaders=pl.DataLoader(train_dataset, batch_size=100),
val_dataloaders=pl.DataLoader(test_dataset, batch_size=100),
max_epochs=1, limit_train_batches=0.2)
Advanced users can also implement their own trainers by inheriting ``MultiModelSupervisedLearningModule``.
Sometimes, a mutated model cannot be executed (e.g., due to shape mismatch). When a trial running multiple models contains
a bad model, CGO execution engine will re-run each model independently in separate trials without cross-model optimizations.
docs/source/nas/exploration_strategy.rst 0 → 100644
View file @ 9f73153f
Exploration Strategy
====================
There are two types of model space exploration approach: **Multi-trial strategy** and **One-shot strategy**. When the model space has been constructed, users can use either exploration approach to explore the model space.
* :ref:`Mutli-trial strategy <multi-trial-nas>` trains each sampled model in the model space independently.
* :ref:`One-shot strategy <one-shot-nas>` samples the model from a super model.
Here is the list of exploration strategies that NNI has supported.
.. list-table::
:header-rows: 1
:widths: auto
* - Name
- Category
- Brief Description
* - :class:`Random <nni.retiarii.strategy.Random>`
- :ref:`Multi-trial <multi-trial-nas>`
- Randomly sample an architecture each time
* - :class:`GridSearch <nni.retiarii.strategy.GridSearch>`
- :ref:`Multi-trial <multi-trial-nas>`
- Traverse the search space and try all possibilities
* - :class:`RegularizedEvolution <nni.retiarii.strategy.RegularizedEvolution>`
- :ref:`Multi-trial <multi-trial-nas>`
- Evolution algorithm for NAS. `Reference <https://arxiv.org/abs/1802.01548>`__
* - :class:`TPE <nni.retiarii.strategy.TPE>`
- :ref:`Multi-trial <multi-trial-nas>`
- Tree-structured Parzen Estimator (TPE). `Reference <https://papers.nips.cc/paper/4443-algorithms-for-hyper-parameter-optimization.pdf>`__
* - :class:`PolicyBasedRL <nni.retiarii.strategy.PolicyBasedRL>`
- :ref:`Multi-trial <multi-trial-nas>`
- Policy-based reinforcement learning, based on implementation of tianshou. `Reference <https://arxiv.org/abs/1611.01578>`__
* - :class:`DARTS <nni.retiarii.strategy.DARTS>`
- :ref:`One-shot <one-shot-nas>`
- Continuous relaxation of the architecture representation, allowing efficient search of the architecture using gradient descent. `Reference <https://arxiv.org/abs/1806.09055>`__
* - :class:`ENAS <nni.retiarii.strategy.ENAS>`
- :ref:`One-shot <one-shot-nas>`
- RL controller learns to generate the best network on a super-net. `Reference <https://arxiv.org/abs/1802.03268>`__
* - :class:`GumbelDARTS <nni.retiarii.strategy.GumbelDARTS>`
- :ref:`One-shot <one-shot-nas>`
- Choose the best block by using Gumbel Softmax random sampling and differentiable training. `Reference <https://arxiv.org/abs/1812.03443>`__
* - :class:`RandomOneShot <nni.retiarii.strategy.RandomOneShot>`
- :ref:`One-shot <one-shot-nas>`
- Train a super-net with uniform path sampling. `Reference <https://arxiv.org/abs/1904.00420>`__
* - :class:`Proxyless <nni.retiarii.strategy.Proxyless>`
- :ref:`One-shot <one-shot-nas>`
- A low-memory-consuming optimized version of differentiable architecture search. `Reference <https://arxiv.org/abs/1812.00332>`__
.. _multi-trial-nas:
Multi-trial strategy
--------------------
Multi-trial NAS means each sampled model from model space is trained independently. A typical multi-trial NAS is `NASNet <https://arxiv.org/abs/1707.07012>`__. In multi-trial NAS, users need model evaluator to evaluate the performance of each sampled model, and need an exploration strategy to sample models from a defined model space. Here, users could use NNI provided model evaluators or write their own model evalutor. They can simply choose a exploration strategy. Advanced users can also customize new exploration strategy.
To use an exploration strategy, users simply instantiate an exploration strategy and pass the instantiated object to :class:`~nni.retiarii.experiment.pytorch.RetiariiExperiment`. Below is a simple example.
.. code-block:: python
import nni.retiarii.strategy as strategy
exploration_strategy = strategy.Random(dedup=True)
Rather than using :class:`strategy.Random <nni.retiarii.strategy.Random>`, users can choose one of the strategies from the table above.
.. _one-shot-nas:
One-shot strategy
-----------------
One-shot NAS algorithms leverage weight sharing among models in neural architecture search space to train a supernet, and use this supernet to guide the selection of better models. This type of algorihtms greatly reduces computational resource compared to independently training each model from scratch (which we call "Multi-trial NAS").
Starting from v2.8, the usage of one-shot strategies are much alike to multi-trial strategies. Users simply need to create a strategy and run :class:`~nni.retiarii.experiment.pytorch.RetiariiExperiment`. Since one-shot strategies will manipulate the training recipe, to use a one-shot strategy, the evaluator needs to be one of the :ref:`PyTorch-Lightning evaluators <lightning-evaluator>`, either built-in or customized. Last but not least, don't forget to set execution engine to ``oneshot``. Example follows:
.. code-block:: python
import nni.retiarii.strategy as strategy
import nni.retiarii.evaluator.pytorch.lightning as pl
evaluator = pl.Classification(
# Need to use `pl.DataLoader` instead of `torch.utils.data.DataLoader` here,
# or use `nni.trace` to wrap `torch.utils.data.DataLoader`.
train_dataloaders=pl.DataLoader(train_dataset, batch_size=100),
val_dataloaders=pl.DataLoader(test_dataset, batch_size=100),
# Other keyword arguments passed to pytorch_lightning.Trainer.
max_epochs=10,
gpus=1,
)
exploration_strategy = strategy.DARTS()
exp_config.execution_engine = 'oneshot'
One-shot strategies only support a limited set of :ref:`mutation-primitives`, and does not support :doc:`customizing mutators <mutator>` at all. See the :ref:`reference <one-shot-strategy-reference>` for the detailed support list of each algorithm.
.. versionadded:: 2.8
One-shot strategy is now compatible with `Lightning accelerators <https://pytorch-lightning.readthedocs.io/en/stable/accelerators/gpu.html>`__. It means that, you can accelerate one-shot strategies on hardwares like multiple GPUs. To enable this feature, you only need to pass the keyword arguments which used to be set in ``pytorch_lightning.Trainer``, to your evaluator. See :doc:`this reference </reference/nas/evaluator>` for more details.
One-shot strategy (legacy)
--------------------------
.. warning::
.. deprecated:: 2.8
The following usages are deprecated and will be removed in future releases. If you intend to use them, the references can be found :doc:`here </deprecated/oneshot_legacy>`.
The usage of one-shot NAS strategy is a little different from multi-trial strategy. One-shot strategy is implemented with a special type of objects named *Trainer*. Following the common practice of one-shot NAS, *Trainer* trains the super-net and searches for the optimal architecture in a single run. For example,
.. code-block:: python
from nni.retiarii.oneshot.pytorch import DartsTrainer
trainer = DartsTrainer(
model=model,
loss=criterion,
metrics=lambda output, target: accuracy(output, target, topk=(1,)),
optimizer=optim,
dataset=dataset_train,
batch_size=32,
log_frequency=50
)
trainer.fit()
One-shot strategy can be used without :class:`~nni.retiarii.experiment.pytorch.RetiariiExperiment`. Thus, the ``trainer.fit()`` here runs the experiment locally.
After ``trainer.fit()`` completes, we can use ``trainer.export()`` to export the searched architecture (a dict of choices) to a file.
.. code-block:: python
final_architecture = trainer.export()
print('Final architecture:', trainer.export())
json.dump(trainer.export(), open('checkpoint.json', 'w'))
.. tip:: The trained super-net (neither the weights or exported JSON) can't be used directly. It's only an intermediate result used for deriving the final architecture. The exported architecture (can be retrieved with :meth:`nni.retiarii.fixed_arch`) needs to be *retrained* with a standard training recipe to get the final model.
docs/source/nas/hardware_aware_nas.rst 0 → 100644
View file @ 9f73153f
Hardware-aware NAS
==================
.. This file should be rewritten as a tutorial
End-to-end Multi-trial SPOS Demo
--------------------------------
To empower affordable DNN on the edge and mobile devices, hardware-aware NAS searches both high accuracy and low latency models. In particular, the search algorithm only considers the models within the target latency constraints during the search process.
To run this demo, first install nn-Meter by running:
.. code-block:: bash
pip install nn-meter
Then run multi-trail SPOS demo:
.. code-block:: bash
cd ${NNI_ROOT}/examples/nas/oneshot/spos/
python search.py --latency-filter cortexA76cpu_tflite21
How the demo works
^^^^^^^^^^^^^^^^^^
To support hardware-aware NAS, you first need a ``Strategy`` that supports filtering the models by latency. We provide such a filter named ``LatencyFilter`` in NNI and initialize a ``RegularizedEvolution`` strategy with the filter:
.. code-block:: python
evolution_strategy = strategy.RegularizedEvolution(
model_filter=latency_filter,
sample_size=args.evolution_sample_size, population_size=args.evolution_population_size, cycles=args.evolution_cycles
)
``LatencyFilter`` will predict the models\' latency by using nn-Meter and filter out the models whose latency are larger than the threshold (i.e., ``100`` in this example).
You can also build your own strategies and filters to support more flexible NAS such as sorting the models according to latency.
Then, pass this strategy to ``RetiariiExperiment``:
.. code-block:: python
exp = RetiariiExperiment(base_model, evaluator, strategy=evolution_strategy)
exp_config = RetiariiExeConfig('local')
...
exp_config.dummy_input = [1, 3, 224, 224]
exp.run(exp_config, args.port)
In ``exp_config``, ``dummy_input`` is required for tracing shape info in latency predictor.
End-to-end ProxylessNAS with Latency Constraints
------------------------------------------------
`ProxylessNAS <https://arxiv.org/abs/1812.00332>`__ is a hardware-aware one-shot NAS algorithm. ProxylessNAS applies the expected latency of the model to build a differentiable metric and design efficient neural network architectures for hardware. The latency loss is added as a regularization term for architecture parameter optimization. In this example, nn-Meter provides a latency estimator to predict expected latency for the mixed operation on other types of mobile and edge hardware.
To run the one-shot ProxylessNAS demo, first install nn-Meter by running:
.. code-block:: bash
pip install nn-meter
Then run one-shot ProxylessNAS demo:
.. code-block:: bash
python ${NNI_ROOT}/examples/nas/oneshot/proxylessnas/main.py --applied_hardware HARDWARE --reference_latency REFERENCE_LATENCY_MS
How the demo works
^^^^^^^^^^^^^^^^^^
In the implementation of ProxylessNAS ``trainer``, we provide a ``HardwareLatencyEstimator`` which currently builds a lookup table, that stores the measured latency of each candidate building block in the search space. The latency sum of all building blocks in a candidate model will be treated as the model inference latency. The latency prediction is obtained by ``nn-Meter``. ``HardwareLatencyEstimator`` predicts expected latency for the mixed operation based on the path weight of ``ProxylessLayerChoice``. With leveraging ``nn-Meter`` in NNI, users can apply ProxylessNAS to search efficient DNN models on more types of edge devices.
Despite of ``applied_hardware`` and ``reference_latency``, There are some other parameters related to hardware-aware ProxylessNAS training in this :githublink:`example <examples/nas/oneshot/proxylessnas/main.py>`:
* ``grad_reg_loss_type``: Regularization type to add hardware related loss. Allowed types include ``"mul#log"`` and ``"add#linear"``. Type of ``mul#log`` is calculate by ``(torch.log(expected_latency) / math.log(reference_latency)) ** beta``. Type of ``"add#linear"`` is calculate by ``reg_lambda * (expected_latency - reference_latency) / reference_latency``.
* ``grad_reg_loss_lambda``: Regularization params, is set to ``0.1`` by default.
* ``grad_reg_loss_alpha``: Regularization params, is set to ``0.2`` by default.
* ``grad_reg_loss_beta``: Regularization params, is set to ``0.3`` by default.
* ``dummy_input``: The dummy input shape when applied to the target hardware. This parameter is set as (1, 3, 224, 224) by default.
docs/source/nas/mutator.rst 0 → 100644
View file @ 9f73153f
Construct Space with Mutator
============================
Besides the mutation primitives demonstrated in the :doc:`basic tutorial <construct_space>`, NNI provides a more general approach to express a model space, i.e., *Mutator*, to cover more complex model spaces. The high-level APIs are also implemented with mutator in the underlying system, which can be seen as a special case of model mutation.
.. warning:: Mutator and inline mutation APIs can NOT be used together.
A mutator is a piece of logic to express how to mutate a given model. Users are free to write their own mutators. Then a model space is expressed with a base model and a list of mutators. A model in the model space is sampled by applying the mutators on the base model one after another. An example is shown below.
.. code-block:: python
applied_mutators = []
applied_mutators.append(BlockMutator('mutable_0'))
applied_mutators.append(BlockMutator('mutable_1'))
``BlockMutator`` is defined by users to express how to mutate the base model.
Write a mutator
---------------
User-defined mutator should inherit :class:`nni.retiarii.Mutator` class, and implement mutation logic in the member function :meth:`nni.retiarii.Mutator.mutate`.
.. code-block:: python
from nni.retiarii import Mutator
class BlockMutator(Mutator):
def __init__(self, target: str, candidates: List):
super(BlockMutator, self).__init__()
self.target = target
self.candidate_op_list = candidates
def mutate(self, model):
nodes = model.get_nodes_by_label(self.target)
for node in nodes:
chosen_op = self.choice(self.candidate_op_list)
node.update_operation(chosen_op.type, chosen_op.params)
The input of :meth:`nni.retiarii.Mutator.mutate` is graph IR (Intermediate Representation) of the base model, users can mutate the graph using the graph's member functions (e.g., :meth:`nni.retiarii.Model.get_nodes_by_label`). The mutation operations can be combined with the API ``self.choice``, in order to express a set of possible mutations. In the above example, the node's operation can be changed to any operation from ``candidate_op_list``.
Use placeholder to make mutation easier: :class:`nni.retiarii.nn.pytorch.Placeholder`. If you want to mutate a subgraph or node of your model, you can define a placeholder in this model to represent the subgraph or node. Then, use mutator to mutate this placeholder to make it real modules.
.. code-block:: python
ph = nn.Placeholder(
label='mutable_0',
kernel_size_options=[1, 3, 5],
n_layer_options=[1, 2, 3, 4],
exp_ratio=exp_ratio,
stride=stride
)
``label`` is used by mutator to identify this placeholder. The other parameters are the information that is required by mutator. They can be accessed from ``node.operation.parameters`` as a dict, it could include any information that users want to put to pass it to user defined mutator. The complete example code can be found in :githublink:`Mnasnet base model <examples/nas/multi-trial/mnasnet/base_mnasnet.py>`.
Starting an experiment is almost the same as using inline mutation APIs. The only difference is that the applied mutators should be passed to :class:`nni.retiarii.experiment.pytorch.RetiariiExperiment`. Below is a simple example.
.. code-block:: python
exp = RetiariiExperiment(base_model, trainer, applied_mutators, simple_strategy)
exp_config = RetiariiExeConfig('local')
exp_config.experiment_name = 'mnasnet_search'
exp_config.trial_concurrency = 2
exp_config.max_trial_number = 10
exp_config.training_service.use_active_gpu = False
exp.run(exp_config, 8081)
docs/source/nas/overview.rst 0 → 100644
View file @ 9f73153f
Overview
========
.. attention:: NNI's latest NAS supports are all based on Retiarii Framework, users who are still on `early version using NNI NAS v1.0 <https://nni.readthedocs.io/en/v2.2/nas.html>`__ shall migrate your work to Retiarii as soon as possible. We plan to remove the legacy NAS framework in the next few releases.
.. note:: PyTorch is the **only supported framework on Retiarii**. Inquiries of NAS support on Tensorflow is in `this discussion <https://github.com/microsoft/nni/discussions/4605>`__. If you intend to run NAS with DL frameworks other than PyTorch and Tensorflow, please `open new issues <https://github.com/microsoft/nni/issues>`__ to let us know.
Basics
------
Automatic neural architecture search is playing an increasingly important role in finding better models. Recent research has proven the feasibility of automatic NAS and has led to models that beat many manually designed and tuned models. Representative works include `NASNet <https://arxiv.org/abs/1707.07012>`__, `ENAS <https://arxiv.org/abs/1802.03268>`__, `DARTS <https://arxiv.org/abs/1806.09055>`__, `Network Morphism <https://arxiv.org/abs/1806.10282>`__, and `Evolution <https://arxiv.org/abs/1703.01041>`__. In addition, new innovations continue to emerge.
High-level speaking, aiming to solve any particular task with neural architecture search typically requires: search space design, search strategy selection, and performance evaluation. The three components work together with the following loop (from the famous `NAS survey <https://arxiv.org/abs/1808.05377>`__):
.. image:: ../../img/nas_abstract_illustration.png
:align: center
:width: 700
In this figure:
* *Model search space* means a set of models from which the best model is explored/searched. Sometimes we use *search space* or *model space* in short.
* *Exploration strategy* is the algorithm that is used to explore a model search space. Sometimes we also call it *search strategy*.
* *Model evaluator* is responsible for training a model and evaluating its performance.
The process is similar to :doc:`Hyperparameter Optimization </hpo/overview>`, except that the target is the best architecture rather than hyperparameter. Concretely, an exploration strategy selects an architecture from a predefined search space. The architecture is passed to a performance evaluation to get a score, which represents how well this architecture performs on a particular task. This process is repeated until the search process is able to find the best architecture.
Key Features
------------
The current NAS framework in NNI is powered by the research of `Retiarii: A Deep Learning Exploratory-Training Framework <https://www.usenix.org/system/files/osdi20-zhang_quanlu.pdf>`__, where we highlight the following features:
* :doc:`Simple APIs to construct search space easily <construct_space>`
* :doc:`SOTA NAS algorithms to explore search space <exploration_strategy>`
* :doc:`Experiment backend support to scale up experiments on large-scale AI platforms </experiment/overview>`
Why NAS with NNI
----------------
We list out the three perspectives where NAS can be particularly challegning without NNI. NNI provides solutions to relieve users' engineering effort when they want to try NAS techniques in their own scenario.
Search Space Design
^^^^^^^^^^^^^^^^^^^
The search space defines which architectures can be represented in principle. Incorporating prior knowledge about typical properties of architectures well-suited for a task can reduce the size of the search space and simplify the search. However, this also introduces a human bias, which may prevent finding novel architectural building blocks that go beyond the current human knowledge. Search space design can be very challenging for beginners, who might not possess the experience to balance the richness and simplicity.
In NNI, we provide a wide range of APIs to build the search space. There are :doc:`high-level APIs <construct_space>`, that enables the possibility to incorporate human knowledge about what makes a good architecture or search space. There are also :doc:`low-level APIs <mutator>`, that is a list of primitives to construct a network from operation to operation.
Exploration strategy
^^^^^^^^^^^^^^^^^^^^
The exploration strategy details how to explore the search space (which is often exponentially large). It encompasses the classical exploration-exploitation trade-off since, on the one hand, it is desirable to find well-performing architectures quickly, while on the other hand, premature convergence to a region of suboptimal architectures should be avoided. The "best" exploration strategy for a particular scenario is usually found via trial-and-error. As many state-of-the-art strategies are implemented with their own code-base, it becomes very troublesome to switch from one to another.
In NNI, we have also provided :doc:`a list of strategies <exploration_strategy>`. Some of them are powerful yet time consuming, while others might be suboptimal but really efficient. Given that all strategies are implemented with a unified interface, users can always find one that matches their need.
Performance estimation
^^^^^^^^^^^^^^^^^^^^^^
The objective of NAS is typically to find architectures that achieve high predictive performance on unseen data. Performance estimation refers to the process of estimating this performance. The problem with performance estimation is mostly its scalability, i.e., how can I run and manage multiple trials simultaneously.
In NNI, we standardize this process is implemented with :doc:`evaluator <evaluator>`, which is responsible of estimating a model's performance. NNI has quite a few built-in supports of evaluators, ranging from the simplest option, e.g., to perform a standard training and validation of the architecture on data, to complex configurations and implementations. Evaluators are run in *trials*, where trials can be spawn onto distributed platforms with our powerful :doc:`training service </experiment/training_service/overview>`.
Tutorials
---------
To start using NNI NAS framework, we recommend at least going through the following tutorials:
* :doc:`Quickstart </tutorials/hello_nas>`
* :doc:`construct_space`
* :doc:`exploration_strategy`
* :doc:`evaluator`
Resources
---------
The following articles will help with a better understanding of the current arts of NAS:
* `Neural Architecture Search: A Survey <https://arxiv.org/abs/1808.05377>`__
* `A Comprehensive Survey of Neural Architecture Search: Challenges and Solutions <https://arxiv.org/abs/2006.02903>`__
docs/source/nas/overview_zh.rst 0 → 100644
View file @ 9f73153f
.. 48c39585a539a877461aadef63078c48
神经架构搜索
===========================
.. toctree::
:hidden:
快速入门 </tutorials/hello_nas>
构建搜索空间 <construct_space>
探索策略 <exploration_strategy>
评估器 <evaluator>
高级用法 <advanced_usage>
.. attention:: NNI 最新的架构搜索支持都是基于 Retiarii 框架，还在使用 `NNI 架构搜索的早期版本 <https://nni.readthedocs.io/en/v2.2/nas.html>`__ 的用户应尽快将您的工作迁移到 Retiarii。我们计划在接下来的几个版本中删除旧的架构搜索框架。
.. attention:: PyTorch 是 **Retiarii 唯一支持的框架**。有关 Tensorflow 上架构搜索支持的需求在 `此讨论 <https://github.com/microsoft/nni/discussions/4605>`__ 中。另外，如果您打算使用 PyTorch 和 Tensorflow 以外的 DL 框架运行 NAS，请 `创建新 issue <https://github.com/microsoft/nni/issues>`__ 让我们知道。
概述
------
自动神经架构搜索 (Neural Architecture Search, NAS）在寻找更好的模型方面发挥着越来越重要的作用。最近的研究证明了自动架构搜索的可行性，并导致模型击败了许多手动设计和调整的模型。其中具有代表性的有 `NASNet <https://arxiv.org/abs/1707.07012>`__、 `ENAS <https://arxiv.org/abs/1802.03268>`__、 `DARTS <https://arxiv.org/ abs/1806.09055>`__、 `Network Morphism <https://arxiv.org/abs/1806.10282>`__ 和 `进化算法 <https://arxiv.org/abs/1703.01041>`__。此外，新的创新正不断涌现。
总的来说，使用神经架构搜索解决任何特定任务通常需要：搜索空间设计、搜索策略选择和性能评估。这三个组件形成如下的循环（图来自于 `架构搜索综述 <https://arxiv.org/abs/1808.05377>`__）：
.. image:: ../../img/nas_abstract_illustration.png
:align: center
:width: 700
在这个图中：
* *模型搜索空间* 是指一组模型，从中探索/搜索最佳模型，简称为 *搜索空间* 或 *模型空间*。
* *探索策略* 是用于探索模型搜索空间的算法。有时我们也称它为 *搜索策略*。
* *模型评估者* 负责训练模型并评估其性能。
该过程类似于 :doc:`超参数优化 </hpo/overview>`，只不过目标是最佳网络结构而不是最优超参数。具体来说，探索策略从预定义的搜索空间中选择架构。该架构被传递给性能评估以获得评分，该评分表示这个网络结构在特定任务上的表现。重复此过程，直到搜索过程能够找到最优的网络结构。
主要特点
------------
NNI 中当前的架构搜索框架由 `Retiarii: A Deep Learning Exploratory-Training Framework <https://www.usenix.org/system/files/osdi20-zhang_quanlu.pdf>`__ 的研究支撑，具有以下特点：
* :doc:`简单的 API，让您轻松构建搜索空间 <construct_space>`
* :doc:`SOTA 架构搜索算法，以高效探索搜索空间 <exploration_strategy>`
* :doc:`后端支持，在大规模 AI 平台上运行实验 </experiment/overview>`
为什么使用 NNI 的架构搜索
-------------------------------
若没有 NNI，实现架构搜索将极具挑战性，主要包含以下三个方面。当用户想在自己的场景中尝试架构搜索技术时，NNI 提供的解决方案可以极大程度上减轻用户的工作量。
搜索空间设计
^^^^^^^^^^^^^^^^^^^
搜索空间定义了架构的可行域集合。为了简化搜索，我们通常需要结合任务相关的先验知识，减小搜索空间的规模。然而，这也引入了人类的偏见，在某种程度上可能会丧失突破人类认知的可能性。无论如何，对于初学者来说，搜索空间设计是一个极具挑战性的任务，因为他们可能无法在简单的空间和丰富的想象力之间取得平衡。
在 NNI 中，我们提供了不同层级的 API 来构建搜索空间。有 :doc:`高层 API <construct_space>`，引入大量先验，帮助用户迅速了解什么是好的架构或搜索空间；也有 :doc:`底层 API <mutator>`，提供了最底层的算子和图变换原语。
探索策略
^^^^^^^^^^^^^^^^^^^^
探索策略定义了如何探索搜索空间（通常是指数级规模的）。它包含经典的探索-利用权衡。一方面，我们希望快速找到性能良好的架构；而另一方面，我们也应避免过早收敛到次优架构的区域。我们往往需要通常通过反复试验找到特定场景的“最佳”探索策略。由于许多近期发表的探索策略都是使用自己的代码库实现的，因此从一个切换到另一个变得非常麻烦。
在 NNI 中，我们还提供了 :doc:`一系列的探索策略 <exploration_strategy>`。其中一些功能强大但耗时，而另一些可能不能找到最优架构但非常高效。鉴于所有策略都使用统一的用户接口实现，用户可以轻松找到符合他们需求的策略。
性能评估
^^^^^^^^^^^^^^^^^^^^^^
架构搜索的目标通常是找到能够在测试数据集表现理想的网络结构。性能评估的作用便是量化每个网络的好坏。其主要难点在于可扩展性，即如何在大规模训练平台上同时运行和管理多个试验。
在 NNI 中，我们使用 :doc:`evaluator <evaluator>` 来标准化性能评估流程。它负责估计模型的性能。NNI 内建了不少性能评估器，从最简单的交叉验证，到复杂的自定义配置。评估器在 *试验 (trials)* 中运行，可以通过我们强大的 :doc:`训练平台 </experiment/training_service/overview>` 将试验分发到大规模训练平台上。
教程
---------
要开始使用 NNI 架构搜索框架，我们建议至少阅读以下教程：
* :doc:`快速入门 </tutorials/hello_nas>`
* :doc:`构建搜索空间 <construct_space>`
* :doc:`探索策略 <exploration_strategy>`
* :doc:`评估器 <evaluator>`
资源
---------
以下文章将有助于更好地了解 NAS 的最新发展：
* `神经架构搜索：综述 <https://arxiv.org/abs/1808.05377>`__
* `神经架构搜索的综述：挑战和解决方案 <https://arxiv.org/abs/2006.02903>`__
docs/source/nas/serialization.rst 0 → 100644
View file @ 9f73153f
Serialization
=============
In multi-trial NAS, a sampled model should be able to be executed on a remote machine or a training platform (e.g., AzureML, OpenPAI). "Serialization" enables re-instantiation of model evaluator in another process or machine, such that, both the model and its model evaluator should be correctly serialized. To make NNI correctly serialize model evaluator, users should apply :func:`nni.trace <nni.common.serializer.trace>` on some of their functions and objects. API references can be found in :func:`nni.trace <nni.common.serializer.trace>`.
Serialization is implemented as a combination of `json-tricks <https://json-tricks.readthedocs.io/en/latest/>`_ and `cloudpickle <https://github.com/cloudpipe/cloudpickle>`_. Essentially, it is json-tricks, that is a enhanced version of Python JSON, enabling handling of serialization of numpy arrays, date/times, decimal, fraction and etc. The difference lies in the handling of class instances. Json-tricks deals with class instances with ``__dict__`` and ``__class__``, which in most of our cases are not reliable (e.g., datasets, dataloaders). Rather, our serialization deals with class instances with two methods:
1. If the class / factory that creates the object is decorated with :func:`nni.trace <nni.common.serializer.trace>`, we can serialize the class / factory function, along with the parameters, such that the instance can be re-instantiated.
2. Otherwise, cloudpickle is used to serialize the object into a binary.
The recommendation is, unless you are absolutely certain that there is no problem and extra burden to serialize the object into binary, always add :func:`nni.trace <nni.common.serializer.trace>`. In most cases, it will be more clean and neat, and enables possibilities such as mutation of parameters (will be supported in future).
.. warning::
**What will happen if I forget to "trace" my objects?**
It is likely that the program can still run. NNI will try to serialize the untraced object into a binary. It might fail in complex cases. For example, when the object is too large. Even if it succeeds, the result might be a substantially large object. For example, if you forgot to add :func:`nni.trace <nni.common.serializer.trace>` on ``MNIST``, the MNIST dataset object wil be serialized into binary, which will be dozens of megabytes because the object has the whole 60k images stored inside. You might see warnings and even errors when running experiments. To avoid such issues, the easiest way is to always remember to add :func:`nni.trace <nni.common.serializer.trace>` to non-primitive objects.
.. note:: In Retiarii, serializer will throw exception when one of an single object in the recursive serialization is larger than 64 KB when binary serialized. This indicates that such object needs to be wrapped by :func:`nni.trace <nni.common.serializer.trace>`. In rare cases, if you insist on pickling large data, the limit can be overridden by setting an environment variable ``PICKLE_SIZE_LIMIT``, whose unit is byte. Please note that even if the experiment might be able to run, this can still cause performance issues and even the crash of NNI experiment.
To trace a function or class, users can use decorator like,
.. code-block:: python
@nni.trace
class MyClass:
...
Inline trace that traces instantly on the object instantiation or function invoke is also acceptable:
.. code-block:: python
nni.trace(MyClass)(parameters)
Assuming a class ``cls`` is already traced, when it is serialized, its class type along with initialization parameters will be dumped. As the parameters are possibly class instances (if not primitive types like ``int`` and ``str``), their serialization will be a similar problem. We recommend decorate them with :func:`nni.trace <nni.common.serializer.trace>` as well. In other words, :func:`nni.trace <nni.common.serializer.trace>` should be applied recursively if necessary.
Below is an example, ``transforms.Compose``, ``transforms.Normalize``, and ``MNIST`` are serialized manually using :func:`nni.trace <nni.common.serializer.trace>`. :func:`nni.trace <nni.common.serializer.trace>` takes a class / function as its argument, and returns a wrapped class and function that has the same behavior with the original class / function. The usage of the wrapped class / function is also identical to the original one, except that the arguments are recorded. No need to apply :func:`nni.trace <nni.common.serializer.trace>` to :class:`pl.Classification <nni.retiarii.evaluator.pytorch.Classification>` and :class:`pl.DataLoader <nni.retiarii.evaluator.pytorch.DataLoader>` because they are already traced.
.. code-block:: python
import nni
import nni.retiarii.evaluator.pytorch.lightning as pl
from torchvision import transforms
def create_mnist_dataset(root, transform):
return MNIST(root='data/mnist', train=False, download=True, transform=transform)
transform = nni.trace(transforms.Compose)([nni.trace(transforms.ToTensor)(), nni.trace(transforms.Normalize)((0.1307,), (0.3081,))])
# If you write like following, the whole transform will be serialized into a pickle.
# This actually works fine, but we do NOT recommend such practice.
# transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.1307,), (0.3081,))])
train_dataset = nni.trace(MNIST)(root='data/mnist', train=True, download=True, transform=transform)
test_dataset = nni.trace(create_mnist_dataset)('data/mnist', transform=transform) # factory is also acceptable
evaluator = pl.Classification(train_dataloaders=pl.DataLoader(train_dataset, batch_size=100),
val_dataloaders=pl.DataLoader(test_dataset, batch_size=100),
max_epochs=10)
.. note::
**What's the relationship between model_wrapper, basic_unit and nni.trace?**
They are fundamentally different. :func:`model_wrapper <nni.retiarii.model_wrapper>` is used to wrap a base model (search space), :func:`basic_unit <nni.retiarii.basic_unit>` to annotate a module as primitive. :func:`nni.trace <nni.common.serializer.trace>` is to enable serialization of general objects. Though they share similar underlying implementations, but do keep in mind that you will experience errors if you mix them up.
Please refer to API reference of :meth:`nni.retiarii.model_wrapper`, :meth:`nni.retiarii.basic_unit`, and :func:`nni.trace <nni.common.serializer.trace>`.
docs/source/nas/space_hub.rst 0 → 100644
View file @ 9f73153f
Model Space Hub
===============
NNI model space hub contains a curated list of well-known NAS search spaces, along with a number of famous model space building blocks. Consider reading this document or try the models / spaces provided in the hub if you intend to:
1. Use a pre-defined model space as a starting point for your model development.
2. Try the state-of-the-art searched architecture along with its associated weights in your own task.
3. Learn the performance of NNI's built-in NAS search strategies on some well-recognized model spaces.
4. Build and test your NAS algorithm on the space hub and fairly compare them with other baselines.
List of supported model spaces
------------------------------
The model spaces provided so far are all built for image classification tasks, though they can serve as backbones for downstream tasks.
.. list-table::
:header-rows: 1
:widths: auto
* - Name
- Brief Description
* - :class:`~nni.retiarii.hub.pytorch.NasBench101`
- Search space benchmarked by `NAS-Bench-101 <http://proceedings.mlr.press/v97/ying19a/ying19a.pdf>`__
* - :class:`~nni.retiarii.hub.pytorch.NasBench201`
- Search space benchmarked by `NAS-Bench-201 <https://arxiv.org/abs/2001.00326>`__
* - :class:`~nni.retiarii.hub.pytorch.NASNet`
- Proposed by `Learning Transferable Architectures for Scalable Image Recognition <https://arxiv.org/abs/1707.07012>`__
* - :class:`~nni.retiarii.hub.pytorch.ENAS`
- Proposed by `Efficient neural architecture search via parameter sharing <https://arxiv.org/abs/1802.03268>`__, subtly different from NASNet
* - :class:`~nni.retiarii.hub.pytorch.AmoebaNet`
- Proposed by `Regularized evolution for image classifier architecture search <https://arxiv.org/abs/1802.01548>`__, subtly different from NASNet
* - :class:`~nni.retiarii.hub.pytorch.PNAS`
- Proposed by `Progressive neural architecture search <https://arxiv.org/abs/1712.00559>`__, subtly different from NASNet
* - :class:`~nni.retiarii.hub.pytorch.DARTS`
- Proposed by `Darts: Differentiable architecture search <https://arxiv.org/abs/1806.09055>`__, most popularly used in evaluating one-shot algorithms
* - :class:`~nni.retiarii.hub.pytorch.ProxylessNAS`
- Proposed by `ProxylessNAS <https://arxiv.org/abs/1812.00332>`__, based on MobileNetV2.
* - :class:`~nni.retiarii.hub.pytorch.MobileNetV3Space`
- The largest space in `TuNAS <https://arxiv.org/abs/2008.06120>`__.
* - :class:`~nni.retiarii.hub.pytorch.ShuffleNetSpace`
- Based on ShuffleNetV2, proposed by `Single Path One-shot <https://www.ecva.net/papers/eccv_2020/papers_ECCV/papers/123610528.pdf>`__
* - :class:`~nni.retiarii.hub.pytorch.AutoformerSpace`
- Based on ViT, proposed by `Autoformer <https://arxiv.org/abs/2107.00651>`__
.. note::
We are actively enriching the model space hub. Planned model spaces include:
- `NAS-BERT <https://arxiv.org/abs/2105.14444>`__
- `LightSpeech <https://arxiv.org/abs/2102.04040>`__
We welcome suggestions and contributions.
Using pre-searched models
-------------------------
One way to use the model space is to directly leverage the searched results. Note that some of them have already been well-known neural networks and widely used.
.. code-block:: python
import torch
from nni.retiarii.hub.pytorch import MobileNetV3Space
from torch.utils.data import DataLoader
from torchvision import transforms
from torchvision.datasets import ImageNet
# Load one of the searched results from MobileNetV3 search space.
mobilenetv3 = MobileNetV3Space.load_searched_model(
'mobilenetv3-small-100', # Available model alias are listed in the table below.
pretrained=True, download=True # download and load the pretrained checkpoint
)
# MobileNetV3 model can be directly evaluated on ImageNet
transform = transforms.Compose([
transforms.Resize(256, interpolation=transforms.InterpolationMode.BICUBIC),
transforms.CenterCrop(224),
transforms.ToTensor(),
transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
])
dataset = ImageNet('/path/to/your/imagenet', 'val', transform=transform)
dataloader = DataLoader(dataset, batch_size=64)
mobilenetv3.eval()
with torch.no_grad():
correct = total = 0
for inputs, targets in dataloader:
logits = mobilenetv3(inputs)
_, predict = torch.max(logits, 1)
correct += (predict == targets).sum().item()
total += targets.size(0)
print('Accuracy:', correct / total)
In the example above, ``MobileNetV3Space`` can be replaced with any model spaces in the hub, and ``mobilenetv3-small-100`` can be any model alias listed below.
+-------------------+------------------------+----------+---------+-------------------------------+
| Search space | Model | Dataset | Metric | Eval configurations |
+===================+========================+==========+=========+===============================+
| ProxylessNAS | acenas-m1 | ImageNet | 75.176 | Default |
+-------------------+------------------------+----------+---------+-------------------------------+
| ProxylessNAS | acenas-m2 | ImageNet | 75.0 | Default |
+-------------------+------------------------+----------+---------+-------------------------------+
| ProxylessNAS | acenas-m3 | ImageNet | 75.118 | Default |
+-------------------+------------------------+----------+---------+-------------------------------+
| ProxylessNAS | proxyless-cpu | ImageNet | 75.29 | Default |
+-------------------+------------------------+----------+---------+-------------------------------+
| ProxylessNAS | proxyless-gpu | ImageNet | 75.084 | Default |
+-------------------+------------------------+----------+---------+-------------------------------+
| ProxylessNAS | proxyless-mobile | ImageNet | 74.594 | Default |
+-------------------+------------------------+----------+---------+-------------------------------+
| MobileNetV3Space | mobilenetv3-large-100 | ImageNet | 75.768 | Bicubic interpolation |
+-------------------+------------------------+----------+---------+-------------------------------+
| MobileNetV3Space | mobilenetv3-small-050 | ImageNet | 57.906 | Bicubic interpolation |
+-------------------+------------------------+----------+---------+-------------------------------+
| MobileNetV3Space | mobilenetv3-small-075 | ImageNet | 65.24 | Bicubic interpolation |
+-------------------+------------------------+----------+---------+-------------------------------+
| MobileNetV3Space | mobilenetv3-small-100 | ImageNet | 67.652 | Bicubic interpolation |
+-------------------+------------------------+----------+---------+-------------------------------+
| MobileNetV3Space | cream-014 | ImageNet | 53.74 | Test image size = 64 |
+-------------------+------------------------+----------+---------+-------------------------------+
| MobileNetV3Space | cream-043 | ImageNet | 66.256 | Test image size = 96 |
+-------------------+------------------------+----------+---------+-------------------------------+
| MobileNetV3Space | cream-114 | ImageNet | 72.514 | Test image size = 160 |
+-------------------+------------------------+----------+---------+-------------------------------+
| MobileNetV3Space | cream-287 | ImageNet | 77.52 | Default |
+-------------------+------------------------+----------+---------+-------------------------------+
| MobileNetV3Space | cream-481 | ImageNet | 79.078 | Default |
+-------------------+------------------------+----------+---------+-------------------------------+
| MobileNetV3Space | cream-604 | ImageNet | 79.92 | Default |
+-------------------+------------------------+----------+---------+-------------------------------+
| DARTS | darts-v2 | CIFAR-10 | 97.37 | Default |
+-------------------+------------------------+----------+---------+-------------------------------+
| ShuffleNetSpace | spos | ImageNet | 74.14 | BGR tensor; no normalization |
+-------------------+------------------------+----------+---------+-------------------------------+
.. note::
1. The metrics listed above are obtained by evaluating the checkpoints provided by the original author and converted to NNI NAS format with `these scripts <https://github.com/ultmaster/spacehub-conversion>`__. Do note that some metrics can be higher / lower than the original report, because there could be subtle differences between data preprocessing, operation implementation (e.g., 3rd-party hswish vs ``nn.Hardswish``), or even library versions we are using. But most of these errors are acceptable (~0.1%).
2. The default metric for ImageNet and CIFAR-10 is top-1 accuracy.
3. Refer to `timm <https://github.com/rwightman/pytorch-image-models>`__ for the evaluation configurations.
.. todos: measure latencies and flops, reproduce training.
Searching within model spaces
-----------------------------
To search within a model space for a new architecture on a particular dataset,
users need to create model space, search strategy, and evaluator following the :doc:`standard procedures </tutorials/hello_nas>`.
Here is a short sample code snippet for reference.
.. code-block:: python
# Create the model space
from nni.retiarii.hub.pytorch import MobileNetV3Space
model_space = MobileNetV3Space()
# Pick a search strategy
from nni.retiarii.strategy import Evolution
strategy = Evolution() # It can be any strategy, including one-shot strategies.
# Define an evaluator
from nni.retiarii.evaluator.pytorch import Classification
evaluator = Classification(train_dataloaders=DataLoader(train_dataset, batch_size=batch_size),
val_dataloaders=DataLoader(test_dataset, batch_size=batch_size))
# Launch the experiment, start the search process
experiment = RetiariiExperiment(model_space, evaluator, [], strategy)
experiment.run(experiment_config)
.. todo: search reproduction results
docs/source/nas/toctree.rst 0 → 100644
View file @ 9f73153f
Neural Architecture Search
==========================
.. toctree::
:hidden:
overview
Tutorials <tutorials>
construct_space
space_hub
exploration_strategy
evaluator
advanced_usage
docs/source/nas/tutorials.rst 0 → 100644
View file @ 9f73153f
NAS Tutorials
=============
.. toctree::
:hidden:
Hello NAS! </tutorials/hello_nas>
Search in DARTS </tutorials/darts>
docs/source/notes/architecture_overview.rst 0 → 100644
View file @ 9f73153f
:orphan:
Architecture Overview
=====================
NNI (Neural Network Intelligence) is a toolkit to help users design and tune machine learning models (e.g., hyperparameters), neural network architectures, or complex system's parameters, in an efficient and automatic way. NNI has several appealing properties: ease-of-use, scalability, flexibility, and efficiency.
* **Ease-of-use**: NNI can be easily installed through python pip. Only several lines need to be added to your code in order to use NNI's power. You can use both the commandline tool and WebUI to work with your experiments.
* **Scalability**: Tuning hyperparameters or the neural architecture often demands a large number of computational resources, while NNI is designed to fully leverage different computation resources, such as remote machines, training platforms (e.g., OpenPAI, Kubernetes). Hundreds of trials could run in parallel by depending on the capacity of your configured training platforms.
* **Flexibility**: Besides rich built-in algorithms, NNI allows users to customize various hyperparameter tuning algorithms, neural architecture search algorithms, early stopping algorithms, etc. Users can also extend NNI with more training platforms, such as virtual machines, kubernetes service on the cloud. Moreover, NNI can connect to external environments to tune special applications/models on them.
* **Efficiency**: We are intensively working on more efficient model tuning on both the system and algorithm level. For example, we leverage early feedback to speedup the tuning procedure.
The figure below shows high-level architecture of NNI.
.. image:: https://user-images.githubusercontent.com/16907603/92089316-94147200-ee00-11ea-9944-bf3c4544257f.png
:width: 700
Key Concepts
------------
* *Experiment*: One task of, for example, finding out the best hyperparameters of a model, finding out the best neural network architecture, etc. It consists of trials and AutoML algorithms.
* *Search Space*: The feasible region for tuning the model. For example, the value range of each hyperparameter.
* *Configuration*: An instance from the search space, that is, each hyperparameter has a specific value.
* *Trial*: An individual attempt at applying a new configuration (e.g., a set of hyperparameter values, a specific neural architecture, etc.). Trial code should be able to run with the provided configuration.
* *Tuner*: An AutoML algorithm, which generates a new configuration for the next try. A new trial will run with this configuration.
* *Assessor*: Analyze a trial's intermediate results (e.g., periodically evaluated accuracy on test dataset) to tell whether this trial can be early stopped or not.
* *Training Platform*: Where trials are executed. Depending on your experiment's configuration, it could be your local machine, or remote servers, or large-scale training platform (e.g., OpenPAI, Kubernetes).
Basically, an experiment runs as follows: Tuner receives search space and generates configurations. These configurations will be submitted to training platforms, such as the local machine, remote machines, or training clusters. Their performances are reported back to Tuner. Then, new configurations are generated and submitted.
For each experiment, the user only needs to define a search space and update a few lines of code, and then leverage NNI built-in Tuner/Assessor and training platforms to search the best hyperparameters and/or neural architecture. There are basically 3 steps:
* Step 1: :doc:`Define search space <../hpo/search_space>`
* Step 2: Update model codes
* Step 3: :doc:`Define Experiment <../reference/experiment_config>`
.. image:: https://user-images.githubusercontent.com/23273522/51816627-5d13db80-2302-11e9-8f3e-627e260203d5.jpg
For more details about how to run an experiment, please refer to :doc:`Quickstart <../tutorials/hpo_quickstart_pytorch/main>`.
Core Features
-------------
NNI provides a key capacity to run multiple instances in parallel to find the best combinations of parameters. This feature can be used in various domains, like finding the best hyperparameters for a deep learning model or finding the best configuration for database and other complex systems with real data.
NNI also provides algorithm toolkits for machine learning and deep learning, especially neural architecture search (NAS) algorithms, model compression algorithms, and feature engineering algorithms.
Hyperparameter Tuning
^^^^^^^^^^^^^^^^^^^^^
This is a core and basic feature of NNI, we provide many popular :doc:`automatic tuning algorithms <../hpo/tuners>` (i.e., tuner) and :doc:`early stop algorithms <../hpo/assessors>` (i.e., assessor). You can follow :doc:`Quickstart <../tutorials/hpo_quickstart_pytorch/main>` to tune your model (or system). Basically, there are the above three steps and then starting an NNI experiment.
General NAS Framework
^^^^^^^^^^^^^^^^^^^^^
This NAS framework is for users to easily specify candidate neural architectures, for example, one can specify multiple candidate operations (e.g., separable conv, dilated conv) for a single layer, and specify possible skip connections. NNI will find the best candidate automatically. On the other hand, the NAS framework provides a simple interface for another type of user (e.g., NAS algorithm researchers) to implement new NAS algorithms. A detailed description of NAS and its usage can be found :doc:`here </nas/overview>`.
NNI has support for many one-shot NAS algorithms such as ENAS and DARTS through NNI trial SDK. To use these algorithms you do not have to start an NNI experiment. Instead, import an algorithm in your trial code and simply run your trial code. If you want to tune the hyperparameters in the algorithms or want to run multiple instances, you can choose a tuner and start an NNI experiment.
Other than one-shot NAS, NAS can also run in a classic mode where each candidate architecture runs as an independent trial job. In this mode, similar to hyperparameter tuning, users have to start an NNI experiment and choose a tuner for NAS.
Model Compression
^^^^^^^^^^^^^^^^^
NNI provides an easy-to-use model compression framework to compress deep neural networks, the compressed networks typically have much smaller model size and much faster
inference speed without losing performance significantlly. Model compression on NNI includes pruning algorithms and quantization algorithms. NNI provides many pruning and
quantization algorithms through NNI trial SDK. Users can directly use them in their trial code and run the trial code without starting an NNI experiment. Users can also use NNI model compression framework to customize their own pruning and quantization algorithms.
A detailed description of model compression and its usage can be found :doc:`here <../compression/overview>`.
Automatic Feature Engineering
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Automatic feature engineering is for users to find the best features for their tasks. A detailed description of automatic feature engineering and its usage can be found :doc:`here <../feature_engineering/overview>`. It is supported through NNI trial SDK, which means you do not have to create an NNI experiment. Instead, simply import a built-in auto-feature-engineering algorithm in your trial code and directly run your trial code.
The auto-feature-engineering algorithms usually have a bunch of hyperparameters themselves. If you want to automatically tune those hyperparameters, you can leverage hyperparameter tuning of NNI, that is, choose a tuning algorithm (i.e., tuner) and start an NNI experiment for it.
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment