Skip to content

GitLab

Unverified Commit 51d261e7 authored by J-shang's avatar J-shang Committed by GitHub
Browse files

Merge pull request #4668 from microsoft/doc-refactor

parents d63a2ea3 b469e1c1
Hide whitespace changes
Inline Side-by-side
Showing with 325 additions and 174 deletions
docs/source/CommunitySharings/model_compression_zh.rst → docs/source/misc/model_compression_zh.rst
View file @ 51d261e7
.. 8a4c54c8127199ad4c95e20247e33fe2
.. ba9409544c5c42e8424e7e2694ab7bd3
#################
模型压缩
#################
========
以下介绍了如何将知识蒸馏应用于 NNI 模型压缩。 将来会添加更多用例和解决方案。
.. toctree::
:maxdepth: 1
使用 NNI 模型压缩进行知识蒸馏<../TrialExample/KDExample>
\ No newline at end of file
使用 NNI 模型压缩进行知识蒸馏<kd_example>
\ No newline at end of file
docs/source/CommunitySharings/perf_compare.rst → docs/source/misc/perf_compare.rst
View file @ 51d261e7
################################################
Performance Measurement, Comparison and Analysis
################################################
================================================
Performance comparison and analysis can help users decide a proper algorithm (e.g., tuner, NAS algorithm) for their scenario. The following are some measurement and comparison data for users' reference.
.. toctree::
:maxdepth: 1
Neural Architecture Search Comparison <NasComparison>
Hyper-parameter Tuning Algorithm Comparsion <HpoComparison>
Model Compression Algorithm Comparsion <ModelCompressionComparison>
\ No newline at end of file
Neural Architecture Search Comparison <nas_comparison>
Hyper-parameter Tuning Algorithm Comparsion <hpo_comparison>
Model Compression Algorithm Comparsion <model_compress_comp>
\ No newline at end of file
docs/source/CommunitySharings/perf_compare_zh.rst → docs/source/misc/perf_compare_zh.rst
View file @ 51d261e7
.. 7d625bd21018f53834e9db08a619c494
.. 8f6869e44d85db2e2a969194c1828580
################################################
性能测量，比较和分析
################################################
====================
性能比较和分析可以帮助用户在他们的场景中选择适合的算法（例如 Tuner，NAS 算法）。 以下是一些供用户参考的测量和比较数据。
.. toctree::
:maxdepth: 1
神经网络结构搜索（NAS）的对比<NasComparison>
超参调优算法的对比<HpoComparison>
模型压缩算法的对比<ModelCompressionComparison>
\ No newline at end of file
神经网络结构搜索（NAS）的对比<nas_comparison>
超参调优算法的对比<hpo_comparison>
模型压缩算法的对比<model_compress_comp>
\ No newline at end of file
docs/source/model_compression.rst deleted 100644 → 0
View file @ d63a2ea3
#################
Model Compression
#################
Deep neural networks (DNNs) have achieved great success in many tasks. However, typical neural networks are both
computationally expensive and energy intensive, can be difficult to be deployed on devices with low computation
resources or with strict latency requirements. Therefore, a natural thought is to perform model compression to
reduce model size and accelerate model training/inference without losing performance significantly. Model compression
techniques can be divided into two categories: pruning and quantization. The pruning methods explore the redundancy
in the model weights and try to remove/prune the redundant and uncritical weights. Quantization refers to compressing
models by reducing the number of bits required to represent weights or activations.
NNI provides an easy-to-use toolkit to help user design and use model pruning and quantization algorithms.
It supports Tensorflow and PyTorch with unified interface.
For users to compress their models, they only need to add several lines in their code.
There are some popular model compression algorithms built-in in NNI.
Users could further use NNI's auto tuning power to find the best compressed model,
which is detailed in Auto Model Compression.
On the other hand, users could easily customize their new compression algorithms using NNI's interface.
For details, please refer to the following tutorials:
.. toctree::
:maxdepth: 2
Overview <Compression/Overview>
Quick Start <Compression/QuickStart>
Tutorial <Compression/Tutorial>
Pruning <Compression/pruning>
Pruning V2 <Compression/v2_pruning>
Quantization <Compression/quantization>
Utilities <Compression/CompressionUtils>
Advanced Usage <Compression/advanced>
API Reference <Compression/CompressionReference>
docs/source/nas.rst deleted 100644 → 0
View file @ d63a2ea3
#############################################
Retiarii for Neural Architecture Search (NAS)
#############################################
Automatic neural architecture search is taking an increasingly important role on finding better models.
Recent research works have proved the feasibility of automatic NAS, and also found some models that could beat manually tuned models.
Some of representative works are NASNet, ENAS, DARTS, Network Morphism, and Evolution. Moreover, new innovations keep emerging.
However, it takes great efforts to implement NAS algorithms, and it is hard to reuse code base of existing algorithms in a new one.
To facilitate NAS innovations (e.g., design and implement new NAS models, compare different NAS models side-by-side),
an easy-to-use and flexible programming interface is crucial.
Thus, we design `Retiarii <https://www.usenix.org/system/files/osdi20-zhang_quanlu.pdf>`__. It is a deep learning framework that supports the exploratory training on a neural network model space, rather than on a single neural network model.
Exploratory training with Retiarii allows user to express various search spaces for *Neural Architecture Search* and *Hyper-Parameter Tuning* with high flexibility.
Some frequently used terminologies in this document:
* *Model search space*: it means a set of models from which the best model is explored/searched. Sometimes we use *search space* or *model space* in short.
* *Exploration strategy*: the algorithm that is used to explore a model search space.
* *Model evaluator*: it is used to train a model and evaluate the model's performance.
Follow the instructions below to start your journey with Retiarii.
.. toctree::
:maxdepth: 2
Overview <NAS/Overview>
Quick Start <NAS/QuickStart>
Construct Model Space <NAS/construct_space>
Multi-trial NAS <NAS/multi_trial_nas>
One-shot NAS <NAS/one_shot_nas>
Hardware-aware NAS <NAS/HardwareAwareNAS>
NAS Benchmarks <NAS/Benchmarks>
NAS API References <NAS/ApiReference>
docs/source/Compression/advanced.rst → docs/source/nas/advanced_usage.rst
View file @ 51d261e7
......@@ -4,6 +4,9 @@ Advanced Usage
.. toctree::
:maxdepth: 2
Framework <./Framework>
Customize a new algorithm <./CustomizeCompressor>
Automatic Model Compression (Beta) <./AutoCompression>
execution_engine
hardware_aware_nas
mutator
customize_strategy
serialization
benchmarks
docs/source/NAS/Benchmarks.rst → docs/source/nas/benchmarks.rst
View file @ 51d261e7
NAS Benchmarks
==============
.. code-block::
.. toctree::
:hidden:
.. toctree::
:hidden:
Example usage of NAS benchmarks </tutorials/nasbench_as_dataset>
Example Usages <BenchmarksExample>
.. note:: :doc:`Example usage of NAS benchmarks </tutorials/nasbench_as_dataset>`.
Introduction
------------
To improve the reproducibility of NAS algorithms as well as reducing computing resource requirements, researchers proposed a series of NAS benchmarks such as `NAS-Bench-101 <https://arxiv.org/abs/1902.09635>`__\ , `NAS-Bench-201 <https://arxiv.org/abs/2001.00326>`__\ , `NDS <https://arxiv.org/abs/1905.13214>`__\ , etc. NNI provides a query interface for users to acquire these benchmarks. Within just a few lines of code, researcher are able to evaluate their NAS algorithms easily and fairly by utilizing these benchmarks.
To improve the reproducibility of NAS algorithms as well as reducing computing resource requirements, researchers proposed a series of NAS benchmarks such as `NAS-Bench-101 <https://arxiv.org/abs/1902.09635>`__, `NAS-Bench-201 <https://arxiv.org/abs/2001.00326>`__, `NDS <https://arxiv.org/abs/1905.13214>`__, etc. NNI provides a query interface for users to acquire these benchmarks. Within just a few lines of code, researcher are able to evaluate their NAS algorithms easily and fairly by utilizing these benchmarks.
Prerequisites
-------------
* Please prepare a folder to household all the benchmark databases. By default, it can be found at ``${HOME}/.cache/nni/nasbenchmark``. Or you can place it anywhere you like, and specify it in ``NASBENCHMARK_DIR`` via ``export NASBENCHMARK_DIR=/path/to/your/nasbenchmark`` before importing NNI.
* Please install ``peewee`` via ``pip3 install peewee``\ , which NNI uses to connect to database.
* Please install ``peewee`` via ``pip3 install peewee``, which NNI uses to connect to database.
Data Preparation
----------------
......@@ -33,8 +33,7 @@ Option 2
.. note:: If you have files that are processed before v2.5, it is recommended that you delete them and try option 1.
#.
Clone NNI to your machine and enter ``examples/nas/benchmarks`` directory.
#. Clone NNI to your machine and enter ``examples/nas/benchmarks`` directory.
.. code-block:: bash
......@@ -43,8 +42,7 @@ Option 2
Replace ``${NNI_VERSION}`` with a released version name or branch name, e.g., ``v2.4``.
#.
Install dependencies via ``pip3 install -r xxx.requirements.txt``. ``xxx`` can be ``nasbench101``\ , ``nasbench201`` or ``nds``.
#. Install dependencies via ``pip3 install -r xxx.requirements.txt``. ``xxx`` can be ``nasbench101``, ``nasbench201`` or ``nds``.
#. Generate the database via ``./xxx.sh``. The directory that stores the benchmark file can be configured with ``NASBENCHMARK_DIR`` environment variable, which defaults to ``~/.nni/nasbenchmark``. Note that the NAS-Bench-201 dataset will be downloaded from a google drive.
......@@ -53,7 +51,7 @@ Please make sure there is at least 10GB free disk space and note that the conver
Example Usages
--------------
Please refer to `examples usages of Benchmarks API <./BenchmarksExample.rst>`__.
Please refer to `examples usages of Benchmarks API <../BenchmarksExample.rst>`__.
NAS-Bench-101
-------------
......@@ -61,36 +59,16 @@ NAS-Bench-101
* `Paper link <https://arxiv.org/abs/1902.09635>`__
* `Open-source <https://github.com/google-research/nasbench>`__
NAS-Bench-101 contains 423,624 unique neural networks, combined with 4 variations in number of epochs (4, 12, 36, 108), each of which is trained 3 times. It is a cell-wise search space, which constructs and stacks a cell by enumerating DAGs with at most 7 operators, and no more than 9 connections. All operators can be chosen from ``CONV3X3_BN_RELU``\ , ``CONV1X1_BN_RELU`` and ``MAXPOOL3X3``\ , except the first operator (always ``INPUT``\ ) and last operator (always ``OUTPUT``\ ).
NAS-Bench-101 contains 423,624 unique neural networks, combined with 4 variations in number of epochs (4, 12, 36, 108), each of which is trained 3 times. It is a cell-wise search space, which constructs and stacks a cell by enumerating DAGs with at most 7 operators, and no more than 9 connections. All operators can be chosen from ``CONV3X3_BN_RELU``, ``CONV1X1_BN_RELU`` and ``MAXPOOL3X3``, except the first operator (always ``INPUT``\ ) and last operator (always ``OUTPUT``\ ).
Notably, NAS-Bench-101 eliminates invalid cells (e.g., there is no path from input to output, or there is redundant computation). Furthermore, isomorphic cells are de-duplicated, i.e., all the remaining cells are computationally unique.
API Documentation
^^^^^^^^^^^^^^^^^
.. autofunction:: nni.nas.benchmarks.nasbench101.query_nb101_trial_stats
.. autoattribute:: nni.nas.benchmarks.nasbench101.INPUT
.. autoattribute:: nni.nas.benchmarks.nasbench101.OUTPUT
.. autoattribute:: nni.nas.benchmarks.nasbench101.CONV3X3_BN_RELU
.. autoattribute:: nni.nas.benchmarks.nasbench101.CONV1X1_BN_RELU
.. autoattribute:: nni.nas.benchmarks.nasbench101.MAXPOOL3X3
.. autoclass:: nni.nas.benchmarks.nasbench101.Nb101TrialConfig
.. autoclass:: nni.nas.benchmarks.nasbench101.Nb101TrialStats
.. autoclass:: nni.nas.benchmarks.nasbench101.Nb101IntermediateStats
.. autofunction:: nni.nas.benchmarks.nasbench101.graph_util.nasbench_format_to_architecture_repr
.. autofunction:: nni.nas.benchmarks.nasbench101.graph_util.infer_num_vertices
.. autofunction:: nni.nas.benchmarks.nasbench101.graph_util.hash_module
.. automodule:: nni.nas.benchmarks.nasbench101
:members:
:imported-members:
NAS-Bench-201
-------------
......@@ -99,28 +77,14 @@ NAS-Bench-201
* `Open-source API <https://github.com/D-X-Y/NAS-Bench-201>`__
* `Implementations <https://github.com/D-X-Y/AutoDL-Projects>`__
NAS-Bench-201 is a cell-wise search space that views nodes as tensors and edges as operators. The search space contains all possible densely-connected DAGs with 4 nodes, resulting in 15,625 candidates in total. Each operator (i.e., edge) is selected from a pre-defined operator set (\ ``NONE``\ , ``SKIP_CONNECT``\ , ``CONV_1X1``\ , ``CONV_3X3`` and ``AVG_POOL_3X3``\ ). Training appraoches vary in the dataset used (CIFAR-10, CIFAR-100, ImageNet) and number of epochs scheduled (12 and 200). Each combination of architecture and training approach is repeated 1 - 3 times with different random seeds.
NAS-Bench-201 is a cell-wise search space that views nodes as tensors and edges as operators. The search space contains all possible densely-connected DAGs with 4 nodes, resulting in 15,625 candidates in total. Each operator (i.e., edge) is selected from a pre-defined operator set (\ ``NONE``, ``SKIP_CONNECT``, ``CONV_1X1``, ``CONV_3X3`` and ``AVG_POOL_3X3``\ ). Training appraoches vary in the dataset used (CIFAR-10, CIFAR-100, ImageNet) and number of epochs scheduled (12 and 200). Each combination of architecture and training approach is repeated 1 - 3 times with different random seeds.
API Documentation
^^^^^^^^^^^^^^^^^
.. autofunction:: nni.nas.benchmarks.nasbench201.query_nb201_trial_stats
.. autoattribute:: nni.nas.benchmarks.nasbench201.NONE
.. autoattribute:: nni.nas.benchmarks.nasbench201.SKIP_CONNECT
.. autoattribute:: nni.nas.benchmarks.nasbench201.CONV_1X1
.. autoattribute:: nni.nas.benchmarks.nasbench201.CONV_3X3
.. autoattribute:: nni.nas.benchmarks.nasbench201.AVG_POOL_3X3
.. autoclass:: nni.nas.benchmarks.nasbench201.Nb201TrialConfig
.. autoclass:: nni.nas.benchmarks.nasbench201.Nb201TrialStats
.. autoclass:: nni.nas.benchmarks.nasbench201.Nb201IntermediateStats
.. automodule:: nni.nas.benchmarks.nasbench201
:members:
:imported-members:
NDS
---
......@@ -137,45 +101,12 @@ Available Operators
Here is a list of available operators used in NDS.
.. autoattribute:: nni.nas.benchmarks.nds.constants.NONE
.. autoattribute:: nni.nas.benchmarks.nds.constants.SKIP_CONNECT
.. autoattribute:: nni.nas.benchmarks.nds.constants.AVG_POOL_3X3
.. autoattribute:: nni.nas.benchmarks.nds.constants.MAX_POOL_3X3
.. autoattribute:: nni.nas.benchmarks.nds.constants.MAX_POOL_5X5
.. autoattribute:: nni.nas.benchmarks.nds.constants.MAX_POOL_7X7
.. autoattribute:: nni.nas.benchmarks.nds.constants.CONV_1X1
.. autoattribute:: nni.nas.benchmarks.nds.constants.CONV_3X3
.. autoattribute:: nni.nas.benchmarks.nds.constants.CONV_3X1_1X3
.. autoattribute:: nni.nas.benchmarks.nds.constants.CONV_7X1_1X7
.. autoattribute:: nni.nas.benchmarks.nds.constants.DIL_CONV_3X3
.. autoattribute:: nni.nas.benchmarks.nds.constants.DIL_CONV_5X5
.. autoattribute:: nni.nas.benchmarks.nds.constants.SEP_CONV_3X3
.. autoattribute:: nni.nas.benchmarks.nds.constants.SEP_CONV_5X5
.. autoattribute:: nni.nas.benchmarks.nds.constants.SEP_CONV_7X7
.. autoattribute:: nni.nas.benchmarks.nds.constants.DIL_SEP_CONV_3X3
.. automodule:: nni.nas.benchmarks.nds.constants
:noindex:
API Documentation
^^^^^^^^^^^^^^^^^
.. autofunction:: nni.nas.benchmarks.nds.query_nds_trial_stats
.. autoclass:: nni.nas.benchmarks.nds.NdsTrialConfig
.. autoclass:: nni.nas.benchmarks.nds.NdsTrialStats
.. autoclass:: nni.nas.benchmarks.nds.NdsIntermediateStats
.. automodule:: nni.nas.benchmarks.nds
:members:
:imported-members:
docs/source/nas/construct_space.rst 0 → 100644
View file @ 51d261e7
Construct Model Space
=====================
NNI provides powerful APIs for users to easily express model space (or search space).
Firstly, users can use high-level APIs (e.g., ValueChoice, LayerChoice) which are building blocks / skeletons of building blocks to construct their search space.
For advanced cases, NNI also provides interface to customize new mutators for expressing more complicated model spaces.
.. tip:: In most cases, this should be simple but expressive enough. We strongly recommend users to try them first, and report issues if those APIs are not satisfying.
.. _mutation-primitives:
Mutation Primitives
-------------------
To make users easily express a model space within their PyTorch/TensorFlow model, NNI provides some inline mutation APIs as shown below.
.. note:: We can actively adding more mutation primitives. If you have any suggestions, feel free to `ask here <https://github.com/microsoft/nni/issues>`__.
.. _nas-layer-choice:
LayerChoice
^^^^^^^^^^^
.. autoclass:: nni.retiarii.nn.pytorch.LayerChoice
:members:
.. _nas-input-choice:
InputChoice
^^^^^^^^^^^
.. autoclass:: nni.retiarii.nn.pytorch.InputChoice
:members:
.. autoclass:: nni.retiarii.nn.pytorch.ChosenInputs
:members:
.. _nas-value-choice:
ValueChoice
^^^^^^^^^^^
.. autoclass:: nni.retiarii.nn.pytorch.ValueChoice
:members:
:inherited-members: Module
.. _nas-repeat:
Repeat
^^^^^^
.. autoclass:: nni.retiarii.nn.pytorch.Repeat
:members:
.. _nas-cell:
Cell
^^^^
.. autoclass:: nni.retiarii.nn.pytorch.Cell
:members:
.. footbibliography::
.. _nas-cell-101:
NasBench101Cell
^^^^^^^^^^^^^^^
.. autoclass:: nni.retiarii.nn.pytorch.NasBench101Cell
:members:
.. footbibliography::
.. _nas-cell-201:
NasBench201Cell
^^^^^^^^^^^^^^^
.. autoclass:: nni.retiarii.nn.pytorch.NasBench201Cell
:members:
.. footbibliography::
.. _hyper-modules:
Hyper-module Library (experimental)
-----------------------------------
Hyper-module is a (PyTorch) module which contains many architecture/hyperparameter candidates for this module. By using hypermodule in user defined model, NNI will help users automatically find the best architecture/hyperparameter of the hyper-modules for this model. This follows the design philosophy of Retiarii that users write DNN model as a space.
We are planning to support some of the hyper-modules commonly used in the community, such as AutoDropout, AutoActivation. These are considered complementary to :ref:`mutation-primitives`, as they are often more concrete, specific, and tailored for particular needs.
.. _nas-autoactivation:
AutoActivation
^^^^^^^^^^^^^^
.. autoclass:: nni.retiarii.nn.pytorch.AutoActivation
:members:
Mutators (advanced)
-------------------
Besides the inline mutation APIs demonstrated :ref:`above <mutation-primitives>`, NNI provides a more general approach to express a model space, i.e., *Mutator*, to cover more complex model spaces. Those inline mutation APIs are also implemented with mutator in the underlying system, which can be seen as a special case of model mutation. Please read :doc:`./mutator` for details.
docs/source/NAS/WriteOneshot.rst → docs/source/nas/customize_strategy.rst
View file @ 51d261e7
Customize a New One-shot Trainer
================================
Customize Exploration Strategy
==============================
One-shot trainers should inherit ``nni.retiarii.oneshot.BaseOneShotTrainer``, and need to implement ``fit()`` (used to conduct the fitting and searching process) and ``export()`` method (used to return the searched best architecture).
Customize Multi-trial Strategy
------------------------------
If users want to innovate a new exploration strategy, they can easily customize a new one following the interface provided by NNI. Specifically, users should inherit the base strategy class ``BaseStrategy``, then implement the member function ``run``. This member function takes ``base_model`` and ``applied_mutators`` as its input arguments. It can simply apply the user specified mutators in ``applied_mutators`` onto ``base_model`` to generate a new model. When a mutator is applied, it should be bound with a sampler (e.g., ``RandomSampler``). Every sampler implements the ``choice`` function which chooses value(s) from candidate values. The ``choice`` functions invoked in mutators are executed with the sampler.
Below is a very simple random strategy, which makes the choices completely random.
.. code-block:: python
from nni.retiarii import Sampler
class RandomSampler(Sampler):
def choice(self, candidates, mutator, model, index):
return random.choice(candidates)
class RandomStrategy(BaseStrategy):
def __init__(self):
self.random_sampler = RandomSampler()
def run(self, base_model, applied_mutators):
_logger.info('stargety start...')
while True:
avail_resource = query_available_resources()
if avail_resource > 0:
model = base_model
_logger.info('apply mutators...')
_logger.info('mutators: %s', str(applied_mutators))
for mutator in applied_mutators:
mutator.bind_sampler(self.random_sampler)
model = mutator.apply(model)
# run models
submit_models(model)
else:
time.sleep(2)
You can find that this strategy does not know the search space beforehand, it passively makes decisions every time ``choice`` is invoked from mutators. If a strategy wants to know the whole search space before making any decision (e.g., TPE, SMAC), it can use ``dry_run`` function provided by ``Mutator`` to obtain the space. An example strategy can be found :githublink:`here <nni/retiarii/strategy/tpe_strategy.py>`.
After generating a new model, the strategy can use our provided APIs (e.g., ``submit_models``, ``is_stopped_exec``) to submit the model and get its reported results.
References
^^^^^^^^^^
.. autoclass:: nni.retiarii.Sampler
:members:
:noindex:
.. autoclass:: nni.retiarii.strategy.BaseStrategy
:members:
:noindex:
Customize a New One-shot Trainer (legacy)
-----------------------------------------
One-shot trainers should inherit :class:`nni.retiarii.oneshot.BaseOneShotTrainer`, and need to implement ``fit()`` (used to conduct the fitting and searching process) and ``export()`` method (used to return the searched best architecture).
Writing a one-shot trainer is very different to single-arch evaluator. First of all, there are no more restrictions on init method arguments, any Python arguments are acceptable. Secondly, the model fed into one-shot trainers might be a model with Retiarii-specific modules, such as LayerChoice and InputChoice. Such model cannot directly forward-propagate and trainers need to decide how to handle those modules.
......@@ -54,3 +107,10 @@ A typical example is DartsTrainer, where learnable-parameters are used to combin
return result
The full code of DartsTrainer is available to Retiarii source code. Please have a check at :githublink:`DartsTrainer <nni/retiarii/oneshot/pytorch/darts.py>`.
References
^^^^^^^^^^
.. autoclass:: nni.retiarii.oneshot.BaseOneShotTrainer
:members:
:noindex:
docs/source/NAS/ModelEvaluators.rst → docs/source/nas/evaluator.rst
View file @ 51d261e7
......@@ -3,10 +3,12 @@ Model Evaluators
A model evaluator is for training and validating each generated model. They are necessary to evaluate the performance of new explored models.
.. _functional-evaluator:
Customize Evaluator with Any Function
-------------------------------------
The simplest way to customize a new evaluator is with functional APIs, which is very easy when training code is already available. Users only need to write a fit function that wraps everything, which usually includes training, validating and testing of a single model. This function takes one positional arguments (``model_cls``) and possible keyword arguments. The keyword arguments (other than ``model_cls``) are fed to FunctionEvaluator as its initialization parameters (note that they will be `serialized <./Serialization.rst>`__). In this way, users get everything under their control, but expose less information to the framework and as a result, further optimizations like `CGO <./ExecutionEngines.rst#cgo-execution-engine-experimental>`__ might be not feasible. An example is as belows:
The simplest way to customize a new evaluator is with functional APIs, which is very easy when training code is already available. Users only need to write a fit function that wraps everything, which usually includes training, validating and testing of a single model. This function takes one positional arguments (``model_cls``) and possible keyword arguments. The keyword arguments (other than ``model_cls``) are fed to FunctionEvaluator as its initialization parameters (note that they will be :doc:`serialized <./serialization>`). In this way, users get everything under their control, but expose less information to the framework and as a result, further optimizations like :ref:`CGO <cgo-execution-engine>` might be not feasible. An example is as belows:
.. code-block:: python
......@@ -50,8 +52,8 @@ NNI provides some commonly used model evaluators for users' convenience. These e
We recommend to read the `serialization tutorial <./Serialization.rst>`__ before using these evaluators. A few notes to summarize the tutorial:
1. ``pl.DataLoader`` should be used in place of ``torch.utils.data.DataLoader``.
2. The datasets used in data-loader should be decorated with ``nni.trace`` recursively.
1. :class:`nni.retarii.evaluator.pytorch.DataLoader`` should be used in place of ``torch.utils.data.DataLoader``.
2. The datasets used in data-loader should be decorated with :meth:`nni.trace` recursively.
For example,
......@@ -69,18 +71,12 @@ For example,
val_dataloaders=pl.DataLoader(test_dataset, batch_size=100),
max_epochs=10)
.. autoclass:: nni.retiarii.evaluator.pytorch.lightning.Classification
:noindex:
.. autoclass:: nni.retiarii.evaluator.pytorch.lightning.Regression
:noindex:
Customize Evaluator with PyTorch-Lightning
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Another approach is to write training code in PyTorch-Lightning style, that is, to write a LightningModule that defines all elements needed for training (e.g., loss function, optimizer) and to define a trainer that takes (optional) dataloaders to execute the training. Before that, please read the `document of PyTorch-lightning <https://pytorch-lightning.readthedocs.io/>`__ to learn the basic concepts and components provided by PyTorch-lightning.
In practice, writing a new training module in Retiarii should inherit ``nni.retiarii.evaluator.pytorch.lightning.LightningModule``, which has a ``set_model`` that will be called after ``__init__`` to save the candidate model (generated by strategy) as ``self.model``. The rest of the process (like ``training_step``) should be the same as writing any other lightning module. Evaluators should also communicate with strategies via two API calls (``nni.report_intermediate_result`` for periodical metrics and ``nni.report_final_result`` for final metrics), added in ``on_validation_epoch_end`` and ``teardown`` respectively.
In practice, writing a new training module in Retiarii should inherit :class:`nni.retiarii.evaluator.pytorch.LightningModule`, which has a ``set_model`` that will be called after ``__init__`` to save the candidate model (generated by strategy) as ``self.model``. The rest of the process (like ``training_step``) should be the same as writing any other lightning module. Evaluators should also communicate with strategies via two API calls (:meth:`nni.report_intermediate_result` for periodical metrics and :meth:`nni.report_final_result` for final metrics), added in ``on_validation_epoch_end`` and ``teardown`` respectively.
An example is as follows:
......@@ -133,7 +129,7 @@ An example is as follows:
if stage == 'fit':
nni.report_final_result(self.trainer.callback_metrics['val_loss'].item())
Then, users need to wrap everything (including LightningModule, trainer and dataloaders) into a ``Lightning`` object, and pass this object into a Retiarii experiment.
Then, users need to wrap everything (including LightningModule, trainer and dataloaders) into a :class:`nni.retiarii.evaluator.pytorch.Lightning` object, and pass this object into a Retiarii experiment.
.. code-block:: python
......@@ -145,3 +141,39 @@ Then, users need to wrap everything (including LightningModule, trainer and data
train_dataloader=pl.DataLoader(train_dataset, batch_size=100),
val_dataloaders=pl.DataLoader(test_dataset, batch_size=100))
experiment = RetiariiExperiment(base_model, lightning, mutators, strategy)
References
----------
FunctionalEvaluator
^^^^^^^^^^^^^^^^^^^
.. autoclass:: nni.retiarii.evaluator.FunctionalEvaluator
:members:
:noindex:
.. _classification-evaluator:
Classification
^^^^^^^^^^^^^^
.. autoclass:: nni.retiarii.evaluator.pytorch.lightning.Classification
:members:
:noindex:
.. _regression-evaluator:
Regression
^^^^^^^^^^
.. autoclass:: nni.retiarii.evaluator.pytorch.lightning.Regression
:members:
:noindex:
Lightning
^^^^^^^^^
.. autoclass:: nni.retiarii.evaluator.pytorch.lightning.Lightning
:members:
:noindex:
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment