Skip to content

GitLab

Switch branch/tag
Find file Normal viewHistoryPermalink
construct_space.rst 3.03 KB
Newer Older
Yuge Zhang's avatar
[Doc] NAS (#4584)  
Yuge Zhang committed
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104 
Construct Model Space
=====================

NNI provides powerful APIs for users to easily express model space (or search space).
Firstly, users can use high-level APIs (e.g., ValueChoice, LayerChoice) which are building blocks / skeletons of building blocks to construct their search space.
For advanced cases, NNI also provides interface to customize new mutators for expressing more complicated model spaces.

.. tip:: In most cases, this should be simple but expressive enough. We strongly recommend users to try them first, and report issues if those APIs are not satisfying.

.. _mutation-primitives:

Mutation Primitives
-------------------

To make users easily express a model space within their PyTorch/TensorFlow model, NNI provides some inline mutation APIs as shown below.

.. note:: We can actively adding more mutation primitives. If you have any suggestions, feel free to `ask here <https://github.com/microsoft/nni/issues>`__.

.. _nas-layer-choice:

LayerChoice
^^^^^^^^^^^

.. autoclass:: nni.retiarii.nn.pytorch.LayerChoice
   :members:

.. _nas-input-choice:

InputChoice
^^^^^^^^^^^

.. autoclass:: nni.retiarii.nn.pytorch.InputChoice
   :members:

.. autoclass:: nni.retiarii.nn.pytorch.ChosenInputs
   :members:

.. _nas-value-choice:

ValueChoice
^^^^^^^^^^^

.. autoclass:: nni.retiarii.nn.pytorch.ValueChoice
   :members:

.. _nas-repeat:

Repeat
^^^^^^

.. autoclass:: nni.retiarii.nn.pytorch.Repeat
   :members:

.. _nas-cell:

Cell
^^^^

.. autoclass:: nni.retiarii.nn.pytorch.Cell
   :members:

.. footbibliography::

.. _nas-cell-101:

NasBench101Cell
^^^^^^^^^^^^^^^

.. autoclass:: nni.retiarii.nn.pytorch.NasBench101Cell
   :members:

.. footbibliography::

.. _nas-cell-201:

NasBench201Cell
^^^^^^^^^^^^^^^

.. autoclass:: nni.retiarii.nn.pytorch.NasBench201Cell
   :members:

.. footbibliography::

.. _hyper-modules:

Hyper-module Library (experimental)
-----------------------------------

Hyper-module is a (PyTorch) module which contains many architecture/hyperparameter candidates for this module. By using hypermodule in user defined model, NNI will help users automatically find the best architecture/hyperparameter of the hyper-modules for this model. This follows the design philosophy of Retiarii that users write DNN model as a space.

We are planning to support some of the hyper-modules commonly used in the community, such as AutoDropout, AutoActivation. These are considered complementary to :ref:`mutation-primitives`, as they are often more concrete, specific, and tailored for particular needs.

.. _nas-autoactivation:

AutoActivation
^^^^^^^^^^^^^^

..  autoclass:: nni.retiarii.nn.pytorch.AutoActivation
    :members:

Mutators (advanced)
-------------------

Besides the inline mutation APIs demonstrated :ref:`above <mutation-primitives>`, NNI provides a more general approach to express a model space, i.e., *Mutator*, to cover more complex model spaces. Those inline mutation APIs are also implemented with mutator in the underlying system, which can be seen as a special case of model mutation. Please read :doc:`./mutator` for details.