ENAS and DRATS search space zoo (#2589)

docs/en_US/NAS/Overview.md

@@ -54,6 +54,17 @@ Please refer to [here](NasGuide.md) for the usage of one-shot NAS algorithms.
 One-shot NAS can be visualized with our visualization tool. Learn more details [here](./Visualization.md).
 
 
+
+## Search Space Zoo
+
+NNI provides some predefined search space which can be easily reused. By stacking the extracted cells, user can quickly reproduce those NAS models.
+
+Search Space Zoo contains the following NAS cells:
+
+* [DartsCell](./SearchSpaceZoo.md#DartsCell)
+* [ENAS micro](./SearchSpaceZoo.md#ENASMicroLayer)
+* [ENAS macro](./SearchSpaceZoo.md#ENASMacroLayer)
+
 ## Using NNI API to Write Your Search Space
 
 The programming interface of designing and searching a model is often demanded in two scenarios.

docs/en_US/NAS/SearchSpaceZoo.md

+# Search Space Zoo
+
+## DartsCell
+
+DartsCell is extracted from [CNN model](./DARTS.md) designed [here](https://github.com/microsoft/nni/tree/master/examples/nas/darts). A DartsCell is a directed acyclic graph containing an ordered sequence of N nodes and each node stands for a latent representation (e.g. feature map in a convolutional network). Directed edges from Node 1 to Node 2 are associated with some operations that transform Node 1 and the result is stored on Node 2. The [operations](#darts-predefined-operations) between nodes is predefined and unchangeable. One edge represents an operation that chosen from the predefined ones to be applied to the starting node of the edge. One cell contains two input nodes, a single output node, and other `n_node` nodes. The input nodes are defined as the cell outputs in the previous two layers. The output of the cell is obtained by applying a reduction operation (e.g. concatenation) to all the intermediate nodes. To make the search space continuous, the categorical choice of a particular operation is relaxed to a softmax over all possible operations. By adjusting the weight of softmax on every node, the operation with the highest probability is chosen to be part of the final structure. A CNN model can be formed by stacking several cells together, which builds a search space. Note that, in DARTS paper all cells in the model share the same structure.
+
+One structure in the Darts search space is shown below. Note that, NNI merges the last one of the four intermediate nodes and the output node.
+
+![](../../img/NAS_Darts_cell.svg)
+
+The predefined operations are shown in [references](#predefined-operations-darts).
+
+```eval_rst
+..  autoclass:: nni.nas.pytorch.search_space_zoo.DartsCell
+    :members:
+```
+
+### Example code
+
+[example code](https://github.com/microsoft/nni/tree/master/examples/nas/search_space_zoo/darts_example.py)
+
+```bash
+git clone https://github.com/Microsoft/nni.git
+cd nni/examples/nas/search_space_zoo
+# search the best structure
+python3 darts_example.py
+```
+
+<a name="predefined-operations-darts"></a>
+
+### References
+
+All supported operations for Darts are listed below.
+
+* MaxPool / AvgPool
+  * MaxPool: Call `torch.nn.MaxPool2d`. This operation applies a 2D max pooling over all input channels. Its parameters `kernel_size=3` and `padding=1` are fixed. The pooling result will pass through a BatchNorm2d then return as the result.
+  * AvgPool: Call `torch.nn.AvgPool2d`. This operation applies a 2D average pooling over all input channels. Its parameters `kernel_size=3` and `padding=1` are fixed. The pooling result will pass through a BatchNorm2d then return as the result.
+
+    MaxPool / AvgPool with `kernel_size=3` and `padding=1` followed by BatchNorm2d
+    ```eval_rst
+    ..  autoclass:: nni.nas.pytorch.search_space_zoo.darts_ops.PoolBN
+    ```
+* SkipConnect
+
+    There is no operation between two nodes. Call `torch.nn.Identity` to forward what it gets to the output.
+* Zero operation
+
+    There is no connection between two nodes.
+* DilConv3x3 / DilConv5x5
+
+    <a name="DilConv"></a>DilConv3x3: (Dilated) depthwise separable Conv. It's a 3x3 depthwise convolution with `C_in` groups, followed by a 1x1 pointwise convolution. It reduces the amount of parameters. Input is first passed through relu, then DilConv and finally batchNorm2d. **Note that the operation is not Dilated Convolution, but we follow the convention in NAS papers to name it DilConv.** 3x3 DilConv has parameters `kernel_size=3`, `padding=1` and 5x5 DilConv has parameters `kernel_size=5`, `padding=4`.
+    ```eval_rst
+    ..  autoclass:: nni.nas.pytorch.search_space_zoo.darts_ops.DilConv
+    ```
+* SepConv3x3 / SepConv5x5
+
+    Composed of two DilConvs with fixed `kernel_size=3`, `padding=1` or `kernel_size=5`, `padding=2` sequentially.
+    ```eval_rst
+    ..  autoclass:: nni.nas.pytorch.search_space_zoo.darts_ops.SepConv
+    ```
+
+## ENASMicroLayer
+
+This layer is extracted from the model designed [here](https://github.com/microsoft/nni/tree/master/examples/nas/enas). A model contains several blocks that share the same architecture. A block is made up of some normal layers and reduction layers, `ENASMicroLayer` is a unified implementation of the two types of layers. The only difference between the two layers is that reduction layers apply all operations with `stride=2`.
+
+ENAS Micro employs a DAG with N nodes in one cell, where the nodes represent local computations, and the edges represent the flow of information between the N nodes. One cell contains two input nodes and a single output node. The following nodes choose two previous nodes as input and apply two operations from [predefined ones](#predefined-operations-enas) then add them as the output of this node. For example, Node 4 chooses Node 1 and Node 3 as inputs then applies `MaxPool` and `AvgPool` on the inputs respectively, then adds and sums them as the output of Node 4. Nodes that are not served as input for any other node are viewed as the output of the layer. If there are multiple output nodes, the model will calculate the average of these nodes as the layer output.
+
+One structure in the ENAS micro search space is shown below.
+
+![](../../img/NAS_ENAS_micro.svg) 
+
+The predefined operations can be seen [here](#predefined-operations-enas).
+
+```eval_rst
+..  autoclass:: nni.nas.pytorch.search_space_zoo.ENASMicroLayer
+    :members:
+```
+
+The Reduction Layer is made up of two Conv operations followed by BatchNorm, each of them will output `C_out//2` channels and concat them in channels as the output. The Convolution has `kernel_size=1` and `stride=2`, and they perform alternate sampling on the input to reduce the resolution without loss of information. This layer is wrapped in `ENASMicroLayer`.
+
+### Example code
+
+[example code](https://github.com/microsoft/nni/tree/master/examples/nas/search_space_zoo/enas_micro_example.py)
+
+```bash
+git clone https://github.com/Microsoft/nni.git
+cd nni/examples/nas/search_space_zoo
+# search the best cell structure
+python3 enas_micro_example.py
+```
+
+<a name="predefined-operations-enas"></a>
+
+### References
+
+All supported operations for ENAS micro search are listed below.
+
+* MaxPool / AvgPool
+    * MaxPool: Call `torch.nn.MaxPool2d`. This operation applies a 2D max pooling over all input channels followed by BatchNorm2d. Its parameters are fixed to `kernel_size=3`, `stride=1` and `padding=1`.
+    * AvgPool: Call `torch.nn.AvgPool2d`. This operation applies a 2D average pooling over all input channels followed by BatchNorm2d. Its parameters are fixed to `kernel_size=3`, `stride=1` and `padding=1`.
+    ```eval_rst
+    ..  autoclass:: nni.nas.pytorch.search_space_zoo.enas_ops.Pool
+    ```
+
+* SepConv
+    * SepConvBN3x3: ReLU followed by a [DilConv](#DilConv) and BatchNorm. Convolution parameters are `kernel_size=3`, `stride=1` and `padding=1`.
+    * SepConvBN5x5: Do the same operation as the previous one but it has different kernel sizes and paddings, which is set to 5 and 2 respectively.
+    
+    ```eval_rst
+    ..  autoclass:: nni.nas.pytorch.search_space_zoo.enas_ops.SepConvBN
+    ```
+
+* SkipConnect
+
+    Call `torch.nn.Identity` to connect directly to the next cell.
+
+## ENASMacroLayer
+
+In Macro search, the controller makes two decisions for each layer: i) the [operation](#macro-operations) to perform on the result of the previous layer, ii) which the previous layer to connect to for SkipConnects. ENAS uses a controller to design the whole model architecture instead of one of its components. The output of operations is going to concat with the tensor of the chosen layer for SkipConnect. NNI provides [predefined operations](#macro-operations) for macro search, which are listed in [references](#macro-operations).
+
+Part of one structure in the ENAS macro search space is shown below.
+
+![](../../img/NAS_ENAS_macro.svg)
+
+```eval_rst
+..  autoclass:: nni.nas.pytorch.search_space_zoo.ENASMacroLayer
+    :members:
+```
+
+To describe the whole search space, NNI provides a model, which is built by stacking the layers.
+
+```eval_rst
+..  autoclass:: nni.nas.pytorch.search_space_zoo.ENASMacroGeneralModel
+    :members:
+```
+
+### Example code
+
+[example code](https://github.com/microsoft/nni/tree/master/examples/nas/search_space_zoo/enas_macro_example.py)
+
+```bash
+git clone https://github.com/Microsoft/nni.git
+cd nni/examples/nas/search_space_zoo
+# search the best cell structure
+python3 enas_macro_example.py
+```
+
+<a name="macro-operations"></a>
+
+### References
+
+All supported operations for ENAS macro search are listed below.
+
+* ConvBranch
+    
+    All input first passes into a StdConv, which is made up of a 1x1Conv followed by BatchNorm2d and ReLU. Then the intermediate result goes through one of the operations listed below. The final result is calculated through a BatchNorm2d and ReLU as post-procedure.
+    * Separable Conv3x3: If `separable=True`, the cell will use [SepConv](#DilConv) instead of normal Conv operation. SepConv's `kernel_size=3`, `stride=1` and `padding=1`.
+    * Separable Conv5x5: SepConv's `kernel_size=5`, `stride=1` and `padding=2`.
+    * Normal Conv3x3: If `separable=False`, the cell will use a normal Conv operations with `kernel_size=3`, `stride=1` and `padding=1`.
+    * Normal Conv5x5: Conv's `kernel_size=5`, `stride=1` and `padding=2`.
+
+    ```eval_rst
+    ..  autoclass:: nni.nas.pytorch.search_space_zoo.enas_ops.ConvBranch
+    ```
+* PoolBranch
+
+    All input first passes into a StdConv, which is made up of a 1x1Conv followed by BatchNorm2d and ReLU. Then the intermediate goes through pooling operation followed by BatchNorm.
+    * AvgPool: Call `torch.nn.AvgPool2d`. This operation applies a 2D average pooling over all input channels. Its parameters are fixed to `kernel_size=3`, `stride=1` and `padding=1`.
+    * MaxPool: Call `torch.nn.MaxPool2d`. This operation applies a 2D max pooling over all input channels. Its parameters are fixed to `kernel_size=3`, `stride=1` and `padding=1`.
+
+    ```eval_rst
+    ..  autoclass:: nni.nas.pytorch.search_space_zoo.enas_ops.PoolBranch
+    ```
+
+<!-- push -->

docs/en_US/nas.rst

@@ -23,5 +23,6 @@ For details, please refer to the following tutorials:
     One-shot NAS <NAS/one_shot_nas>
     Customize a NAS Algorithm <NAS/Advanced>
     NAS Visualization <NAS/Visualization>
+    Search Space Zoo <NAS/SearchSpaceZoo>
     NAS Benchmarks <NAS/Benchmarks>
     API Reference <NAS/NasReference>

docs/img/NAS_Darts_cell.svg

+<svg id="SvgjsSvg1006" width="825.4666442871094" height="869.1333465576172" xmlns="http://www.w3.org/2000/svg" version="1.1" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:svgjs="http://svgjs.com/svgjs"><defs id="SvgjsDefs1007"><marker id="SvgjsMarker1046" markerWidth="16" markerHeight="12" refX="16" refY="6" viewBox="0 0 16 12" orient="auto" markerUnits="userSpaceOnUse" stroke-dasharray="0,0"><path id="SvgjsPath1047" d="M0,2 L14,6 L0,11 L0,2" fill="#323232" stroke="#323232" stroke-width="2"></path></marker><marker id="SvgjsMarker1050" markerWidth="16" markerHeight="12" refX="16" refY="6" viewBox="0 0 16 12" orient="auto" markerUnits="userSpaceOnUse" stroke-dasharray="0,0"><path id="SvgjsPath1051" d="M0,2 L14,6 L0,11 L0,2" fill="#323232" stroke="#323232" stroke-width="2"></path></marker><marker id="SvgjsMarker1054" markerWidth="16" markerHeight="12" refX="16" refY="6" viewBox="0 0 16 12" orient="auto" markerUnits="userSpaceOnUse" stroke-dasharray="0,0"><path id="SvgjsPath1055" d="M0,2 L14,6 L0,11 L0,2" fill="#323232" stroke="#323232" stroke-width="2"></path></marker><marker id="SvgjsMarker1058" markerWidth="16" markerHeight="12" refX="16" refY="6" viewBox="0 0 16 12" orient="auto" markerUnits="userSpaceOnUse" stroke-dasharray="0,0"><path id="SvgjsPath1059" d="M0,2 L14,6 L0,11 L0,2" fill="#323232" stroke="#323232" stroke-width="2"></path></marker><marker id="SvgjsMarker1062" markerWidth="16" markerHeight="12" refX="16" refY="6" viewBox="0 0 16 12" orient="auto" markerUnits="userSpaceOnUse" stroke-dasharray="0,0"><path id="SvgjsPath1063" d="M0,2 L14,6 L0,11 L0,2" fill="#323232" stroke="#323232" stroke-width="2"></path></marker><marker id="SvgjsMarker1066" markerWidth="16" markerHeight="12" refX="16" refY="6" viewBox="0 0 16 12" orient="auto" markerUnits="userSpaceOnUse" stroke-dasharray="0,0"><path id="SvgjsPath1067" d="M0,2 L14,6 L0,11 L0,2" fill="#323232" stroke="#323232" stroke-width="2"></path></marker><marker id="SvgjsMarker1070" markerWidth="16" markerHeight="12" refX="16" refY="6" viewBox="0 0 16 12" orient="auto" markerUnits="userSpaceOnUse" stroke-dasharray="0,0"><path id="SvgjsPath1071" d="M0,2 L14,6 L0,11 L0,2" fill="#323232" stroke="#323232" stroke-width="2"></path></marker><marker id="SvgjsMarker1074" markerWidth="16" markerHeight="12" refX="16" refY="6" viewBox="0 0 16 12" orient="auto" markerUnits="userSpaceOnUse" stroke-dasharray="0,0"><path id="SvgjsPath1075" d="M0,2 L14,6 L0,11 L0,2" fill="#323232" stroke="#323232" stroke-width="2"></path></marker><marker id="SvgjsMarker1078" markerWidth="16" markerHeight="12" refX="16" refY="6" viewBox="0 0 16 12" orient="auto" markerUnits="userSpaceOnUse" stroke-dasharray="0,0"><path id="SvgjsPath1079" d="M0,2 L14,6 L0,11 L0,2" fill="#323232" stroke="#323232" stroke-width="2"></path></marker><marker id="SvgjsMarker1082" markerWidth="16" markerHeight="12" refX="16" refY="6" viewBox="0 0 16 12" orient="auto" markerUnits="userSpaceOnUse" stroke-dasharray="0,0"><path id="SvgjsPath1083" d="M0,2 L14,6 L0,11 L0,2" fill="#323232" stroke="#323232" stroke-width="2"></path></marker><marker id="SvgjsMarker1086" markerWidth="16" markerHeight="12" refX="16" refY="6" viewBox="0 0 16 12" orient="auto" markerUnits="userSpaceOnUse" stroke-dasharray="0,0"><path id="SvgjsPath1087" d="M0,2 L14,6 L0,11 L0,2" fill="#323232" stroke="#323232" stroke-width="2"></path></marker><marker id="SvgjsMarker1090" markerWidth="16" markerHeight="12" refX="16" refY="6" viewBox="0 0 16 12" orient="auto" markerUnits="userSpaceOnUse" stroke-dasharray="0,0"><path id="SvgjsPath1091" d="M0,2 L14,6 L0,11 L0,2" fill="#323232" stroke="#323232" stroke-width="2"></path></marker><marker id="SvgjsMarker1094" markerWidth="16" markerHeight="12" refX="16" refY="6" viewBox="0 0 16 12" orient="auto" markerUnits="userSpaceOnUse" stroke-dasharray="0,0"><path id="SvgjsPath1095" d="M0,2 L14,6 L0,11 L0,2" fill="#323232" stroke="#323232" stroke-width="2"></path></marker><marker id="SvgjsMarker1098" markerWidth="16" markerHeight="12" refX="16" refY="6" viewBox="0 0 16 12" orient="auto" markerUnits="userSpaceOnUse" stroke-dasharray="0,0"><path id="SvgjsPath1099" d="M0,2 L14,6 L0,11 L0,2" fill="#323232" stroke="#323232" stroke-width="2"></path></marker><marker id="SvgjsMarker1108" markerWidth="16" markerHeight="12" refX="16" refY="6" viewBox="0 0 16 12" orient="auto" markerUnits="userSpaceOnUse" stroke-dasharray="0,0"><path id="SvgjsPath1109" d="M0,2 L14,6 L0,11 L0,2" fill="#323232" stroke="#323232" stroke-width="2"></path></marker><marker id="SvgjsMarker1112" markerWidth="16" markerHeight="12" refX="16" refY="6" viewBox="0 0 16 12" orient="auto" markerUnits="userSpaceOnUse" stroke-dasharray="0,0"><path id="SvgjsPath1113" d="M0,2 L14,6 L0,11 L0,2" fill="#323232" stroke="#323232" stroke-width="2"></path></marker><marker id="SvgjsMarker1116" markerWidth="16" markerHeight="12" refX="16" refY="6" viewBox="0 0 16 12" orient="auto" markerUnits="userSpaceOnUse" stroke-dasharray="0,0"><path id="SvgjsPath1117" d="M0,2 L14,6 L0,11 L0,2" fill="#323232" stroke="#323232" stroke-width="2"></path></marker><marker id="SvgjsMarker1120" markerWidth="16" markerHeight="12" refX="16" refY="6" viewBox="0 0 16 12" orient="auto" markerUnits="userSpaceOnUse" stroke-dasharray="0,0"><path id="SvgjsPath1121" d="M0,2 L14,6 L0,11 L0,2" fill="#323232" stroke="#323232" stroke-width="2"></path></marker><marker id="SvgjsMarker1134" markerWidth="16" markerHeight="12" refX="16" refY="6" viewBox="0 0 16 12" orient="auto" markerUnits="userSpaceOnUse" stroke-dasharray="0,0"><path id="SvgjsPath1135" d="M0,2 L14,6 L0,11 L0,2" fill="#323232" stroke="#323232" stroke-width="2"></path></marker><marker id="SvgjsMarker1170" markerWidth="16" markerHeight="12" refX="16" refY="6" viewBox="0 0 16 12" orient="auto" markerUnits="userSpaceOnUse" stroke-dasharray="0,0"><path id="SvgjsPath1171" d="M0,2 L14,6 L0,11 L0,2" fill="#323232" stroke="#323232" stroke-width="2"></path></marker></defs><g id="SvgjsG1008" transform="translate(102.03997802734375,25.008010864257812)"><path id="SvgjsPath1009" d="M 0 35C 0 -11.666666666666666 70 -11.666666666666666 70 35C 70 81.66666666666667 0 81.66666666666667 0 35Z" stroke="rgba(50,50,50,1)" stroke-width="2" fill-opacity="1" fill="#00cc66"></path><g id="SvgjsG1010"><text id="SvgjsText1011" font-family="微软雅黑" text-anchor="middle" font-size="13px" width="50px" fill="#323232" font-weight="400" align="middle" anchor="middle" family="微软雅黑" size="13px" weight="400" font-style="" opacity="1" y="25.55" transform="rotate(0)"><tspan id="SvgjsTspan1012" dy="16" x="35"><tspan id="SvgjsTspan1013" style="text-decoration:;">input 0</tspan></tspan></text></g></g><g id="SvgjsG1014" transform="translate(346.03997802734375,91.00801086425781)"><path id="SvgjsPath1015" d="M 0 35C 0 -11.666666666666666 70 -11.666666666666666 70 35C 70 81.66666666666667 0 81.66666666666667 0 35Z" stroke="rgba(50,50,50,1)" stroke-width="2" fill-opacity="1" fill="#00cc66"></path><g id="SvgjsG1016"><text id="SvgjsText1017" font-family="微软雅黑" text-anchor="middle" font-size="13px" width="50px" fill="#323232" font-weight="400" align="middle" anchor="middle" family="微软雅黑" size="13px" weight="400" font-style="" opacity="1" y="25.55" transform="rotate(0)"><tspan id="SvgjsTspan1018" dy="16" x="35"><tspan id="SvgjsTspan1019" style="text-decoration:;">input 1</tspan></tspan></text></g></g><g id="SvgjsG1020" transform="translate(102.03997802734375,247.0080108642578)"><path id="SvgjsPath1021" d="M 0 35C 0 -11.666666666666666 70 -11.666666666666666 70 35C 70 81.66666666666667 0 81.66666666666667 0 35Z" stroke="rgba(50,50,50,1)" stroke-width="2" fill-opacity="1" fill="#ffff00"></path><g id="SvgjsG1022"><text id="SvgjsText1023" font-family="微软雅黑" text-anchor="middle" font-size="13px" width="50px" fill="#323232" font-weight="400" align="middle" anchor="middle" family="微软雅黑" size="13px" weight="400" font-style="" opacity="1" y="25.55" transform="rotate(0)"><tspan id="SvgjsTspan1024" dy="16" x="35"><tspan id="SvgjsTspan1025" style="text-decoration:;">node 0</tspan></tspan></text></g></g><g id="SvgjsG1026" transform="translate(346.03997802734375,386.24000549316406)"><path id="SvgjsPath1027" d="M 0 35C 0 -11.666666666666666 70 -11.666666666666666 70 35C 70 81.66666666666667 0 81.66666666666667 0 35Z" stroke="rgba(50,50,50,1)" stroke-width="2" fill-opacity="1" fill="#ffff33"></path><g id="SvgjsG1028"><text id="SvgjsText1029" font-family="微软雅黑" text-anchor="middle" font-size="13px" width="50px" fill="#323232" font-weight="400" align="middle" anchor="middle" family="微软雅黑" size="13px" weight="400" font-style="" opacity="1" y="25.55" transform="rotate(0)"><tspan id="SvgjsTspan1030" dy="16" x="35"><tspan id="SvgjsTspan1031" style="text-decoration:;">node 1</tspan></tspan></text></g></g><g id="SvgjsG1032" transform="translate(102.03997802734375,503.24000549316406)"><path id="SvgjsPath1033" d="M 0 35C 0 -11.666666666666666 70 -11.666666666666666 70 35C 70 81.66666666666667 0 81.66666666666667 0 35Z" stroke="rgba(50,50,50,1)" stroke-width="2" fill-opacity="1" fill="#ffff33"></path><g id="SvgjsG1034"><text id="SvgjsText1035" font-family="微软雅黑" text-anchor="middle" font-size="13px" width="50px" fill="#323232" font-weight="400" align="middle" anchor="middle" family="微软雅黑" size="13px" weight="400" font-style="" opacity="1" y="25.55" transform="rotate(0)"><tspan id="SvgjsTspan1036" dy="16" x="35"><tspan id="SvgjsTspan1037" style="text-decoration:;">node 2</tspan></tspan></text></g></g><g id="SvgjsG1038" transform="translate(346.03997802734375,660.2400054931641)"><path id="SvgjsPath1039" d="M 0 35C 0 -11.666666666666666 70 -11.666666666666666 70 35C 70 81.66666666666667 0 81.66666666666667 0 35Z" stroke="rgba(50,50,50,1)" stroke-width="2" fill-opacity="1" fill="#ffff33"></path><g id="SvgjsG1040"><text id="SvgjsText1041" font-family="微软雅黑" text-anchor="middle" font-size="13px" width="50px" fill="#323232" font-weight="400" align="middle" anchor="middle" family="微软雅黑" size="13px" weight="400" font-style="" opacity="1" y="25.55" transform="rotate(0)"><tspan id="SvgjsTspan1042" dy="16" x="35"><tspan id="SvgjsTspan1043" style="text-decoration:;">node 3</tspan></tspan></text></g></g><g id="SvgjsG1044"><path id="SvgjsPath1045" d="M137.03997802734375 95.00801086425781C 142.3390471864014 155.57664850823596 142.3390471864014 186.43937322027966 137.03997802734375 247.0080108642578" stroke-dasharray="8,5" stroke="#323232" stroke-width="2" fill="none" marker-end="url(#SvgjsMarker1046)"></path></g><g id="SvgjsG1048"><path id="SvgjsPath1049" d="M137.03997802734375 95.00801086425781C 150.28545233965042 246.404475028805 394.2854523396504 234.84354132861688 381.03997802734375 386.24000549316406" stroke-dasharray="8,5" stroke="#323232" stroke-width="2" fill="none" marker-end="url(#SvgjsMarker1050)"></path></g><g id="SvgjsG1052"><path id="SvgjsPath1053" d="M381.03997802734375 161.0080108642578C 388.89208274031125 250.75797842023576 388.89208274031137 296.4900379371861 381.03997802734375 386.24000549316406" stroke-dasharray="8,5" stroke="#323232" stroke-width="2" fill="none" marker-end="url(#SvgjsMarker1054)"></path></g><g id="SvgjsG1056"><path id="SvgjsPath1057" d="M381.03997802734375 161.0080108642578C 390.059279443249 264.0990977824248 146.05927944324907 143.91692394609086 137.03997802734375 247.0080108642578" stroke-dasharray="8,5" stroke="#323232" stroke-width="2" fill="none" marker-end="url(#SvgjsMarker1058)"></path></g><g id="SvgjsG1060"><path id="SvgjsPath1061" d="M137.03997802734375 317.0080108642578C 143.53245315344782 391.21734113000457 143.53245315344788 429.0306752274173 137.03997802734375 503.24000549316406" stroke-dasharray="8,5" stroke="#323232" stroke-width="2" fill="none" marker-end="url(#SvgjsMarker1062)"></path></g><g id="SvgjsG1064"><path id="SvgjsPath1065" d="M137.03997802734375 317.0080108642578C 145.8821637569588 418.0746562244878 389.8821637569588 285.1733601329341 381.03997802734375 386.24000549316406" stroke-dasharray="8,5" stroke="#323232" stroke-width="2" fill="none" marker-end="url(#SvgjsMarker1066)"></path></g><g id="SvgjsG1068"><path id="SvgjsPath1069" d="M137.03997802734375 573.2400054931641C 146.07092807311756 676.4642368599839 390.07092807311756 557.0157741263442 381.03997802734375 660.2400054931641" stroke-dasharray="8,5" stroke="#323232" stroke-width="2" fill="none" marker-end="url(#SvgjsMarker1070)"></path></g><g id="SvgjsG1072"><path id="SvgjsPath1073" d="M381.03997802734375 456.24000549316406C 388.1518866355526 537.5294928574505 388.1518866355527 578.9505181288777 381.03997802734375 660.2400054931641" stroke-dasharray="8,5" stroke="#323232" stroke-width="2" fill="none" marker-end="url(#SvgjsMarker1074)"></path></g><g id="SvgjsG1076"><path id="SvgjsPath1077" d="M137.03997802734375 317.0080108642578C 151.72127748132198 484.81603149573016 395.721277481322 492.4319848616917 381.03997802734375 660.2400054931641" stroke-dasharray="8,5" stroke="#323232" stroke-width="2" fill="none" marker-end="url(#SvgjsMarker1078)"></path></g><g id="SvgjsG1080"><path id="SvgjsPath1081" d="M381.03997802734375 456.24000549316406C 389.70275028689764 555.2559455067756 145.7027502868977 404.22406547955256 137.03997802734375 503.24000549316406" stroke-dasharray="8,5" stroke="#323232" stroke-width="2" fill="none" marker-end="url(#SvgjsMarker1082)"></path></g><g id="SvgjsG1084"><path id="SvgjsPath1085" d="M381.03997802734375 161.0080108642578C 395.6928772148526 328.491414964573 151.69287721485261 335.7566013928489 137.03997802734375 503.24000549316406" stroke-dasharray="8,5" stroke="#323232" stroke-width="2" fill="none" marker-end="url(#SvgjsMarker1086)"></path></g><g id="SvgjsG1088"><path id="SvgjsPath1089" d="M137.03997802734375 95.00801086425781C 24.999984741210938 260.9920015335083 77.99998474121094 362.9920015335083 137.03997802734375 503.24000549316406" stroke-dasharray="8,5" stroke="#323232" stroke-width="2" fill="none" marker-end="url(#SvgjsMarker1090)"></path></g><g id="SvgjsG1092"><path id="SvgjsPath1093" d="M137.03997802734375 95.00801086425781C 158.5029026859753 340.33036228264245 402.5029026859754 414.9176540747794 381.03997802734375 660.2400054931641" stroke-dasharray="8,5" stroke="#323232" stroke-width="2" fill="none" marker-end="url(#SvgjsMarker1094)"></path></g><g id="SvgjsG1096"><path id="SvgjsPath1097" d="M381.03997802734375 161.0080108642578C 474.99998474121094 357.7520065307617 436.99998474121094 463.7520065307617 381.03997802734375 660.2400054931641" stroke-dasharray="8,5" stroke="#323232" stroke-width="2" fill="none" marker-end="url(#SvgjsMarker1098)"></path></g><g id="SvgjsG1100" transform="translate(213.99998474121094,774.1439895629883)"><path id="SvgjsPath1101" d="M 0 35C 0 -11.666666666666666 70 -11.666666666666666 70 35C 70 81.66666666666667 0 81.66666666666667 0 35Z" stroke="rgba(50,50,50,1)" stroke-width="2" fill-opacity="1" fill="#cc99ff"></path><g id="SvgjsG1102"><text id="SvgjsText1103" font-family="微软雅黑" text-anchor="middle" font-size="13px" width="50px" fill="#323232" font-weight="400" align="middle" anchor="middle" family="微软雅黑" size="13px" weight="400" font-style="" opacity="1" y="25.55" transform="rotate(0)"><tspan id="SvgjsTspan1104" dy="16" x="35"><tspan id="SvgjsTspan1105" style="text-decoration:;">output</tspan></tspan></text></g></g><g id="SvgjsG1106"><path id="SvgjsPath1107" d="M137.03997802734375 317.0080108642578C 153.44780421895462 504.5503224089872 265.40781093282186 586.6016780182589 248.99998474121094 774.1439895629883" stroke="#323232" stroke-width="2" fill="none" marker-end="url(#SvgjsMarker1108)"></path></g><g id="SvgjsG1110"><path id="SvgjsPath1111" d="M137.03997802734375 573.2400054931641C 145.05811339639388 664.8877121320274 257.01812011026107 682.4962829241249 248.99998474121094 774.1439895629883" stroke="#323232" stroke-width="2" fill="none" marker-end="url(#SvgjsMarker1112)"></path></g><g id="SvgjsG1114"><path id="SvgjsPath1115" d="M381.03997802734375 456.24000549316406C 393.0407891056441 593.409903793695 261.0007958195113 636.9740912624574 248.99998474121094 774.1439895629883" stroke="#323232" stroke-width="2" fill="none" marker-end="url(#SvgjsMarker1116)"></path></g><g id="SvgjsG1118"><path id="SvgjsPath1119" d="M381.03997802734375 730.2400054931641C 385.89099114909015 785.6873391961069 253.85099786295734 718.6966558600454 248.99998474121094 774.1439895629883" stroke="#323232" stroke-width="2" fill="none" marker-end="url(#SvgjsMarker1120)"></path></g><g id="SvgjsG1122" transform="translate(543.471981048584,627.2400054931641)"><path id="SvgjsPath1123" d="M 0 0L 255 0L 255 33L 0 33Z" stroke="none" fill="none"></path><g id="SvgjsG1124"><text id="SvgjsText1125" font-family="微软雅黑" text-anchor="start" font-size="13px" width="255px" fill="#323232" font-weight="400" align="top" anchor="start" family="微软雅黑" size="13px" weight="400" font-style="" opacity="1" y="-1.95" transform="rotate(0)"><tspan id="SvgjsTspan1126" dy="16" x="0"><tspan id="SvgjsTspan1127" style="text-decoration:;">choose an operation from AvgPool, </tspan></tspan><tspan id="SvgjsTspan1128" dy="16" x="0"><tspan id="SvgjsTspan1129" style="text-decoration:;">MaxPool, 3x3 SepConv, 5x5SepConv, </tspan></tspan><tspan id="SvgjsTspan1130" dy="16" x="0"><tspan id="SvgjsTspan1131" style="text-decoration:;">3x3DilConv, 5x5DilConv, SkipConnect</tspan></tspan></text></g></g><g id="SvgjsG1132"><path id="SvgjsPath1133" d="M473.471981048584 655.6359786987305L523.471981048584 655.6359786987305" stroke-dasharray="8,5" stroke="#323232" stroke-width="2" fill="none" marker-end="url(#SvgjsMarker1134)"></path></g><g id="SvgjsG1136" transform="translate(484.471981048584,682.1359786987305)"><path id="SvgjsPath1137" d="M 0 13.104026794433594C 0 -4.368008931477864 29 -4.368008931477864 29 13.104026794433594C 29 30.57606252034505 0 30.57606252034505 0 13.104026794433594Z" stroke="rgba(50,50,50,1)" stroke-width="2" fill-opacity="1" fill="#00cc00"></path><g id="SvgjsG1138"><text id="SvgjsText1139" font-family="微软雅黑" text-anchor="middle" font-size="13px" width="9px" fill="#323232" font-weight="400" align="middle" anchor="middle" family="微软雅黑" size="13px" weight="400" font-style="" opacity="1" y="3.6540267944335936" transform="rotate(0)"></text></g></g><g id="SvgjsG1140" transform="translate(484.471981048584,720.2400054931641)"><path id="SvgjsPath1141" d="M 0 13.104026794433594C 0 -4.368008931477864 29 -4.368008931477864 29 13.104026794433594C 29 30.57606252034505 0 30.57606252034505 0 13.104026794433594Z" stroke="rgba(50,50,50,1)" stroke-width="2" fill-opacity="1" fill="#ffff00"></path><g id="SvgjsG1142"><text id="SvgjsText1143" font-family="微软雅黑" text-anchor="middle" font-size="13px" width="9px" fill="#323232" font-weight="400" align="middle" anchor="middle" family="微软雅黑" size="13px" weight="400" font-style="" opacity="1" y="3.6540267944335936" transform="rotate(0)"></text></g></g><g id="SvgjsG1144" transform="translate(484.471981048584,760.2400054931641)"><path id="SvgjsPath1145" d="M 0 13.104026794433594C 0 -4.368008931477864 29 -4.368008931477864 29 13.104026794433594C 29 30.57606252034505 0 30.57606252034505 0 13.104026794433594Z" stroke="rgba(50,50,50,1)" stroke-width="2" fill-opacity="1" fill="#cc99ff"></path><g id="SvgjsG1146"><text id="SvgjsText1147" font-family="微软雅黑" text-anchor="middle" font-size="13px" width="9px" fill="#323232" font-weight="400" align="middle" anchor="middle" family="微软雅黑" size="13px" weight="400" font-style="" opacity="1" y="3.6540267944335936" transform="rotate(0)"></text></g></g><g id="SvgjsG1148" transform="translate(544.471981048584,684.4480590820312)"><path id="SvgjsPath1149" d="M 0 0L 232 0L 232 35.79194641113281L 0 35.79194641113281Z" stroke="none" fill="none"></path><g id="SvgjsG1150"><text id="SvgjsText1151" font-family="微软雅黑" text-anchor="start" font-size="13px" width="232px" fill="#323232" font-weight="400" align="top" anchor="start" family="微软雅黑" size="13px" weight="400" font-style="" opacity="1" y="-1.95" transform="rotate(0)"><tspan id="SvgjsTspan1152" dy="16" x="0"><tspan id="SvgjsTspan1153" style="text-decoration:;">results of previous layers as input</tspan></tspan></text></g></g><g id="SvgjsG1154" transform="translate(544.471981048584,720.2400054931641)"><path id="SvgjsPath1155" d="M 0 0L 251 0L 251 37.79194641113281L 0 37.79194641113281Z" stroke="none" fill="none"></path><g id="SvgjsG1156"><text id="SvgjsText1157" font-family="微软雅黑" text-anchor="start" font-size="13px" width="251px" fill="#323232" font-weight="400" align="top" anchor="start" family="微软雅黑" size="13px" weight="400" font-style="" opacity="1" y="-1.95" transform="rotate(0)"><tspan id="SvgjsTspan1158" dy="16" x="0"><tspan id="SvgjsTspan1159" style="text-decoration:;">accept two operations from all inputs, </tspan></tspan><tspan id="SvgjsTspan1160" dy="16" x="0"><tspan id="SvgjsTspan1161" style="text-decoration:;">then add the results as output</tspan></tspan></text></g></g><g id="SvgjsG1162" transform="translate(545.471981048584,763.2400054931641)"><path id="SvgjsPath1163" d="M 0 0L 251 0L 251 37.79194641113281L 0 37.79194641113281Z" stroke="none" fill="none"></path><g id="SvgjsG1164"><text id="SvgjsText1165" font-family="微软雅黑" text-anchor="start" font-size="13px" width="251px" fill="#323232" font-weight="400" align="top" anchor="start" family="微软雅黑" size="13px" weight="400" font-style="" opacity="1" y="-1.95" transform="rotate(0)"><tspan id="SvgjsTspan1166" dy="16" x="0"><tspan id="SvgjsTspan1167" style="text-decoration:;">concat all inputs in channels</tspan></tspan></text></g></g><g id="SvgjsG1168"><path id="SvgjsPath1169" d="M472.471981048584 599.6359786987305L522.471981048584 599.6359786987305" stroke="#323232" stroke-width="2" fill="none" marker-end="url(#SvgjsMarker1170)"></path></g><g id="SvgjsG1172" transform="translate(545.471981048584,586.1359786987305)"><path id="SvgjsPath1173" d="M 0 0L 255 0L 255 33L 0 33Z" stroke="none" fill="none"></path><g id="SvgjsG1174"><text id="SvgjsText1175" font-family="微软雅黑" text-anchor="start" font-size="13px" width="255px" fill="#323232" font-weight="400" align="middle" anchor="start" family="微软雅黑" size="13px" weight="400" font-style="" opacity="1" y="-0.95" transform="rotate(0)"><tspan id="SvgjsTspan1176" dy="16" x="0"><tspan id="SvgjsTspan1177" style="text-decoration:;">pass the result directly to the output </tspan></tspan><tspan id="SvgjsTspan1178" dy="16" x="0"><tspan id="SvgjsTspan1179" style="text-decoration:;">node</tspan></tspan></text></g></g></svg>
\ No newline at end of file

docs/img/NAS_ENAS_macro.svg

+<svg id="SvgjsSvg1006" width="837.0800476074219" height="917.6933898925781" xmlns="http://www.w3.org/2000/svg" version="1.1" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:svgjs="http://svgjs.com/svgjs"><defs id="SvgjsDefs1007"><marker id="SvgjsMarker1022" markerWidth="16" markerHeight="12" refX="16" refY="6" viewBox="0 0 16 12" orient="auto" markerUnits="userSpaceOnUse" stroke-dasharray="0,0"><path id="SvgjsPath1023" d="M0,2 L14,6 L0,11 L0,2" fill="#323232" stroke="#323232" stroke-width="2"></path></marker><marker id="SvgjsMarker1036" markerWidth="16" markerHeight="12" refX="16" refY="6" viewBox="0 0 16 12" orient="auto" markerUnits="userSpaceOnUse" stroke-dasharray="0,0"><path id="SvgjsPath1037" d="M0,2 L14,6 L0,11 L0,2" fill="#323232" stroke="#323232" stroke-width="2"></path></marker><marker id="SvgjsMarker1040" markerWidth="16" markerHeight="12" refX="16" refY="6" viewBox="0 0 16 12" orient="auto" markerUnits="userSpaceOnUse" stroke-dasharray="0,0"><path id="SvgjsPath1041" d="M0,2 L14,6 L0,11 L0,2" fill="#323232" stroke="#323232" stroke-width="2"></path></marker><marker id="SvgjsMarker1044" markerWidth="16" markerHeight="12" refX="16" refY="6" viewBox="0 0 16 12" orient="auto" markerUnits="userSpaceOnUse" stroke-dasharray="0,0"><path id="SvgjsPath1045" d="M0,2 L14,6 L0,11 L0,2" fill="#323232" stroke="#323232" stroke-width="2"></path></marker><marker id="SvgjsMarker1053" markerWidth="16" markerHeight="12" refX="16" refY="6" viewBox="0 0 16 12" orient="auto" markerUnits="userSpaceOnUse" stroke-dasharray="0,0"><path id="SvgjsPath1054" d="M0,2 L14,6 L0,11 L0,2" fill="#323232" stroke="#323232" stroke-width="2"></path></marker><marker id="SvgjsMarker1089" markerWidth="16" markerHeight="12" refX="16" refY="6" viewBox="0 0 16 12" orient="auto" markerUnits="userSpaceOnUse" stroke-dasharray="0,0"><path id="SvgjsPath1090" d="M0,2 L14,6 L0,11 L0,2" fill="#323232" stroke="#323232" stroke-width="2"></path></marker><marker id="SvgjsMarker1141" markerWidth="16" markerHeight="12" refX="16" refY="6" viewBox="0 0 16 12" orient="auto" markerUnits="userSpaceOnUse" stroke-dasharray="0,0"><path id="SvgjsPath1142" d="M0,2 L14,6 L0,11 L0,2" fill="#323232" stroke="#323232" stroke-width="2"></path></marker><marker id="SvgjsMarker1145" markerWidth="16" markerHeight="12" refX="16" refY="6" viewBox="0 0 16 12" orient="auto" markerUnits="userSpaceOnUse" stroke-dasharray="0,0"><path id="SvgjsPath1146" d="M0,2 L14,6 L0,11 L0,2" fill="#323232" stroke="#323232" stroke-width="2"></path></marker><marker id="SvgjsMarker1149" markerWidth="16" markerHeight="12" refX="16" refY="6" viewBox="0 0 16 12" orient="auto" markerUnits="userSpaceOnUse" stroke-dasharray="0,0"><path id="SvgjsPath1150" d="M0,2 L14,6 L0,11 L0,2" fill="#323232" stroke="#323232" stroke-width="2"></path></marker><marker id="SvgjsMarker1153" markerWidth="16" markerHeight="12" refX="16" refY="6" viewBox="0 0 16 12" orient="auto" markerUnits="userSpaceOnUse" stroke-dasharray="0,0"><path id="SvgjsPath1154" d="M0,2 L14,6 L0,11 L0,2" fill="#323232" stroke="#323232" stroke-width="2"></path></marker><marker id="SvgjsMarker1157" markerWidth="16" markerHeight="12" refX="16" refY="6" viewBox="0 0 16 12" orient="auto" markerUnits="userSpaceOnUse" stroke-dasharray="0,0"><path id="SvgjsPath1158" d="M0,2 L14,6 L0,11 L0,2" fill="#323232" stroke="#323232" stroke-width="2"></path></marker><marker id="SvgjsMarker1161" markerWidth="16" markerHeight="12" refX="16" refY="6" viewBox="0 0 16 12" orient="auto" markerUnits="userSpaceOnUse" stroke-dasharray="0,0"><path id="SvgjsPath1162" d="M0,2 L14,6 L0,11 L0,2" fill="#323232" stroke="#323232" stroke-width="2"></path></marker><marker id="SvgjsMarker1165" markerWidth="16" markerHeight="12" refX="16" refY="6" viewBox="0 0 16 12" orient="auto" markerUnits="userSpaceOnUse" stroke-dasharray="0,0"><path id="SvgjsPath1166" d="M0,2 L14,6 L0,11 L0,2" fill="#323232" stroke="#323232" stroke-width="2"></path></marker><marker id="SvgjsMarker1169" markerWidth="16" markerHeight="12" refX="16" refY="6" viewBox="0 0 16 12" orient="auto" markerUnits="userSpaceOnUse" stroke-dasharray="0,0"><path id="SvgjsPath1170" d="M0,2 L14,6 L0,11 L0,2" fill="#323232" stroke="#323232" stroke-width="2"></path></marker><marker id="SvgjsMarker1173" markerWidth="16" markerHeight="12" refX="16" refY="6" viewBox="0 0 16 12" orient="auto" markerUnits="userSpaceOnUse" stroke-dasharray="0,0"><path id="SvgjsPath1174" d="M0,2 L14,6 L0,11 L0,2" fill="#323232" stroke="#323232" stroke-width="2"></path></marker><marker id="SvgjsMarker1177" markerWidth="16" markerHeight="12" refX="16" refY="6" viewBox="0 0 16 12" orient="auto" markerUnits="userSpaceOnUse" stroke-dasharray="0,0"><path id="SvgjsPath1178" d="M0,2 L14,6 L0,11 L0,2" fill="#323232" stroke="#323232" stroke-width="2"></path></marker><marker id="SvgjsMarker1181" markerWidth="16" markerHeight="12" refX="16" refY="6" viewBox="0 0 16 12" orient="auto" markerUnits="userSpaceOnUse" stroke-dasharray="0,0"><path id="SvgjsPath1182" d="M0,2 L14,6 L0,11 L0,2" fill="#323232" stroke="#323232" stroke-width="2"></path></marker><marker id="SvgjsMarker1191" markerWidth="16" markerHeight="12" refX="16" refY="6" viewBox="0 0 16 12" orient="auto" markerUnits="userSpaceOnUse" stroke-dasharray="0,0"><path id="SvgjsPath1192" d="M0,2 L14,6 L0,11 L0,2" fill="#323232" stroke="#323232" stroke-width="2"></path></marker><marker id="SvgjsMarker1205" markerWidth="16" markerHeight="12" refX="16" refY="6" viewBox="0 0 16 12" orient="auto" markerUnits="userSpaceOnUse" stroke-dasharray="0,0"><path id="SvgjsPath1206" d="M0,2 L14,6 L0,11 L0,2" fill="#323232" stroke="#323232" stroke-width="2"></path></marker><marker id="SvgjsMarker1225" markerWidth="16" markerHeight="12" refX="16" refY="6" viewBox="0 0 16 12" orient="auto" markerUnits="userSpaceOnUse" stroke-dasharray="0,0"><path id="SvgjsPath1226" d="M0,2 L14,6 L0,11 L0,2" fill="#323232" stroke="#323232" stroke-width="2"></path></marker><marker id="SvgjsMarker1237" markerWidth="16" markerHeight="12" refX="16" refY="6" viewBox="0 0 16 12" orient="auto" markerUnits="userSpaceOnUse" stroke-dasharray="0,0"><path id="SvgjsPath1238" d="M0,2 L14,6 L0,11 L0,2" fill="#323232" stroke="#323232" stroke-width="2"></path></marker><marker id="SvgjsMarker1241" markerWidth="16" markerHeight="12" refX="16" refY="6" viewBox="0 0 16 12" orient="auto" markerUnits="userSpaceOnUse" stroke-dasharray="0,0"><path id="SvgjsPath1242" d="M0,2 L14,6 L0,11 L0,2" fill="#323232" stroke="#323232" stroke-width="2"></path></marker></defs><g id="SvgjsG1008" transform="translate(74.002685546875,148.85643214220443)"><path id="SvgjsPath1009" d="M 0 133.86033519553072C 0 -44.62011173184357 267.72067039106145 -44.62011173184357 267.72067039106145 133.86033519553072C 267.72067039106145 312.340782122905 0 312.340782122905 0 133.86033519553072Z" stroke="rgba(50,50,50,1)" stroke-width="2" fill-opacity="1" fill="#ffffff"></path><g id="SvgjsG1010"><text id="SvgjsText1011" font-family="微软雅黑" text-anchor="middle" font-size="13px" width="248px" fill="#323232" font-weight="400" align="middle" anchor="middle" family="微软雅黑" size="13px" weight="400" font-style="" opacity="1" y="124.41033519553072" transform="rotate(0)"></text></g></g><g id="SvgjsG1012" transform="translate(131.37140063067386,187.10224219807034)"><path id="SvgjsPath1013" d="M 0 31.871508379888265C 0 -10.623836126629422 63.74301675977653 -10.623836126629422 63.74301675977653 31.871508379888265C 63.74301675977653 74.36685288640595 0 74.36685288640595 0 31.871508379888265Z" stroke="rgba(50,50,50,1)" stroke-width="2" fill-opacity="1" fill="#ffff00"></path><g id="SvgjsG1014"><text id="SvgjsText1015" font-family="微软雅黑" text-anchor="middle" font-size="13px" width="44px" fill="#323232" font-weight="400" align="middle" anchor="middle" family="微软雅黑" size="13px" weight="400" font-style="" opacity="1" y="22.421508379888266" transform="rotate(0)"></text></g></g><g id="SvgjsG1016" transform="translate(209.68424979268502,311.8564321422044)"><path id="SvgjsPath1017" d="M 0 31.871508379888265C 0 -10.623836126629422 63.74301675977653 -10.623836126629422 63.74301675977653 31.871508379888265C 63.74301675977653 74.36685288640595 0 74.36685288640595 0 31.871508379888265Z" stroke="rgba(50,50,50,1)" stroke-width="2" fill-opacity="1" fill="#cc99ff"></path><g id="SvgjsG1018"><text id="SvgjsText1019" font-family="微软雅黑" text-anchor="middle" font-size="13px" width="44px" fill="#323232" font-weight="400" align="middle" anchor="middle" family="微软雅黑" size="13px" weight="400" font-style="" opacity="1" y="22.421508379888266" transform="rotate(0)"></text></g></g><g id="SvgjsG1020"><path id="SvgjsPath1021" d="M109.51665202732192 96.04078968410386C 131.00715482061798 132.46537068969045 142.81869506835938 152.7813491821289 163.24290901056213 187.10224219807034" stroke-dasharray="10,5,3,5" stroke="#323232" stroke-width="2" fill="none" marker-end="url(#SvgjsMarker1022)"></path></g><g id="SvgjsG1024" transform="translate(301.6563168317912,25.923471248349685)"><path id="SvgjsPath1025" d="M 0 35.05865921787709C 0 -11.686219739292364 70.11731843575419 -11.686219739292364 70.11731843575419 35.05865921787709C 70.11731843575419 81.80353817504655 0 81.80353817504655 0 35.05865921787709Z" stroke="rgba(50,50,50,1)" stroke-width="2" fill-opacity="1" fill="#00cc66"></path><g id="SvgjsG1026"><text id="SvgjsText1027" font-family="微软雅黑" text-anchor="middle" font-size="13px" width="51px" fill="#323232" font-weight="400" align="middle" anchor="middle" family="微软雅黑" size="13px" weight="400" font-style="" opacity="1" y="10.108659217877094" transform="rotate(0)"><tspan id="SvgjsTspan1028" dy="16" x="35.5"><tspan id="SvgjsTspan1029" style="text-decoration:;">Layer </tspan></tspan><tspan id="SvgjsTspan1030" dy="16" x="35.5"><tspan id="SvgjsTspan1031" style="text-decoration:;">N-1's</tspan></tspan><tspan id="SvgjsTspan1032" dy="16" x="35.5"><tspan id="SvgjsTspan1033" style="text-decoration:;">output</tspan></tspan></text></g></g><g id="SvgjsG1034"><path id="SvgjsPath1035" d="M336.7149760496683 96.04078968410386C 344.9377251996825 190.02724254125366 249.77850732258764 217.86997928505463 241.5557581725733 311.8564321422044" stroke-dasharray="8,5" stroke="#323232" stroke-width="2" fill="none" marker-end="url(#SvgjsMarker1036)"></path></g><g id="SvgjsG1038"><path id="SvgjsPath1039" d="M132.28201515581355 94.21956063382453C 144.66637269771297 131.37263325952284 152.81869506835938 149.7813491821289 163.24290901056213 187.10224219807034" stroke-dasharray="10,5,3,5" stroke="#323232" stroke-width="2" fill="none" marker-end="url(#SvgjsMarker1040)"></path></g><g id="SvgjsG1042"><path id="SvgjsPath1043" d="M163.24290901056213 93.30894610868489C 163.24290901056213 130.82626454443908 166.51275876473073 149.7276884857523 163.24290901056213 187.10224219807034" stroke-dasharray="10,5,3,5" stroke="#323232" stroke-width="2" fill="none" marker-end="url(#SvgjsMarker1044)"></path></g><g id="SvgjsG1046" transform="matrix(6.123233995736766e-17,1,-1,6.123233995736766e-17,175.99151428052167,25.012857064188523)"><path id="SvgjsPath1047" d="M 61.9217877094972 0Q 55.729608938547486 0 55.729608938547486 6.73854748603352L 55.729608938547486 26.95418994413408Q 55.729608938547486 33.6927374301676 39.9217877094972 33.6927374301676Q 55.729608938547486 33.6927374301676 55.729608938547486 40.431284916201115L 55.729608938547486 60.646927374301676Q 55.729608938547486 67.3854748603352 61.9217877094972 67.3854748603352" stroke="rgba(50,50,50,1)" stroke-width="2" fill="none"></path><path id="SvgjsPath1048" d="M 0 0L 61.9217877094972 0L 61.9217877094972 67.3854748603352L 0 67.3854748603352Z" stroke="none" fill="none"></path><g id="SvgjsG1049"><text id="SvgjsText1050" font-family="微软雅黑" text-anchor="end" font-size="13px" width="35px" fill="#323232" font-weight="400" align="middle" anchor="end" family="微软雅黑" size="13px" weight="400" font-style="" opacity="1" y="24.242737430167598" transform="rotate(0)"></text></g></g><g id="SvgjsG1051"><path id="SvgjsPath1052" d="M427.321121301065 258.1301751589642L472.8518475580481 258.1301751589642" stroke-dasharray="8,5" stroke="#323232" stroke-width="2" fill="none" marker-end="url(#SvgjsMarker1053)"></path></g><g id="SvgjsG1055" transform="translate(489.24290901056213,239.00727013103125)"><path id="SvgjsPath1056" d="M 0 0L 236.75977653631284 0L 236.75977653631284 46.44134078212291L 0 46.44134078212291Z" stroke="none" fill="none"></path><g id="SvgjsG1057"><text id="SvgjsText1058" font-family="微软雅黑" text-anchor="start" font-size="13px" width="237px" fill="#323232" font-weight="400" align="middle" anchor="start" family="微软雅黑" size="13px" weight="400" font-style="" opacity="1" y="-1.7293296089385464" transform="rotate(0)"><tspan id="SvgjsTspan1059" dy="16" x="0"><tspan id="SvgjsTspan1060" style="text-decoration:;">choose an operation from: MaxPool, </tspan></tspan><tspan id="SvgjsTspan1061" dy="16" x="0"><tspan id="SvgjsTspan1062" style="text-decoration:;">AvgPool, SepConvBN3x3, SepConvBN</tspan></tspan><tspan id="SvgjsTspan1063" dy="16" x="0"><tspan id="SvgjsTspan1064" style="text-decoration:;">5x5 and SkipConnect</tspan></tspan></text></g></g><g id="SvgjsG1065" transform="translate(424.58927772564596,293.6441416394111)"><path id="SvgjsPath1066" d="M 0 25.952513966480446C 0 -8.650837988826815 51.90502793296089 -8.650837988826815 51.90502793296089 25.952513966480446C 51.90502793296089 60.555865921787706 0 60.555865921787706 0 25.952513966480446Z" stroke="rgba(50,50,50,1)" stroke-width="2" fill-opacity="1" fill="#ffff00"></path><g id="SvgjsG1067"><text id="SvgjsText1068" font-family="微软雅黑" text-anchor="middle" font-size="13px" width="32px" fill="#323232" font-weight="400" align="middle" anchor="middle" family="微软雅黑" size="13px" weight="400" font-style="" opacity="1" y="16.502513966480446" transform="rotate(0)"></text></g></g><g id="SvgjsG1069" transform="translate(489.24290901056213,301.3843651030983)"><path id="SvgjsPath1070" d="M 0 0L 236.75977653631284 0L 236.75977653631284 36.42458100558659L 0 36.42458100558659Z" stroke="none" fill="none"></path><g id="SvgjsG1071"><text id="SvgjsText1072" font-family="微软雅黑" text-anchor="start" font-size="13px" width="237px" fill="#323232" font-weight="400" align="middle" anchor="start" family="微软雅黑" size="13px" weight="400" font-style="" opacity="1" y="8.762290502793295" transform="rotate(0)"><tspan id="SvgjsTspan1073" dy="16" x="0"><tspan id="SvgjsTspan1074" style="text-decoration:;">accept none/one/multiple input(s)</tspan></tspan></text></g></g><g id="SvgjsG1075" transform="translate(425.9551995133554,367.40391817572396)"><path id="SvgjsPath1076" d="M 0 24.58659217877095C 0 -8.195530726256983 49.1731843575419 -8.195530726256983 49.1731843575419 24.58659217877095C 49.1731843575419 57.36871508379888 0 57.36871508379888 0 24.58659217877095Z" stroke="rgba(50,50,50,1)" stroke-width="2" fill-opacity="1" fill="#cc99ff"></path><g id="SvgjsG1077"><text id="SvgjsText1078" font-family="微软雅黑" text-anchor="middle" font-size="13px" width="30px" fill="#323232" font-weight="400" align="middle" anchor="middle" family="微软雅黑" size="13px" weight="400" font-style="" opacity="1" y="15.13659217877095" transform="rotate(0)"></text></g></g><g id="SvgjsG1079" transform="translate(489.24290901056213,373.7782198517016)"><path id="SvgjsPath1080" d="M 0 0L 236.75977653631284 0L 236.75977653631284 36.42458100558659L 0 36.42458100558659Z" stroke="none" fill="none"></path><g id="SvgjsG1081"><text id="SvgjsText1082" font-family="微软雅黑" text-anchor="start" font-size="13px" width="237px" fill="#323232" font-weight="400" align="middle" anchor="start" family="微软雅黑" size="13px" weight="400" font-style="" opacity="1" y="0.7622905027932945" transform="rotate(0)"><tspan id="SvgjsTspan1083" dy="16" x="0"><tspan id="SvgjsTspan1084" style="text-decoration:;">add all input as the output of this </tspan></tspan><tspan id="SvgjsTspan1085" dy="16" x="0"><tspan id="SvgjsTspan1086" style="text-decoration:;">layer</tspan></tspan></text></g></g><g id="SvgjsG1087"><path id="SvgjsPath1088" d="M426.41050677592534 208.04637627628261L471.9412330329085 208.04637627628261" stroke-dasharray="10,5,3,5" stroke="#323232" stroke-width="2" fill="none" marker-end="url(#SvgjsMarker1089)"></path></g><g id="SvgjsG1091" transform="translate(489.24290901056213,184.37039862265135)"><path id="SvgjsPath1092" d="M 0 0L 205.79888268156424 0L 205.79888268156424 43.70949720670391L 0 43.70949720670391Z" stroke="none" fill="none"></path><g id="SvgjsG1093"><text id="SvgjsText1094" font-family="微软雅黑" text-anchor="start" font-size="13px" width="206px" fill="#323232" font-weight="400" align="middle" anchor="start" family="微软雅黑" size="13px" weight="400" font-style="" opacity="1" y="4.404748603351954" transform="rotate(0)"><tspan id="SvgjsTspan1095" dy="16" x="0"><tspan id="SvgjsTspan1096" style="text-decoration:;">(suppose this layer is the Nth </tspan></tspan><tspan id="SvgjsTspan1097" dy="16" x="0"><tspan id="SvgjsTspan1098" style="text-decoration:;">layer) outputs of all N-2 layers</tspan></tspan></text></g></g><g id="SvgjsG1099" transform="translate(100.41050677592528,33.208387449467)"><path id="SvgjsPath1100" d="M 0 0L 109.27374301675977 0L 109.27374301675977 36.42458100558659L 0 36.42458100558659Z" stroke="none" fill="none"></path><g id="SvgjsG1101"><text id="SvgjsText1102" font-family="微软雅黑" text-anchor="middle" font-size="13px" width="110px" fill="#323232" font-weight="400" align="middle" anchor="middle" family="微软雅黑" size="13px" weight="400" font-style="" opacity="1" y="8.762290502793295" transform="rotate(0)"><tspan id="SvgjsTspan1103" dy="16" x="55"><tspan id="SvgjsTspan1104" style="text-decoration:;">x (N-2)</tspan></tspan></text></g></g><g id="SvgjsG1105" transform="translate(150.49430565860683,439.196758792387)"><path id="SvgjsPath1106" d="M 0 0L 109.27374301675977 0L 109.27374301675977 36.42458100558659L 0 36.42458100558659Z" stroke="none" fill="none"></path><g id="SvgjsG1107"><text id="SvgjsText1108" font-family="微软雅黑" text-anchor="middle" font-size="13px" width="110px" fill="#323232" font-weight="400" align="middle" anchor="middle" family="微软雅黑" size="13px" weight="400" font-style="" opacity="1" y="8.762290502793295" transform="rotate(0)"><tspan id="SvgjsTspan1109" dy="16" x="55"><tspan id="SvgjsTspan1110" style="text-decoration:;">ENASMacroLayer</tspan></tspan></text></g></g><g id="SvgjsG1111" transform="translate(25.002685546875,608.5613342285156)"><path id="SvgjsPath1112" d="M 0 0L 67 0L 67 46.9L 0 46.9Z" stroke="rgba(50,50,50,1)" stroke-width="2" fill-opacity="1" fill="#00cc66"></path><g id="SvgjsG1113"><text id="SvgjsText1114" font-family="微软雅黑" text-anchor="middle" font-size="13px" width="47px" fill="#323232" font-weight="400" align="middle" anchor="middle" family="微软雅黑" size="13px" weight="400" font-style="" opacity="1" y="14" transform="rotate(0)"><tspan id="SvgjsTspan1115" dy="16" x="33.5"><tspan id="SvgjsTspan1116" style="text-decoration:;">input</tspan></tspan></text></g></g><g id="SvgjsG1117" transform="translate(145.49430565860683,601.7863342285157)"><path id="SvgjsPath1118" d="M 0 29.224999999999966C 0 -9.741666666666655 59.84357541899442 -9.741666666666655 59.84357541899442 29.224999999999966C 59.84357541899442 68.19166666666659 0 68.19166666666659 0 29.224999999999966Z" stroke="rgba(50,50,50,1)" stroke-width="2" fill-opacity="1" fill="#00ffff"></path><g id="SvgjsG1119"><text id="SvgjsText1120" font-family="微软雅黑" text-anchor="middle" font-size="13px" width="40px" fill="#323232" font-weight="400" align="middle" anchor="middle" family="微软雅黑" size="13px" weight="400" font-style="" opacity="1" y="19.774999999999967" transform="rotate(0)"></text></g></g><g id="SvgjsG1121" transform="translate(264.49430565860683,602.7863342285157)"><path id="SvgjsPath1122" d="M 0 29.224999999999966C 0 -9.741666666666655 59.84357541899442 -9.741666666666655 59.84357541899442 29.224999999999966C 59.84357541899442 68.19166666666659 0 68.19166666666659 0 29.224999999999966Z" stroke="rgba(50,50,50,1)" stroke-width="2" fill-opacity="1" fill="#00ffff"></path><g id="SvgjsG1123"><text id="SvgjsText1124" font-family="微软雅黑" text-anchor="middle" font-size="13px" width="40px" fill="#323232" font-weight="400" align="middle" anchor="middle" family="微软雅黑" size="13px" weight="400" font-style="" opacity="1" y="19.774999999999967" transform="rotate(0)"></text></g></g><g id="SvgjsG1125" transform="translate(384.87978051894197,602.7863342285157)"><path id="SvgjsPath1126" d="M 0 29.224999999999966C 0 -9.741666666666655 59.84357541899442 -9.741666666666655 59.84357541899442 29.224999999999966C 59.84357541899442 68.19166666666659 0 68.19166666666659 0 29.224999999999966Z" stroke="rgba(50,50,50,1)" stroke-width="2" fill-opacity="1" fill="#00ffff"></path><g id="SvgjsG1127"><text id="SvgjsText1128" font-family="微软雅黑" text-anchor="middle" font-size="13px" width="40px" fill="#323232" font-weight="400" align="middle" anchor="middle" family="微软雅黑" size="13px" weight="400" font-style="" opacity="1" y="19.774999999999967" transform="rotate(0)"></text></g></g><g id="SvgjsG1129" transform="translate(504.87978051894197,603.7863342285157)"><path id="SvgjsPath1130" d="M 0 29.224999999999966C 0 -9.741666666666655 59.84357541899442 -9.741666666666655 59.84357541899442 29.224999999999966C 59.84357541899442 68.19166666666659 0 68.19166666666659 0 29.224999999999966Z" stroke="rgba(50,50,50,1)" stroke-width="2" fill-opacity="1" fill="#00ffff"></path><g id="SvgjsG1131"><text id="SvgjsText1132" font-family="微软雅黑" text-anchor="middle" font-size="13px" width="40px" fill="#323232" font-weight="400" align="middle" anchor="middle" family="微软雅黑" size="13px" weight="400" font-style="" opacity="1" y="19.774999999999967" transform="rotate(0)"></text></g></g><g id="SvgjsG1133" transform="translate(621.002685546875,610.5613342285156)"><path id="SvgjsPath1134" d="M 0 0L 83 0L 83 44.899999999999864L 0 44.899999999999864Z" stroke="rgba(50,50,50,1)" stroke-width="2" fill-opacity="1" fill="#cc99ff"></path><g id="SvgjsG1135"><text id="SvgjsText1136" font-family="微软雅黑" text-anchor="middle" font-size="13px" width="63px" fill="#323232" font-weight="400" align="middle" anchor="middle" family="微软雅黑" size="13px" weight="400" font-style="" opacity="1" y="12.999999999999932" transform="rotate(0)"><tspan id="SvgjsTspan1137" dy="16" x="41.5"><tspan id="SvgjsTspan1138" style="text-decoration:;">softmax</tspan></tspan></text></g></g><g id="SvgjsG1139"><path id="SvgjsPath1140" d="M92.002685546875 632.0113342285157L145.49430565860683 631.0113342285156" stroke="#323232" stroke-width="2" fill="none" marker-end="url(#SvgjsMarker1141)"></path></g><g id="SvgjsG1143"><path id="SvgjsPath1144" d="M205.33788107760125 631.0113342285156L264.49430565860683 632.0113342285156" stroke-dasharray="8,5" stroke="#323232" stroke-width="2" fill="none" marker-end="url(#SvgjsMarker1145)"></path></g><g id="SvgjsG1147"><path id="SvgjsPath1148" d="M324.3378810776012 632.0113342285156L384.87978051894197 632.0113342285156" stroke-dasharray="8,5" stroke="#323232" stroke-width="2" fill="none" marker-end="url(#SvgjsMarker1149)"></path></g><g id="SvgjsG1151"><path id="SvgjsPath1152" d="M444.7233559379365 632.0113342285156L504.87978051894197 633.0113342285156" stroke-dasharray="8,5" stroke="#323232" stroke-width="2" fill="none" marker-end="url(#SvgjsMarker1153)"></path></g><g id="SvgjsG1155"><path id="SvgjsPath1156" d="M564.7233559379364 633.0113342285156L621.002685546875 633.0113342285156" stroke="#323232" stroke-width="2" fill="none" marker-end="url(#SvgjsMarker1157)"></path></g><g id="SvgjsG1159"><path id="SvgjsPath1160" d="M205.33788107760125 631.0113342285156C 232.002685546875 616.6213397979736 235.002685546875 606.6213397979736 264.49430565860683 632.0113342285156" stroke-dasharray="10,5,3,5" stroke="#323232" stroke-width="2" fill="none" marker-end="url(#SvgjsMarker1161)"></path></g><g id="SvgjsG1163"><path id="SvgjsPath1164" d="M205.33788107760125 631.0113342285156C 272.002685546875 564.6213397979736 290.002685546875 572.6213397979736 384.87978051894197 632.0113342285156" stroke-dasharray="10,5,3,5" stroke="#323232" stroke-width="2" fill="none" marker-end="url(#SvgjsMarker1165)"></path></g><g id="SvgjsG1167"><path id="SvgjsPath1168" d="M205.33788107760125 631.0113342285156C 285.002685546875 531.6213397979736 416.002685546875 545.6213397979736 504.87978051894197 633.0113342285156" stroke-dasharray="10,5,3,5" stroke="#323232" stroke-width="2" fill="none" marker-end="url(#SvgjsMarker1169)"></path></g><g id="SvgjsG1171"><path id="SvgjsPath1172" d="M324.3378810776012 632.0113342285156C 353.002685546875 653.6213397979736 361.002685546875 652.6213397979736 384.87978051894197 632.0113342285156" stroke-dasharray="10,5,3,5" stroke="#323232" stroke-width="2" fill="none" marker-end="url(#SvgjsMarker1173)"></path></g><g id="SvgjsG1175"><path id="SvgjsPath1176" d="M324.3378810776012 632.0113342285156C 393.002685546875 696.6213397979736 435.002685546875 690.6213397979736 504.87978051894197 633.0113342285156" stroke-dasharray="10,5,3,5" stroke="#323232" stroke-width="2" fill="none" marker-end="url(#SvgjsMarker1177)"></path></g><g id="SvgjsG1179"><path id="SvgjsPath1180" d="M444.7233559379365 632.0113342285156C 465.002685546875 613.6213397979736 475.002685546875 615.6213397979736 504.87978051894197 633.0113342285156" stroke-dasharray="10,5,3,5" stroke="#323232" stroke-width="2" fill="none" marker-end="url(#SvgjsMarker1181)"></path></g><g id="SvgjsG1183" transform="translate(254.42726655246156,734.6213397979736)"><path id="SvgjsPath1184" d="M 0 0L 177.28770949720672 0L 177.28770949720672 43L 0 43Z" stroke="none" fill="none"></path><g id="SvgjsG1185"><text id="SvgjsText1186" font-family="微软雅黑" text-anchor="middle" font-size="13px" width="178px" fill="#323232" font-weight="400" align="middle" anchor="middle" family="微软雅黑" size="13px" weight="400" font-style="" opacity="1" y="12.05" transform="rotate(0)"><tspan id="SvgjsTspan1187" dy="16" x="89"><tspan id="SvgjsTspan1188" style="text-decoration:;">GeneralModel</tspan></tspan></text></g></g><g id="SvgjsG1189"><path id="SvgjsPath1190" d="M513.16190342397 711.1301751589642L558.6926296809531 711.1301751589642" stroke-dasharray="8,5" stroke="#323232" stroke-width="2" fill="none" marker-end="url(#SvgjsMarker1191)"></path></g><g id="SvgjsG1193" transform="translate(575.0836911334671,692.0072701310312)"><path id="SvgjsPath1194" d="M 0 0L 236.75977653631284 0L 236.75977653631284 46.44134078212291L 0 46.44134078212291Z" stroke="none" fill="none"></path><g id="SvgjsG1195"><text id="SvgjsText1196" font-family="微软雅黑" text-anchor="start" font-size="13px" width="237px" fill="#323232" font-weight="400" align="middle" anchor="start" family="微软雅黑" size="13px" weight="400" font-style="" opacity="1" y="-1.7293296089385464" transform="rotate(0)"><tspan id="SvgjsTspan1197" dy="16" x="0"><tspan id="SvgjsTspan1198" style="text-decoration:;">choose an operation from: MaxPool, </tspan></tspan><tspan id="SvgjsTspan1199" dy="16" x="0"><tspan id="SvgjsTspan1200" style="text-decoration:;">AvgPool, SepConvBN3x3, SepConvBN</tspan></tspan><tspan id="SvgjsTspan1201" dy="16" x="0"><tspan id="SvgjsTspan1202" style="text-decoration:;">5x5 and SkipConnect</tspan></tspan></text></g></g><g id="SvgjsG1203"><path id="SvgjsPath1204" d="M512.002685546875 764.0113342285156L558.002685546875 764.0113342285156" stroke="#323232" stroke-width="2" fill="none" marker-end="url(#SvgjsMarker1205)"></path></g><g id="SvgjsG1207" transform="translate(575.0836911334671,746.4613342285156)"><path id="SvgjsPath1208" d="M 0 0L 206.91899441340786 0L 206.91899441340786 30L 0 30Z" stroke="none" fill="none"></path><g id="SvgjsG1209"><text id="SvgjsText1210" font-family="微软雅黑" text-anchor="start" font-size="13px" width="207px" fill="#323232" font-weight="400" align="middle" anchor="start" family="微软雅黑" size="13px" weight="400" font-style="" opacity="1" y="5.55" transform="rotate(0)"><tspan id="SvgjsTspan1211" dy="16" x="0"><tspan id="SvgjsTspan1212" style="text-decoration:;">pass tensor without operation</tspan></tspan></text></g></g><g id="SvgjsG1213" transform="translate(515.2401157144727,840.7024446654088)"><path id="SvgjsPath1214" d="M 0 25.766944781553388C 0 -8.588981593851129 52.76256983240228 -8.588981593851129 52.76256983240228 25.766944781553388C 52.76256983240228 60.122871156957906 0 60.122871156957906 0 25.766944781553388Z" stroke="rgba(50,50,50,1)" stroke-width="2" fill-opacity="1" fill="#00ffff"></path><g id="SvgjsG1215"><text id="SvgjsText1216" font-family="微软雅黑" text-anchor="middle" font-size="13px" width="33px" fill="#323232" font-weight="400" align="middle" anchor="middle" family="微软雅黑" size="13px" weight="400" font-style="" opacity="1" y="16.31694478155339" transform="rotate(0)"></text></g></g><g id="SvgjsG1217" transform="translate(586.4943056586069,848.257098944169)"><path id="SvgjsPath1218" d="M 0 0L 109.27374301675977 0L 109.27374301675977 36.42458100558659L 0 36.42458100558659Z" stroke="none" fill="none"></path><g id="SvgjsG1219"><text id="SvgjsText1220" font-family="微软雅黑" text-anchor="middle" font-size="13px" width="110px" fill="#323232" font-weight="400" align="middle" anchor="middle" family="微软雅黑" size="13px" weight="400" font-style="" opacity="1" y="8.762290502793295" transform="rotate(0)"><tspan id="SvgjsTspan1221" dy="16" x="55"><tspan id="SvgjsTspan1222" style="text-decoration:;">ENASMacroLayer</tspan></tspan></text></g></g><g id="SvgjsG1223"><path id="SvgjsPath1224" d="M515.2401157144727 810.0463762762827L560.7708419714559 810.0463762762827" stroke-dasharray="10,5,3,5" stroke="#323232" stroke-width="2" fill="none" marker-end="url(#SvgjsMarker1225)"></path></g><g id="SvgjsG1227" transform="translate(578.0725179491095,786.3703986226515)"><path id="SvgjsPath1228" d="M 0 0L 205.79888268156424 0L 205.79888268156424 43.70949720670391L 0 43.70949720670391Z" stroke="none" fill="none"></path><g id="SvgjsG1229"><text id="SvgjsText1230" font-family="微软雅黑" text-anchor="start" font-size="13px" width="206px" fill="#323232" font-weight="400" align="middle" anchor="start" family="微软雅黑" size="13px" weight="400" font-style="" opacity="1" y="4.404748603351954" transform="rotate(0)"><tspan id="SvgjsTspan1231" dy="16" x="0"><tspan id="SvgjsTspan1232" style="text-decoration:;">(suppose this layer is the Nth </tspan></tspan><tspan id="SvgjsTspan1233" dy="16" x="0"><tspan id="SvgjsTspan1234" style="text-decoration:;">layer) outputs of all N-1 layers</tspan></tspan></text></g></g><g id="SvgjsG1235"><path id="SvgjsPath1236" d="M163.24290901056213 250.84525895784685C 166.7038178979269 290.40362855551774 245.01666705993807 272.29806254453354 241.5557581725733 311.8564321422044" stroke="#323232" stroke-width="2" fill="none" marker-end="url(#SvgjsMarker1237)"></path></g><g id="SvgjsG1239"><path id="SvgjsPath1240" d="M424.41050677592534 163.04637627628261L469.9412330329085 163.04637627628261" stroke="#323232" stroke-width="2" fill="none" marker-end="url(#SvgjsMarker1241)"></path></g><g id="SvgjsG1243" transform="translate(489.24290901056213,136.37039862265135)"><path id="SvgjsPath1244" d="M 0 0L 205.79888268156424 0L 205.79888268156424 43.70949720670391L 0 43.70949720670391Z" stroke="none" fill="none"></path><g id="SvgjsG1245"><text id="SvgjsText1246" font-family="微软雅黑" text-anchor="start" font-size="13px" width="206px" fill="#323232" font-weight="400" align="middle" anchor="start" family="微软雅黑" size="13px" weight="400" font-style="" opacity="1" y="4.404748603351954" transform="rotate(0)"><tspan id="SvgjsTspan1247" dy="16" x="0"><tspan id="SvgjsTspan1248" style="text-decoration:;">pass the tensor input directly to </tspan></tspan><tspan id="SvgjsTspan1249" dy="16" x="0"><tspan id="SvgjsTspan1250" style="text-decoration:;">the next node</tspan></tspan></text></g></g></svg>
\ No newline at end of file

docs/img/NAS_ENAS_micro.svg

+<svg id="SvgjsSvg1006" width="704.9999694824219" height="824.0800323486328" xmlns="http://www.w3.org/2000/svg" version="1.1" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:svgjs="http://svgjs.com/svgjs"><defs id="SvgjsDefs1007"><marker id="SvgjsMarker1046" markerWidth="16" markerHeight="12" refX="16" refY="6" viewBox="0 0 16 12" orient="auto" markerUnits="userSpaceOnUse" stroke-dasharray="0,0"><path id="SvgjsPath1047" d="M0,2 L14,6 L0,11 L0,2" fill="#323232" stroke="#323232" stroke-width="2"></path></marker><marker id="SvgjsMarker1050" markerWidth="16" markerHeight="12" refX="16" refY="6" viewBox="0 0 16 12" orient="auto" markerUnits="userSpaceOnUse" stroke-dasharray="0,0"><path id="SvgjsPath1051" d="M0,2 L14,6 L0,11 L0,2" fill="#323232" stroke="#323232" stroke-width="2"></path></marker><marker id="SvgjsMarker1054" markerWidth="16" markerHeight="12" refX="16" refY="6" viewBox="0 0 16 12" orient="auto" markerUnits="userSpaceOnUse" stroke-dasharray="0,0"><path id="SvgjsPath1055" d="M0,2 L14,6 L0,11 L0,2" fill="#323232" stroke="#323232" stroke-width="2"></path></marker><marker id="SvgjsMarker1058" markerWidth="16" markerHeight="12" refX="16" refY="6" viewBox="0 0 16 12" orient="auto" markerUnits="userSpaceOnUse" stroke-dasharray="0,0"><path id="SvgjsPath1059" d="M0,2 L14,6 L0,11 L0,2" fill="#323232" stroke="#323232" stroke-width="2"></path></marker><marker id="SvgjsMarker1062" markerWidth="16" markerHeight="12" refX="16" refY="6" viewBox="0 0 16 12" orient="auto" markerUnits="userSpaceOnUse" stroke-dasharray="0,0"><path id="SvgjsPath1063" d="M0,2 L14,6 L0,11 L0,2" fill="#323232" stroke="#323232" stroke-width="2"></path></marker><marker id="SvgjsMarker1066" markerWidth="16" markerHeight="12" refX="16" refY="6" viewBox="0 0 16 12" orient="auto" markerUnits="userSpaceOnUse" stroke-dasharray="0,0"><path id="SvgjsPath1067" d="M0,2 L14,6 L0,11 L0,2" fill="#323232" stroke="#323232" stroke-width="2"></path></marker><marker id="SvgjsMarker1070" markerWidth="16" markerHeight="12" refX="16" refY="6" viewBox="0 0 16 12" orient="auto" markerUnits="userSpaceOnUse" stroke-dasharray="0,0"><path id="SvgjsPath1071" d="M0,2 L14,6 L0,11 L0,2" fill="#323232" stroke="#323232" stroke-width="2"></path></marker><marker id="SvgjsMarker1074" markerWidth="16" markerHeight="12" refX="16" refY="6" viewBox="0 0 16 12" orient="auto" markerUnits="userSpaceOnUse" stroke-dasharray="0,0"><path id="SvgjsPath1075" d="M0,2 L14,6 L0,11 L0,2" fill="#323232" stroke="#323232" stroke-width="2"></path></marker><marker id="SvgjsMarker1078" markerWidth="16" markerHeight="12" refX="16" refY="6" viewBox="0 0 16 12" orient="auto" markerUnits="userSpaceOnUse" stroke-dasharray="0,0"><path id="SvgjsPath1079" d="M0,2 L14,6 L0,11 L0,2" fill="#323232" stroke="#323232" stroke-width="2"></path></marker><marker id="SvgjsMarker1082" markerWidth="16" markerHeight="12" refX="16" refY="6" viewBox="0 0 16 12" orient="auto" markerUnits="userSpaceOnUse" stroke-dasharray="0,0"><path id="SvgjsPath1083" d="M0,2 L14,6 L0,11 L0,2" fill="#323232" stroke="#323232" stroke-width="2"></path></marker><marker id="SvgjsMarker1086" markerWidth="16" markerHeight="12" refX="16" refY="6" viewBox="0 0 16 12" orient="auto" markerUnits="userSpaceOnUse" stroke-dasharray="0,0"><path id="SvgjsPath1087" d="M0,2 L14,6 L0,11 L0,2" fill="#323232" stroke="#323232" stroke-width="2"></path></marker><marker id="SvgjsMarker1090" markerWidth="16" markerHeight="12" refX="16" refY="6" viewBox="0 0 16 12" orient="auto" markerUnits="userSpaceOnUse" stroke-dasharray="0,0"><path id="SvgjsPath1091" d="M0,2 L14,6 L0,11 L0,2" fill="#323232" stroke="#323232" stroke-width="2"></path></marker><marker id="SvgjsMarker1094" markerWidth="16" markerHeight="12" refX="16" refY="6" viewBox="0 0 16 12" orient="auto" markerUnits="userSpaceOnUse" stroke-dasharray="0,0"><path id="SvgjsPath1095" d="M0,2 L14,6 L0,11 L0,2" fill="#323232" stroke="#323232" stroke-width="2"></path></marker><marker id="SvgjsMarker1098" markerWidth="16" markerHeight="12" refX="16" refY="6" viewBox="0 0 16 12" orient="auto" markerUnits="userSpaceOnUse" stroke-dasharray="0,0"><path id="SvgjsPath1099" d="M0,2 L14,6 L0,11 L0,2" fill="#323232" stroke="#323232" stroke-width="2"></path></marker><marker id="SvgjsMarker1134" markerWidth="16" markerHeight="12" refX="16" refY="6" viewBox="0 0 16 12" orient="auto" markerUnits="userSpaceOnUse" stroke-dasharray="0,0"><path id="SvgjsPath1135" d="M0,2 L14,6 L0,11 L0,2" fill="#323232" stroke="#323232" stroke-width="2"></path></marker><marker id="SvgjsMarker1144" markerWidth="16" markerHeight="12" refX="16" refY="6" viewBox="0 0 16 12" orient="auto" markerUnits="userSpaceOnUse" stroke-dasharray="0,0"><path id="SvgjsPath1145" d="M0,2 L14,6 L0,11 L0,2" fill="#323232" stroke="#323232" stroke-width="2"></path></marker><marker id="SvgjsMarker1148" markerWidth="16" markerHeight="12" refX="16" refY="6" viewBox="0 0 16 12" orient="auto" markerUnits="userSpaceOnUse" stroke-dasharray="0,0"><path id="SvgjsPath1149" d="M0,2 L14,6 L0,11 L0,2" fill="#323232" stroke="#323232" stroke-width="2"></path></marker><marker id="SvgjsMarker1152" markerWidth="16" markerHeight="12" refX="16" refY="6" viewBox="0 0 16 12" orient="auto" markerUnits="userSpaceOnUse" stroke-dasharray="0,0"><path id="SvgjsPath1153" d="M0,2 L14,6 L0,11 L0,2" fill="#323232" stroke="#323232" stroke-width="2"></path></marker><marker id="SvgjsMarker1156" markerWidth="16" markerHeight="12" refX="16" refY="6" viewBox="0 0 16 12" orient="auto" markerUnits="userSpaceOnUse" stroke-dasharray="0,0"><path id="SvgjsPath1157" d="M0,2 L14,6 L0,11 L0,2" fill="#323232" stroke="#323232" stroke-width="2"></path></marker><marker id="SvgjsMarker1160" markerWidth="16" markerHeight="12" refX="16" refY="6" viewBox="0 0 16 12" orient="auto" markerUnits="userSpaceOnUse" stroke-dasharray="0,0"><path id="SvgjsPath1161" d="M0,2 L14,6 L0,11 L0,2" fill="#323232" stroke="#323232" stroke-width="2"></path></marker></defs><g id="SvgjsG1008" transform="translate(67.99998474121094,25.008005142211914)"><path id="SvgjsPath1009" d="M 0 35C 0 -11.666666666666666 70 -11.666666666666666 70 35C 70 81.66666666666667 0 81.66666666666667 0 35Z" stroke="rgba(50,50,50,1)" stroke-width="2" fill-opacity="1" fill="#00cc66"></path><g id="SvgjsG1010"><text id="SvgjsText1011" font-family="微软雅黑" text-anchor="middle" font-size="13px" width="50px" fill="#323232" font-weight="400" align="middle" anchor="middle" family="微软雅黑" size="13px" weight="400" font-style="" opacity="1" y="25.55" transform="rotate(0)"><tspan id="SvgjsTspan1012" dy="16" x="35"><tspan id="SvgjsTspan1013" style="text-decoration:;">input_0</tspan></tspan></text></g></g><g id="SvgjsG1014" transform="translate(238.99998474121094,111.00800514221191)"><path id="SvgjsPath1015" d="M 0 35C 0 -11.666666666666666 70 -11.666666666666666 70 35C 70 81.66666666666667 0 81.66666666666667 0 35Z" stroke="rgba(50,50,50,1)" stroke-width="2" fill-opacity="1" fill="#00cc66"></path><g id="SvgjsG1016"><text id="SvgjsText1017" font-family="微软雅黑" text-anchor="middle" font-size="13px" width="50px" fill="#323232" font-weight="400" align="middle" anchor="middle" family="微软雅黑" size="13px" weight="400" font-style="" opacity="1" y="25.55" transform="rotate(0)"><tspan id="SvgjsTspan1018" dy="16" x="35"><tspan id="SvgjsTspan1019" style="text-decoration:;">input_1</tspan></tspan></text></g></g><g id="SvgjsG1020" transform="translate(238.99998474121094,279.0080051422119)"><path id="SvgjsPath1021" d="M 0 35C 0 -11.666666666666666 70 -11.666666666666666 70 35C 70 81.66666666666667 0 81.66666666666667 0 35Z" stroke="rgba(50,50,50,1)" stroke-width="2" fill-opacity="1" fill="#ffff00"></path><g id="SvgjsG1022"><text id="SvgjsText1023" font-family="微软雅黑" text-anchor="middle" font-size="13px" width="50px" fill="#323232" font-weight="400" align="middle" anchor="middle" family="微软雅黑" size="13px" weight="400" font-style="" opacity="1" y="25.55" transform="rotate(0)"><tspan id="SvgjsTspan1024" dy="16" x="35"><tspan id="SvgjsTspan1025" style="text-decoration:;">node 0</tspan></tspan></text></g></g><g id="SvgjsG1026" transform="translate(238.99998474121094,492.0080051422119)"><path id="SvgjsPath1027" d="M 0 35C 0 -11.666666666666666 70 -11.666666666666666 70 35C 70 81.66666666666667 0 81.66666666666667 0 35Z" stroke="rgba(50,50,50,1)" stroke-width="2" fill-opacity="1" fill="#ffff00"></path><g id="SvgjsG1028"><text id="SvgjsText1029" font-family="微软雅黑" text-anchor="middle" font-size="13px" width="50px" fill="#323232" font-weight="400" align="middle" anchor="middle" family="微软雅黑" size="13px" weight="400" font-style="" opacity="1" y="25.55" transform="rotate(0)"><tspan id="SvgjsTspan1030" dy="16" x="35"><tspan id="SvgjsTspan1031" style="text-decoration:;">node 2</tspan></tspan></text></g></g><g id="SvgjsG1032" transform="translate(67.99998474121094,372.0080051422119)"><path id="SvgjsPath1033" d="M 0 35C 0 -11.666666666666666 70 -11.666666666666666 70 35C 70 81.66666666666667 0 81.66666666666667 0 35Z" stroke="rgba(50,50,50,1)" stroke-width="2" fill-opacity="1" fill="#ffff00"></path><g id="SvgjsG1034"><text id="SvgjsText1035" font-family="微软雅黑" text-anchor="middle" font-size="13px" width="50px" fill="#323232" font-weight="400" align="middle" anchor="middle" family="微软雅黑" size="13px" weight="400" font-style="" opacity="1" y="25.55" transform="rotate(0)"><tspan id="SvgjsTspan1036" dy="16" x="35"><tspan id="SvgjsTspan1037" style="text-decoration:;">node 1</tspan></tspan></text></g></g><g id="SvgjsG1038" transform="translate(67.99998474121094,594.0080051422119)"><path id="SvgjsPath1039" d="M 0 35C 0 -11.666666666666666 70 -11.666666666666666 70 35C 70 81.66666666666667 0 81.66666666666667 0 35Z" stroke="rgba(50,50,50,1)" stroke-width="2" fill-opacity="1" fill="#ffff00"></path><g id="SvgjsG1040"><text id="SvgjsText1041" font-family="微软雅黑" text-anchor="middle" font-size="13px" width="50px" fill="#323232" font-weight="400" align="middle" anchor="middle" family="微软雅黑" size="13px" weight="400" font-style="" opacity="1" y="25.55" transform="rotate(0)"><tspan id="SvgjsTspan1042" dy="16" x="35"><tspan id="SvgjsTspan1043" style="text-decoration:;">node 3</tspan></tspan></text></g></g><g id="SvgjsG1044"><path id="SvgjsPath1045" d="M102.99998474121094 95.00800514221191C 111.75708566874948 195.1021267645375 282.7570856687495 178.91388351988633 273.99998474121094 279.0080051422119" stroke-dasharray="8,5" stroke="#323232" stroke-width="2" fill="none" marker-end="url(#SvgjsMarker1046)"></path></g><g id="SvgjsG1048"><path id="SvgjsPath1049" d="M273.99998474121094 181.0080051422119C 277.41648985691916 220.05883730740834 277.41648985691916 239.9571729770155 273.99998474121094 279.0080051422119" stroke-dasharray="8,5" stroke="#323232" stroke-width="2" fill="none" marker-end="url(#SvgjsMarker1050)"></path></g><g id="SvgjsG1052"><path id="SvgjsPath1053" d="M273.99998474121094 349.0080051422119C 278.98529322637694 405.9903418730597 278.98529322637705 435.0256684113641 273.99998474121094 492.0080051422119" stroke-dasharray="8,5" stroke="#323232" stroke-width="2" fill="none" marker-end="url(#SvgjsMarker1054)"></path></g><g id="SvgjsG1056"><path id="SvgjsPath1057" d="M102.99998474121094 95.00800514221191C 112.65684103765147 205.38637769077735 112.65684103765147 261.6296325936465 102.99998474121094 372.0080051422119" stroke-dasharray="8,5" stroke="#323232" stroke-width="2" fill="none" marker-end="url(#SvgjsMarker1058)"></path></g><g id="SvgjsG1060"><path id="SvgjsPath1061" d="M273.99998474121094 349.0080051422119C 280.01512026970283 417.7613188410721 109.01512026970283 303.2546914433517 102.99998474121094 372.0080051422119" stroke-dasharray="8,5" stroke="#323232" stroke-width="2" fill="none" marker-end="url(#SvgjsMarker1062)"></path></g><g id="SvgjsG1064"><path id="SvgjsPath1065" d="M102.99998474121094 442.0080051422119C 108.29905390026858 502.57664278619006 108.29905390026858 533.4393674982338 102.99998474121094 594.0080051422119" stroke-dasharray="8,5" stroke="#323232" stroke-width="2" fill="none" marker-end="url(#SvgjsMarker1066)"></path></g><g id="SvgjsG1068"><path id="SvgjsPath1069" d="M273.99998474121094 349.0080051422119C 284.41593834944956 468.0628996675149 113.41593834944956 474.95311061690893 102.99998474121094 594.0080051422119" stroke-dasharray="8,5" stroke="#323232" stroke-width="2" fill="none" marker-end="url(#SvgjsMarker1070)"></path></g><g id="SvgjsG1072"><path id="SvgjsPath1073" d="M273.99998474121094 181.0080051422119C 212.99998474121094 264.0080051422119 111.93738864405282 269.8530110818261 102.99998474121094 372.0080051422119" stroke-dasharray="8,5" stroke="#323232" stroke-width="2" fill="none" marker-end="url(#SvgjsMarker1074)"></path></g><g id="SvgjsG1076"><path id="SvgjsPath1077" d="M102.99998474121094 442.0080051422119C 109.2110530012738 513.0008402107513 280.21105300127385 421.0151700736725 273.99998474121094 492.0080051422119" stroke-dasharray="8,5" stroke="#323232" stroke-width="2" fill="none" marker-end="url(#SvgjsMarker1078)"></path></g><g id="SvgjsG1080"><path id="SvgjsPath1081" d="M273.99998474121094 562.0080051422119C 280.06492236717565 631.3305594199732 109.0649223671756 524.6854508644507 102.99998474121094 594.0080051422119" stroke-dasharray="8,5" stroke="#323232" stroke-width="2" fill="none" marker-end="url(#SvgjsMarker1082)"></path></g><g id="SvgjsG1084"><path id="SvgjsPath1085" d="M102.99998474121094 95.00800514221191C 24.999984741210938 329.0080051422119 30.999984741210938 425.0080051422119 102.99998474121094 594.0080051422119" stroke-dasharray="8,5" stroke="#323232" stroke-width="2" fill="none" marker-end="url(#SvgjsMarker1086)"></path></g><g id="SvgjsG1088"><path id="SvgjsPath1089" d="M273.99998474121094 181.0080051422119C 222.99998474121094 348.0080051422119 118.5834705403015 415.88794739926834 102.99998474121094 594.0080051422119" stroke-dasharray="8,5" stroke="#323232" stroke-width="2" fill="none" marker-end="url(#SvgjsMarker1090)"></path></g><g id="SvgjsG1092"><path id="SvgjsPath1093" d="M273.99998474121094 181.0080051422119C 319.99998474121094 315.0080051422119 358.99998474121094 344.0080051422119 273.99998474121094 492.0080051422119" stroke-dasharray="8,5" stroke="#323232" stroke-width="2" fill="none" marker-end="url(#SvgjsMarker1094)"></path></g><g id="SvgjsG1096"><path id="SvgjsPath1097" d="M371.99998474121094 529.0080051422119L417.99998474121094 529.0080051422119" stroke-dasharray="8,5" stroke="#323232" stroke-width="2" fill="none" marker-end="url(#SvgjsMarker1098)"></path></g><g id="SvgjsG1100" transform="translate(435.99998474121094,504.0080051422119)"><path id="SvgjsPath1101" d="M 0 0L 238 0L 238 46L 0 46Z" stroke="none" fill="none"></path><g id="SvgjsG1102"><text id="SvgjsText1103" font-family="微软雅黑" text-anchor="start" font-size="13px" width="238px" fill="#323232" font-weight="400" align="middle" anchor="start" family="微软雅黑" size="13px" weight="400" font-style="" opacity="1" y="-1.95" transform="rotate(0)"><tspan id="SvgjsTspan1104" dy="16" x="0"><tspan id="SvgjsTspan1105" style="text-decoration:;">choose one operation from MaxPool, </tspan></tspan><tspan id="SvgjsTspan1106" dy="16" x="0"><tspan id="SvgjsTspan1107" style="text-decoration:;">AvgPool, SepConvBN3x3, SepConvBN</tspan></tspan><tspan id="SvgjsTspan1108" dy="16" x="0"><tspan id="SvgjsTspan1109" style="text-decoration:;">5x5, SkipConnect</tspan></tspan></text></g></g><g id="SvgjsG1110" transform="translate(371.99998474121094,562.0080051422119)"><path id="SvgjsPath1111" d="M 0 25.5C 0 -8.5 51 -8.5 51 25.5C 51 59.5 0 59.5 0 25.5Z" stroke="rgba(50,50,50,1)" stroke-width="2" fill-opacity="1" fill="#ffff00"></path><g id="SvgjsG1112"><text id="SvgjsText1113" font-family="微软雅黑" text-anchor="middle" font-size="13px" width="31px" fill="#323232" font-weight="400" align="middle" anchor="middle" family="微软雅黑" size="13px" weight="400" font-style="" opacity="1" y="16.05" transform="rotate(0)"></text></g></g><g id="SvgjsG1114" transform="translate(371.99998474121094,637.0080051422119)"><path id="SvgjsPath1115" d="M 0 25.5C 0 -8.5 51 -8.5 51 25.5C 51 59.5 0 59.5 0 25.5Z" stroke="rgba(50,50,50,1)" stroke-width="2" fill-opacity="1" fill="#00cc66"></path><g id="SvgjsG1116"><text id="SvgjsText1117" font-family="微软雅黑" text-anchor="middle" font-size="13px" width="31px" fill="#323232" font-weight="400" align="middle" anchor="middle" family="微软雅黑" size="13px" weight="400" font-style="" opacity="1" y="16.05" transform="rotate(0)"></text></g></g><g id="SvgjsG1118" transform="translate(435.99998474121094,566.5080051422119)"><path id="SvgjsPath1119" d="M 0 0L 244 0L 244 42L 0 42Z" stroke="none" fill="none"></path><g id="SvgjsG1120"><text id="SvgjsText1121" font-family="微软雅黑" text-anchor="start" font-size="13px" width="244px" fill="#323232" font-weight="400" align="middle" anchor="start" family="微软雅黑" size="13px" weight="400" font-style="" opacity="1" y="3.55" transform="rotate(0)"><tspan id="SvgjsTspan1122" dy="16" x="0"><tspan id="SvgjsTspan1123" style="text-decoration:;">accept two operations from all inputs, </tspan></tspan><tspan id="SvgjsTspan1124" dy="16" x="0"><tspan id="SvgjsTspan1125" style="text-decoration:;">then add the results as output</tspan></tspan></text></g></g><g id="SvgjsG1126" transform="translate(435.99998474121094,642.5080051422119)"><path id="SvgjsPath1127" d="M 0 0L 224 0L 224 40L 0 40Z" stroke="none" fill="none"></path><g id="SvgjsG1128"><text id="SvgjsText1129" font-family="微软雅黑" text-anchor="start" font-size="13px" width="224px" fill="#323232" font-weight="400" align="middle" anchor="start" family="微软雅黑" size="13px" weight="400" font-style="" opacity="1" y="10.55" transform="rotate(0)"><tspan id="SvgjsTspan1130" dy="16" x="0"><tspan id="SvgjsTspan1131" style="text-decoration:;">results of previous layers as input</tspan></tspan></text></g></g><g id="SvgjsG1132"><path id="SvgjsPath1133" d="M102.99998474121094 95.00800514221191C 118.06961340198137 267.2546489180095 289.0696134019814 319.76136136641435 273.99998474121094 492.0080051422119" stroke-dasharray="8,5" stroke="#323232" stroke-width="2" fill="none" marker-end="url(#SvgjsMarker1134)"></path></g><g id="SvgjsG1136" transform="translate(150.99998474121094,729.0880031585693)"><path id="SvgjsPath1137" d="M 0 35C 0 -11.666666666666666 70 -11.666666666666666 70 35C 70 81.66666666666667 0 81.66666666666667 0 35Z" stroke-dasharray="3,4" stroke="rgba(50,50,50,1)" stroke-width="2" fill-opacity="1" fill="#cc99ff"></path><g id="SvgjsG1138"><text id="SvgjsText1139" font-family="微软雅黑" text-anchor="middle" font-size="13px" width="50px" fill="#323232" font-weight="400" align="middle" anchor="middle" family="微软雅黑" size="13px" weight="400" font-style="" opacity="1" y="25.55" transform="rotate(0)"><tspan id="SvgjsTspan1140" dy="16" x="35"><tspan id="SvgjsTspan1141" style="text-decoration:;">output</tspan></tspan></text></g></g><g id="SvgjsG1142"><path id="SvgjsPath1143" d="M102.99998474121094 442.0080051422119C 113.41815075849257 561.0881876185914 196.41815075849257 610.0078206821898 185.99998474121094 729.0880031585693" stroke-dasharray="3,3" stroke="#323232" stroke-width="2" fill="none" marker-end="url(#SvgjsMarker1144)"></path></g><g id="SvgjsG1146"><path id="SvgjsPath1147" d="M102.99998474121094 664.0080051422119C 106.67699141184806 706.0363837051965 189.67699141184806 687.0596245955848 185.99998474121094 729.0880031585693" stroke-dasharray="3,3" stroke="#323232" stroke-width="2" fill="none" marker-end="url(#SvgjsMarker1148)"></path></g><g id="SvgjsG1150"><path id="SvgjsPath1151" d="M273.99998474121094 562.0080051422119C 280.58330524633993 637.2557028416771 192.58330524633993 653.8403054591041 185.99998474121094 729.0880031585693" stroke-dasharray="3,3" stroke="#323232" stroke-width="2" fill="none" marker-end="url(#SvgjsMarker1152)"></path></g><g id="SvgjsG1154"><path id="SvgjsPath1155" d="M273.99998474121094 349.0080051422119C 233.99998474121094 500.92799377441406 199.60096408433088 573.6280978979321 185.99998474121094 729.0880031585693" stroke-dasharray="3,3" stroke="#323232" stroke-width="2" fill="none" marker-end="url(#SvgjsMarker1156)"></path></g><g id="SvgjsG1158"><path id="SvgjsPath1159" d="M369.99998474121094 477.92799377441406L394.49998474121094 477.92799377441406L394.49998474121094 477.92799377441406L418.99998474121094 477.92799377441406" stroke-dasharray="3,3" stroke="#323232" stroke-width="2" fill="none" marker-end="url(#SvgjsMarker1160)"></path></g><g id="SvgjsG1162" transform="translate(435.99998474121094,448.0080051422119)"><path id="SvgjsPath1163" d="M 0 0L 243 0L 243 50.91998863220215L 0 50.91998863220215Z" stroke="none" fill="none"></path><g id="SvgjsG1164"><text id="SvgjsText1165" font-family="微软雅黑" text-anchor="start" font-size="13px" width="243px" fill="#323232" font-weight="400" align="middle" anchor="start" family="微软雅黑" size="13px" weight="400" font-style="" opacity="1" y="16.009994316101075" transform="rotate(0)"><tspan id="SvgjsTspan1166" dy="16" x="0"><tspan id="SvgjsTspan1167" style="text-decoration:;">pass the result directly to the output</tspan></tspan></text></g></g><g id="SvgjsG1168" transform="translate(371.99998474121094,709.0080051422119)"><path id="SvgjsPath1169" d="M 0 25.5C 0 -8.5 51 -8.5 51 25.5C 51 59.5 0 59.5 0 25.5Z" stroke-dasharray="3,4" stroke="rgba(50,50,50,1)" stroke-width="2" fill-opacity="1" fill="#cc99ff"></path><g id="SvgjsG1170"><text id="SvgjsText1171" font-family="微软雅黑" text-anchor="middle" font-size="13px" width="31px" fill="#323232" font-weight="400" align="middle" anchor="middle" family="微软雅黑" size="13px" weight="400" font-style="" opacity="1" y="16.05" transform="rotate(0)"></text></g></g><g id="SvgjsG1172" transform="translate(438.99998474121094,729.0880031585693)"><path id="SvgjsPath1173" d="M 0 0L 240 0L 240 19.920001983642578L 0 19.920001983642578Z" stroke="none" fill="none"></path><g id="SvgjsG1174"><text id="SvgjsText1175" font-family="微软雅黑" text-anchor="start" font-size="13px" width="240px" fill="#323232" font-weight="400" align="middle" anchor="start" family="微软雅黑" size="13px" weight="400" font-style="" opacity="1" y="-30.48999900817871" transform="rotate(0)"><tspan id="SvgjsTspan1176" dy="16" x="0"><tspan id="SvgjsTspan1177" style="text-decoration:;">(the node does not exist) </tspan><tspan id="SvgjsTspan1178" style="text-decoration:;font-weight: bold;">Only accept </tspan></tspan><tspan id="SvgjsTspan1179" dy="16" x="0"><tspan id="SvgjsTspan1180" style="text-decoration:;font-weight: bold;">input whose starting point is not </tspan></tspan><tspan id="SvgjsTspan1181" dy="16" x="0"><tspan id="SvgjsTspan1182" style="text-decoration:;font-weight: bold;">served as input for any other node.</tspan><tspan id="SvgjsTspan1183" style="text-decoration:;"> </tspan></tspan><tspan id="SvgjsTspan1184" dy="16" x="0"><tspan id="SvgjsTspan1185" style="text-decoration:;">It calculates inputs' average value as </tspan></tspan><tspan id="SvgjsTspan1186" dy="16" x="0"><tspan id="SvgjsTspan1187" style="text-decoration:;">the output of this cell.</tspan></tspan></text></g></g></svg>
\ No newline at end of file

examples/nas/search_space_zoo/darts_example.py

+# copyright (c) Microsoft Corporation.
+# Licensed under the MIT license.
+
+import logging
+import time
+from argparse import ArgumentParser
+
+import torch
+import torch.nn as nn
+
+import datasets
+from nni.nas.pytorch.callbacks import ArchitectureCheckpoint, LRSchedulerCallback
+from nni.nas.pytorch.darts import DartsTrainer
+from utils import accuracy
+
+from nni.nas.pytorch.search_space_zoo import DartsCell
+from darts_search_space import DartsStackedCells
+
+logger = logging.getLogger('nni')
+
+if __name__ == "__main__":
+    parser = ArgumentParser("darts")
+    parser.add_argument("--layers", default=8, type=int)
+    parser.add_argument("--batch-size", default=64, type=int)
+    parser.add_argument("--log-frequency", default=10, type=int)
+    parser.add_argument("--epochs", default=50, type=int)
+    parser.add_argument("--channels", default=16, type=int)
+    parser.add_argument("--unrolled", default=False, action="store_true")
+    parser.add_argument("--visualization", default=False, action="store_true")
+    args = parser.parse_args()
+
+    dataset_train, dataset_valid = datasets.get_dataset("cifar10")
+
+    model = DartsStackedCells(3, args.channels, 10, args.layers, DartsCell)
+    criterion = nn.CrossEntropyLoss()
+
+    optim = torch.optim.SGD(model.parameters(), 0.025, momentum=0.9, weight_decay=3.0E-4)
+    lr_scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optim, args.epochs, eta_min=0.001)
+
+    trainer = DartsTrainer(model,
+                           loss=criterion,
+                           metrics=lambda output, target: accuracy(output, target, topk=(1,)),
+                           optimizer=optim,
+                           num_epochs=args.epochs,
+                           dataset_train=dataset_train,
+                           dataset_valid=dataset_valid,
+                           batch_size=args.batch_size,
+                           log_frequency=args.log_frequency,
+                           unrolled=args.unrolled,
+                           callbacks=[LRSchedulerCallback(lr_scheduler), ArchitectureCheckpoint("./checkpoints")])
+    if args.visualization:
+        trainer.enable_visualization()
+    trainer.train()

examples/nas/search_space_zoo/darts_stack_cells.py

+# Copyright (c) Microsoft Corporation.
+# Licensed under the MIT license.
+
+import torch.nn as nn
+import ops
+
+
+class DartsStackedCells(nn.Module):
+    """
+    builtin Darts Search Space
+    Compared to Darts example, DartsSearchSpace removes Auxiliary Head, which
+    is considered as a trick rather than part of model.
+
+    Attributes
+    ---
+    in_channels: int
+        the number of input channels
+    channels: int
+        the number of initial channels expected
+    n_classes: int
+        classes for final classification
+    n_layers: int
+        the number of cells contained in this network
+    factory_func: function
+        return a callable instance for demand cell structure.
+        user should pass in ``__init__`` of the cell class with required parameters (see nni.nas.DartsCell for detail)
+    n_nodes: int
+        the number of nodes contained in each cell
+    stem_multiplier: int
+        channels multiply coefficient when passing a cell
+    """
+
+    def __init__(self, in_channels, channels, n_classes, n_layers, factory_func, n_nodes=4,
+                 stem_multiplier=3):
+        super().__init__()
+        self.in_channels = in_channels
+        self.channels = channels
+        self.n_classes = n_classes
+        self.n_layers = n_layers
+
+        c_cur = stem_multiplier * self.channels
+        self.stem = nn.Sequential(
+            nn.Conv2d(in_channels, c_cur, 3, 1, 1, bias=False),
+            nn.BatchNorm2d(c_cur)
+        )
+
+        # for the first cell, stem is used for both s0 and s1
+        # [!] channels_pp and channels_p is output channel size, but c_cur is input channel size.
+        channels_pp, channels_p, c_cur = c_cur, c_cur, channels
+
+        self.cells = nn.ModuleList()
+        reduction_p, reduction = False, False
+        for i in range(n_layers):
+            reduction_p, reduction = reduction, False
+            # Reduce featuremap size and double channels in 1/3 and 2/3 layer.
+            if i in [n_layers // 3, 2 * n_layers // 3]:
+                c_cur *= 2
+                reduction = True
+
+            cell = factory_func(n_nodes, channels_pp, channels_p, c_cur, reduction_p, reduction)
+            self.cells.append(cell)
+            c_cur_out = c_cur * n_nodes
+            channels_pp, channels_p = channels_p, c_cur_out
+
+        self.gap = nn.AdaptiveAvgPool2d(1)
+        self.linear = nn.Linear(channels_p, n_classes)
+
+    def forward(self, x):
+        s0 = s1 = self.stem(x)
+
+        for cell in self.cells:
+            s0, s1 = s1, cell(s0, s1)
+
+        out = self.gap(s1)
+        out = out.view(out.size(0), -1)  # flatten
+        logits = self.linear(out)
+
+        return logits
+
+    def drop_path_prob(self, p):
+        for module in self.modules():
+            if isinstance(module, ops.DropPath):
+                module.p = p

examples/nas/search_space_zoo/datasets.py

+# Copyright (c) Microsoft Corporation.
+# Licensed under the MIT license.
+
+import numpy as np
+import torch
+from torchvision import transforms
+from torchvision.datasets import CIFAR10
+
+
+class Cutout(object):
+    def __init__(self, length):
+        self.length = length
+
+    def __call__(self, img):
+        h, w = img.size(1), img.size(2)
+        mask = np.ones((h, w), np.float32)
+        y = np.random.randint(h)
+        x = np.random.randint(w)
+
+        y1 = np.clip(y - self.length // 2, 0, h)
+        y2 = np.clip(y + self.length // 2, 0, h)
+        x1 = np.clip(x - self.length // 2, 0, w)
+        x2 = np.clip(x + self.length // 2, 0, w)
+
+        mask[y1: y2, x1: x2] = 0.
+        mask = torch.from_numpy(mask)
+        mask = mask.expand_as(img)
+        img *= mask
+
+        return img
+
+
+def get_dataset(cls, cutout_length=0):
+    MEAN = [0.49139968, 0.48215827, 0.44653124]
+    STD = [0.24703233, 0.24348505, 0.26158768]
+    transf = [
+        transforms.RandomCrop(32, padding=4),
+        transforms.RandomHorizontalFlip()
+    ]
+    normalize = [
+        transforms.ToTensor(),
+        transforms.Normalize(MEAN, STD)
+    ]
+    cutout = []
+    if cutout_length > 0:
+        cutout.append(Cutout(cutout_length))
+
+    train_transform = transforms.Compose(transf + normalize + cutout)
+    valid_transform = transforms.Compose(normalize)
+
+    if cls == "cifar10":
+        dataset_train = CIFAR10(root="./data", train=True, download=True, transform=train_transform)
+        dataset_valid = CIFAR10(root="./data", train=False, download=True, transform=valid_transform)
+    else:
+        raise NotImplementedError
+    return dataset_train, dataset_valid

examples/nas/search_space_zoo/enas_macro_example.py

+import torch
+import logging
+import torch.nn as nn
+import torch.nn.functional as F
+
+from argparse import ArgumentParser
+from torchvision import transforms
+from torchvision.datasets import CIFAR10
+
+from nni.nas.pytorch import mutables
+from nni.nas.pytorch import enas
+from utils import accuracy, reward_accuracy
+from nni.nas.pytorch.callbacks import (ArchitectureCheckpoint,
+                                       LRSchedulerCallback)
+from nni.nas.pytorch.search_space_zoo import ENASMacroLayer
+from nni.nas.pytorch.search_space_zoo import ENASMacroGeneralModel
+
+logger = logging.getLogger('nni')
+
+
+def get_dataset(cls):
+    MEAN = [0.49139968, 0.48215827, 0.44653124]
+    STD = [0.24703233, 0.24348505, 0.26158768]
+    transf = [
+        transforms.RandomCrop(32, padding=4),
+        transforms.RandomHorizontalFlip()
+    ]
+    normalize = [
+        transforms.ToTensor(),
+        transforms.Normalize(MEAN, STD)
+    ]
+
+    train_transform = transforms.Compose(transf + normalize)
+    valid_transform = transforms.Compose(normalize)
+
+    if cls == "cifar10":
+        dataset_train = CIFAR10(root="./data", train=True, download=True, transform=train_transform)
+        dataset_valid = CIFAR10(root="./data", train=False, download=True, transform=valid_transform)
+    else:
+        raise NotImplementedError
+    return dataset_train, dataset_valid
+
+
+class FactorizedReduce(nn.Module):
+    def __init__(self, C_in, C_out, affine=False):
+        super().__init__()
+        self.conv1 = nn.Conv2d(C_in, C_out // 2, 1, stride=2, padding=0, bias=False)
+        self.conv2 = nn.Conv2d(C_in, C_out // 2, 1, stride=2, padding=0, bias=False)
+        self.bn = nn.BatchNorm2d(C_out, affine=affine)
+
+    def forward(self, x):
+        out = torch.cat([self.conv1(x), self.conv2(x[:, :, 1:, 1:])], dim=1)
+        out = self.bn(out)
+        return out
+
+
+if __name__ == "__main__":
+    parser = ArgumentParser("enas")
+    parser.add_argument("--batch-size", default=128, type=int)
+    parser.add_argument("--log-frequency", default=10, type=int)
+    # parser.add_argument("--search-for", choices=["macro", "micro"], default="macro")
+    parser.add_argument("--epochs", default=None, type=int, help="Number of epochs (default: macro 310, micro 150)")
+    parser.add_argument("--visualization", default=False, action="store_true")
+    args = parser.parse_args()
+
+    dataset_train, dataset_valid = get_dataset("cifar10")
+    model = ENASMacroGeneralModel()
+    num_epochs = args.epochs or 310
+    mutator = None
+
+    criterion = nn.CrossEntropyLoss()
+    optimizer = torch.optim.SGD(model.parameters(), 0.05, momentum=0.9, weight_decay=1.0E-4)
+    lr_scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=num_epochs, eta_min=0.001)
+
+    trainer = enas.EnasTrainer(model,
+                               loss=criterion,
+                               metrics=accuracy,
+                               reward_function=reward_accuracy,
+                               optimizer=optimizer,
+                               callbacks=[LRSchedulerCallback(lr_scheduler), ArchitectureCheckpoint("./checkpoints")],
+                               batch_size=args.batch_size,
+                               num_epochs=num_epochs,
+                               dataset_train=dataset_train,
+                               dataset_valid=dataset_valid,
+                               log_frequency=args.log_frequency,
+                               mutator=mutator)
+    if args.visualization:
+        trainer.enable_visualization()
+    trainer.train()

examples/nas/search_space_zoo/enas_micro_example.py

+import torch
+import logging
+import torch.nn as nn
+import torch.nn.functional as F
+
+from argparse import ArgumentParser
+from torchvision import transforms
+from torchvision.datasets import CIFAR10
+
+from nni.nas.pytorch import enas
+from utils import accuracy, reward_accuracy
+from nni.nas.pytorch.callbacks import (ArchitectureCheckpoint,
+                                       LRSchedulerCallback)
+
+from nni.nas.pytorch.search_space_zoo import ENASMicroLayer
+
+logger = logging.getLogger('nni')
+
+
+def get_dataset(cls):
+    MEAN = [0.49139968, 0.48215827, 0.44653124]
+    STD = [0.24703233, 0.24348505, 0.26158768]
+    transf = [
+        transforms.RandomCrop(32, padding=4),
+        transforms.RandomHorizontalFlip()
+    ]
+    normalize = [
+        transforms.ToTensor(),
+        transforms.Normalize(MEAN, STD)
+    ]
+
+    train_transform = transforms.Compose(transf + normalize)
+    valid_transform = transforms.Compose(normalize)
+
+    if cls == "cifar10":
+        dataset_train = CIFAR10(root="./data", train=True, download=True, transform=train_transform)
+        dataset_valid = CIFAR10(root="./data", train=False, download=True, transform=valid_transform)
+    else:
+        raise NotImplementedError
+    return dataset_train, dataset_valid
+
+
+class MicroNetwork(nn.Module):
+    def __init__(self, num_layers=2, num_nodes=5, out_channels=24, in_channels=3, num_classes=10,
+                 dropout_rate=0.0):
+        super().__init__()
+        self.num_layers = num_layers
+
+        self.stem = nn.Sequential(
+            nn.Conv2d(in_channels, out_channels * 3, 3, 1, 1, bias=False),
+            nn.BatchNorm2d(out_channels * 3)
+        )
+
+        pool_distance = self.num_layers // 3
+        pool_layers = [pool_distance, 2 * pool_distance + 1]
+        self.dropout = nn.Dropout(dropout_rate)
+
+        self.layers = nn.ModuleList()
+        c_pp = c_p = out_channels * 3
+        c_cur = out_channels
+        for layer_id in range(self.num_layers + 2):
+            reduction = False
+            if layer_id in pool_layers:
+                c_cur, reduction = c_p * 2, True
+            self.layers.append(ENASMicroLayer(self.layers, num_nodes, c_pp, c_p, c_cur, reduction))
+            if reduction:
+                c_pp = c_p = c_cur
+            c_pp, c_p = c_p, c_cur
+
+        self.gap = nn.AdaptiveAvgPool2d(1)
+        self.dense = nn.Linear(c_cur, num_classes)
+
+        self.reset_parameters()
+
+    def reset_parameters(self):
+        for m in self.modules():
+            if isinstance(m, nn.Conv2d):
+                nn.init.kaiming_normal_(m.weight)
+
+    def forward(self, x):
+        bs = x.size(0)
+        prev = cur = self.stem(x)
+        # aux_logits = None
+
+        for layer in self.layers:
+            prev, cur = layer(prev, cur)
+
+        cur = self.gap(F.relu(cur)).view(bs, -1)
+        cur = self.dropout(cur)
+        logits = self.dense(cur)
+
+        # if aux_logits is not None:
+        #     return logits, aux_logits
+        return logits
+
+
+if __name__ == "__main__":
+    parser = ArgumentParser("enas")
+    parser.add_argument("--batch-size", default=128, type=int)
+    parser.add_argument("--log-frequency", default=10, type=int)
+    # parser.add_argument("--search-for", choices=["macro", "micro"], default="macro")
+    parser.add_argument("--epochs", default=None, type=int, help="Number of epochs (default: macro 310, micro 150)")
+    parser.add_argument("--visualization", default=False, action="store_true")
+    args = parser.parse_args()
+
+    dataset_train, dataset_valid = get_dataset("cifar10")
+
+    model = MicroNetwork(num_layers=6, out_channels=20, num_nodes=5, dropout_rate=0.1)
+    num_epochs = args.epochs or 150
+    mutator = enas.EnasMutator(model, tanh_constant=1.1, cell_exit_extra_step=True)
+
+    criterion = nn.CrossEntropyLoss()
+    optimizer = torch.optim.SGD(model.parameters(), 0.05, momentum=0.9, weight_decay=1.0E-4)
+    lr_scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=num_epochs, eta_min=0.001)
+
+    trainer = enas.EnasTrainer(model,
+                               loss=criterion,
+                               metrics=accuracy,
+                               reward_function=reward_accuracy,
+                               optimizer=optimizer,
+                               callbacks=[LRSchedulerCallback(lr_scheduler), ArchitectureCheckpoint("./checkpoints")],
+                               batch_size=args.batch_size,
+                               num_epochs=num_epochs,
+                               dataset_train=dataset_train,
+                               dataset_valid=dataset_valid,
+                               log_frequency=args.log_frequency,
+                               mutator=mutator)
+    if args.visualization:
+        trainer.enable_visualization()
+    trainer.train()
+

examples/nas/search_space_zoo/utils.py

+# Copyright (c) Microsoft Corporation.
+# Licensed under the MIT license.
+
+import torch
+
+
+def accuracy(output, target, topk=(1,)):
+    """ Computes the precision@k for the specified values of k """
+    maxk = max(topk)
+    batch_size = target.size(0)
+
+    _, pred = output.topk(maxk, 1, True, True)
+    pred = pred.t()
+    # one-hot case
+    if target.ndimension() > 1:
+        target = target.max(1)[1]
+
+    correct = pred.eq(target.view(1, -1).expand_as(pred))
+
+    res = dict()
+    for k in topk:
+        correct_k = correct[:k].view(-1).float().sum(0)
+        res["acc{}".format(k)] = correct_k.mul_(1.0 / batch_size).item()
+    return res
+
+
+def reward_accuracy(output, target, topk=(1,)):
+    batch_size = target.size(0)
+    _, predicted = torch.max(output.data, 1)
+    return (predicted == target).sum().item() / batch_size

src/sdk/pynni/nni/nas/pytorch/darts/__init__.py

@@ -2,4 +2,4 @@
 # Licensed under the MIT license.
 
 from .mutator import DartsMutator
-from .trainer import DartsTrainer
\ No newline at end of file
+from .trainer import DartsTrainer

src/sdk/pynni/nni/nas/pytorch/search_space_zoo/__init__.py

+from .darts_cell import DartsCell
+from .enas_cell import ENASMicroLayer
+from .enas_cell import ENASMacroLayer
+from .enas_cell import ENASMacroGeneralModel

src/sdk/pynni/nni/nas/pytorch/search_space_zoo/darts_cell.py

+# Copyright (c) Microsoft Corporation.
+# Licensed under the MIT license.
+
+from collections import OrderedDict
+
+import torch
+import torch.nn as nn
+from nni.nas.pytorch import mutables
+
+from .darts_ops import PoolBN, SepConv, DilConv, FactorizedReduce, DropPath, StdConv
+
+
+class Node(nn.Module):
+    def __init__(self, node_id, num_prev_nodes, channels, num_downsample_connect):
+        """
+        builtin Darts Node structure
+
+        Parameters
+        ---
+        node_id: str
+        num_prev_nodes: int
+            the number of previous nodes in this cell
+        channels: int
+            output channels
+        num_downsample_connect: int
+            downsample the input node if this cell is reduction cell
+        """
+        super().__init__()
+        self.ops = nn.ModuleList()
+        choice_keys = []
+        for i in range(num_prev_nodes):
+            stride = 2 if i < num_downsample_connect else 1
+            choice_keys.append("{}_p{}".format(node_id, i))
+            self.ops.append(
+                mutables.LayerChoice(OrderedDict([
+                    ("maxpool", PoolBN('max', channels, 3, stride, 1, affine=False)),
+                    ("avgpool", PoolBN('avg', channels, 3, stride, 1, affine=False)),
+                    ("skipconnect",
+                     nn.Identity() if stride == 1 else FactorizedReduce(channels, channels, affine=False)),
+                    ("sepconv3x3", SepConv(channels, channels, 3, stride, 1, affine=False)),
+                    ("sepconv5x5", SepConv(channels, channels, 5, stride, 2, affine=False)),
+                    ("dilconv3x3", DilConv(channels, channels, 3, stride, 2, 2, affine=False)),
+                    ("dilconv5x5", DilConv(channels, channels, 5, stride, 4, 2, affine=False))
+                ]), key=choice_keys[-1]))
+        self.drop_path = DropPath()
+        self.input_switch = mutables.InputChoice(choose_from=choice_keys, n_chosen=2, key="{}_switch".format(node_id))
+
+    def forward(self, prev_nodes):
+        assert len(self.ops) == len(prev_nodes)
+        out = [op(node) for op, node in zip(self.ops, prev_nodes)]
+        out = [self.drop_path(o) if o is not None else None for o in out]
+        return self.input_switch(out)
+
+
+class DartsCell(nn.Module):
+    """
+    Builtin Darts Cell structure. There are ``n_nodes`` nodes in one cell, in which the first two nodes' values are
+    fixed to the results of previous previous cell and previous cell respectively. One node will connect all
+    the nodes after with predefined operations in a mutable way. The last node accepts five inputs from nodes
+    before and it concats all inputs in channels as the output of the current cell, and the number of output
+    channels is ``n_nodes`` times ``channels``.
+
+    Parameters
+    ---
+    n_nodes: int
+        the number of nodes contained in this cell
+    channels_pp: int
+        the number of previous previous cell's output channels
+    channels_p: int
+        the number of previous cell's output channels
+    channels: int
+        the number of output channels for each node
+    reduction_p: bool
+        Is previous cell a reduction cell
+    reduction: bool
+        is current cell a reduction cell
+    """
+    def __init__(self, n_nodes, channels_pp, channels_p, channels, reduction_p, reduction):
+        super().__init__()
+        self.reduction = reduction
+        self.n_nodes = n_nodes
+
+        # If previous cell is reduction cell, current input size does not match with
+        # output size of cell[k-2]. So the output[k-2] should be reduced by preprocessing.
+        if reduction_p:
+            self.preproc0 = FactorizedReduce(channels_pp, channels, affine=False)
+        else:
+            self.preproc0 = StdConv(channels_pp, channels, 1, 1, 0, affine=False)
+        self.preproc1 = StdConv(channels_p, channels, 1, 1, 0, affine=False)
+
+        # generate dag
+        self.mutable_ops = nn.ModuleList()
+        for depth in range(2, self.n_nodes + 2):
+            self.mutable_ops.append(Node("{}_n{}".format("reduce" if reduction else "normal", depth),
+                                         depth, channels, 2 if reduction else 0))
+
+    def forward(self, pprev, prev):
+        """
+        Parameters
+        ---
+        pprev: torch.Tensor
+            the output of the previous previous layer
+        prev: torch.Tensor
+            the output of the previous layer
+        """
+        tensors = [self.preproc0(pprev), self.preproc1(prev)]
+        for node in self.mutable_ops:
+            cur_tensor = node(tensors)
+            tensors.append(cur_tensor)
+
+        output = torch.cat(tensors[2:], dim=1)
+        return output

src/sdk/pynni/nni/nas/pytorch/search_space_zoo/darts_ops.py

+# Copyright (c) Microsoft Corporation.
+# Licensed under the MIT license.
+
+import torch
+import torch.nn as nn
+
+
+class DropPath(nn.Module):
+    def __init__(self, p=0.):
+        """
+        Drop path with probability.
+
+        Parameters
+        ----------
+        p : float
+            Probability of an path to be zeroed.
+        """
+        super().__init__()
+        self.p = p
+
+    def forward(self, x):
+        if self.training and self.p > 0.:
+            keep_prob = 1. - self.p
+            # per data point mask
+            mask = torch.zeros((x.size(0), 1, 1, 1), device=x.device).bernoulli_(keep_prob)
+            return x / keep_prob * mask
+
+        return x
+
+
+class PoolBN(nn.Module):
+    """
+    AvgPool or MaxPool with BN. ``pool_type`` must be ``max`` or ``avg``.
+
+    Parameters
+    ---
+    pool_type: str
+        choose operation
+    C: int
+        number of channels
+    kernal_size: int
+	    size of the convolving kernel
+    stride: int
+	    stride of the convolution
+    padding: int
+	    zero-padding added to both sides of the input
+    affine: bool
+        is using affine in BatchNorm
+    """
+
+    def __init__(self, pool_type, C, kernel_size, stride, padding, affine=True):
+        super().__init__()
+        if pool_type.lower() == 'max':
+            self.pool = nn.MaxPool2d(kernel_size, stride, padding)
+        elif pool_type.lower() == 'avg':
+            self.pool = nn.AvgPool2d(kernel_size, stride, padding, count_include_pad=False)
+        else:
+            raise ValueError()
+
+        self.bn = nn.BatchNorm2d(C, affine=affine)
+
+    def forward(self, x):
+        out = self.pool(x)
+        out = self.bn(out)
+        return out
+
+
+class StdConv(nn.Sequential):
+    """
+    Standard conv: ReLU - Conv - BN
+
+    Parameters
+    ---
+    C_in: int
+        the number of input channels
+    C_out: int
+        the number of output channels
+    kernel_size: int
+        size of the convolution kernel
+    padding:
+        zero-padding added to both sides of the input
+    affine: bool
+        is using affine in BatchNorm
+    """
+
+    def __init__(self, C_in, C_out, kernel_size, stride, padding, affine=True):
+        super().__init__()
+        self.net = nn.Sequential
+        for idx, ops in enumerate((nn.ReLU(), nn.Conv2d(C_in, C_out, kernel_size, stride, padding, bias=False),
+                                   nn.BatchNorm2d(C_out, affine=affine))):
+            self.add_module(str(idx), ops)
+
+
+class FacConv(nn.Module):
+    """
+    Factorized conv: ReLU - Conv(Kx1) - Conv(1xK) - BN
+    """
+
+    def __init__(self, C_in, C_out, kernel_length, stride, padding, affine=True):
+        super().__init__()
+        self.net = nn.Sequential(
+            nn.ReLU(),
+            nn.Conv2d(C_in, C_in, (kernel_length, 1), stride, padding, bias=False),
+            nn.Conv2d(C_in, C_out, (1, kernel_length), stride, padding, bias=False),
+            nn.BatchNorm2d(C_out, affine=affine)
+        )
+
+    def forward(self, x):
+        return self.net(x)
+
+
+class DilConv(nn.Module):
+    """
+    (Dilated) depthwise separable conv.
+    ReLU - (Dilated) depthwise separable - Pointwise - BN.
+    If dilation == 2, 3x3 conv => 5x5 receptive field, 5x5 conv => 9x9 receptive field.
+
+    Parameters
+    ---
+    C_in: int
+        the number of input channels
+    C_out: int
+        the number of output channels
+    kernal_size:
+        size of the convolving kernel
+    padding:
+        zero-padding added to both sides of the input
+    dilation: int
+        spacing between kernel elements.
+    affine: bool
+        is using affine in BatchNorm
+    """
+
+    def __init__(self, C_in, C_out, kernel_size, stride, padding, dilation, affine=True):
+        super().__init__()
+        self.net = nn.Sequential(
+            nn.ReLU(),
+            nn.Conv2d(C_in, C_in, kernel_size, stride, padding, dilation=dilation, groups=C_in,
+                      bias=False),
+            nn.Conv2d(C_in, C_out, 1, stride=1, padding=0, bias=False),
+            nn.BatchNorm2d(C_out, affine=affine)
+        )
+
+    def forward(self, x):
+        return self.net(x)
+
+
+class SepConv(nn.Module):
+    """
+    Depthwise separable conv.
+    DilConv(dilation=1) * 2.
+
+    Parameters
+    ---
+    C_in: int
+        the number of input channels
+    C_out: int
+        the number of output channels
+    kernal_size:
+        size of the convolving kernel
+    padding:
+        zero-padding added to both sides of the input
+    dilation: int
+        spacing between kernel elements.
+    affine: bool
+        is using affine in BatchNorm
+    """
+
+    def __init__(self, C_in, C_out, kernel_size, stride, padding, affine=True):
+        super().__init__()
+        self.net = nn.Sequential(
+            DilConv(C_in, C_in, kernel_size, stride, padding, dilation=1, affine=affine),
+            DilConv(C_in, C_out, kernel_size, 1, padding, dilation=1, affine=affine)
+        )
+
+    def forward(self, x):
+        return self.net(x)
+
+
+class FactorizedReduce(nn.Module):
+    """
+    Reduce feature map size by factorized pointwise (stride=2).
+    """
+
+    def __init__(self, C_in, C_out, affine=True):
+        super().__init__()
+        self.relu = nn.ReLU()
+        self.conv1 = nn.Conv2d(C_in, C_out // 2, 1, stride=2, padding=0, bias=False)
+        self.conv2 = nn.Conv2d(C_in, C_out // 2, 1, stride=2, padding=0, bias=False)
+        self.bn = nn.BatchNorm2d(C_out, affine=affine)
+
+    def forward(self, x):
+        x = self.relu(x)
+        out = torch.cat([self.conv1(x), self.conv2(x[:, :, 1:, 1:])], dim=1)
+        out = self.bn(out)
+        return out

src/sdk/pynni/nni/nas/pytorch/search_space_zoo/enas_cell.py

+# Copyright (c) Microsoft Corporation.
+# Licensed under the MIT license.
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+
+from nni.nas.pytorch import mutables
+from .enas_ops import FactorizedReduce, StdConv, SepConvBN, Pool, ConvBranch, PoolBranch
+
+
+class Cell(nn.Module):
+    def __init__(self, cell_name, prev_labels, channels):
+        super().__init__()
+        self.input_choice = mutables.InputChoice(choose_from=prev_labels, n_chosen=1, return_mask=True,
+                                                 key=cell_name + "_input")
+        self.op_choice = mutables.LayerChoice([
+            SepConvBN(channels, channels, 3, 1),
+            SepConvBN(channels, channels, 5, 2),
+            Pool("avg", 3, 1, 1),
+            Pool("max", 3, 1, 1),
+            nn.Identity()
+        ], key=cell_name + "_op")
+
+    def forward(self, prev_layers):
+        chosen_input, chosen_mask = self.input_choice(prev_layers)
+        cell_out = self.op_choice(chosen_input)
+        return cell_out, chosen_mask
+
+
+class Node(mutables.MutableScope):
+    def __init__(self, node_name, prev_node_names, channels):
+        super().__init__(node_name)
+        self.cell_x = Cell(node_name + "_x", prev_node_names, channels)
+        self.cell_y = Cell(node_name + "_y", prev_node_names, channels)
+
+    def forward(self, prev_layers):
+        out_x, mask_x = self.cell_x(prev_layers)
+        out_y, mask_y = self.cell_y(prev_layers)
+        return out_x + out_y, mask_x | mask_y
+
+
+class Calibration(nn.Module):
+    def __init__(self, in_channels, out_channels):
+        super().__init__()
+        self.process = None
+        if in_channels != out_channels:
+            self.process = StdConv(in_channels, out_channels)
+
+    def forward(self, x):
+        if self.process is None:
+            return x
+        return self.process(x)
+
+
+class ENASMicroLayer(nn.Module):
+    """
+    Builtin EnasMicroLayer. Micro search designs only one building block whose architecture is repeated
+    throughout the final architecture. A cell has ``num_nodes`` nodes and searches the topology and
+    operations among them in RL way. The first two nodes in a layer stand for the outputs from previous
+    previous layer and previous layer respectively. For the following nodes, the controller chooses
+    two previous nodes and applies two operations respectively for each node. Nodes that are not served
+    as input for any other node are viewed as the output of the layer. If there are multiple output nodes,
+    the model will calculate the average of these nodes as the layer output. Every node's output has ``out_channels``
+    channels so the result of the layer has the same number of channels as each node.
+
+    Parameters
+    ---
+    num_nodes: int
+        the number of nodes contained in this layer
+    in_channles_pp: int
+        the number of previous previous layer's output channels
+    in_channels_p: int
+        the number of previous layer's output channels
+    out_channels: int
+        output channels of this layer
+    reduction: bool
+        is reduction operation empolyed before this layer
+    """
+    def __init__(self, num_nodes, in_channels_pp, in_channels_p, out_channels, reduction):
+        super().__init__()
+        print(in_channels_pp, in_channels_p, out_channels, reduction)
+        self.reduction = reduction
+        if self.reduction:
+            self.reduce0 = FactorizedReduce(in_channels_pp, out_channels, affine=False)
+            self.reduce1 = FactorizedReduce(in_channels_p, out_channels, affine=False)
+            in_channels_pp = in_channels_p = out_channels
+        self.preproc0 = Calibration(in_channels_pp, out_channels)
+        self.preproc1 = Calibration(in_channels_p, out_channels)
+
+        self.num_nodes = num_nodes
+        name_prefix = "reduce" if reduction else "normal"
+        self.nodes = nn.ModuleList()
+        node_labels = [mutables.InputChoice.NO_KEY, mutables.InputChoice.NO_KEY]
+        for i in range(num_nodes):
+            node_labels.append("{}_node_{}".format(name_prefix, i))
+            self.nodes.append(Node(node_labels[-1], node_labels[:-1], out_channels))
+        self.final_conv_w = nn.Parameter(torch.zeros(out_channels, self.num_nodes + 2, out_channels, 1, 1),
+                                         requires_grad=True)
+        self.bn = nn.BatchNorm2d(out_channels, affine=False)
+        self.reset_parameters()
+
+    def reset_parameters(self):
+        nn.init.kaiming_normal_(self.final_conv_w)
+
+    def forward(self, pprev, prev):
+        """
+        Parameters
+        ---
+        pprev: torch.Tensor
+            the output of the previous previous layer
+        prev: torch.Tensor
+            the output of the previous previous layer
+        """
+        if self.reduction:
+            pprev, prev = self.reduce0(pprev), self.reduce1(prev)
+        pprev_, prev_ = self.preproc0(pprev), self.preproc1(prev)
+
+        prev_nodes_out = [pprev_, prev_]
+        nodes_used_mask = torch.zeros(self.num_nodes + 2, dtype=torch.bool, device=prev.device)
+        for i in range(self.num_nodes):
+            node_out, mask = self.nodes[i](prev_nodes_out)
+            nodes_used_mask[:mask.size(0)] |= mask.to(node_out.device)
+            prev_nodes_out.append(node_out)
+
+        unused_nodes = torch.cat([out for used, out in zip(nodes_used_mask, prev_nodes_out) if not used], 1)
+        unused_nodes = F.relu(unused_nodes)
+        conv_weight = self.final_conv_w[:, ~nodes_used_mask, :, :, :]
+        conv_weight = conv_weight.view(conv_weight.size(0), -1, 1, 1)
+        out = F.conv2d(unused_nodes, conv_weight)
+        return prev, self.bn(out)
+
+
+class ENASMacroLayer(mutables.MutableScope):
+    """
+    Builtin ENAS Marco Layer. With search space changing to layer level, the controller decides
+    what operation is employed and the previous layer to connect to for skip connections. The model
+    is made up of the same layers but the choice of each layer may be different.
+
+    Parameters
+    ---
+    key: str
+        the name of this layer
+    prev_labels: str
+        names of all previous layers
+    in_filters: int
+        the number of input channels
+    out_filters:
+        the number of output channels
+    """
+    def __init__(self, key, prev_labels, in_filters, out_filters):
+        super().__init__(key)
+        self.in_filters = in_filters
+        self.out_filters = out_filters
+        self.mutable = mutables.LayerChoice([
+            ConvBranch(in_filters, out_filters, 3, 1, 1, separable=False),
+            ConvBranch(in_filters, out_filters, 3, 1, 1, separable=True),
+            ConvBranch(in_filters, out_filters, 5, 1, 2, separable=False),
+            ConvBranch(in_filters, out_filters, 5, 1, 2, separable=True),
+            PoolBranch('avg', in_filters, out_filters, 3, 1, 1),
+            PoolBranch('max', in_filters, out_filters, 3, 1, 1)
+        ])
+        if prev_labels > 0:
+            self.skipconnect = mutables.InputChoice(choose_from=prev_labels, n_chosen=None)
+        else:
+            self.skipconnect = None
+        self.batch_norm = nn.BatchNorm2d(out_filters, affine=False)
+
+    def forward(self, prev_list):
+        """
+        Parameters
+        ---
+        prev_list: list
+            The cell selects the last element of the list as input and applies an operation on it.
+            The cell chooses none/one/multiple tensor(s) as SkipConnect(s) from the list excluding
+            the last element.
+        """
+        out = self.mutable(prev_list[-1])
+        if self.skipconnect is not None:
+            connection = self.skipconnect(prev_list[:-1])
+            if connection is not None:
+                out += connection
+        return self.batch_norm(out)
+
+
+class ENASMacroGeneralModel(nn.Module):
+    """
+    The network is made up by stacking ENASMacroLayer. The Macro search space contains these layers.
+    Each layer chooses an operation from predefined ones and SkipConnect then forms a network.
+
+    Parameters
+    ---
+    num_layers: int
+        The number of layers contained in the network.
+    out_filters: int
+        The number of each layer's output channels.
+    in_channel: int
+        The number of input's channels.
+    num_classes: int
+        The number of classes for classification.
+    dropout_rate: float
+        Dropout layer's dropout rate before the final dense layer.
+    """
+    def __init__(self, num_layers=12, out_filters=24, in_channels=3, num_classes=10,
+                 dropout_rate=0.0):
+        super().__init__()
+        self.num_layers = num_layers
+        self.num_classes = num_classes
+        self.out_filters = out_filters
+
+        self.stem = nn.Sequential(
+            nn.Conv2d(in_channels, out_filters, 3, 1, 1, bias=False),
+            nn.BatchNorm2d(out_filters)
+        )
+
+        pool_distance = self.num_layers // 3
+        self.pool_layers_idx = [pool_distance - 1, 2 * pool_distance - 1]
+        self.dropout_rate = dropout_rate
+        self.dropout = nn.Dropout(self.dropout_rate)
+
+        self.layers = nn.ModuleList()
+        self.pool_layers = nn.ModuleList()
+        labels = []
+        for layer_id in range(self.num_layers):
+            labels.append("layer_{}".format(layer_id))
+            if layer_id in self.pool_layers_idx:
+                self.pool_layers.append(FactorizedReduce(self.out_filters, self.out_filters))
+            self.layers.append(ENASMacroLayer(labels[-1], labels[:-1], self.out_filters, self.out_filters))
+
+        self.gap = nn.AdaptiveAvgPool2d(1)
+        self.dense = nn.Linear(self.out_filters, self.num_classes)
+
+    def forward(self, x):
+        """
+        Parameters
+        ---
+        x: torch.Tensor
+            the input of the network
+        """
+        bs = x.size(0)
+        cur = self.stem(x)
+
+        layers = [cur]
+
+        for layer_id in range(self.num_layers):
+            cur = self.layers[layer_id](layers)
+            layers.append(cur)
+            if layer_id in self.pool_layers_idx:
+                for i, layer in enumerate(layers):
+                    layers[i] = self.pool_layers[self.pool_layers_idx.index(layer_id)](layer)
+                cur = layers[-1]
+
+        cur = self.gap(cur).view(bs, -1)
+        cur = self.dropout(cur)
+        logits = self.dense(cur)
+        return logits

src/sdk/pynni/nni/nas/pytorch/search_space_zoo/enas_ops.py

+# Copyright (c) Microsoft Corporation.
+# Licensed under the MIT license.
+
+import torch
+import torch.nn as nn
+
+
+class StdConv(nn.Module):
+    def __init__(self, C_in, C_out):
+        super(StdConv, self).__init__()
+        self.conv = nn.Sequential(
+            nn.Conv2d(C_in, C_out, 1, stride=1, padding=0, bias=False),
+            nn.BatchNorm2d(C_out, affine=False),
+            nn.ReLU()
+        )
+
+    def forward(self, x):
+        return self.conv(x)
+
+
+class PoolBranch(nn.Module):
+    """
+    Pooling structure for Macro search. First pass through a 1x1 Conv, then pooling operation followed by BatchNorm2d.
+
+    Parameters
+    ---
+    pool_type: str
+        only accept ``max`` for MaxPool and ``avg`` for AvgPool
+    C_in: int
+        the number of input channels
+    C_out: int
+        the number of output channels
+    kernal_size: int
+        size of the convolving kernel
+    stride: int
+	    stride of the convolution
+    padding: int
+	    zero-padding added to both sides of the input
+    """
+    def __init__(self, pool_type, C_in, C_out, kernel_size, stride, padding, affine=False):
+        super().__init__()
+        self.preproc = StdConv(C_in, C_out)
+        self.pool = Pool(pool_type, kernel_size, stride, padding)
+        self.bn = nn.BatchNorm2d(C_out, affine=affine)
+
+    def forward(self, x):
+        out = self.preproc(x)
+        out = self.pool(out)
+        out = self.bn(out)
+        return out
+
+
+class SeparableConv(nn.Module):
+    def __init__(self, C_in, C_out, kernel_size, stride, padding):
+        super(SeparableConv, self).__init__()
+        self.depthwise = nn.Conv2d(C_in, C_in, kernel_size=kernel_size, padding=padding, stride=stride,
+                                   groups=C_in, bias=False)
+        self.pointwise = nn.Conv2d(C_in, C_out, kernel_size=1, bias=False)
+
+    def forward(self, x):
+        out = self.depthwise(x)
+        out = self.pointwise(out)
+        return out
+
+
+class ConvBranch(nn.Module):
+    """
+    Conv structure for Macro search. First pass through a 1x1 Conv,
+    then Conv operation with kernal_size equals 3 or 5 followed by BatchNorm and ReLU.
+
+    Parameters
+    ---
+    C_in: int
+        the number of input channels
+    C_out: int
+        the number of output channels
+    kernal_size: int
+        size of the convolving kernel
+    stride: int
+	    stride of the convolution
+    padding: int
+	    zero-padding added to both sides of the input
+    separable: True
+        is separable Conv is used
+    """
+    def __init__(self, C_in, C_out, kernel_size, stride, padding, separable):
+        super(ConvBranch, self).__init__()
+        self.preproc = StdConv(C_in, C_out)
+        if separable:
+            self.conv = SeparableConv(C_out, C_out, kernel_size, stride, padding)
+        else:
+            self.conv = nn.Conv2d(C_out, C_out, kernel_size, stride=stride, padding=padding)
+        self.postproc = nn.Sequential(
+            nn.BatchNorm2d(C_out, affine=False),
+            nn.ReLU()
+        )
+
+    def forward(self, x):
+        out = self.preproc(x)
+        out = self.conv(out)
+        out = self.postproc(out)
+        return out
+
+
+class FactorizedReduce(nn.Module):
+    def __init__(self, C_in, C_out, affine=False):
+        super().__init__()
+        self.conv1 = nn.Conv2d(C_in, C_out // 2, 1, stride=2, padding=0, bias=False)
+        self.conv2 = nn.Conv2d(C_in, C_out // 2, 1, stride=2, padding=0, bias=False)
+        self.bn = nn.BatchNorm2d(C_out, affine=affine)
+
+    def forward(self, x):
+        out = torch.cat([self.conv1(x), self.conv2(x[:, :, 1:, 1:])], dim=1)
+        out = self.bn(out)
+        return out
+
+
+class Pool(nn.Module):
+    """
+    Pooling structure
+
+    Parameters
+    ---
+    pool_type: str
+        only accept ``max`` for MaxPool and ``avg`` for AvgPool
+    kernal_size: int
+        size of the convolving kernel
+    stride: int
+	    stride of the convolution
+    padding: int
+	    zero-padding added to both sides of the input
+    """
+    def __init__(self, pool_type, kernel_size, stride, padding):
+        super().__init__()
+        if pool_type.lower() == 'max':
+            self.pool = nn.MaxPool2d(kernel_size, stride, padding)
+        elif pool_type.lower() == 'avg':
+            self.pool = nn.AvgPool2d(kernel_size, stride, padding, count_include_pad=False)
+        else:
+            raise ValueError()
+
+    def forward(self, x):
+        return self.pool(x)
+
+
+class SepConvBN(nn.Module):
+    """
+    Implement SepConv followed by BatchNorm. The structure is ReLU ==> SepConv ==> BN.
+
+    Parameters
+    ---
+    C_in: int
+        the number of imput channels
+    C_out: int
+        the number of output channels
+    kernal_size: int
+        size of the convolving kernel
+    padding: int
+        zero-padding added to both sides of the input
+    """
+    def __init__(self, C_in, C_out, kernel_size, padding):
+        super().__init__()
+        self.relu = nn.ReLU()
+        self.conv = SeparableConv(C_in, C_out, kernel_size, 1, padding)
+        self.bn = nn.BatchNorm2d(C_out, affine=True)
+
+    def forward(self, x):
+        x = self.relu(x)
+        x = self.conv(x)
+        x = self.bn(x)
+        return x