Proposed in `Single Path One-Shot Neural Architecture Search with Uniform Sampling <https://arxiv.org/abs/1904.00420>`__ is a one-shot NAS method that addresses the difficulties in training One-Shot NAS models by constructing a simplified supernet trained with an uniform path sampling method, so that all underlying architectures (and their weights) get trained fully and equally. An evolutionary algorithm is then applied to efficiently search for the best-performing architectures without any fine tuning.
Proposed in `Single Path One-Shot Neural Architecture Search with Uniform Sampling <https://arxiv.org/abs/1904.00420>`__ is a one-shot NAS method that addresses the difficulties in training One-Shot NAS models by constructing a simplified supernet trained with an uniform path sampling method, so that all underlying architectures (and their weights) get trained fully and equally. An evolutionary algorithm is then applied to efficiently search for the best-performing architectures without any fine tuning.
Implementation on NNI is based on `official repo <https://github.com/megvii-model/SinglePathOneShot>`__. We implement a trainer that trains the supernet and a evolution tuner that leverages the power of NNI framework that speeds up the evolutionary search phase. We have also shown
Implementation on NNI is based on `official repo <https://github.com/megvii-model/SinglePathOneShot>`__. We implement a trainer that trains the supernet and a evolution tuner that leverages the power of NNI framework that speeds up the evolutionary search phase.
Examples
Examples
--------
--------
Here is a use case, which is the search space in paper, and the way to use flops limit to perform uniform sampling.
Here is a use case, which is the search space in paper. However, we applied latency limit instead of flops limit to perform the architecture search phase.
NVIDIA DALI >= 0.16 is needed as we use DALI to accelerate the data loading of ImageNet. `Installation guide <https://docs.nvidia.com/deeplearning/sdk/dali-developer-guide/docs/installation.html>`__
Prepare ImageNet in the standard format (follow the script `here <https://gist.github.com/BIGBALLON/8a71d225eff18d88e469e6ea9b39cef4>`__\ ). Linking it to ``data/imagenet`` will be more convenient.
Download the flops lookup table from `here <https://1drv.ms/u/s!Am_mmG2-KsrnajesvSdfsq_cN48?e=aHVppN>`__ (maintained by `Megvii <https://github.com/megvii-model>`__\ ).
Download the checkpoint file from `here <https://1drv.ms/u/s!Am_mmG2-KsrnajesvSdfsq_cN48?e=aHVppN>`__ (maintained by `Megvii <https://github.com/megvii-model>`__\ ) if you don't want to retrain the supernet.
Put ``op_flops_dict.pkl`` and ``checkpoint-150000.pth.tar`` (if you don't want to retrain the supernet) under ``data`` directory.
Put ``checkpoint-150000.pth.tar`` under ``data`` directory.
Prepare ImageNet in the standard format (follow the script `here <https://gist.github.com/BIGBALLON/8a71d225eff18d88e469e6ea9b39cef4>`__\ ). Linking it to ``data/imagenet`` will be more convenient.
After preparation, it's expected to have the following code structure:
After preparation, it's expected to have the following code structure:
...
@@ -32,19 +31,16 @@ After preparation, it's expected to have the following code structure:
...
@@ -32,19 +31,16 @@ After preparation, it's expected to have the following code structure:
spos
spos
├── architecture_final.json
├── architecture_final.json
├── blocks.py
├── blocks.py
├── config_search.yml
├── data
├── data
│ ├── imagenet
│ ├── imagenet
│ │ ├── train
│ │ ├── train
│ │ └── val
│ │ └── val
│ └── op_flops_dict.pkl
│ └── checkpoint-150000.pth.tar
├── dataloader.py
├── network.py
├── network.py
├── readme.md
├── readme.md
├── scratch.py
├── supernet.py
├── supernet.py
├── tester.py
├── evaluation.py
├── tuner.py
├── search.py
└── utils.py
└── utils.py
Step 1. Train Supernet
Step 1. Train Supernet
...
@@ -61,30 +57,18 @@ NOTE: The data loading used in the official repo is `slightly different from usu
...
@@ -61,30 +57,18 @@ NOTE: The data loading used in the official repo is `slightly different from usu
Step 2. Evolution Search
Step 2. Evolution Search
^^^^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^^^^
Single Path One-Shot leverages evolution algorithm to search for the best architecture. The tester, which is responsible for testing the sampled architecture, recalculates all the batch norm for a subset of training images, and evaluates the architecture on the full validation set.
Single Path One-Shot leverages evolution algorithm to search for the best architecture. In the paper, the search module, which is responsible for testing the sampled architecture, recalculates all the batch norm for a subset of training images, and evaluates the architecture on the full validation set.
In order to make the tuner aware of the flops limit and have the ability to calculate the flops, we created a new tuner called ``EvolutionWithFlops`` in ``tuner.py``\ , inheriting the tuner in SDK.
To have a search space ready for NNI framework, first run
.. code-block:: bash
nnictl ss_gen -t "python tester.py"
This will generate a file called ``nni_auto_gen_search_space.json``\ , which is a serialized representation of your search space.
By default, it will use ``checkpoint-150000.pth.tar`` downloaded previously. In case you want to use the checkpoint trained by yourself from the last step, specify ``--checkpoint`` in the command in ``config_search.yml``.
Then search with evolution tuner.
In this example, we have an incomplete implementation of the evolution search. The example only support training from scratch. Inheriting weights from pretrained supernet is not supported yet. To search with the regularized evolution strategy, run
.. code-block:: bash
.. code-block:: bash
nnictl create --config config_search.yml
python search.py
The final architecture exported from every epoch of evolution can be found in ``checkpoints`` under the working directory of your tuner, which, by default, is ``$HOME/nni-experiments/your_experiment_id/log``.
The final architecture exported from every epoch of evolution can be found in ``trials`` under the working directory of your tuner, which, by default, is ``$HOME/nni-experiments/your_experiment_id/trials``.
Step 3. Train from Scratch
Step 3. Train for Evaluation
^^^^^^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. code-block:: bash
.. code-block:: bash
...
@@ -106,7 +90,7 @@ Known Limitations
...
@@ -106,7 +90,7 @@ Known Limitations
* Block search only. Channel search is not supported yet.
* Block search only. Channel search is not supported yet.
* Only GPU version is provided here.
* In the search phase, training from the scratch is required. Inheriting weights from supernet is not supported yet.
