Unverified Commit 8c6a6407 authored by Yuge Zhang's avatar Yuge Zhang Committed by GitHub
Browse files

Refine NAS benchmark docs and examples (#2800)

* Refine NAS benchmark docs and examples

* Improve docs of NAS benchmark
parent 64c7f298
# NAS Benchmarks (experimental) # NAS Benchmarks
```eval_rst ```eval_rst
.. toctree:: .. toctree::
...@@ -8,12 +8,13 @@ ...@@ -8,12 +8,13 @@
``` ```
## Introduction ## Introduction
To imporve the reproducibility of NAS algorithms as well as reducing computing resource requirements, researchers proposed a series of NAS benchmarks such as [NAS-Bench-101](https://arxiv.org/abs/1902.09635), [NAS-Bench-201](https://arxiv.org/abs/2001.00326), [NDS](https://arxiv.org/abs/1905.13214), etc. NNI provides a query interface for users to acquire these benchmarks. Within just a few lines of code, researcher are able to evaluate their NAS algorithms easily and fairly by utilizing these benchmarks. To imporve the reproducibility of NAS algorithms as well as reducing computing resource requirements, researchers proposed a series of NAS benchmarks such as [NAS-Bench-101](https://arxiv.org/abs/1902.09635), [NAS-Bench-201](https://arxiv.org/abs/2001.00326), [NDS](https://arxiv.org/abs/1905.13214), etc. NNI provides a query interface for users to acquire these benchmarks. Within just a few lines of code, researcher are able to evaluate their NAS algorithms easily and fairly by utilizing these benchmarks.
## Prerequisites ## Prerequisites
* Please prepare a folder to household all the benchmark databases. By default, it can be found at `${HOME}/.nni/nasbenchmark`. You can place it anywhere you like, and specify it in `NASBENCHMARK_DIR` before importing NNI. * Please prepare a folder to household all the benchmark databases. By default, it can be found at `${HOME}/.nni/nasbenchmark`. You can place it anywhere you like, and specify it in `NASBENCHMARK_DIR` via `export NASBENCHMARK_DIR=/path/to/your/nasbenchmark` before importing NNI.
* Please install `peewee` via `pip install peewee`, which NNI uses to connect to database. * Please install `peewee` via `pip3 install peewee`, which NNI uses to connect to database.
## Data Preparation ## Data Preparation
......
...@@ -15,5 +15,5 @@ fi ...@@ -15,5 +15,5 @@ fi
echo "Generating database..." echo "Generating database..."
rm -f ${NASBENCHMARK_DIR}/nasbench101.db ${NASBENCHMARK_DIR}/nasbench101.db-journal rm -f ${NASBENCHMARK_DIR}/nasbench101.db ${NASBENCHMARK_DIR}/nasbench101.db-journal
mkdir -p ${NASBENCHMARK_DIR} mkdir -p ${NASBENCHMARK_DIR}
python -m nni.nas.benchmarks.nasbench101.db_gen nasbench_full.tfrecord python3 -m nni.nas.benchmarks.nasbench101.db_gen nasbench_full.tfrecord
rm -f nasbench_full.tfrecord rm -f nasbench_full.tfrecord
...@@ -15,5 +15,5 @@ fi ...@@ -15,5 +15,5 @@ fi
echo "Generating database..." echo "Generating database..."
rm -f ${NASBENCHMARK_DIR}/nasbench201.db ${NASBENCHMARK_DIR}/nasbench201.db-journal rm -f ${NASBENCHMARK_DIR}/nasbench201.db ${NASBENCHMARK_DIR}/nasbench201.db-journal
mkdir -p ${NASBENCHMARK_DIR} mkdir -p ${NASBENCHMARK_DIR}
python -m nni.nas.benchmarks.nasbench201.db_gen a.pth python3 -m nni.nas.benchmarks.nasbench201.db_gen a.pth
rm -f a.pth rm -f a.pth
...@@ -16,5 +16,5 @@ unzip data.zip ...@@ -16,5 +16,5 @@ unzip data.zip
echo "Generating database..." echo "Generating database..."
rm -f ${NASBENCHMARK_DIR}/nds.db ${NASBENCHMARK_DIR}/nds.db-journal rm -f ${NASBENCHMARK_DIR}/nds.db ${NASBENCHMARK_DIR}/nds.db-journal
mkdir -p ${NASBENCHMARK_DIR} mkdir -p ${NASBENCHMARK_DIR}
python -m nni.nas.benchmarks.nds.db_gen nds_data python3 -m nni.nas.benchmarks.nds.db_gen nds_data
rm -rf data.zip nds_data rm -rf data.zip nds_data
...@@ -15,7 +15,7 @@ def query_nb101_trial_stats(arch, num_epochs, isomorphism=True, reduction=None, ...@@ -15,7 +15,7 @@ def query_nb101_trial_stats(arch, num_epochs, isomorphism=True, reduction=None,
arch : dict or None arch : dict or None
If a dict, it is in the format that is described in If a dict, it is in the format that is described in
:class:`nni.nas.benchmark.nasbench101.Nb101TrialConfig`. Only trial stats :class:`nni.nas.benchmark.nasbench101.Nb101TrialConfig`. Only trial stats
matched will be returned. If none, architecture will be a wildcard. matched will be returned. If none, all architectures in the database will be matched.
num_epochs : int or None num_epochs : int or None
If int, matching results will be returned. Otherwise a wildcard. If int, matching results will be returned. Otherwise a wildcard.
isomorphism : boolean isomorphism : boolean
......
...@@ -14,7 +14,7 @@ def query_nb201_trial_stats(arch, num_epochs, dataset, reduction=None, include_i ...@@ -14,7 +14,7 @@ def query_nb201_trial_stats(arch, num_epochs, dataset, reduction=None, include_i
arch : dict or None arch : dict or None
If a dict, it is in the format that is described in If a dict, it is in the format that is described in
:class:`nni.nas.benchmark.nasbench201.Nb201TrialConfig`. Only trial stats :class:`nni.nas.benchmark.nasbench201.Nb201TrialConfig`. Only trial stats
matched will be returned. If none, architecture will be a wildcard. matched will be returned. If none, all architectures in the database will be matched.
num_epochs : int or None num_epochs : int or None
If int, matching results will be returned. Otherwise a wildcard. If int, matching results will be returned. Otherwise a wildcard.
dataset : str or None dataset : str or None
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment