"src/vscode:/vscode.git/clone" did not exist on "f8ae600ce4b3e728e7e852908603a225f2fe5932"
Unverified Commit 8c6a6407 authored by Yuge Zhang's avatar Yuge Zhang Committed by GitHub
Browse files

Refine NAS benchmark docs and examples (#2800)

* Refine NAS benchmark docs and examples

* Improve docs of NAS benchmark
parent 64c7f298
# NAS Benchmarks (experimental)
# NAS Benchmarks
```eval_rst
.. toctree::
......@@ -8,12 +8,13 @@
```
## Introduction
To imporve the reproducibility of NAS algorithms as well as reducing computing resource requirements, researchers proposed a series of NAS benchmarks such as [NAS-Bench-101](https://arxiv.org/abs/1902.09635), [NAS-Bench-201](https://arxiv.org/abs/2001.00326), [NDS](https://arxiv.org/abs/1905.13214), etc. NNI provides a query interface for users to acquire these benchmarks. Within just a few lines of code, researcher are able to evaluate their NAS algorithms easily and fairly by utilizing these benchmarks.
## Prerequisites
* Please prepare a folder to household all the benchmark databases. By default, it can be found at `${HOME}/.nni/nasbenchmark`. You can place it anywhere you like, and specify it in `NASBENCHMARK_DIR` before importing NNI.
* Please install `peewee` via `pip install peewee`, which NNI uses to connect to database.
* Please prepare a folder to household all the benchmark databases. By default, it can be found at `${HOME}/.nni/nasbenchmark`. You can place it anywhere you like, and specify it in `NASBENCHMARK_DIR` via `export NASBENCHMARK_DIR=/path/to/your/nasbenchmark` before importing NNI.
* Please install `peewee` via `pip3 install peewee`, which NNI uses to connect to database.
## Data Preparation
......
......@@ -15,5 +15,5 @@ fi
echo "Generating database..."
rm -f ${NASBENCHMARK_DIR}/nasbench101.db ${NASBENCHMARK_DIR}/nasbench101.db-journal
mkdir -p ${NASBENCHMARK_DIR}
python -m nni.nas.benchmarks.nasbench101.db_gen nasbench_full.tfrecord
python3 -m nni.nas.benchmarks.nasbench101.db_gen nasbench_full.tfrecord
rm -f nasbench_full.tfrecord
......@@ -15,5 +15,5 @@ fi
echo "Generating database..."
rm -f ${NASBENCHMARK_DIR}/nasbench201.db ${NASBENCHMARK_DIR}/nasbench201.db-journal
mkdir -p ${NASBENCHMARK_DIR}
python -m nni.nas.benchmarks.nasbench201.db_gen a.pth
python3 -m nni.nas.benchmarks.nasbench201.db_gen a.pth
rm -f a.pth
......@@ -16,5 +16,5 @@ unzip data.zip
echo "Generating database..."
rm -f ${NASBENCHMARK_DIR}/nds.db ${NASBENCHMARK_DIR}/nds.db-journal
mkdir -p ${NASBENCHMARK_DIR}
python -m nni.nas.benchmarks.nds.db_gen nds_data
python3 -m nni.nas.benchmarks.nds.db_gen nds_data
rm -rf data.zip nds_data
......@@ -15,7 +15,7 @@ def query_nb101_trial_stats(arch, num_epochs, isomorphism=True, reduction=None,
arch : dict or None
If a dict, it is in the format that is described in
:class:`nni.nas.benchmark.nasbench101.Nb101TrialConfig`. Only trial stats
matched will be returned. If none, architecture will be a wildcard.
matched will be returned. If none, all architectures in the database will be matched.
num_epochs : int or None
If int, matching results will be returned. Otherwise a wildcard.
isomorphism : boolean
......
......@@ -14,7 +14,7 @@ def query_nb201_trial_stats(arch, num_epochs, dataset, reduction=None, include_i
arch : dict or None
If a dict, it is in the format that is described in
:class:`nni.nas.benchmark.nasbench201.Nb201TrialConfig`. Only trial stats
matched will be returned. If none, architecture will be a wildcard.
matched will be returned. If none, all architectures in the database will be matched.
num_epochs : int or None
If int, matching results will be returned. Otherwise a wildcard.
dataset : str or None
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment