nasbench_as_dataset.ipynb

{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "%matplotlib inline"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\n# Use NAS Benchmarks as Datasets\n\nIn this tutorial, we show how to use NAS Benchmarks as datasets.\nFor research purposes we sometimes desire to query the benchmarks for architecture accuracies,\nrather than train them one by one from scratch.\nNNI has provided query tools so that users can easily get the retrieve the data in NAS benchmarks.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Prerequisites\nThis tutorial assumes that you have already prepared your NAS benchmarks under cache directory\n(by default, ``~/.cache/nni/nasbenchmark``).\nIf you haven't, please follow the data preparation guide in :doc:`/nas/benchmarks`.\n\nAs a result, the directory should look like:\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "import os\nos.listdir(os.path.expanduser('~/.cache/nni/nasbenchmark'))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "import pprint\n\nfrom nni.nas.benchmarks.nasbench101 import query_nb101_trial_stats\nfrom nni.nas.benchmarks.nasbench201 import query_nb201_trial_stats\nfrom nni.nas.benchmarks.nds import query_nds_trial_stats"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## NAS-Bench-101\n\nUse the following architecture as an example:\n\n<img src=\"file://../../img/nas-bench-101-example.png\">\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "arch = {\n    'op1': 'conv3x3-bn-relu',\n    'op2': 'maxpool3x3',\n    'op3': 'conv3x3-bn-relu',\n    'op4': 'conv3x3-bn-relu',\n    'op5': 'conv1x1-bn-relu',\n    'input1': [0],\n    'input2': [1],\n    'input3': [2],\n    'input4': [0],\n    'input5': [0, 3, 4],\n    'input6': [2, 5]\n}\nfor t in query_nb101_trial_stats(arch, 108, include_intermediates=True):\n    pprint.pprint(t)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "An architecture of NAS-Bench-101 could be trained more than once.\nEach element of the returned generator is a dict which contains one of the training results of this trial config\n(architecture + hyper-parameters) including train/valid/test accuracy,\ntraining time, number of epochs, etc. The results of NAS-Bench-201 and NDS follow similar formats.\n\n## NAS-Bench-201\n\nUse the following architecture as an example:\n\n<img src=\"file://../../img/nas-bench-201-example.png\">\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "arch = {\n    '0_1': 'avg_pool_3x3',\n    '0_2': 'conv_1x1',\n    '1_2': 'skip_connect',\n    '0_3': 'conv_1x1',\n    '1_3': 'skip_connect',\n    '2_3': 'skip_connect'\n}\nfor t in query_nb201_trial_stats(arch, 200, 'cifar100'):\n    pprint.pprint(t)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Intermediate results are also available.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "for t in query_nb201_trial_stats(arch, None, 'imagenet16-120', include_intermediates=True):\n    print(t['config'])\n    print('Intermediates:', len(t['intermediates']))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## NDS\n\nUse the following architecture as an example:\n\n<img src=\"file://../../img/nas-bench-nds-example.png\">\n\nHere, ``bot_muls``, ``ds``, ``num_gs``, ``ss`` and ``ws`` stand for \"bottleneck multipliers\",\n\"depths\", \"number of groups\", \"strides\" and \"widths\" respectively.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "model_spec = {\n    'bot_muls': [0.0, 0.25, 0.25, 0.25],\n    'ds': [1, 16, 1, 4],\n    'num_gs': [1, 2, 1, 2],\n    'ss': [1, 1, 2, 2],\n    'ws': [16, 64, 128, 16]\n}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Use none as a wildcard.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "for t in query_nds_trial_stats('residual_bottleneck', None, None, model_spec, None, 'cifar10'):\n    pprint.pprint(t)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "model_spec = {\n    'bot_muls': [0.0, 0.25, 0.25, 0.25],\n    'ds': [1, 16, 1, 4],\n    'num_gs': [1, 2, 1, 2],\n    'ss': [1, 1, 2, 2],\n    'ws': [16, 64, 128, 16]\n}\nfor t in query_nds_trial_stats('residual_bottleneck', None, None, model_spec, None, 'cifar10', include_intermediates=True):\n    pprint.pprint(t['intermediates'][:10])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "model_spec = {'ds': [1, 12, 12, 12], 'ss': [1, 1, 2, 2], 'ws': [16, 24, 24, 40]}\nfor t in query_nds_trial_stats('residual_basic', 'resnet', 'random', model_spec, {}, 'cifar10'):\n    pprint.pprint(t)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Get the first one.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "pprint.pprint(next(query_nds_trial_stats('vanilla', None, None, None, None, None)))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Count number.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "model_spec = {'num_nodes_normal': 5, 'num_nodes_reduce': 5, 'depth': 12, 'width': 32, 'aux': False, 'drop_prob': 0.0}\ncell_spec = {\n    'normal_0_op_x': 'avg_pool_3x3',\n    'normal_0_input_x': 0,\n    'normal_0_op_y': 'conv_7x1_1x7',\n    'normal_0_input_y': 1,\n    'normal_1_op_x': 'sep_conv_3x3',\n    'normal_1_input_x': 2,\n    'normal_1_op_y': 'sep_conv_5x5',\n    'normal_1_input_y': 0,\n    'normal_2_op_x': 'dil_sep_conv_3x3',\n    'normal_2_input_x': 2,\n    'normal_2_op_y': 'dil_sep_conv_3x3',\n    'normal_2_input_y': 2,\n    'normal_3_op_x': 'skip_connect',\n    'normal_3_input_x': 4,\n    'normal_3_op_y': 'dil_sep_conv_3x3',\n    'normal_3_input_y': 4,\n    'normal_4_op_x': 'conv_7x1_1x7',\n    'normal_4_input_x': 2,\n    'normal_4_op_y': 'sep_conv_3x3',\n    'normal_4_input_y': 4,\n    'normal_concat': [3, 5, 6],\n    'reduce_0_op_x': 'avg_pool_3x3',\n    'reduce_0_input_x': 0,\n    'reduce_0_op_y': 'dil_sep_conv_3x3',\n    'reduce_0_input_y': 1,\n    'reduce_1_op_x': 'sep_conv_3x3',\n    'reduce_1_input_x': 0,\n    'reduce_1_op_y': 'sep_conv_3x3',\n    'reduce_1_input_y': 0,\n    'reduce_2_op_x': 'skip_connect',\n    'reduce_2_input_x': 2,\n    'reduce_2_op_y': 'sep_conv_7x7',\n    'reduce_2_input_y': 0,\n    'reduce_3_op_x': 'conv_7x1_1x7',\n    'reduce_3_input_x': 4,\n    'reduce_3_op_y': 'skip_connect',\n    'reduce_3_input_y': 4,\n    'reduce_4_op_x': 'conv_7x1_1x7',\n    'reduce_4_input_x': 0,\n    'reduce_4_op_y': 'conv_7x1_1x7',\n    'reduce_4_input_y': 5,\n    'reduce_concat': [3, 6]\n}\n\nfor t in query_nds_trial_stats('nas_cell', None, None, model_spec, cell_spec, 'cifar10'):\n    assert t['config']['model_spec'] == model_spec\n    assert t['config']['cell_spec'] == cell_spec\n    pprint.pprint(t)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Count number.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "print('NDS (amoeba) count:', len(list(query_nds_trial_stats(None, 'amoeba', None, None, None, None, None))))"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.8"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}