{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Python wrapper for nni restful APIs\n",
    "\n",
    "nni provides nnicli module as a python wrapper for its restful APIs, which can be used to retrieve nni experiment and trial job information in your python code. This notebook shows how to use nnicli module.\n",
    "\n",
    "Following are the functions available in nnicli module:\n",
    "\n",
    "#### start_nni(config_file)\n",
    "Starts nni experiment with specified configuration file\n",
    "\n",
    "#### stop_nni()\n",
    "Stop nni experiment.\n",
    "\n",
    "#### set_endpoint(endpoint)\n",
    "Set nni endpoint for nnicli, the endpoint is showed while nni experiment is started successfully using nnictl command or start_nni function\n",
    "\n",
    "#### version()\n",
    "Returns nni version\n",
    "\n",
    "#### get_experiment_profile()\n",
    "Returns experiment profile.\n",
    "\n",
    "#### get_experiment_status()\n",
    "Returns nni experiment status.\n",
    "\n",
    "#### get_job_metrics(trial_job_id)\n",
    "Returns specified trial job metrics, including final results and intermediate results.\n",
    "\n",
    "#### get_job_statistics()\n",
    "Returns trial job statistics information\n",
    "\n",
    "#### get_trial_job(trial_job_id)\n",
    "Returns information of a specified trial job.\n",
    "\n",
    "#### list_trial_jobs()\n",
    "Returns information of all trial jobs of current experiment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Start nni experiment using specified configuration file\n",
    "Let's use a configruation file in nni examples directory to start an experiment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "authorName: default\r\n",
      "experimentName: example_mnist\r\n",
      "trialConcurrency: 1\r\n",
      "maxExecDuration: 1h\r\n",
      "maxTrialNum: 10\r\n",
      "#choice: local, remote, pai\r\n",
      "trainingServicePlatform: local\r\n",
      "searchSpacePath: search_space.json\r\n",
      "#choice: true, false\r\n",
      "useAnnotation: false\r\n",
      "tuner:\r\n",
      "  #choice: TPE, Random, Anneal, Evolution, BatchTuner, MetisTuner, GPTuner\r\n",
      "  #SMAC (SMAC should be installed through nnictl)\r\n",
      "  builtinTunerName: TPE\r\n",
      "  classArgs:\r\n",
      "    #choice: maximize, minimize\r\n",
      "    optimize_mode: maximize\r\n",
      "trial:\r\n",
      "  command: python3 mnist.py\r\n",
      "  codeDir: .\r\n",
      "  gpuNum: 0\r\n"
     ]
    }
   ],
   "source": [
    "! cat ../trials/mnist/config.yml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO: expand searchSpacePath: search_space.json to /mnt/d/Repos/nni/examples/trials/mnist/search_space.json\n",
      "INFO: expand codeDir: . to /mnt/d/Repos/nni/examples/trials/mnist/.\n",
      "INFO: Starting restful server...\n",
      "INFO: Successfully started Restful server!\n",
      "INFO: Setting local config...\n",
      "INFO: Successfully set local config!\n",
      "INFO: Starting experiment...\n",
      "INFO: Successfully started experiment!\n",
      "-----------------------------------------------------------------------\n",
      "The experiment id is PlUIfDTR\n",
      "The Web UI urls are: http://172.18.17.1:8080   http://10.172.121.40:8080   http://10.0.75.1:8080   http://127.0.0.1:8080\n",
      "-----------------------------------------------------------------------\n",
      "\n",
      "You can use these commands to get more information about the experiment\n",
      "-----------------------------------------------------------------------\n",
      "commands                       description\n",
      "1. nnictl experiment show        show the information of experiments\n",
      "2. nnictl trial ls               list all of trial jobs\n",
      "3. nnictl top                    monitor the status of running experiments\n",
      "4. nnictl log stderr             show stderr log content\n",
      "5. nnictl log stdout             show stdout log content\n",
      "6. nnictl stop                   stop an experiment\n",
      "7. nnictl trial kill             kill a trial job by id\n",
      "8. nnictl --help                 get help information about nnictl\n",
      "-----------------------------------------------------------------------\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import nnicli as nc\n",
    "nc.start_nni(config_file='../trials/mnist/config.yml')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Connect nnicli module to started nni experiment\n",
    "Call set_endpoint to connect nnicli moduele to the rest server of started nni experiment. Local mode training serviced is used in this notebook, but nnicli module can connect to any started nni experiment. The endpoint can be found in the output of start_nni function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "nc.set_endpoint('http://127.0.0.1:8080')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Retrieve nni experiment and trial job information"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'errors': [], 'status': 'RUNNING'}"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nc.get_experiment_status()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'trialJobNumber': 4, 'trialJobStatus': 'SUCCEEDED'},\n",
       " {'trialJobNumber': 1, 'trialJobStatus': 'RUNNING'}]"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nc.get_job_statistics()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'execDuration': 1117,\n",
       " 'id': 'PlUIfDTR',\n",
       " 'logDir': '/home/chicm/nni/experiments/PlUIfDTR',\n",
       " 'maxSequenceId': 3,\n",
       " 'params': {'authorName': 'default',\n",
       "  'clusterMetaData': [{'key': 'codeDir',\n",
       "    'value': '/mnt/d/Repos/nni/examples/trials/mnist/.'},\n",
       "   {'key': 'command', 'value': 'python3 mnist.py'}],\n",
       "  'experimentName': 'example_mnist',\n",
       "  'maxExecDuration': 3600,\n",
       "  'maxTrialNum': 10,\n",
       "  'searchSpace': '{\"hidden_size\": {\"_value\": [124, 512, 1024], \"_type\": \"choice\"}, \"batch_size\": {\"_value\": [1, 4, 8, 16, 32], \"_type\": \"choice\"}, \"conv_size\": {\"_value\": [2, 3, 5, 7], \"_type\": \"choice\"}, \"dropout_rate\": {\"_value\": [0.5, 0.9], \"_type\": \"uniform\"}, \"learning_rate\": {\"_value\": [0.0001, 0.001, 0.01, 0.1], \"_type\": \"choice\"}}',\n",
       "  'trainingServicePlatform': 'local',\n",
       "  'trialConcurrency': 1,\n",
       "  'tuner': {'builtinTunerName': 'TPE',\n",
       "   'checkpointDir': '/home/chicm/nni/experiments/PlUIfDTR/checkpoint',\n",
       "   'classArgs': {'optimize_mode': 'maximize'},\n",
       "   'className': 'TPE'},\n",
       "  'versionCheck': True},\n",
       " 'revision': 116,\n",
       " 'startTime': 1564484985839}"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nc.get_experiment_profile()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's define an utility function to format json string returned by nnicli module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "def show_json(res):\n",
    "    print(json.dumps(res, indent=4))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "    \"params\": {\n",
      "        \"searchSpace\": \"{\\\"hidden_size\\\": {\\\"_value\\\": [124, 512, 1024], \\\"_type\\\": \\\"choice\\\"}, \\\"batch_size\\\": {\\\"_value\\\": [1, 4, 8, 16, 32], \\\"_type\\\": \\\"choice\\\"}, \\\"conv_size\\\": {\\\"_value\\\": [2, 3, 5, 7], \\\"_type\\\": \\\"choice\\\"}, \\\"dropout_rate\\\": {\\\"_value\\\": [0.5, 0.9], \\\"_type\\\": \\\"uniform\\\"}, \\\"learning_rate\\\": {\\\"_value\\\": [0.0001, 0.001, 0.01, 0.1], \\\"_type\\\": \\\"choice\\\"}}\",\n",
      "        \"clusterMetaData\": [\n",
      "            {\n",
      "                \"key\": \"codeDir\",\n",
      "                \"value\": \"/mnt/d/Repos/nni/examples/trials/mnist/.\"\n",
      "            },\n",
      "            {\n",
      "                \"key\": \"command\",\n",
      "                \"value\": \"python3 mnist.py\"\n",
      "            }\n",
      "        ],\n",
      "        \"tuner\": {\n",
      "            \"classArgs\": {\n",
      "                \"optimize_mode\": \"maximize\"\n",
      "            },\n",
      "            \"builtinTunerName\": \"TPE\",\n",
      "            \"checkpointDir\": \"/home/chicm/nni/experiments/PlUIfDTR/checkpoint\",\n",
      "            \"className\": \"TPE\"\n",
      "        },\n",
      "        \"maxTrialNum\": 10,\n",
      "        \"maxExecDuration\": 3600,\n",
      "        \"experimentName\": \"example_mnist\",\n",
      "        \"authorName\": \"default\",\n",
      "        \"trialConcurrency\": 1,\n",
      "        \"trainingServicePlatform\": \"local\",\n",
      "        \"versionCheck\": true\n",
      "    },\n",
      "    \"execDuration\": 1192,\n",
      "    \"revision\": 124,\n",
      "    \"logDir\": \"/home/chicm/nni/experiments/PlUIfDTR\",\n",
      "    \"maxSequenceId\": 3,\n",
      "    \"id\": \"PlUIfDTR\",\n",
      "    \"startTime\": 1564484985839\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "show_json(nc.get_experiment_profile())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[\n",
      "    {\n",
      "        \"startTime\": 1564484995992,\n",
      "        \"hyperParameters\": [\n",
      "            \"{\\\"parameter_source\\\":\\\"algorithm\\\",\\\"parameter_id\\\":0,\\\"parameter_index\\\":0,\\\"parameters\\\":{\\\"batch_size\\\":8,\\\"conv_size\\\":3,\\\"hidden_size\\\":1024,\\\"learning_rate\\\":0.0001,\\\"dropout_rate\\\":0.8055724367106529}}\"\n",
      "        ],\n",
      "        \"id\": \"BW0NR\",\n",
      "        \"endTime\": 1564485259753,\n",
      "        \"status\": \"SUCCEEDED\",\n",
      "        \"sequenceId\": 0,\n",
      "        \"finalMetricData\": [\n",
      "            {\n",
      "                \"parameterId\": \"0\",\n",
      "                \"type\": \"FINAL\",\n",
      "                \"trialJobId\": \"BW0NR\",\n",
      "                \"timestamp\": 1564485258774,\n",
      "                \"data\": \"0.9078999757766724\",\n",
      "                \"sequence\": 0\n",
      "            }\n",
      "        ],\n",
      "        \"logPath\": \"file://localhost:/home/chicm/nni/experiments/PlUIfDTR/trials/BW0NR\"\n",
      "    },\n",
      "    {\n",
      "        \"startTime\": 1564485271947,\n",
      "        \"hyperParameters\": [\n",
      "            \"{\\\"parameter_source\\\":\\\"algorithm\\\",\\\"parameter_id\\\":1,\\\"parameter_index\\\":0,\\\"parameters\\\":{\\\"batch_size\\\":4,\\\"conv_size\\\":5,\\\"hidden_size\\\":512,\\\"learning_rate\\\":0.01,\\\"dropout_rate\\\":0.5547528540531742}}\"\n",
      "        ],\n",
      "        \"id\": \"x0P5w\",\n",
      "        \"endTime\": 1564485642784,\n",
      "        \"status\": \"SUCCEEDED\",\n",
      "        \"sequenceId\": 1,\n",
      "        \"finalMetricData\": [\n",
      "            {\n",
      "                \"parameterId\": \"1\",\n",
      "                \"type\": \"FINAL\",\n",
      "                \"trialJobId\": \"x0P5w\",\n",
      "                \"timestamp\": 1564485642072,\n",
      "                \"data\": \"0.10100000351667404\",\n",
      "                \"sequence\": 0\n",
      "            }\n",
      "        ],\n",
      "        \"logPath\": \"file://localhost:/home/chicm/nni/experiments/PlUIfDTR/trials/x0P5w\"\n",
      "    },\n",
      "    {\n",
      "        \"startTime\": 1564485652151,\n",
      "        \"hyperParameters\": [\n",
      "            \"{\\\"parameter_source\\\":\\\"algorithm\\\",\\\"parameter_id\\\":2,\\\"parameter_index\\\":0,\\\"parameters\\\":{\\\"batch_size\\\":8,\\\"conv_size\\\":3,\\\"hidden_size\\\":512,\\\"learning_rate\\\":0.0001,\\\"dropout_rate\\\":0.5584485925416655}}\"\n",
      "        ],\n",
      "        \"id\": \"V9jSG\",\n",
      "        \"endTime\": 1564485917057,\n",
      "        \"status\": \"SUCCEEDED\",\n",
      "        \"sequenceId\": 2,\n",
      "        \"finalMetricData\": [\n",
      "            {\n",
      "                \"parameterId\": \"2\",\n",
      "                \"type\": \"FINAL\",\n",
      "                \"trialJobId\": \"V9jSG\",\n",
      "                \"timestamp\": 1564485916403,\n",
      "                \"data\": \"0.928600013256073\",\n",
      "                \"sequence\": 0\n",
      "            }\n",
      "        ],\n",
      "        \"logPath\": \"file://localhost:/home/chicm/nni/experiments/PlUIfDTR/trials/V9jSG\"\n",
      "    },\n",
      "    {\n",
      "        \"startTime\": 1564485927295,\n",
      "        \"hyperParameters\": [\n",
      "            \"{\\\"parameter_source\\\":\\\"algorithm\\\",\\\"parameter_id\\\":3,\\\"parameter_index\\\":0,\\\"parameters\\\":{\\\"batch_size\\\":8,\\\"conv_size\\\":7,\\\"hidden_size\\\":124,\\\"learning_rate\\\":0.001,\\\"dropout_rate\\\":0.6281630602835235}}\"\n",
      "        ],\n",
      "        \"id\": \"CDlRX\",\n",
      "        \"status\": \"RUNNING\",\n",
      "        \"sequenceId\": 3,\n",
      "        \"logPath\": \"file://localhost:/home/chicm/nni/experiments/PlUIfDTR/trials/CDlRX\"\n",
      "    }\n",
      "]\n"
     ]
    }
   ],
   "source": [
    "show_json(nc.list_trial_jobs())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualizing nni experiment result\n",
    "\n",
    "With the retrieved trial job information, we can do some analysis by visualizing the metric data, below is a simple example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 24696 characters]\n",
      "text/plain": [
       "<Figure size 1080x432 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "sns.set(style=\"whitegrid\")\n",
    "\n",
    "jobs = nc.list_trial_jobs()\n",
    "job_ids = [x['id'] for x in jobs]\n",
    "final_metrics = [float(x['finalMetricData'][0]['data']) for x in jobs]\n",
    "\n",
    "data = {'job id': job_ids, 'final metrics': final_metrics}\n",
    "sns.set(rc={'figure.figsize':(15, 6)})\n",
    "\n",
    "plt.title('Trial job final results')\n",
    "ax = sns.barplot(x='job id', y='final metrics', data=data) \n",
    "\n",
    "for i,p in enumerate(ax.patches):\n",
    "    ax.annotate('{:.4f}'.format(p.get_height()), (p.get_x() + p.get_width() / 2., p.get_height()),\n",
    "                 ha='center', va='center', fontsize=11, color='black', rotation=0, xytext=(0, 5),\n",
    "                 textcoords='offset points') "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Stop nni experiment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO: Stoping experiment PlUIfDTR\n",
      "INFO: Stop experiment success.\n"
     ]
    }
   ],
   "source": [
    "nc.stop_nni()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
