GettingStarted.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A short tutorial how to use the mlperf inference reference benchmark"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We wrapped all inference models into a single benchmark app. The benchmark app will read the propper dataset, preprocesses it and interface with the backend. Traffic is generated by loadgen, which depending on the desired mode drives the desired traffic to the benchmark app. \n",
    "\n",
    "To run this notebook, pick a directory and clone the mlperf source tree:\n",
    "```\n",
    "cd /tmp\n",
    "git clone --recurse-submodules https://github.com/mlcommons/inference.git --depth 1\n",
    "cd inference/vision/classification_and_detection\n",
    "jupyter notebook \n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "root = os.getcwd()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "running develop\n",
      "running egg_info\n",
      "writing mlperf_loadgen.egg-info/PKG-INFO\n",
      "writing dependency_links to mlperf_loadgen.egg-info/dependency_links.txt\n",
      "writing top-level names to mlperf_loadgen.egg-info/top_level.txt\n",
      "reading manifest file 'mlperf_loadgen.egg-info/SOURCES.txt'\n",
      "writing manifest file 'mlperf_loadgen.egg-info/SOURCES.txt'\n",
      "running build_ext\n",
      "building 'mlperf_loadgen' extension\n",
      "gcc -pthread -B /opt/anaconda3/compiler_compat -Wl,--sysroot=/ -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -std=c++14 -fPIC -DMAJOR_VERSION=0 -DMINOR_VERSION=5 -I. -I../third_party/pybind/include -I/opt/anaconda3/include/python3.7m -c loadgen.cc -o build/temp.linux-x86_64-3.7/loadgen.o\n",
      "\u001b[01m\u001b[Kcc1plus:\u001b[m\u001b[K \u001b[01;35m\u001b[Kwarning: \u001b[m\u001b[Kcommand line option ‘\u001b[01m\u001b[K-Wstrict-prototypes\u001b[m\u001b[K’ is valid for C/ObjC but not for C++\n",
      "...\n",
      "Using /opt/anaconda3/lib/python3.7/site-packages\n",
      "Finished processing dependencies for mlperf-inference==0.1.0\n"
     ]
    }
   ],
   "source": [
    "!cd ../../loadgen; CFLAGS=\"-std=c++14\" python setup.py develop; cd {root}\n",
    "!python setup.py develop"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The benchmark app uses a shell script to simplify command line options and the user can pick backend, model and device:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "usage: ./run_local.sh tf|onnxruntime|pytorch|tflite [resnet50|mobilenet|ssd-mobilenet|ssd-resnet34|ssd-resnet34] [cpu|gpu]\r\n"
     ]
    }
   ],
   "source": [
    "!./run_local.sh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before running the benchmark, device on model and dataset and set the environment variable ```MODEL_DIR``` and ```DATA_DIR```. \n",
    "\n",
    "For this tutorial we use onnxruntime (tensorflow and pytorch will work as well), mobilenet and a fake imagetnet dataset with a few images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: onnxruntime in /opt/anaconda3/lib/python3.7/site-packages (0.4.0)\r\n"
     ]
    }
   ],
   "source": [
    "!pip install onnxruntime pycocotools opencv-python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Step 1 - download the model. You find the links to the models [here](https://github.com/mlcommons/inference/blob/master/vision/classification_and_detection/README.md)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget -q https://zenodo.org/record/3157894/files/mobilenet_v1_1.0_224.onnx"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Step 2 - download the dataset. For this tutorial we create a small, fake dataset that pretends to be imagenet.\n",
    "Normally you'd need to download imagenet2012/valiation for image classification or coco2017/valiation for object detections.\n",
    "\n",
    "Links and instructions how to download the datasets can be found in the [README](https://github.com/mlperf/inference/tree/master/v0.5/classification_and_detection#datasets)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "!tools/make_fake_imagenet.sh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Step 3 - tell the benchmark where to find model and data "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "os.environ['MODEL_DIR'] = root\n",
    "os.environ['DATA_DIR'] = os.path.join(root, \"fake_imagenet\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For mlperf submission number of queries, time, latencies and percentiles are given and we default to those settings. But for this tuturial we pass in some extra options to make things go quicker.\n",
    "run_local.sh will look for the evironment variable EXTRA_OPS and add this to the arguments. You can also add additional arguments in the command line.\n",
    "The options below will limit the time that the benchmarks run to 10 seconds and adds accuracy reporting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "os.environ['EXTRA_OPS'] =\"--time 10 --max-latency 0.2\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Step 4 - run the benchmark."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:main:Namespace(accuracy=True, backend='onnxruntime', cache=0, count=None, data_format=None, dataset='imagenet_mobilenet', dataset_list=None, dataset_path='/home/gs/inference/v0.5/classification_and_detection/fake_imagenet', inputs=None, max_batchsize=32, max_latency=[0.2], model='/home/gs/inference/v0.5/classification_and_detection/mobilenet_v1_1.0_224.onnx', output='/home/gs/inference/v0.5/classification_and_detection/output/mobilenet-onnxruntime-cpu/results.json', outputs=['MobilenetV1/Predictions/Reshape_1:0'], profile='mobilenet-onnxruntime', qps=10, queries_multi=24576, queries_offline=20, queries_single=1024, scenario=[TestScenario.SingleStream], threads=8, time=10)\n",
      "INFO:imagenet:loaded 8 images, cache=0, took=0.2sec\n",
      "INFO:main:starting TestScenario.SingleStream\n",
      "TestScenario.SingleStream qps=69.75, mean=0.0115, time=0.11, acc=62.50, queries=8, tiles=50.0:0.0112,80.0:0.0115,90.0:0.0121,95.0:0.0129,99.0:0.0135,99.9:0.0137\n"
     ]
    }
   ],
   "source": [
    "!./run_local.sh onnxruntime mobilenet cpu --accuracy "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The line ```Accuracy``` reports accuracy or mAP together with some latencies in various percentiles so you can insight how this run was. Above accuracy was 87.5%.\n",
    "\n",
    "The line ```TestScenario.SingleStream-1.0``` reports the latency and qps seen during the benchmark.\n",
    "\n",
    "For submission the official logging is found in [mlperf_log_summary.txt](mlperf_log_summary.txt) and [mlperf_log_detail.txt](mlperf_log_detail.txt)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you read over the mlperf inference rules guide you'll find multiple scenarios to be run for the inference benchmarks:\n",
    "\n",
    "|scenario|description|\n",
    "|:---|:---|\n",
    "|SingleStream|The LoadGen sends the next query as soon as the SUT completes the previous one.|\n",
    "|MultiStream|The LoadGen sends the next query as soon as the SUT completes the previous one. Each query contains multiple samples.|\n",
    "|Server|The LoadGen sends new queries to the SUT according to a Poisson distribution. Overtime queries must not exceed 2x the latency bound.|\n",
    "|Offline|The LoadGen sends all queries to the SUT at one time.|\n",
    "\n",
    "We can run those scenario using the ```--scenario``` option in the command line, for example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:main:Namespace(accuracy=False, backend='onnxruntime', cache=0, count=None, data_format=None, dataset='imagenet_mobilenet', dataset_list=None, dataset_path='/home/gs/inference/v0.5/classification_and_detection/fake_imagenet', inputs=None, max_batchsize=32, max_latency=[0.2], model='/home/gs/inference/v0.5/classification_and_detection/mobilenet_v1_1.0_224.onnx', output='/home/gs/inference/v0.5/classification_and_detection/output/mobilenet-onnxruntime-cpu/results.json', outputs=['MobilenetV1/Predictions/Reshape_1:0'], profile='mobilenet-onnxruntime', qps=10, queries_multi=24576, queries_offline=20, queries_single=1024, scenario=[TestScenario.Offline], threads=8, time=10)\n",
      "INFO:imagenet:loaded 8 images, cache=0, took=0.0sec\n",
      "INFO:main:starting TestScenario.Offline\n",
      "TestScenario.Offline qps=44.11, mean=2.3486, time=2.49, queries=110, tiles=50.0:2.4500,80.0:2.4687,90.0:2.4687,95.0:2.4687,99.0:2.4687,99.9:2.4687\n"
     ]
    }
   ],
   "source": [
    "!./run_local.sh onnxruntime mobilenet cpu --scenario Offline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Additional logfiles\n",
    "\n",
    "We log some additional information [here](output/mobilenet-onnxruntime-cpu/results.json) which can be used to plot graphs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Under the hood\n",
    "\n",
    "In case you wonder what the run_local.sh does, it only assembles the command line for the python based benchmark app. Command ine options for the app are documented [here](https://github.com/mlperf/inference/blob/master/cloud/image_classification)\n",
    "\n",
    "Calling\n",
    "```\n",
    "!bash -x ./run_local.sh onnxruntime mobilenet cpu  --accuracy \n",
    "```\n",
    "will results in the following command line:\n",
    "```\n",
    "python python/main.py --profile mobilenet-onnxruntime --model /tmp/inference/cloud/image_classification/mobilenet_v1_1.0_224.onnx --dataset-path /tmp/inference/cloud/image_classification/fake_imagenet --output /tmp/inference/cloud/image_classification/output/mobilenet-onnxruntime-cpu/results.json --queries-offline 20 --time 10 --max-latency 0.2 --accuracy\n",
    "```\n",
    "During testing you can change some of the options to have faster test cycles but for final submission use the defaults."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using docker\n",
    "\n",
    "Instead of run_local.sh you can use run_and_time.sh which does have the same options but instead of running local will run the benchmark under docker."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sending build context to Docker daemon  18.54MB\n",
      "Step 1/15 : FROM ubuntu:16.04\n",
      " ---> bd3d4369aebc\n",
      "Step 2/15 : ENV PYTHON_VERSION=3.7\n",
      " ---> Using cache\n",
      " ---> e25f214201a2\n",
      "Step 3/15 : ENV LANG C.UTF-8\n",
      " ---> Using cache\n",
      " ---> 12986ee696e1\n",
      "Step 4/15 : ENV LC_ALL C.UTF-8\n",
      " ---> Using cache\n",
      " ---> 1460535b24e1\n",
      "Step 5/15 : ENV PATH /opt/anaconda3/bin:$PATH\n",
      " ---> Using cache\n",
      " ---> f4c922578fdf\n",
      "Step 6/15 : WORKDIR /root\n",
      " ---> Using cache\n",
      " ---> fb0ec9a436a5\n",
      "Step 7/15 : ENV HOME /root\n",
      " ---> Using cache\n",
      " ---> edeb7c15ebfb\n",
      "Step 8/15 : RUN apt-get update\n",
      " ---> Using cache\n",
      " ---> 42da1a4fa3fd\n",
      "Step 9/15 : RUN apt-get install -y --no-install-recommends       git       build-essential       software-properties-common       ca-certificates       wget       curl       htop       zip       unzip\n",
      " ---> Using cache\n",
      " ---> a1de66a3c7bd\n",
      "Step 10/15 : RUN cd /opt &&     wget --quiet https://repo.anaconda.com/miniconda/Miniconda3-4.6.14-Linux-x86_64.sh -O miniconda.sh &&     /bin/bash ./miniconda.sh -b -p /opt/anaconda3 &&     rm miniconda.sh &&     /opt/anaconda3/bin/conda clean -tipsy &&     ln -s /opt/anaconda3/etc/profile.d/conda.sh /etc/profile.d/conda.sh &&     echo \". /opt/anaconda3/etc/profile.d/conda.sh\" >> ~/.bashrc &&     echo \"conda activate base\" >> ~/.bashrc &&     conda config --set always_yes yes --set changeps1 no\n",
      " ---> Using cache\n",
      " ---> b3a1fa068421\n",
      "Step 11/15 : RUN conda install pytorch-cpu torchvision-cpu -c pytorch\n",
      " ---> Using cache\n",
      " ---> 0f7c294fe4c8\n",
      "Step 12/15 : RUN pip install future pillow onnx opencv-python-headless tensorflow onnxruntime\n",
      " ---> Using cache\n",
      " ---> 160977b84ece\n",
      "Step 13/15 : RUN pip install Cython && pip install pycocotools\n",
      " ---> Using cache\n",
      " ---> ffc479fc7d11\n",
      "Step 14/15 : RUN cd /tmp &&     git clone https://github.com/mlperf/inference &&     cd inference/loadgen &&     pip install pybind11 &&     CFLAGS=\"-std=c++14\" python setup.py install &&     rm -rf mlperf\n",
      " ---> Using cache\n",
      " ---> 20eb0ce678b0\n",
      "Step 15/15 : ENTRYPOINT [\"/bin/bash\"]\n",
      " ---> Using cache\n",
      " ---> 9440a8884457\n",
      "Successfully built 9440a8884457\n",
      "Successfully tagged mlperf-infer-imgclassify-cpu:latest\n",
      "Clearing caches.\n",
      "3\n",
      "STARTING RUN AT 2019-07-23 04:09:29 PM\n",
      "INFO:main:Namespace(accuracy=False, backend='onnxruntime', cache=0, count=None, data_format=None, dataset='imagenet_mobilenet', dataset_list=None, dataset_path='/home/gs/inference/v0.5/classification_and_detection/fake_imagenet', inputs=None, max_batchsize=32, max_latency=[0.2], model='/home/gs/inference/v0.5/classification_and_detection/mobilenet_v1_1.0_224.onnx', output='/output/results.json', outputs=['MobilenetV1/Predictions/Reshape_1:0'], profile='mobilenet-onnxruntime', qps=10, queries_multi=24576, queries_offline=20, queries_single=1024, scenario=[TestScenario.SingleStream], threads=8, time=10)\n",
      "INFO:imagenet:loaded 8 images, cache=0, took=0.3sec\n",
      "INFO:main:starting TestScenario.SingleStream\n",
      "TestScenario.SingleStream qps=37.18, mean=0.0268, time=10.09, queries=375, tiles=50.0:0.0261,80.0:0.0262,90.0:0.0266,95.0:0.0271,99.0:0.0385,99.9:0.0823\n",
      "ENDING RUN AT 2019-07-23 04:09:45 PM\n"
     ]
    }
   ],
   "source": [
    "!./run_and_time.sh onnxruntime mobilenet cpu "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preparing for offical submision\n",
    "\n",
    "TODO."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}