{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# End to End MLPerf Submission example\n",
    "\n",
    "This is following the [General MLPerf Submission Rules](https://github.com/mlcommons/policies/blob/master/submission_rules.adoc).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get the MLPerf Inference Benchmark Suite source code\n",
    "\n",
    "You run this notebook from the root of the 'mlcommons/inference' repo that you cloned with\n",
    "```\n",
    "git clone --recurse-submodules https://github.com/mlcommons/inference.git --depth 1\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Build loadgen"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# build loadgen\n",
    "!pip install pybind11\n",
    "!cd loadgen; CFLAGS=\"-std=c++14 -O3\" python setup.py develop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!cd vision/classification_and_detection; python setup.py develop"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Set Working Directory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%cd vision/classification_and_detection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Download data\n",
    "\n",
    "For this example, the ImageNet and/or COCO validation data should already be on the host system. See the [MLPerf Image Classification task](https://github.com/mlcommons/inference/tree/master/vision/classification_and_detection#datasets) for more details on obtaining this. For the following step each validation dataset is stored in /workspace/data/. You should change this to the location in your setup."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "mkdir data\n",
    "ln -s /workspace/data/imagenet2012 data/\n",
    "ln -s /workspace/data/coco data/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Download models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "mkdir models\n",
    "\n",
    "# resnet50\n",
    "wget -q https://zenodo.org/record/2535873/files/resnet50_v1.pb -O models/resnet50_v1.pb \n",
    "wget -q https://zenodo.org/record/2592612/files/resnet50_v1.onnx -O models/resnet50_v1.onnx\n",
    "\n",
    "# ssd-mobilenet\n",
    "wget -q http://download.tensorflow.org/models/object_detection/ssd_mobilenet_v1_coco_2018_01_28.tar.gz -O models/ssd_mobilenet_v1_coco_2018_01_28.tar.gz\n",
    "tar zxvf ./models/ssd_mobilenet_v1_coco_2018_01_28.tar.gz -C ./models; mv models/ssd_mobilenet_v1_coco_2018_01_28/frozen_inference_graph.pb ./models/ssd_mobilenet_v1_coco_2018_01_28.pb\n",
    "wget -q https://zenodo.org/record/3163026/files/ssd_mobilenet_v1_coco_2018_01_28.onnx -O models/ssd_mobilenet_v1_coco_2018_01_28.onnx \n",
    "\n",
    "# ssd-resnet34\n",
    "wget -q https://zenodo.org/record/3345892/files/tf_ssd_resnet34_22.1.zip -O models/tf_ssd_resnet34_22.1.zip\n",
    "unzip ./models/tf_ssd_resnet34_22.1.zip -d ./models; mv models/tf_ssd_resnet34_22.1/resnet34_tf.22.1.pb ./models\n",
    "wget -q https://zenodo.org/record/3228411/files/resnet34-ssd1200.onnx -O models/resnet34-ssd1200.onnx"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run benchmarks using the reference implementation\n",
    "\n",
    "Lets prepare a submission for ResNet-50 on a cloud datacenter server with a NVIDIA T4 GPU using TensorFlow. \n",
    "\n",
    "The following script will run those combinations and prepare a submission directory, following the general submission rules documented [here](https://github.com/mlcommons/policies/blob/master/submission_rules.adoc)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import logging\n",
    "import os\n",
    "logger = logging.getLogger()\n",
    "logger.setLevel(logging.CRITICAL)\n",
    "\n",
    "os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3' \n",
    "os.environ['CUDA_VISIBLE_DEVICES'] = \"0\"\n",
    "\n",
    "# final results go here\n",
    "ORG = \"mlperf-org\"\n",
    "DIVISION = \"closed\"\n",
    "SUBMISSION_ROOT = \"/tmp/mlperf-submission\"\n",
    "SUBMISSION_DIR = os.path.join(SUBMISSION_ROOT, DIVISION, ORG)\n",
    "os.environ['SUBMISSION_ROOT'] = SUBMISSION_ROOT\n",
    "os.environ['SUBMISSION_DIR'] = SUBMISSION_DIR\n",
    "os.makedirs(SUBMISSION_DIR, exist_ok=True)\n",
    "os.makedirs(os.path.join(SUBMISSION_DIR, \"measurements\"), exist_ok=True)\n",
    "os.makedirs(os.path.join(SUBMISSION_DIR, \"code\"), exist_ok=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "====== resnet50/SingleStream =====\n",
      "TestScenario.SingleStream qps=7322.28, mean=0.0078, time=6.828, acc=76.456%, queries=50000, tiles=50.0:0.0077,80.0:0.0078,90.0:0.0078,95.0:0.0079,99.0:0.0131,99.9:0.0135\n",
      "accuracy=76.456%, good=38228, total=50000\n",
      "TestScenario.SingleStream qps=125.88, mean=0.0079, time=600.138, queries=75546, tiles=50.0:0.0079,80.0:0.0080,90.0:0.0080,95.0:0.0081,99.0:0.0081,99.9:0.0082\n",
      "====== resnet50/Server =====\n",
      "TestScenario.Server qps=7528.79, mean=0.0832, time=6.641, acc=76.456%, queries=50000, tiles=50.0:0.0809,80.0:0.0922,90.0:0.0932,95.0:0.0941,99.0:0.0963,99.9:0.1022\n",
      "accuracy=76.456%, good=38228, total=50000\n",
      "TestScenario.Server qps=128.84, mean=116.7138, time=2098.285, queries=270336, tiles=50.0:115.9511,80.0:185.2868,90.0:209.0362,95.0:220.8464,99.0:230.0520,99.9:231.5965\n",
      "====== resnet50/Offline =====\n",
      "TestScenario.Offline qps=2008.52, mean=0.3050, time=3.112, acc=76.456%, queries=6250, tiles=50.0:0.3017,80.0:0.3416,90.0:0.3465,95.0:0.3525,99.0:0.3646,99.9:1.2464\n",
      "accuracy=76.456%, good=38228, total=50000\n",
      "TestScenario.Offline qps=285.33, mean=1157.2775, time=2313.086, queries=660000, tiles=50.0:1157.2701,80.0:1850.5871,90.0:2081.7068,95.0:2197.3040,99.0:2289.7431,99.9:2310.5646\n",
      "====== resnet50/MultiStream =====\n",
      "TestScenario.MultiStream qps=1891.35, mean=0.0879, time=3.357, acc=76.447%, queries=6350, tiles=50.0:0.1002,80.0:0.1265,90.0:0.1311,95.0:0.1321,99.0:0.1356,99.9:0.1422\n",
      "accuracy=76.456%, good=38228, total=50000\n",
      "TestScenario.MultiStream qps=266.63, mean=0.0904, time=40555.550, queries=10813440, tiles=50.0:0.1050,80.0:0.1289,90.0:0.1369,95.0:0.1376,99.0:0.1386,99.9:0.1399\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:main:Namespace(accuracy=True, backend='tensorflow', cache=0, count=None, data_format=None, dataset='imagenet', dataset_list=None, dataset_path='/workspace/inference/vision/classification_and_detection/data/imagenet2012', find_peak_performance=False, inputs=['input_tensor:0'], max_batchsize=8, max_latency=0.0005, mlperf_conf='../../mlperf.conf', model='/workspace/inference/vision/classification_and_detection/models/resnet50_v1.pb', model_name='resnet50', output='/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/SingleStream/accuracy', outputs=['ArgMax:0'], profile='resnet50-tf', qps=145, samples_per_query=40, scenario='SingleStream', threads=2, time=None, user_conf='user.conf')\n",
      "INFO:imagenet:loaded 50000 images, cache=0, took=419.7sec\n",
      "INFO:main:starting TestScenario.SingleStream\n",
      "INFO:main:Namespace(accuracy=False, backend='tensorflow', cache=0, count=None, data_format=None, dataset='imagenet', dataset_list=None, dataset_path='/workspace/inference/vision/classification_and_detection/data/imagenet2012', find_peak_performance=False, inputs=['input_tensor:0'], max_batchsize=8, max_latency=0.0005, mlperf_conf='../../mlperf.conf', model='/workspace/inference/vision/classification_and_detection/models/resnet50_v1.pb', model_name='resnet50', output='/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/SingleStream/performance/run_1', outputs=['ArgMax:0'], profile='resnet50-tf', qps=145, samples_per_query=40, scenario='SingleStream', threads=2, time=None, user_conf='user.conf')\n",
      "INFO:imagenet:loaded 50000 images, cache=0, took=1.1sec\n",
      "INFO:main:starting TestScenario.SingleStream\n",
      "INFO:main:Namespace(accuracy=True, backend='tensorflow', cache=0, count=None, data_format=None, dataset='imagenet', dataset_list=None, dataset_path='/workspace/inference/vision/classification_and_detection/data/imagenet2012', find_peak_performance=False, inputs=['input_tensor:0'], max_batchsize=8, max_latency=None, mlperf_conf='../../mlperf.conf', model='/workspace/inference/vision/classification_and_detection/models/resnet50_v1.pb', model_name='resnet50', output='/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/Server/accuracy', outputs=['ArgMax:0'], profile='resnet50-tf', qps=145, samples_per_query=40, scenario='Server', threads=2, time=None, user_conf='user.conf')\n",
      "INFO:imagenet:loaded 50000 images, cache=0, took=1.1sec\n",
      "INFO:main:starting TestScenario.Server\n",
      "INFO:main:Namespace(accuracy=False, backend='tensorflow', cache=0, count=None, data_format=None, dataset='imagenet', dataset_list=None, dataset_path='/workspace/inference/vision/classification_and_detection/data/imagenet2012', find_peak_performance=False, inputs=['input_tensor:0'], max_batchsize=8, max_latency=None, mlperf_conf='../../mlperf.conf', model='/workspace/inference/vision/classification_and_detection/models/resnet50_v1.pb', model_name='resnet50', output='/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/Server/performance/run_1', outputs=['ArgMax:0'], profile='resnet50-tf', qps=145, samples_per_query=40, scenario='Server', threads=2, time=None, user_conf='user.conf')\n",
      "INFO:imagenet:loaded 50000 images, cache=0, took=1.1sec\n",
      "INFO:main:starting TestScenario.Server\n",
      "INFO:main:Namespace(accuracy=True, backend='tensorflow', cache=0, count=None, data_format=None, dataset='imagenet', dataset_list=None, dataset_path='/workspace/inference/vision/classification_and_detection/data/imagenet2012', find_peak_performance=False, inputs=['input_tensor:0'], max_batchsize=8, max_latency=None, mlperf_conf='../../mlperf.conf', model='/workspace/inference/vision/classification_and_detection/models/resnet50_v1.pb', model_name='resnet50', output='/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/Offline/accuracy', outputs=['ArgMax:0'], profile='resnet50-tf', qps=1000, samples_per_query=40, scenario='Offline', threads=2, time=None, user_conf='user.conf')\n",
      "INFO:imagenet:loaded 50000 images, cache=0, took=1.1sec\n",
      "INFO:main:starting TestScenario.Offline\n",
      "INFO:main:Namespace(accuracy=False, backend='tensorflow', cache=0, count=None, data_format=None, dataset='imagenet', dataset_list=None, dataset_path='/workspace/inference/vision/classification_and_detection/data/imagenet2012', find_peak_performance=False, inputs=['input_tensor:0'], max_batchsize=8, max_latency=None, mlperf_conf='../../mlperf.conf', model='/workspace/inference/vision/classification_and_detection/models/resnet50_v1.pb', model_name='resnet50', output='/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/Offline/performance/run_1', outputs=['ArgMax:0'], profile='resnet50-tf', qps=1000, samples_per_query=40, scenario='Offline', threads=2, time=None, user_conf='user.conf')\n",
      "INFO:imagenet:loaded 50000 images, cache=0, took=1.1sec\n",
      "INFO:main:starting TestScenario.Offline\n",
      "INFO:main:Namespace(accuracy=True, backend='tensorflow', cache=0, count=None, data_format=None, dataset='imagenet', dataset_list=None, dataset_path='/workspace/inference/vision/classification_and_detection/data/imagenet2012', find_peak_performance=False, inputs=['input_tensor:0'], max_batchsize=8, max_latency=None, mlperf_conf='../../mlperf.conf', model='/workspace/inference/vision/classification_and_detection/models/resnet50_v1.pb', model_name='resnet50', output='/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/MultiStream/accuracy', outputs=['ArgMax:0'], profile='resnet50-tf', qps=145, samples_per_query=40, scenario='MultiStream', threads=2, time=None, user_conf='user.conf')\n",
      "INFO:imagenet:loaded 50000 images, cache=0, took=1.1sec\n",
      "INFO:main:starting TestScenario.MultiStream\n",
      "INFO:main:Namespace(accuracy=False, backend='tensorflow', cache=0, count=None, data_format=None, dataset='imagenet', dataset_list=None, dataset_path='/workspace/inference/vision/classification_and_detection/data/imagenet2012', find_peak_performance=False, inputs=['input_tensor:0'], max_batchsize=8, max_latency=None, mlperf_conf='../../mlperf.conf', model='/workspace/inference/vision/classification_and_detection/models/resnet50_v1.pb', model_name='resnet50', output='/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/MultiStream/performance/run_1', outputs=['ArgMax:0'], profile='resnet50-tf', qps=145, samples_per_query=40, scenario='MultiStream', threads=2, time=None, user_conf='user.conf')\n",
      "INFO:imagenet:loaded 50000 images, cache=0, took=1.1sec\n",
      "INFO:main:starting TestScenario.MultiStream\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "# where to find stuff\n",
    "export DATA_ROOT=`pwd`/data\n",
    "export MODEL_DIR=`pwd`/models\n",
    "\n",
    "# options for official runs\n",
    "gopt=\"--max-batchsize 8 --samples-per-query 40 --threads 2 --qps 145\"\n",
    "\n",
    "\n",
    "function one_run {\n",
    "    # args: mode count framework device model ...\n",
    "    scenario=$1; shift\n",
    "    count=$1; shift\n",
    "    framework=$1\n",
    "    device=$2\n",
    "    model=$3\n",
    "    system_id=$framework-$device\n",
    "    echo \"====== $model/$scenario =====\"\n",
    "\n",
    "    case $model in \n",
    "    resnet50) \n",
    "        cmd=\"tools/accuracy-imagenet.py --imagenet-val-file $DATA_ROOT/imagenet2012/val_map.txt\"\n",
    "        offical_name=\"resnet\";;\n",
    "    ssd-mobilenet) \n",
    "        cmd=\"tools/accuracy-coco.py --coco-dir $DATA_ROOT/coco\"\n",
    "        offical_name=\"ssd-small\";;\n",
    "    ssd-resnet34) \n",
    "        cmd=\"tools/accuracy-coco.py --coco-dir $DATA_ROOT/coco\"\n",
    "        offical_name=\"ssd-large\";;\n",
    "    esac\n",
    "    output_dir=$SUBMISSION_DIR/results/$system_id/$offical_name\n",
    "    \n",
    "    # accuracy run\n",
    "    ./run_local.sh $@ --scenario $scenario --accuracy --output $output_dir/$scenario/accuracy\n",
    "    python $cmd --mlperf-accuracy-file $output_dir/$scenario/accuracy/mlperf_log_accuracy.json \\\n",
    "            >  $output_dir/$scenario/accuracy/accuracy.txt\n",
    "    cat $output_dir/$scenario/accuracy/accuracy.txt\n",
    "\n",
    "    # performance run\n",
    "    cnt=0\n",
    "    while [ $cnt -lt $count ]; do\n",
    "        let cnt=cnt+1\n",
    "        ./run_local.sh $@ --scenario $scenario --output $output_dir/$scenario/performance/run_$cnt\n",
    "    done\n",
    "    \n",
    "    # setup the measurements directory\n",
    "    mdir=$SUBMISSION_DIR/measurements/$system_id/$offical_name/$scenario\n",
    "    mkdir -p $mdir\n",
    "    cp ../../mlperf.conf $mdir\n",
    "\n",
    "    # reference app uses command line instead of user.conf\n",
    "    echo \"# empty\" > $mdir/user.conf\n",
    "    touch $mdir/README.md\n",
    "    impid=\"reference\"\n",
    "    cat > $mdir/$system_id\"_\"$impid\"_\"$scenario\".json\" <<EOF\n",
    "    {\n",
    "        \"input_data_types\": \"fp32\",\n",
    "        \"retraining\": \"none\",\n",
    "        \"starting_weights_filename\": \"https://zenodo.org/record/2535873/files/resnet50_v1.pb\",\n",
    "        \"weight_data_types\": \"fp32\",\n",
    "        \"weight_transformations\": \"none\"\n",
    "    }\n",
    "EOF\n",
    "}\n",
    "\n",
    "function one_model {\n",
    "    # args: framework device model ...\n",
    "    one_run SingleStream 1 $@ --max-latency 0.0005\n",
    "    one_run Server 1 $@\n",
    "    one_run Offline 1 $@ --qps 1000\n",
    "    one_run MultiStream 1 $@\n",
    "}\n",
    "\n",
    "\n",
    "# run image classifier benchmarks \n",
    "export DATA_DIR=$DATA_ROOT/imagenet2012\n",
    "one_model tf gpu resnet50 $gopt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There might be large trace files in the submission directory - we can delete them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "!find {SUBMISSION_DIR}/ -name mlperf_log_trace.json -delete"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Complete submission directory\n",
    "\n",
    "Add the required meta data to the submission."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "#\n",
    "# setup systems directory\n",
    "#\n",
    "if [ ! -d ${SUBMISSION_DIR}/systems ]; then\n",
    "    mkdir ${SUBMISSION_DIR}/systems\n",
    "fi\n",
    "\n",
    "cat > ${SUBMISSION_DIR}/systems/tf-gpu.json <<EOF\n",
    "{\n",
    "        \"division\": \"closed\",\n",
    "        \"status\": \"available\",\n",
    "        \"submitter\": \"mlperf-org\",\n",
    "        \"system_name\": \"tf-gpu\",\n",
    "        \"system_type\": \"datacenter\",\n",
    "        \n",
    "        \"number_of_nodes\": 1,\n",
    "        \"host_memory_capacity\": \"32GB\",\n",
    "        \"host_processor_core_count\": 1,\n",
    "        \"host_processor_frequency\": \"3.50GHz\",\n",
    "        \"host_processor_model_name\": \"Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz\",\n",
    "        \"host_processors_per_node\": 1,\n",
    "        \"host_storage_capacity\": \"512GB\",\n",
    "        \"host_storage_type\": \"SSD\",\n",
    "        \n",
    "        \"accelerator_frequency\": \"-\",\n",
    "        \"accelerator_host_interconnect\": \"-\",\n",
    "        \"accelerator_interconnect\": \"-\",\n",
    "        \"accelerator_interconnect_topology\": \"-\",\n",
    "        \"accelerator_memory_capacity\": \"16GB\",\n",
    "        \"accelerator_memory_configuration\": \"none\",\n",
    "        \"accelerator_model_name\": \"T4\",\n",
    "        \"accelerator_on-chip_memories\": \"-\",\n",
    "        \"accelerators_per_node\": 1,\n",
    "\n",
    "        \"framework\": \"v1.14.0-rc1-22-gaf24dc9\",\n",
    "        \"operating_system\": \"ubuntu-18.04\",\n",
    "        \"other_software_stack\": \"cuda-11.2\",\n",
    "        \"sw_notes\": \"\"\n",
    "}\n",
    "EOF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "#\n",
    "# setup code directory\n",
    "#\n",
    "dir=${SUBMISSION_DIR}/code/resnet/reference\n",
    "mkdir -p $dir\n",
    "echo \"git clone https://github.com/mlcommons/inference.git\" > $dir/VERSION.txt\n",
    "git rev-parse HEAD >> $dir/VERSION.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What's in the submission directory now ?\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/tmp/mlperf-submission/closed/mlperf-org/systems/tf-gpu.json\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/measurements/tf-gpu/resnet/Offline/user.conf\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/measurements/tf-gpu/resnet/Offline/mlperf.conf\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/measurements/tf-gpu/resnet/Offline/README.md\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/measurements/tf-gpu/resnet/Offline/tf-gpu_reference_Offline.json\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/measurements/tf-gpu/resnet/SingleStream/user.conf\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/measurements/tf-gpu/resnet/SingleStream/mlperf.conf\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/measurements/tf-gpu/resnet/SingleStream/tf-gpu_reference_SingleStream.json\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/measurements/tf-gpu/resnet/SingleStream/README.md\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/measurements/tf-gpu/resnet/MultiStream/user.conf\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/measurements/tf-gpu/resnet/MultiStream/mlperf.conf\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/measurements/tf-gpu/resnet/MultiStream/tf-gpu_reference_MultiStream.json\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/measurements/tf-gpu/resnet/MultiStream/README.md\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/measurements/tf-gpu/resnet/Server/user.conf\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/measurements/tf-gpu/resnet/Server/mlperf.conf\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/measurements/tf-gpu/resnet/Server/README.md\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/measurements/tf-gpu/resnet/Server/tf-gpu_reference_Server.json\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/Offline/performance/run_1/results.json\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/Offline/performance/run_1/mlperf_log_summary.txt\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/Offline/performance/run_1/mlperf_log_detail.txt\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/Offline/performance/run_1/mlperf_log_accuracy.json\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/Offline/accuracy/results.json\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/Offline/accuracy/accuracy.txt\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/Offline/accuracy/mlperf_log_summary.txt\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/Offline/accuracy/mlperf_log_detail.txt\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/Offline/accuracy/mlperf_log_accuracy.json\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/SingleStream/performance/run_1/results.json\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/SingleStream/performance/run_1/mlperf_log_summary.txt\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/SingleStream/performance/run_1/mlperf_log_detail.txt\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/SingleStream/performance/run_1/mlperf_log_accuracy.json\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/SingleStream/accuracy/results.json\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/SingleStream/accuracy/accuracy.txt\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/SingleStream/accuracy/mlperf_log_summary.txt\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/SingleStream/accuracy/mlperf_log_detail.txt\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/SingleStream/accuracy/mlperf_log_accuracy.json\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/MultiStream/performance/run_1/results.json\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/MultiStream/performance/run_1/mlperf_log_summary.txt\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/MultiStream/performance/run_1/mlperf_log_detail.txt\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/MultiStream/performance/run_1/mlperf_log_accuracy.json\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/MultiStream/accuracy/results.json\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/MultiStream/accuracy/accuracy.txt\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/MultiStream/accuracy/mlperf_log_summary.txt\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/MultiStream/accuracy/mlperf_log_detail.txt\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/MultiStream/accuracy/mlperf_log_accuracy.json\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/Server/performance/run_1/results.json\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/Server/performance/run_1/mlperf_log_summary.txt\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/Server/performance/run_1/mlperf_log_detail.txt\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/Server/performance/run_1/mlperf_log_accuracy.json\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/Server/accuracy/results.json\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/Server/accuracy/accuracy.txt\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/Server/accuracy/mlperf_log_summary.txt\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/Server/accuracy/mlperf_log_detail.txt\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/resnet/Server/accuracy/mlperf_log_accuracy.json\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/code/resnet/reference/VERSION.txt\r\n"
     ]
    }
   ],
   "source": [
    "!find {SUBMISSION_ROOT}/ -type f"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we look at some files:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-- SingleStream Accuracy\n",
      "accuracy=76.456%, good=38228, total=50000\n",
      "\n",
      "-- SingleStream Summary\n",
      "================================================\n",
      "MLPerf Results Summary\n",
      "================================================\n",
      "SUT name : PySUT\n",
      "Scenario : SingleStream\n",
      "Mode     : PerformanceOnly\n",
      "90th percentile latency (ns) : 8030958\n",
      "Result is : VALID\n",
      "  Min duration satisfied : Yes\n",
      "  Min queries satisfied : Yes\n",
      "\n",
      "-- Server Summary\n",
      "================================================\n",
      "MLPerf Results Summary\n",
      "================================================\n",
      "SUT name : PySUT\n",
      "Scenario : Server\n",
      "Mode     : PerformanceOnly\n",
      "Scheduled samples per second : 144.87\n",
      "Result is : INVALID\n",
      "  Performance constraints satisfied : NO\n",
      "  Min duration satisfied : Yes\n"
     ]
    }
   ],
   "source": [
    "!echo \"-- SingleStream Accuracy\"; head {SUBMISSION_DIR}/results/tf-gpu/resnet/SingleStream/accuracy/accuracy.txt\n",
    "!echo \"\\n-- SingleStream Summary\"; head {SUBMISSION_DIR}/results/tf-gpu/resnet/SingleStream/performance/run_1/mlperf_log_summary.txt\n",
    "!echo \"\\n-- Server Summary\"; head {SUBMISSION_DIR}/results/tf-gpu/resnet/Server/performance/run_1/mlperf_log_summary.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run the submission checker\n",
    "\n",
    "Finally, run the submission checker tool that does some sanity checking on your submission.\n",
    "We run it at the end and attach the output to the submission."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python ../../tools/submission/submission-checker.py --input {SUBMISSION_ROOT} > {SUBMISSION_DIR}/submission-checker.log 2>&1 \n",
    "!cat {SUBMISSION_DIR}/submission-checker.log"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}