Nfnet example (#849)

* Initial commit for NFNet inference example.

* updates to notebook and readme

* slice op parser changes for nfnet

* ort comparison script

* initial functional notebook example for nfnet-f0

* requirements file and update on onnx file pointer

* update intro text, add second inference for time measurement

* update to nfnet readme

* removing artifact update to parse_slice.cpp, needed updates are in other branch

* typo in nfnet readme

* update to main examples readme

* Initial commit for NFNet inference example.

* updates to notebook and readme

* slice op parser changes for nfnet

* ort comparison script

* initial functional notebook example for nfnet-f0

* requirements file and update on onnx file pointer

* update intro text, add second inference for time measurement

* update to nfnet readme

* removing artifact update to parse_slice.cpp, needed updates are in other branch

* typo in nfnet readme

* update to main examples readme

* remove duplicate import

* formatting

* updates to readme, notebook cleanup, remove not-needed requirement artifact

* Update README.md

* Apply suggestions from code review
Co-authored-by: kahmed10 <15948690+kahmed10@users.noreply.github.com>
Co-authored-by: kahmed10 <15948690+kahmed10@users.noreply.github.com>
Co-authored-by: mvermeulen <5479696+mvermeulen@users.noreply.github.com>

examples/README.md

@@ -12,4 +12,6 @@ This directory contains examples of common use cases for MIGraphX.
 - [MIGraphX Driver](./migraphx_driver)
 - [Python Resnet50 Inference](./python_api_inference)
 - [Python BERT SQuAD Inference](./python_bert_squad_example)
-- [Python Super Resolution](./python_super_resolution)
+- [Python Super Resolution](./python_super_resolution)
\ No newline at end of file
+- [Python NFNet Inference](./python_nfnet_inference)
+

examples/python_nfnet_inference/README.md

+# NFNet Inference with MIGraphX
+
+## NFNet
+NFNet: Normalizer-Free Nets. An image recognition model that can be trained without batch normalization layers. It instead uses gradient clipping algorithm to provide same affects of BatchNorm.
+
+<ins>**Summary:**</ins>
+- SOTA on ImageNet (86.5% top-1 w/o extra data)
+- Up to 8.7x faster to train than EfficientNets to a given accuracy
+- Normalizer-free (no BatchNorm)
+
+**Paper**: https://arxiv.org/pdf/2102.06171.pdf
+
+**Colab notebook**: https://github.com/deepmind/deepmind-research/tree/master/nfnets
+
+### Why not batch norm?
+
+Batch normalization has three significant practical disadvantages:
+1. It is an expensive computational primitive, which incurs memory overhead and significantly increases the time required to evaluate the gradient in some networks.
+2. It introduces a discrepancy between the behavior of the model during training and at inference time, introducing hidden hyper-parameters that have to be tuned.
+3. Last and most important point, batch normalization breaks the independence between training examples in the minibatch (batch size matters with batch norm, distributed training becomes extremely cumbersome).
+
+Instead:
+
+- Authors provide Adaptive Gradient Clipping (AGC), which clips gradients based on the unit-wise ratio of gradient norms to parameter norms, and they demonstrate that AGC allows them to train normalizer-free networks with larger batch sizes and stronger data augmentations.
+- They design a family of Normalizer-Free ResNets, called NFNets, which set new state-of-the-art validation accuracies on ImageNet for a range of training latencies. Their NFNet-F1 model achieves similar accuracy to EfficientNet-B7 while being 8.7× faster to train, and their largest model sets a new overall state of the art without extra data of 86.5% top-1 accuracy.
+- They show that NFNets achieve substantially higher validation accuracies than batch-normalized networks when fine-tuning on ImageNet after pre-training on a large private dataset of 300 million labelled images. Their best model achieves 89.2% top-1 accuracy after fine-tuning.
+
+## Inference with MIGraphX using NFNet ONNX Model
+
+There is no ONNX model released for NFNet, as of June 2021, however a PyTorch model is available at:
+https://github.com/rwightman/pytorch-image-models. 
+We provide an in-house produced and optimized ONNX model, which can be parsed and compiled using MIGraphX for AMD GPUs. The ONNX model file can be fetched using the Jupyter notebook we provide.
+
+### Requirements:
+1) AMD GPU system with ROCm installed.
+2) Jupyter notebook library.
+
+### How to use NFNet for image recognition:
+Please utilize the notebook example provided:
+1) Install jupyter notebook to your environment if not already installed:
+```
+https://jupyter.org/install
+```
+2) Connect to your jupyter server and utilize `nfnet_inference.ipynb` notebook file.
+
+### How to compare MIGraphX to ONNX Runtime for NFNet ONNX model:
+First install requirements:
+```
+pip3 install -r requirements_nfnet.txt
+```
+
+On your terminal, invoke:
+```
+python3 ort_comparison.py
+````

examples/python_nfnet_inference/nfnet_inference.ipynb

+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# NFNet Inference with AMD MIGraphX\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Normalizer-Free ResNet is a new residual convolutional network providing new state-of-the-art Top-1 accuracy of 86.5% at ImageNet dataset. The most important feature of the model is removing batch normalization. Instead of batch normalization, it uses adaptive gradient clipping to provide same regularization effect of BatchNorm. <br> Details of this network: https://arxiv.org/abs/2102.06171\n",
+    "\n",
+    "In this notebook, we are showing: <br>\n",
+    "- How to optimize NFNet ONNX model with AMD MIGraphX.\n",
+    "- How to run inference on AMD GPU with the optimized ONNX model.\n",
+    "\n",
+    "The NFNet utilized in this example is the smallest NFNet version, F0: 71.5M parameters (83.6% top-1 accuracy on ImageNet)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Requirements"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!apt-get update\n",
+    "!apt-get install ffmpeg libsm6 libxext6  -y "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!pip3 install --upgrade pip\n",
+    "!pip3 install -r requirements_nfnet.txt"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import numpy as np\n",
+    "import cv2\n",
+    "import json\n",
+    "from PIL import Image\n",
+    "import time\n",
+    "from os import path "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Importing AMD MIGraphX Python Module"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import migraphx"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Create NFNet ONNX file\n",
+    "Following repository provides functionality to create NFNet ONNX file from PyTorch model."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!wget -nc https://www.dropbox.com/s/u4ga8zyxtppfzxc/dm_nfnet_f0.onnx"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Load ImageNet labels"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "with open('../python_api_inference/imagenet_simple_labels.json') as json_data:\n",
+    "    labels = json.load(json_data)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "## Load ONNX model using MIGraphX"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "model = migraphx.parse_onnx(\"dm_nfnet_f0.onnx\")\n",
+    "model.compile(migraphx.get_target(\"gpu\"))\n",
+    "\n",
+    "print(model.get_parameter_names())\n",
+    "print(model.get_parameter_shapes())\n",
+    "print(model.get_output_shapes())"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Functions for image processing"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def make_nxn(image, n):\n",
+    "    height, width = image.shape[:2]    \n",
+    "    if height > width:\n",
+    "        dif = height - width\n",
+    "        bar = dif // 2 \n",
+    "        square = image[(bar + (dif % 2)):(height - bar),:]\n",
+    "        return cv2.resize(square, (n, n))\n",
+    "    elif width > height:\n",
+    "        dif = width - height\n",
+    "        bar = dif // 2\n",
+    "        square = image[:,(bar + (dif % 2)):(width - bar)]\n",
+    "        return cv2.resize(square, (n, n))\n",
+    "    else:\n",
+    "        return cv2.resize(image, (n, n))\n",
+    "    \n",
+    "def preprocess(img_data):\n",
+    "    mean_vec = np.array([0.485, 0.456, 0.406])\n",
+    "    stddev_vec = np.array([0.229, 0.224, 0.225])\n",
+    "    norm_img_data = np.zeros(img_data.shape).astype('float32')\n",
+    "    for i in range(img_data.shape[0]):  \n",
+    "        norm_img_data[i,:,:] = (img_data[i,:,:]/255 - mean_vec[i]) / stddev_vec[i]\n",
+    "    return norm_img_data\n",
+    "\n",
+    "def input_process(frame, dim):\n",
+    "    # Crop and resize original image\n",
+    "    cropped = make_nxn(frame, dim)\n",
+    "    # Convert from HWC to CHW\n",
+    "    chw = cropped.transpose(2,0,1)\n",
+    "    # Apply normalization\n",
+    "    pp = preprocess(chw)\n",
+    "    # Add singleton dimension (CHW to NCHW)\n",
+    "    data = np.expand_dims(pp.astype('float32'),0)\n",
+    "    return data"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Download example image"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Fetch example image: traffic light\n",
+    "!wget -nc http://farm5.static.flickr.com/4072/4462811418_8bc2bd42ca_z_d.jpg -O traffic_light.jpg\n",
+    "# Read the image\n",
+    "im = cv2.imread('traffic_light.jpg')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Process the read image to conform input requirements\n",
+    "data_input = input_process(im, 192)\n",
+    "\n",
+    "# Run the model\n",
+    "start = time.time()\n",
+    "results = model.run({'inputs':data_input}) # Your first inference would take longer than the following ones.\n",
+    "print(f\"Time inference took: {100*(time.time() - start):.2f}ms\")\n",
+    "# Extract the index of the top prediction\n",
+    "res_npa = np.array(results[0])\n",
+    "print(f\"\\nResult: {labels[np.argmax(res_npa)]}\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Run the model again, first one would take long\n",
+    "start = time.time()\n",
+    "results = model.run({'inputs':data_input}) # Your first inference would take longer than the following ones.\n",
+    "print(f\"Time inference took: {100*(time.time() - start):.2f}ms\")\n",
+    "# Extract the index of the top prediction\n",
+    "res_npa = np.array(results[0])\n",
+    "print(f\"\\nResult: {labels[np.argmax(res_npa)]}\")"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.6.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}

examples/python_nfnet_inference/ort_comparison.py

+import numpy
+import onnxruntime as rt
+
+sess = rt.InferenceSession("dm_nfnet_f0.onnx")
+
+input_name = sess.get_inputs()[0].name
+print("input name", input_name)
+input_shape = sess.get_inputs()[0].shape
+print("input shape", input_shape)
+input_type = sess.get_inputs()[0].type
+print("input type", input_type)
+
+output_name = sess.get_outputs()[0].name
+print("output name", output_name)
+output_shape = sess.get_outputs()[0].shape
+print("output shape", output_shape)
+output_type = sess.get_outputs()[0].type
+print("output type", output_type)
+
+x = numpy.random.random((1, 3, 192, 192))
+x = x.astype(numpy.float32)
+
+import migraphx
+model = migraphx.parse_onnx("dm_nfnet_f0.onnx")
+model.compile(migraphx.get_target("gpu"))
+print(model.get_parameter_names())
+print(model.get_parameter_shapes())
+print(model.get_output_shapes())
+
+result_migraphx = model.run({"inputs": x})
+result_ort = sess.run([output_name], {input_name: x})
+
+result_migraphx = result_migraphx[0].tolist()
+
+for i in range(10):
+    x = numpy.random.random((1, 3, 192, 192))
+    x = x.astype(numpy.float32)
+
+    result_migraphx = model.run({"inputs": x})
+    result_ort = sess.run([output_name], {input_name: x})
+
+    try:
+        numpy.testing.assert_allclose(result_migraphx[0].tolist(),
+                                      result_ort[0][0],
+                                      rtol=1e-02)
+        print(f"Test #{i} completed.")
+    except AssertionError as e:
+        print(e)

examples/python_nfnet_inference/requirements_nfnet.txt

+opencv-python
+onnxruntime
\ No newline at end of file