Super resolution example (#789)

* initial push for super resolution: notebook + image + requirements

* first version of working notebook

* cleaning up notebook

* added super resolution to main examples readme file

* readme file for super resolution example

* torch is not needed, removed from requirements

* removed not-needed imports and code from notebook

* update to super resolution readme

* New input image. Added PSNR. Related code changes in notebook

* readme file update for step-by-step instructions

* opencv requirement moved to requirements file. updated notebook accordingly

* update to main examples readme file

* typo in readme
Co-authored-by: kahmed10 <15948690+kahmed10@users.noreply.github.com>
Co-authored-by: mvermeulen <5479696+mvermeulen@users.noreply.github.com>

examples/README.md

@@ -11,4 +11,5 @@ This directory contains examples of common use cases for MIGraphX.
 - [MIGraphX Docker Container](./migraphx_docker)
 - [MIGraphX Driver](./migraphx_driver)
 - [Python Resnet50 Inference](./python_api_inference)
-- [Python BERT SQuAD Inference](./python_bert_squad_example)
\ No newline at end of file
+- [Python BERT SQuAD Inference](./python_bert_squad_example)
+- [Python Super Resolution](./python_super_resolution)
\ No newline at end of file

examples/python_super_resolution/README.md

+# Super Resolution with AMD MIGraphX
+
+This example is based on [ONNX run_super_resolution_model notebook](https://github.com/onnx/models/blob/master/vision/super_resolution/sub_pixel_cnn_2016/dependencies/Run_Super_Resolution_Model.ipynb) and modified for MIGraphX.
+
+## Description
+Given an input image, this application resizes the image to 224x224 and then scales it to 672x672, thus it is useful for upscaling low-resolution images.
+
+### Model Utilized
+> "Super Resolution uses efficient  [Sub-pixel convolutional layer](https://arxiv.org/abs/1609.05158) described for increasing spatial resolution within network tasks. By increasing pixel count, images are then clarified, sharpened, and upscaled without losing the input image’s content and characteristics." [[Reference]](https://github.com/onnx/models/blob/master/vision/super_resolution/sub_pixel_cnn_2016/README.md)
+
+Model in PyTorch definitions:
+```
+self.relu = nn.ReLU(inplace=inplace)
+self.conv1 = nn.Conv2d(1, 64, (5, 5), (1, 1), (2, 2))
+self.conv2 = nn.Conv2d(64, 64, (3, 3), (1, 1), (1, 1))
+self.conv3 = nn.Conv2d(64, 32, (3, 3), (1, 1), (1, 1))
+self.conv4 = nn.Conv2d(32, upscale_factor ** 2, (3, 3), (1, 1), (1, 1))
+self.pixel_shuffle = nn.PixelShuffle(upscale_factor)
+```
+## How-to
+If you have jupyter installed, you can simply use the notebook given. Otherwise please follow the step-by-step guide.
+### Jupyter Notebook
+Run Jupyter notebook server on a ROCm and MIGraphX installed system, and run `Run_Super_Resolution_Model.ipynb`
+
+### Step by Step
+1) Upgrade pip3. You may skip this stage if you already have latest pip3. This step is needed for OpenCV installation.
+```
+pip3 install --upgrade pip
+```
+2) Install requirements.
+```
+pip3 install -r requirements.txt
+```
+3) Import required libraries.
+```
+import numpy as np
+import matplotlib.pyplot as plt
+from PIL import Image, ImageDraw, ImageFont
+from resizeimage import resizeimage
+```
+4) Download ONNX model.
+```
+wget -nc https://github.com/onnx/models/raw/master/vision/super_resolution/sub_pixel_cnn_2016/model/super-resolution-10.onnx
+```
+5) Preprocess the sample image `cat.jpg`.
+```
+orig_img = Image.open("./cat.jpg")
+print(orig_img.size)
+img = resizeimage.resize_cover(orig_img, [224,224], validate=False)
+img_ycbcr = img.convert('YCbCr')
+img_y_0, img_cb, img_cr = img_ycbcr.split()
+img_ndarray = np.asarray(img_y_0)
+
+img_4 = np.expand_dims(np.expand_dims(img_ndarray, axis=0), axis=0)
+img_5 = img_4.astype(np.float32) / 255.0
+```
+6) Import MIGraphX, parse & compile the ONNX model with MIGraphX. Print the model.
+```
+model = migraphx.parse_onnx("super-resolution-10.onnx")
+model.compile(migraphx.get_target("gpu"))
+model.print()
+```
+7) You can check the model inputs and outputs with the following functions.
+```
+print(model.get_parameter_names())
+print(model.get_parameter_shapes())
+print(model.get_output_shapes())
+```
+8) Run the image throgh model and get the output data.
+```
+result = model.run({
+         "input": img_5
+     })
+
+data = np.array(result[0])[0]
+```
+9) Post processing image. If matplotlib is installed correctly, it should show up the image. The output image will be stored with filename `output.jpg`.
+```
+img_out_y = Image.fromarray(np.uint8((data* 255.0).clip(0, 255)[0]), mode='L')
+# get the output image follow post-processing step from PyTorch implementation
+final_img = Image.merge(
+    "YCbCr", [
+        img_out_y,
+        img_cb.resize(img_out_y.size, Image.BICUBIC),
+        img_cr.resize(img_out_y.size, Image.BICUBIC),
+    ]).convert("RGB")
+final_img.save("output.jpg")
+print(final_img.size)
+```
+10) Measure the improvement in terms of PSNR and show the both input and super-resolution image:
+```
+import cv2
+
+imgIN = cv2.imread('cat.jpg')
+imgOUT = cv2.imread('output.jpg')
+imgIN = cv2.cvtColor(imgIN, cv2.COLOR_BGR2RGB) #BGR to RGB
+imgOUT = cv2.cvtColor(imgOUT, cv2.COLOR_BGR2RGB)
+
+imgIN_resized = cv2.resize(imgIN, (672,672)) #Resizing input to 672
+
+psnr = cv2.PSNR(imgIN_resized, imgOUT) #dimensions need to be same
+print("PSNR Value = %.3f db"%psnr)
+
+fig = plt.figure(figsize=(16, 16))
+sp1 = fig.add_subplot(1, 2, 1)
+sp1.title.set_text('Output Super Resolution Image (%sx%s)'%(imgOUT.shape[0], imgOUT.shape[1]))
+plt.imshow(imgOUT)
+
+sp2 = fig.add_subplot(1, 2, 2)
+sp2.title.set_text('Input Image (%sx%s)'%(imgIN.shape[0], imgIN.shape[1]))
+plt.imshow(imgIN)
+plt.show()
+```
\ No newline at end of file

examples/python_super_resolution/Run_Super_Resolution_Model.ipynb

+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Super Resolution Inference with AMD MIGraphX\n",
+    "This notebook is inspired from: https://github.com/onnx/models/blob/master/vision/super_resolution/sub_pixel_cnn_2016/dependencies/Run_Super_Resolution_Model.ipynb"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Install Dependencies"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!pip3 install --upgrade pip #needed for opencv-python installation\n",
+    "!pip3 install -r requirements.txt"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import numpy as np\n",
+    "import matplotlib.pyplot as plt\n",
+    "from PIL import Image, ImageDraw, ImageFont\n",
+    "from resizeimage import resizeimage\n",
+    "%matplotlib inline"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Download ONNX Model"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!wget -nc https://github.com/onnx/models/raw/master/vision/super_resolution/sub_pixel_cnn_2016/model/super-resolution-10.onnx"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Import MIGraphX Python Module"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import migraphx"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Preprocessing Image"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "orig_img = Image.open(\"./cat.jpg\")\n",
+    "print(orig_img.size)\n",
+    "img = resizeimage.resize_cover(orig_img, [224,224], validate=False)\n",
+    "img_ycbcr = img.convert('YCbCr')\n",
+    "img_y_0, img_cb, img_cr = img_ycbcr.split()\n",
+    "img_ndarray = np.asarray(img_y_0)\n",
+    "\n",
+    "img_4 = np.expand_dims(np.expand_dims(img_ndarray, axis=0), axis=0)\n",
+    "img_5 = img_4.astype(np.float32) / 255.0\n",
+    "img_5"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Run Model"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "model = migraphx.parse_onnx(\"super-resolution-10.onnx\")\n",
+    "model.compile(migraphx.get_target(\"gpu\"))\n",
+    "#model.print()\n",
+    "\n",
+    "print(model.get_parameter_names())\n",
+    "print(model.get_parameter_shapes())\n",
+    "print(model.get_output_shapes())\n",
+    "\n",
+    "\n",
+    "result = model.run({\n",
+    "         \"input\": img_5\n",
+    "     })\n",
+    "\n",
+    "data = np.array(result[0])[0]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Postprocessing Image"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "img_out_y = Image.fromarray(np.uint8((data* 255.0).clip(0, 255)[0]), mode='L')\n",
+    "# get the output image follow post-processing step from PyTorch implementation\n",
+    "final_img = Image.merge(\n",
+    "    \"YCbCr\", [\n",
+    "        img_out_y,\n",
+    "        img_cb.resize(img_out_y.size, Image.BICUBIC),\n",
+    "        img_cr.resize(img_out_y.size, Image.BICUBIC),\n",
+    "    ]).convert(\"RGB\")\n",
+    "final_img.save(\"output.jpg\")\n",
+    "print(final_img.size)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "## PSNR Comparison Output vs Input"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import cv2\n",
+    "\n",
+    "imgIN = cv2.imread('cat.jpg')\n",
+    "imgOUT = cv2.imread('output.jpg')\n",
+    "imgIN = cv2.cvtColor(imgIN, cv2.COLOR_BGR2RGB) #BGR to RGB\n",
+    "imgOUT = cv2.cvtColor(imgOUT, cv2.COLOR_BGR2RGB)\n",
+    "\n",
+    "imgIN_resized = cv2.resize(imgIN, (672,672)) #Resizing input to 672\n",
+    "\n",
+    "psnr = cv2.PSNR(imgIN_resized, imgOUT) #dimensions need to be same\n",
+    "print(\"PSNR Value = %.3f db\"%psnr)\n",
+    "\n",
+    "fig = plt.figure(figsize=(16, 16))\n",
+    "sp1 = fig.add_subplot(1, 2, 1)\n",
+    "sp1.title.set_text('Output Super Resolution Image (%sx%s)'%(imgOUT.shape[0], imgOUT.shape[1]))\n",
+    "plt.imshow(imgOUT)\n",
+    "\n",
+    "sp2 = fig.add_subplot(1, 2, 2)\n",
+    "sp2.title.set_text('Input Image (%sx%s)'%(imgIN.shape[0], imgIN.shape[1]))\n",
+    "plt.imshow(imgIN)\n",
+    "plt.show()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## "
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.6.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

examples/python_super_resolution/requirements.txt

+matplotlib
+python-resize-image
+opencv-python
\ No newline at end of file