{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rOvvWAVTkMR7"
      },
      "source": [
        "# Waste identification with instance segmentation in TensorFlow\n",
        "\n",
        "Welcome to the Instance Segmentation Colab! This notebook will take you through the steps of running an \"out-of-the-box\" Mask RCNN Instance Segmentation model on images."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HVTXSC07QwfG"
      },
      "source": [
        "Given 3 different Mask RCNN models for the material type, material form type and plastic type, your goal is to inference with any of the models and visualize the results. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AQUsAE0TRkmh"
      },
      "source": [
        "To finish this task, a proper path for the TF Lite models and a single image needs to be provided. The path to the labels on which the models are trained is in the  waste_identification_ml directory inside the Tensorflow Model Garden repository. The label files are inferred automatically once you select the ML model by which you want to do the inference."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vPs64QA1Zdov"
      },
      "source": [
        "## Imports and Setup\n",
        "\n",
        "Let's start with the base imports."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Xk4FU-jx9kc3"
      },
      "outputs": [],
      "source": [
        "# install model-garden official\n",
        "!pip install tf-models-official"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "yn5_uV1HLvaz"
      },
      "outputs": [],
      "source": [
        "import cv2\n",
        "\n",
        "import matplotlib\n",
        "import matplotlib.pyplot as plt\n",
        "from pprint import pprint\n",
        "\n",
        "import numpy as np\n",
        "from six import BytesIO\n",
        "from PIL import Image\n",
        "from six.moves.urllib.request import urlopen\n",
        "\n",
        "from official.vision.ops.preprocess_ops import normalize_image\n",
        "\n",
        "import tensorflow as tf"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "14bNk1gzh0TN"
      },
      "source": [
        "## Visualization tools\n",
        "\n",
        "To visualize the images with the proper detected boxes and segmentation masks, we will use the TensorFlow Object Detection API. To install it we will clone the repo."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "oi28cqGGFWnY",
        "outputId": "0d35a3d1-0615-4a69-b861-c98228bfae26"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Cloning into 'models'...\n",
            "remote: Enumerating objects: 3444, done.\u001b[K\n",
            "remote: Counting objects: 100% (3444/3444), done.\u001b[K\n",
            "remote: Compressing objects: 100% (2888/2888), done.\u001b[K\n",
            "remote: Total 3444 (delta 894), reused 1456 (delta 499), pack-reused 0\u001b[K\n",
            "Receiving objects: 100% (3444/3444), 43.78 MiB | 20.57 MiB/s, done.\n",
            "Resolving deltas: 100% (894/894), done.\n"
          ]
        }
      ],
      "source": [
        "# Clone the tensorflow models repository\n",
        "!git clone --depth 1 https://github.com/tensorflow/models"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yX3pb_pXDjYA"
      },
      "source": [
        "Installing the Object Detection API"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NwdsBdGhFanc",
        "outputId": "534bf43d-6325-463f-bac6-764d0271977b"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Reading package lists...\n",
            "Building dependency tree...\n",
            "Reading state information...\n",
            "protobuf-compiler is already the newest version (3.0.0-9.1ubuntu1).\n",
            "The following package was automatically installed and is no longer required:\n",
            "  libnvidia-common-460\n",
            "Use 'sudo apt autoremove' to remove it.\n",
            "0 upgraded, 0 newly installed, 0 to remove and 19 not upgraded.\n",
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Processing /content/models/research\n",
            "Collecting avro-python3\n",
            "  Downloading avro-python3-1.10.2.tar.gz (38 kB)\n",
            "Collecting apache-beam\n",
            "  Downloading apache_beam-2.40.0-cp37-cp37m-manylinux2010_x86_64.whl (10.9 MB)\n",
            "Requirement already satisfied: pillow in /usr/local/lib/python3.7/dist-packages (from object-detection==0.1) (7.1.2)\n",
            "Requirement already satisfied: lxml in /usr/local/lib/python3.7/dist-packages (from object-detection==0.1) (4.9.1)\n",
            "Requirement already satisfied: matplotlib in /usr/local/lib/python3.7/dist-packages (from object-detection==0.1) (3.2.2)\n",
            "Requirement already satisfied: Cython in /usr/local/lib/python3.7/dist-packages (from object-detection==0.1) (0.29.32)\n",
            "Requirement already satisfied: contextlib2 in /usr/local/lib/python3.7/dist-packages (from object-detection==0.1) (0.5.5)\n",
            "Requirement already satisfied: tf-slim in /usr/local/lib/python3.7/dist-packages (from object-detection==0.1) (1.1.0)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from object-detection==0.1) (1.15.0)\n",
            "Requirement already satisfied: pycocotools in /usr/local/lib/python3.7/dist-packages (from object-detection==0.1) (2.0.4)\n",
            "Collecting lvis\n",
            "  Downloading lvis-0.5.3-py3-none-any.whl (14 kB)\n",
            "Requirement already satisfied: scipy in /usr/local/lib/python3.7/dist-packages (from object-detection==0.1) (1.7.3)\n",
            "Requirement already satisfied: pandas in /usr/local/lib/python3.7/dist-packages (from object-detection==0.1) (1.3.5)\n",
            "Requirement already satisfied: tf-models-official\u003e=2.5.1 in /usr/local/lib/python3.7/dist-packages (from object-detection==0.1) (2.9.2)\n",
            "Collecting tensorflow_io\n",
            "  Downloading tensorflow_io-0.26.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (25.9 MB)\n",
            "Requirement already satisfied: keras in /usr/local/lib/python3.7/dist-packages (from object-detection==0.1) (2.9.0)\n",
            "Collecting pyparsing==2.4.7\n",
            "  Downloading pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)\n",
            "Requirement already satisfied: gin-config in /usr/local/lib/python3.7/dist-packages (from tf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (0.5.0)\n",
            "Requirement already satisfied: seqeval in /usr/local/lib/python3.7/dist-packages (from tf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (1.2.2)\n",
            "Requirement already satisfied: sentencepiece in /usr/local/lib/python3.7/dist-packages (from tf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (0.1.97)\n",
            "Requirement already satisfied: tensorflow-model-optimization\u003e=0.4.1 in /usr/local/lib/python3.7/dist-packages (from tf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (0.7.3)\n",
            "Requirement already satisfied: numpy\u003e=1.20 in /usr/local/lib/python3.7/dist-packages (from tf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (1.21.6)\n",
            "Requirement already satisfied: tensorflow~=2.9.0 in /usr/local/lib/python3.7/dist-packages (from tf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (2.9.1)\n",
            "Requirement already satisfied: psutil\u003e=5.4.3 in /usr/local/lib/python3.7/dist-packages (from tf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (5.4.8)\n",
            "Requirement already satisfied: pyyaml\u003c6.0,\u003e=5.1 in /usr/local/lib/python3.7/dist-packages (from tf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (5.4.1)\n",
            "Requirement already satisfied: kaggle\u003e=1.3.9 in /usr/local/lib/python3.7/dist-packages (from tf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (1.5.12)\n",
            "Requirement already satisfied: py-cpuinfo\u003e=3.3.0 in /usr/local/lib/python3.7/dist-packages (from tf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (8.0.0)\n",
            "Requirement already satisfied: tensorflow-hub\u003e=0.6.0 in /usr/local/lib/python3.7/dist-packages (from tf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (0.12.0)\n",
            "Requirement already satisfied: oauth2client in /usr/local/lib/python3.7/dist-packages (from tf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (4.1.3)\n",
            "Requirement already satisfied: tensorflow-datasets in /usr/local/lib/python3.7/dist-packages (from tf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (4.6.0)\n",
            "Requirement already satisfied: tensorflow-addons in /usr/local/lib/python3.7/dist-packages (from tf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (0.17.1)\n",
            "Requirement already satisfied: google-api-python-client\u003e=1.6.7 in /usr/local/lib/python3.7/dist-packages (from tf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (1.12.11)\n",
            "Requirement already satisfied: sacrebleu in /usr/local/lib/python3.7/dist-packages (from tf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (2.2.0)\n",
            "Requirement already satisfied: tensorflow-text~=2.9.0 in /usr/local/lib/python3.7/dist-packages (from tf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (2.9.0)\n",
            "Requirement already satisfied: opencv-python-headless in /usr/local/lib/python3.7/dist-packages (from tf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (4.6.0.66)\n",
            "Requirement already satisfied: uritemplate\u003c4dev,\u003e=3.0.0 in /usr/local/lib/python3.7/dist-packages (from google-api-python-client\u003e=1.6.7-\u003etf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (3.0.1)\n",
            "Requirement already satisfied: google-api-core\u003c3dev,\u003e=1.21.0 in /usr/local/lib/python3.7/dist-packages (from google-api-python-client\u003e=1.6.7-\u003etf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (1.31.6)\n",
            "Requirement already satisfied: httplib2\u003c1dev,\u003e=0.15.0 in /usr/local/lib/python3.7/dist-packages (from google-api-python-client\u003e=1.6.7-\u003etf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (0.17.4)\n",
            "Requirement already satisfied: google-auth\u003c3dev,\u003e=1.16.0 in /usr/local/lib/python3.7/dist-packages (from google-api-python-client\u003e=1.6.7-\u003etf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (1.35.0)\n",
            "Requirement already satisfied: google-auth-httplib2\u003e=0.0.3 in /usr/local/lib/python3.7/dist-packages (from google-api-python-client\u003e=1.6.7-\u003etf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (0.0.4)\n",
            "Requirement already satisfied: requests\u003c3.0.0dev,\u003e=2.18.0 in /usr/local/lib/python3.7/dist-packages (from google-api-core\u003c3dev,\u003e=1.21.0-\u003egoogle-api-python-client\u003e=1.6.7-\u003etf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (2.23.0)\n",
            "Requirement already satisfied: setuptools\u003e=40.3.0 in /usr/local/lib/python3.7/dist-packages (from google-api-core\u003c3dev,\u003e=1.21.0-\u003egoogle-api-python-client\u003e=1.6.7-\u003etf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (57.4.0)\n",
            "Requirement already satisfied: packaging\u003e=14.3 in /usr/local/lib/python3.7/dist-packages (from google-api-core\u003c3dev,\u003e=1.21.0-\u003egoogle-api-python-client\u003e=1.6.7-\u003etf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (21.3)\n",
            "Requirement already satisfied: protobuf\u003c4.0.0dev,\u003e=3.12.0 in /usr/local/lib/python3.7/dist-packages (from google-api-core\u003c3dev,\u003e=1.21.0-\u003egoogle-api-python-client\u003e=1.6.7-\u003etf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (3.17.3)\n",
            "Requirement already satisfied: googleapis-common-protos\u003c2.0dev,\u003e=1.6.0 in /usr/local/lib/python3.7/dist-packages (from google-api-core\u003c3dev,\u003e=1.21.0-\u003egoogle-api-python-client\u003e=1.6.7-\u003etf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (1.56.4)\n",
            "Requirement already satisfied: pytz in /usr/local/lib/python3.7/dist-packages (from google-api-core\u003c3dev,\u003e=1.21.0-\u003egoogle-api-python-client\u003e=1.6.7-\u003etf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (2022.1)\n",
            "Requirement already satisfied: rsa\u003c5,\u003e=3.1.4 in /usr/local/lib/python3.7/dist-packages (from google-auth\u003c3dev,\u003e=1.16.0-\u003egoogle-api-python-client\u003e=1.6.7-\u003etf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (4.9)\n",
            "Requirement already satisfied: pyasn1-modules\u003e=0.2.1 in /usr/local/lib/python3.7/dist-packages (from google-auth\u003c3dev,\u003e=1.16.0-\u003egoogle-api-python-client\u003e=1.6.7-\u003etf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (0.2.8)\n",
            "Requirement already satisfied: cachetools\u003c5.0,\u003e=2.0.0 in /usr/local/lib/python3.7/dist-packages (from google-auth\u003c3dev,\u003e=1.16.0-\u003egoogle-api-python-client\u003e=1.6.7-\u003etf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (4.2.4)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.7/dist-packages (from kaggle\u003e=1.3.9-\u003etf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (4.64.0)\n",
            "Requirement already satisfied: urllib3 in /usr/local/lib/python3.7/dist-packages (from kaggle\u003e=1.3.9-\u003etf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (1.24.3)\n",
            "Requirement already satisfied: certifi in /usr/local/lib/python3.7/dist-packages (from kaggle\u003e=1.3.9-\u003etf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (2022.6.15)\n",
            "Requirement already satisfied: python-dateutil in /usr/local/lib/python3.7/dist-packages (from kaggle\u003e=1.3.9-\u003etf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (2.8.2)\n",
            "Requirement already satisfied: python-slugify in /usr/local/lib/python3.7/dist-packages (from kaggle\u003e=1.3.9-\u003etf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (6.1.2)\n",
            "Requirement already satisfied: pyasn1\u003c0.5.0,\u003e=0.4.6 in /usr/local/lib/python3.7/dist-packages (from pyasn1-modules\u003e=0.2.1-\u003egoogle-auth\u003c3dev,\u003e=1.16.0-\u003egoogle-api-python-client\u003e=1.6.7-\u003etf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (0.4.8)\n",
            "Requirement already satisfied: idna\u003c3,\u003e=2.5 in /usr/local/lib/python3.7/dist-packages (from requests\u003c3.0.0dev,\u003e=2.18.0-\u003egoogle-api-core\u003c3dev,\u003e=1.21.0-\u003egoogle-api-python-client\u003e=1.6.7-\u003etf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (2.10)\n",
            "Requirement already satisfied: chardet\u003c4,\u003e=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests\u003c3.0.0dev,\u003e=2.18.0-\u003egoogle-api-core\u003c3dev,\u003e=1.21.0-\u003egoogle-api-python-client\u003e=1.6.7-\u003etf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (3.0.4)\n",
            "Requirement already satisfied: libclang\u003e=13.0.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow~=2.9.0-\u003etf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (14.0.6)\n",
            "Requirement already satisfied: tensorflow-estimator\u003c2.10.0,\u003e=2.9.0rc0 in /usr/local/lib/python3.7/dist-packages (from tensorflow~=2.9.0-\u003etf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (2.9.0)\n",
            "Requirement already satisfied: absl-py\u003e=1.0.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow~=2.9.0-\u003etf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (1.2.0)\n",
            "Requirement already satisfied: gast\u003c=0.4.0,\u003e=0.2.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow~=2.9.0-\u003etf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (0.4.0)\n",
            "Requirement already satisfied: keras-preprocessing\u003e=1.1.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow~=2.9.0-\u003etf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (1.1.2)\n",
            "Requirement already satisfied: h5py\u003e=2.9.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow~=2.9.0-\u003etf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (3.1.0)\n",
            "Requirement already satisfied: tensorboard\u003c2.10,\u003e=2.9 in /usr/local/lib/python3.7/dist-packages (from tensorflow~=2.9.0-\u003etf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (2.9.1)\n",
            "Requirement already satisfied: google-pasta\u003e=0.1.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow~=2.9.0-\u003etf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (0.2.0)\n",
            "Requirement already satisfied: wrapt\u003e=1.11.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow~=2.9.0-\u003etf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (1.14.1)\n",
            "Requirement already satisfied: grpcio\u003c2.0,\u003e=1.24.3 in /usr/local/lib/python3.7/dist-packages (from tensorflow~=2.9.0-\u003etf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (1.47.0)\n",
            "Requirement already satisfied: astunparse\u003e=1.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow~=2.9.0-\u003etf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (1.6.3)\n",
            "Requirement already satisfied: flatbuffers\u003c2,\u003e=1.12 in /usr/local/lib/python3.7/dist-packages (from tensorflow~=2.9.0-\u003etf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (1.12)\n",
            "Requirement already satisfied: tensorflow-io-gcs-filesystem\u003e=0.23.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow~=2.9.0-\u003etf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (0.26.0)\n",
            "Requirement already satisfied: opt-einsum\u003e=2.3.2 in /usr/local/lib/python3.7/dist-packages (from tensorflow~=2.9.0-\u003etf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (3.3.0)\n",
            "Requirement already satisfied: termcolor\u003e=1.1.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow~=2.9.0-\u003etf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (1.1.0)\n",
            "Requirement already satisfied: typing-extensions\u003e=3.6.6 in /usr/local/lib/python3.7/dist-packages (from tensorflow~=2.9.0-\u003etf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (4.1.1)\n",
            "Requirement already satisfied: wheel\u003c1.0,\u003e=0.23.0 in /usr/local/lib/python3.7/dist-packages (from astunparse\u003e=1.6.0-\u003etensorflow~=2.9.0-\u003etf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (0.37.1)\n",
            "Requirement already satisfied: cached-property in /usr/local/lib/python3.7/dist-packages (from h5py\u003e=2.9.0-\u003etensorflow~=2.9.0-\u003etf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (1.5.2)\n",
            "Requirement already satisfied: markdown\u003e=2.6.8 in /usr/local/lib/python3.7/dist-packages (from tensorboard\u003c2.10,\u003e=2.9-\u003etensorflow~=2.9.0-\u003etf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (3.4.1)\n",
            "Requirement already satisfied: google-auth-oauthlib\u003c0.5,\u003e=0.4.1 in /usr/local/lib/python3.7/dist-packages (from tensorboard\u003c2.10,\u003e=2.9-\u003etensorflow~=2.9.0-\u003etf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (0.4.6)\n",
            "Requirement already satisfied: tensorboard-data-server\u003c0.7.0,\u003e=0.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard\u003c2.10,\u003e=2.9-\u003etensorflow~=2.9.0-\u003etf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (0.6.1)\n",
            "Requirement already satisfied: tensorboard-plugin-wit\u003e=1.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard\u003c2.10,\u003e=2.9-\u003etensorflow~=2.9.0-\u003etf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (1.8.1)\n",
            "Requirement already satisfied: werkzeug\u003e=1.0.1 in /usr/local/lib/python3.7/dist-packages (from tensorboard\u003c2.10,\u003e=2.9-\u003etensorflow~=2.9.0-\u003etf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (1.0.1)\n",
            "Requirement already satisfied: requests-oauthlib\u003e=0.7.0 in /usr/local/lib/python3.7/dist-packages (from google-auth-oauthlib\u003c0.5,\u003e=0.4.1-\u003etensorboard\u003c2.10,\u003e=2.9-\u003etensorflow~=2.9.0-\u003etf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (1.3.1)\n",
            "Requirement already satisfied: importlib-metadata\u003e=4.4 in /usr/local/lib/python3.7/dist-packages (from markdown\u003e=2.6.8-\u003etensorboard\u003c2.10,\u003e=2.9-\u003etensorflow~=2.9.0-\u003etf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (4.12.0)\n",
            "Requirement already satisfied: zipp\u003e=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata\u003e=4.4-\u003emarkdown\u003e=2.6.8-\u003etensorboard\u003c2.10,\u003e=2.9-\u003etensorflow~=2.9.0-\u003etf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (3.8.1)\n",
            "Requirement already satisfied: oauthlib\u003e=3.0.0 in /usr/local/lib/python3.7/dist-packages (from requests-oauthlib\u003e=0.7.0-\u003egoogle-auth-oauthlib\u003c0.5,\u003e=0.4.1-\u003etensorboard\u003c2.10,\u003e=2.9-\u003etensorflow~=2.9.0-\u003etf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (3.2.0)\n",
            "Requirement already satisfied: dm-tree~=0.1.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow-model-optimization\u003e=0.4.1-\u003etf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (0.1.7)\n",
            "Collecting requests\u003c3.0.0dev,\u003e=2.18.0\n",
            "  Downloading requests-2.28.1-py3-none-any.whl (62 kB)\n",
            "Collecting hdfs\u003c3.0.0,\u003e=2.1.0\n",
            "  Downloading hdfs-2.7.0-py3-none-any.whl (34 kB)\n",
            "Requirement already satisfied: pydot\u003c2,\u003e=1.2.0 in /usr/local/lib/python3.7/dist-packages (from apache-beam-\u003eobject-detection==0.1) (1.3.0)\n",
            "Collecting proto-plus\u003c2,\u003e=1.7.1\n",
            "  Downloading proto_plus-1.22.0-py3-none-any.whl (47 kB)\n",
            "Collecting orjson\u003c4.0\n",
            "  Downloading orjson-3.7.11-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (275 kB)\n",
            "Requirement already satisfied: crcmod\u003c2.0,\u003e=1.7 in /usr/local/lib/python3.7/dist-packages (from apache-beam-\u003eobject-detection==0.1) (1.7)\n",
            "Collecting cloudpickle\u003c3,\u003e=2.1.0\n",
            "  Downloading cloudpickle-2.1.0-py3-none-any.whl (25 kB)\n",
            "Collecting dill\u003c0.3.2,\u003e=0.3.1.1\n",
            "  Downloading dill-0.3.1.1.tar.gz (151 kB)\n",
            "Collecting fastavro\u003c2,\u003e=0.23.6\n",
            "  Downloading fastavro-1.5.4-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.4 MB)\n",
            "Requirement already satisfied: pyarrow\u003c8.0.0,\u003e=0.15.1 in /usr/local/lib/python3.7/dist-packages (from apache-beam-\u003eobject-detection==0.1) (6.0.1)\n",
            "Collecting pymongo\u003c4.0.0,\u003e=3.8.0\n",
            "  Downloading pymongo-3.12.3-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (508 kB)\n",
            "Collecting docopt\n",
            "  Downloading docopt-0.6.2.tar.gz (25 kB)\n",
            "Collecting protobuf\u003c4.0.0dev,\u003e=3.12.0\n",
            "  Downloading protobuf-3.19.4-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.1 MB)\n",
            "Requirement already satisfied: charset-normalizer\u003c3,\u003e=2 in /usr/local/lib/python3.7/dist-packages (from requests\u003c3.0.0dev,\u003e=2.18.0-\u003egoogle-api-core\u003c3dev,\u003e=1.21.0-\u003egoogle-api-python-client\u003e=1.6.7-\u003etf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (2.1.0)\n",
            "Requirement already satisfied: kiwisolver\u003e=1.1.0 in /usr/local/lib/python3.7/dist-packages (from lvis-\u003eobject-detection==0.1) (1.4.4)\n",
            "Requirement already satisfied: opencv-python\u003e=4.1.0.25 in /usr/local/lib/python3.7/dist-packages (from lvis-\u003eobject-detection==0.1) (4.6.0.66)\n",
            "Requirement already satisfied: cycler\u003e=0.10.0 in /usr/local/lib/python3.7/dist-packages (from lvis-\u003eobject-detection==0.1) (0.11.0)\n",
            "Requirement already satisfied: text-unidecode\u003e=1.3 in /usr/local/lib/python3.7/dist-packages (from python-slugify-\u003ekaggle\u003e=1.3.9-\u003etf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (1.3)\n",
            "Requirement already satisfied: portalocker in /usr/local/lib/python3.7/dist-packages (from sacrebleu-\u003etf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (2.5.1)\n",
            "Requirement already satisfied: regex in /usr/local/lib/python3.7/dist-packages (from sacrebleu-\u003etf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (2022.6.2)\n",
            "Requirement already satisfied: colorama in /usr/local/lib/python3.7/dist-packages (from sacrebleu-\u003etf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (0.4.5)\n",
            "Requirement already satisfied: tabulate\u003e=0.8.9 in /usr/local/lib/python3.7/dist-packages (from sacrebleu-\u003etf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (0.8.10)\n",
            "Requirement already satisfied: scikit-learn\u003e=0.21.3 in /usr/local/lib/python3.7/dist-packages (from seqeval-\u003etf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (1.0.2)\n",
            "Requirement already satisfied: joblib\u003e=0.11 in /usr/local/lib/python3.7/dist-packages (from scikit-learn\u003e=0.21.3-\u003eseqeval-\u003etf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (1.1.0)\n",
            "Requirement already satisfied: threadpoolctl\u003e=2.0.0 in /usr/local/lib/python3.7/dist-packages (from scikit-learn\u003e=0.21.3-\u003eseqeval-\u003etf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (3.1.0)\n",
            "Requirement already satisfied: typeguard\u003e=2.7 in /usr/local/lib/python3.7/dist-packages (from tensorflow-addons-\u003etf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (2.7.1)\n",
            "Requirement already satisfied: importlib-resources in /usr/local/lib/python3.7/dist-packages (from tensorflow-datasets-\u003etf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (5.9.0)\n",
            "Requirement already satisfied: etils[epath] in /usr/local/lib/python3.7/dist-packages (from tensorflow-datasets-\u003etf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (0.6.0)\n",
            "Requirement already satisfied: toml in /usr/local/lib/python3.7/dist-packages (from tensorflow-datasets-\u003etf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (0.10.2)\n",
            "Requirement already satisfied: promise in /usr/local/lib/python3.7/dist-packages (from tensorflow-datasets-\u003etf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (2.3)\n",
            "Requirement already satisfied: tensorflow-metadata in /usr/local/lib/python3.7/dist-packages (from tensorflow-datasets-\u003etf-models-official\u003e=2.5.1-\u003eobject-detection==0.1) (1.9.0)\n",
            "Building wheels for collected packages: object-detection, dill, avro-python3, docopt\n",
            "  Building wheel for object-detection (setup.py): started\n",
            "  Building wheel for object-detection (setup.py): finished with status 'done'\n",
            "  Created wheel for object-detection: filename=object_detection-0.1-py3-none-any.whl size=1694955 sha256=bbe6ac88d20695351c8d1ff4cfa94b837bf70b4a1ea34d2500087845a1a518f7\n",
            "  Stored in directory: /tmp/pip-ephem-wheel-cache-fosu29b4/wheels/fa/a4/d2/e9a5057e414fd46c8e543d2706cd836d64e1fcd9eccceb2329\n",
            "  Building wheel for dill (setup.py): started\n",
            "  Building wheel for dill (setup.py): finished with status 'done'\n",
            "  Created wheel for dill: filename=dill-0.3.1.1-py3-none-any.whl size=78544 sha256=8600ea58bf6db3cb069ab8ec62f0ac32b9785185e5c756a953626ba68cd67e08\n",
            "  Stored in directory: /root/.cache/pip/wheels/a4/61/fd/c57e374e580aa78a45ed78d5859b3a44436af17e22ca53284f\n",
            "  Building wheel for avro-python3 (setup.py): started\n",
            "  Building wheel for avro-python3 (setup.py): finished with status 'done'\n",
            "  Created wheel for avro-python3: filename=avro_python3-1.10.2-py3-none-any.whl size=44010 sha256=4b5cf56fe1a1eecb4dc5deeef369333aec8389555dbc4bf86959cc48ca96adab\n",
            "  Stored in directory: /root/.cache/pip/wheels/d6/e5/b1/6b151d9b535ee50aaa6ab27d145a0104b6df02e5636f0376da\n",
            "  Building wheel for docopt (setup.py): started\n",
            "  Building wheel for docopt (setup.py): finished with status 'done'\n",
            "  Created wheel for docopt: filename=docopt-0.6.2-py2.py3-none-any.whl size=13723 sha256=c9e261014ce1176b800ebe8923f8660f712dd8e5dc07d7975998f863881c2b3d\n",
            "  Stored in directory: /root/.cache/pip/wheels/72/b0/3f/1d95f96ff986c7dfffe46ce2be4062f38ebd04b506c77c81b9\n",
            "Successfully built object-detection dill avro-python3 docopt\n",
            "Installing collected packages: requests, pyparsing, protobuf, docopt, dill, pymongo, proto-plus, orjson, hdfs, fastavro, cloudpickle, tensorflow-io, lvis, avro-python3, apache-beam, object-detection\n",
            "  Attempting uninstall: requests\n",
            "    Found existing installation: requests 2.23.0\n",
            "    Uninstalling requests-2.23.0:\n",
            "      Successfully uninstalled requests-2.23.0\n",
            "  Attempting uninstall: pyparsing\n",
            "    Found existing installation: pyparsing 3.0.9\n",
            "    Uninstalling pyparsing-3.0.9:\n",
            "      Successfully uninstalled pyparsing-3.0.9\n",
            "  Attempting uninstall: protobuf\n",
            "    Found existing installation: protobuf 3.17.3\n",
            "    Uninstalling protobuf-3.17.3:\n",
            "      Successfully uninstalled protobuf-3.17.3\n",
            "  Attempting uninstall: dill\n",
            "    Found existing installation: dill 0.3.5.1\n",
            "    Uninstalling dill-0.3.5.1:\n",
            "      Successfully uninstalled dill-0.3.5.1\n",
            "  Attempting uninstall: pymongo\n",
            "    Found existing installation: pymongo 4.2.0\n",
            "    Uninstalling pymongo-4.2.0:\n",
            "      Successfully uninstalled pymongo-4.2.0\n",
            "  Attempting uninstall: cloudpickle\n",
            "    Found existing installation: cloudpickle 1.3.0\n",
            "    Uninstalling cloudpickle-1.3.0:\n",
            "      Successfully uninstalled cloudpickle-1.3.0\n",
            "Successfully installed apache-beam-2.40.0 avro-python3-1.10.2 cloudpickle-2.1.0 dill-0.3.1.1 docopt-0.6.2 fastavro-1.5.4 hdfs-2.7.0 lvis-0.5.3 object-detection-0.1 orjson-3.7.11 proto-plus-1.22.0 protobuf-3.19.4 pymongo-3.12.3 pyparsing-2.4.7 requests-2.28.1 tensorflow-io-0.26.0\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "\n",
            "WARNING: apt does not have a stable CLI interface. Use with caution in scripts.\n",
            "\n",
            "  DEPRECATION: A future pip version will change local packages to be built in-place without first copying to a temporary directory. We recommend you use --use-feature=in-tree-build to test your packages with this new behavior before it becomes the default.\n",
            "   pip 21.3 will remove support for this functionality. You can find discussion regarding this at https://github.com/pypa/pip/issues/7555.\n",
            "ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
            "gym 0.17.3 requires cloudpickle\u003c1.7.0,\u003e=1.2.0, but you have cloudpickle 2.1.0 which is incompatible.\n"
          ]
        }
      ],
      "source": [
        "%%bash\n",
        "sudo apt install -y protobuf-compiler\n",
        "cd models/research/\n",
        "protoc object_detection/protos/*.proto --python_out=.\n",
        "cp object_detection/packages/tf2/setup.py .\n",
        "python -m pip install ."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3yDNgIx-kV7X"
      },
      "source": [
        "Now we can import the dependencies we will need later"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2JCeQU3fkayh"
      },
      "outputs": [],
      "source": [
        "from object_detection.utils import label_map_util\n",
        "from object_detection.utils import visualization_utils as viz_utils\n",
        "from object_detection.utils import ops as utils_ops\n",
        "\n",
        "%matplotlib inline"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XRUr9Aiwuho7"
      },
      "source": [
        "## Import pre-trained models from the Waste Identification project"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ZSLPDKwV7G9i",
        "outputId": "c261518a-7667-472d-e22d-9328160cefa3"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "--2022-08-10 22:46:06--  https://storage.googleapis.com/tf_model_garden/vision/waste_identification_ml/material_model.zip\n",
            "Resolving storage.googleapis.com (storage.googleapis.com)... 142.251.2.128, 2607:f8b0:4023:c0d::80\n",
            "Connecting to storage.googleapis.com (storage.googleapis.com)|142.251.2.128|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 521320844 (497M) [application/zip]\n",
            "Saving to: ‘material_model.zip’\n",
            "\n",
            "material_model.zip  100%[===================\u003e] 497.17M   131MB/s    in 3.8s    \n",
            "\n",
            "2022-08-10 22:46:10 (131 MB/s) - ‘material_model.zip’ saved [521320844/521320844]\n",
            "\n",
            "--2022-08-10 22:46:10--  https://storage.googleapis.com/tf_model_garden/vision/waste_identification_ml/material_form_model.zip\n",
            "Resolving storage.googleapis.com (storage.googleapis.com)... 142.251.2.128, 2607:f8b0:4023:c0d::80\n",
            "Connecting to storage.googleapis.com (storage.googleapis.com)|142.251.2.128|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 523568744 (499M) [application/zip]\n",
            "Saving to: ‘material_form_model.zip’\n",
            "\n",
            "material_form_model 100%[===================\u003e] 499.31M   130MB/s    in 4.4s    \n",
            "\n",
            "2022-08-10 22:46:15 (113 MB/s) - ‘material_form_model.zip’ saved [523568744/523568744]\n",
            "\n",
            "--2022-08-10 22:46:15--  https://storage.googleapis.com/tf_model_garden/vision/waste_identification_ml/plastic_types_model.zip\n",
            "Resolving storage.googleapis.com (storage.googleapis.com)... 142.251.2.128, 2607:f8b0:4023:c0d::80\n",
            "Connecting to storage.googleapis.com (storage.googleapis.com)|142.251.2.128|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 521268394 (497M) [application/zip]\n",
            "Saving to: ‘plastic_types_model.zip’\n",
            "\n",
            "plastic_types_model 100%[===================\u003e] 497.12M   159MB/s    in 3.1s    \n",
            "\n",
            "2022-08-10 22:46:18 (159 MB/s) - ‘plastic_types_model.zip’ saved [521268394/521268394]\n",
            "\n"
          ]
        }
      ],
      "source": [
        "# download the model weights from the Google's repo\n",
        "!wget https://storage.googleapis.com/tf_model_garden/vision/waste_identification_ml/material_model.zip \n",
        "!wget https://storage.googleapis.com/tf_model_garden/vision/waste_identification_ml/material_form_model.zip \n",
        "!wget https://storage.googleapis.com/tf_model_garden/vision/waste_identification_ml/plastic_types_model.zip "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "RkC_Pk197QlC",
        "outputId": "8c1775bf-82c8-44bc-b92f-768b744802b9"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Archive:  material_model.zip\n",
            "   creating: material/saved_model/\n",
            "  inflating: material/saved_model/params.yaml  \n",
            "   creating: material/saved_model/saved_model/\n",
            "  inflating: material/saved_model/saved_model/saved_model.pb  \n",
            "   creating: material/saved_model/saved_model/variables/\n",
            "  inflating: material/saved_model/saved_model/variables/variables.data-00000-of-00001  \n",
            "  inflating: material/saved_model/saved_model/variables/variables.index  \n",
            "   creating: material/saved_model/checkpoint/\n",
            "  inflating: material/saved_model/checkpoint/ckpt-1.data-00000-of-00001  \n",
            "  inflating: material/saved_model/checkpoint/checkpoint  \n",
            "  inflating: material/saved_model/checkpoint/ckpt-1.index  \n",
            "   creating: material/tflite_model/\n",
            "  inflating: material/tflite_model/model.tflite  \n",
            "Archive:  material_form_model.zip\n",
            "   creating: material_form/saved_model/\n",
            "  inflating: material_form/saved_model/params.yaml  \n",
            "   creating: material_form/saved_model/saved_model/\n",
            "  inflating: material_form/saved_model/saved_model/saved_model.pb  \n",
            "   creating: material_form/saved_model/saved_model/variables/\n",
            "  inflating: material_form/saved_model/saved_model/variables/variables.data-00000-of-00001  \n",
            "  inflating: material_form/saved_model/saved_model/variables/variables.index  \n",
            "   creating: material_form/saved_model/checkpoint/\n",
            "  inflating: material_form/saved_model/checkpoint/ckpt-1.data-00000-of-00001  \n",
            "  inflating: material_form/saved_model/checkpoint/checkpoint  \n",
            "  inflating: material_form/saved_model/checkpoint/ckpt-1.index  \n",
            "   creating: material_form/tflite_model/\n",
            "  inflating: material_form/tflite_model/model.tflite  \n",
            "Archive:  plastic_types_model.zip\n",
            "   creating: plastic_type/saved_model/\n",
            "  inflating: plastic_type/saved_model/params.yaml  \n",
            "   creating: plastic_type/saved_model/saved_model/\n",
            "  inflating: plastic_type/saved_model/saved_model/saved_model.pb  \n",
            "   creating: plastic_type/saved_model/saved_model/variables/\n",
            "  inflating: plastic_type/saved_model/saved_model/variables/variables.data-00000-of-00001  \n",
            "  inflating: plastic_type/saved_model/saved_model/variables/variables.index  \n",
            "   creating: plastic_type/saved_model/checkpoint/\n",
            "  inflating: plastic_type/saved_model/checkpoint/ckpt-1.data-00000-of-00001  \n",
            "  inflating: plastic_type/saved_model/checkpoint/checkpoint  \n",
            "  inflating: plastic_type/saved_model/checkpoint/ckpt-1.index  \n",
            "   creating: plastic_type/tflite_model/\n",
            "  inflating: plastic_type/tflite_model/model.tflite  \n"
          ]
        }
      ],
      "source": [
        "# unziping the folders\n",
        "%%bash\n",
        "mkdir material material_form plastic_type\n",
        "unzip material_model.zip -d material/\n",
        "unzip material_form_model.zip -d material_form/\n",
        "unzip plastic_types_model.zip -d plastic_type/"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Ey-8Ij2sKjkD"
      },
      "outputs": [],
      "source": [
        "ALL_MODELS = {\n",
        "'material_model' : 'material/tflite_model/model.tflite',\n",
        "'material_form_model' : 'material_form/tflite_model/model.tflite',\n",
        "'plastic_model' : 'plastic_type/tflite_model/model.tflite'\n",
        "}\n",
        "\n",
        "# path to an image\n",
        "IMAGES_FOR_TEST = {\n",
        "  'Image1' : 'models/official/projects/waste_identification_ml/pre_processing/config/sample_images/image_2.png'\n",
        "}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IogyryF2lFBL"
      },
      "source": [
        "## Utilities\n",
        "\n",
        "Run the following cell to create some utils that will be needed later:\n",
        "\n",
        "- Helper method to load an image"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9XXfEdD9PMKn"
      },
      "outputs": [],
      "source": [
        "# Inputs to preprocess functions\n",
        "\n",
        "def load_image_into_numpy_array(path):\n",
        "  \"\"\"Load an image from file into a numpy array.\n",
        "\n",
        "  Puts image into numpy array to feed into tensorflow graph.\n",
        "  Note that by convention we put it into a numpy array with shape\n",
        "  (height, width, channels), where channels=3 for RGB.\n",
        "\n",
        "  Args:\n",
        "    path: the file path to the image\n",
        "\n",
        "  Returns:\n",
        "    uint8 numpy array with shape (1, h, w, 3)\n",
        "  \"\"\"\n",
        "  image = None\n",
        "  if(path.startswith('http')):\n",
        "    response = urlopen(path)\n",
        "    image_data = response.read()\n",
        "    image_data = BytesIO(image_data)\n",
        "    image = Image.open(image_data)\n",
        "  else:\n",
        "    image_data = tf.io.gfile.GFile(path, 'rb').read()\n",
        "    image = Image.open(BytesIO(image_data))\n",
        "\n",
        "  (im_width, im_height) = image.size\n",
        "  return np.array(image.getdata()).reshape(\n",
        "      (1, im_height, im_width, 3)).astype(np.uint8)\n",
        "\n",
        "\n",
        "def build_inputs_for_segmentation(image):\n",
        "  \"\"\"Builds segmentation model inputs for serving.\"\"\"\n",
        "  # Normalizes image with mean and std pixel values.\n",
        "  image = normalize_image(image)\n",
        "  return image"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6917xnUSlp9x"
      },
      "source": [
        "## Build an instance segmentation model and load pre-trained model weights\n",
        "\n",
        "Here we will choose which Instance Segmentation model we will use.\n",
        "If you want to change the model to try other architectures later, just change the next cell and execute following ones."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "HtwrSqvakTNn",
        "outputId": "7f40f0ed-a0aa-45e3-8d43-e4f857e058d1"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Selected model:material_form_model\n",
            "Model Handle at TensorFlow Hub: material_form/tflite_model/model.tflite\n"
          ]
        }
      ],
      "source": [
        "# @title Model Selection { display-mode: \"form\", run: \"auto\" }\n",
        "model_display_name = 'material_form_model' # @param ['material_model','material_form_model','plastic_model']\n",
        "model_handle = ALL_MODELS[model_display_name]\n",
        "\n",
        "print('Selected model:'+ model_display_name)\n",
        "print('Model Handle at TensorFlow Hub: {}'.format(model_handle))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NKtD0IeclbL5"
      },
      "source": [
        "### Load label map data (for plotting).\n",
        "\n",
        "Label maps correspond index numbers to category names, so that when our convolution network predicts `7`, we know that this corresponds to `tray`.  Here we use internal utility functions, but anything that returns a dictionary mapping integers to appropriate string labels would be fine.\n",
        "\n",
        "We are going, for simplicity, to load from the repository that we loaded the Object Detection API code"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3Kwqa0T1NTUf",
        "outputId": "d8d12557-7308-4227-d33e-1b7514bac381"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Labels selected for material_form_model\n",
            "\n",
            "\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "{1: {'id': 1, 'name': 'Flexibles'},\n",
              " 2: {'id': 2, 'name': 'Bottle'},\n",
              " 3: {'id': 3, 'name': 'Jar'},\n",
              " 4: {'id': 4, 'name': 'Carton'},\n",
              " 5: {'id': 5, 'name': 'Sachets-\u0026-Pouch'},\n",
              " 6: {'id': 6, 'name': 'Blister-pack'},\n",
              " 7: {'id': 7, 'name': 'Tray'},\n",
              " 8: {'id': 8, 'name': 'Tube'},\n",
              " 9: {'id': 9, 'name': 'Can'},\n",
              " 10: {'id': 10, 'name': 'Tub'},\n",
              " 11: {'id': 11, 'name': 'Cosmetic'},\n",
              " 12: {'id': 12, 'name': 'Box'},\n",
              " 13: {'id': 13, 'name': 'Clothes'},\n",
              " 14: {'id': 14, 'name': 'Bulb'},\n",
              " 15: {'id': 15, 'name': 'Cup-\u0026-glass'},\n",
              " 16: {'id': 16, 'name': 'Book-\u0026-magazine'},\n",
              " 17: {'id': 17, 'name': 'Bag'},\n",
              " 18: {'id': 18, 'name': 'Lid'},\n",
              " 19: {'id': 19, 'name': 'Clamshell'},\n",
              " 20: {'id': 20, 'name': 'Mirror'},\n",
              " 21: {'id': 21, 'name': 'Tangler'},\n",
              " 22: {'id': 22, 'name': 'Cutlery'},\n",
              " 23: {'id': 23, 'name': 'Cassette-\u0026-tape'},\n",
              " 24: {'id': 24, 'name': 'Electronic-devices'},\n",
              " 25: {'id': 25, 'name': 'Battery'},\n",
              " 26: {'id': 26, 'name': 'Pen-\u0026-pencil'},\n",
              " 27: {'id': 27, 'name': 'Paper-products'},\n",
              " 28: {'id': 28, 'name': 'Foot-wear'},\n",
              " 29: {'id': 29, 'name': 'Scissor'},\n",
              " 30: {'id': 30, 'name': 'Toys'},\n",
              " 31: {'id': 31, 'name': 'Brush'},\n",
              " 32: {'id': 32, 'name': 'Pipe'},\n",
              " 33: {'id': 33, 'name': 'Foil'},\n",
              " 34: {'id': 34, 'name': 'Hangers'}}"
            ]
          },
          "execution_count": 12,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# @title Labels for the above model { display-mode: \"form\", run: \"auto\" }\n",
        "\n",
        "if model_display_name == 'material_model':\n",
        "  PATH_TO_LABELS = './models/official/projects/waste_identification_ml/pre_processing/config/data/material_labels.pbtxt'\n",
        "elif model_display_name == 'material_form_model':\n",
        "  PATH_TO_LABELS = './models/official/projects/waste_identification_ml/pre_processing/config/data/material_form_labels.pbtxt'\n",
        "elif model_display_name == 'plastic_model':\n",
        "  PATH_TO_LABELS = './models/official/projects/waste_identification_ml/pre_processing/config/data/plastic_type_labels.pbtxt'\n",
        "\n",
        "print('Labels selected for',model_display_name)\n",
        "print('\\n')\n",
        "category_index = label_map_util.create_category_index_from_labelmap(PATH_TO_LABELS, use_display_name=True)\n",
        "category_index"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "muhUt-wWL582"
      },
      "source": [
        "## Loading the selected model \n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "rBuD07fLlcEO",
        "outputId": "9f5392ec-91a0-42c4-a3cb-4976f19eb99e"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "loading model...\n",
            "model loaded!\n"
          ]
        }
      ],
      "source": [
        "print('loading model...')\n",
        "interpreter = tf.lite.Interpreter(model_path=model_handle)\n",
        "runner = interpreter.get_signature_runner()\n",
        "print('model loaded!')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0s0Rne2xA4i1",
        "outputId": "48083ea1-aea7-48f5-9dab-57ac4fa5ed83"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "{'serving_default': {'inputs': ['inputs'],\n",
            "                     'outputs': ['detection_boxes',\n",
            "                                 'detection_classes',\n",
            "                                 'detection_masks',\n",
            "                                 'detection_scores',\n",
            "                                 'image_info',\n",
            "                                 'num_detections']}}\n"
          ]
        }
      ],
      "source": [
        "# get signature list\n",
        "pprint(interpreter.get_signature_list())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GIawRDKPPnd4"
      },
      "source": [
        "## Loading an image\n",
        "\n",
        "Let's try the model on a simple image. \n",
        "\n",
        "Here are some simple things to try out if you are curious:\n",
        "* Try running inference on your own images, just upload them to colab and load the same way it's done in the cell below.\n",
        "* Modify some of the input images and see if detection still works.  Some simple things to try out here include flipping the image horizontally, or converting to grayscale (note that we still expect the input image to have 3 channels).\n",
        "\n",
        "**Be careful:** when using images with an alpha channel, the model expect 3 channels images and the alpha will count as a 4th.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 822
        },
        "id": "hX-AWUQ1wIEr",
        "outputId": "f0ccad9b-e6fc-46fd-c975-8acdf261f482"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "min: 0 max: 255\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 2489020 characters]\n",
            "text/plain": [
              "\u003cFigure size 1728x2304 with 1 Axes\u003e"
            ]
          },
          "metadata": {
            "needs_background": "light"
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "#@title Image Selection (don't forget to execute the cell!) { display-mode: \"form\"}\n",
        "selected_image = 'Image1' # @param ['Image1']\n",
        "flip_image_horizontally = False #@param {type:\"boolean\"}\n",
        "convert_image_to_grayscale = False #@param {type:\"boolean\"}\n",
        "\n",
        "image_path = IMAGES_FOR_TEST[selected_image]\n",
        "image_np = load_image_into_numpy_array(image_path)\n",
        "\n",
        "# Flip horizontally\n",
        "if(flip_image_horizontally):\n",
        "  image_np[0] = np.fliplr(image_np[0]).copy()\n",
        "\n",
        "# Convert image to grayscale\n",
        "if(convert_image_to_grayscale):\n",
        "  image_np[0] = np.tile(\n",
        "    np.mean(image_np[0], 2, keepdims=True), (1, 1, 3)).astype(np.uint8)\n",
        "\n",
        "print('min:',np.min(image_np[0]), 'max:', np.max(image_np[0]))\n",
        "plt.figure(figsize=(24,32))\n",
        "plt.imshow(image_np[0])\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dkkBAgGcX65P"
      },
      "source": [
        "## Pre-processing an image"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "97zIaKAhX-92",
        "outputId": "9156b8c5-6ca4-4e69-fdd1-60f82b0ee4b2"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "height = 512, width = 1024\n"
          ]
        }
      ],
      "source": [
        "# get an input size of images on which an Instance Segmentation model is trained\n",
        "\n",
        "# Get model details.\n",
        "input_details = interpreter.get_input_details()\n",
        "output_details = interpreter.get_output_details()\n",
        "\n",
        "# read height and width\n",
        "height = input_details[0]['shape'][1]\n",
        "width = input_details[0]['shape'][2]\n",
        "\n",
        "# verify input height and width\n",
        "input_size = (height, width)\n",
        "print('height = {}, width = {}'.format(height, width))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-K0V6KWiYYpD",
        "outputId": "f5788aa0-6a0d-48e1-a0a8-cf49738420ec"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "TensorShape([1, 512, 1024, 3])"
            ]
          },
          "execution_count": 17,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# apply pre-processing functions which were applied during training the model\n",
        "image_np_cp = cv2.resize(image_np[0], input_size[::-1], interpolation = cv2.INTER_AREA)\n",
        "image_np = build_inputs_for_segmentation(image_np_cp)\n",
        "image_np = tf.expand_dims(image_np, axis=0)\n",
        "image_np.get_shape()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 738
        },
        "id": "ga1lccBpdxpd",
        "outputId": "3268aa20-3cb3-4f4f-cdcc-c06b8867d016"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "WARNING:matplotlib.image:Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers).\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 975348 characters]\n",
            "text/plain": [
              "\u003cFigure size 1728x2304 with 1 Axes\u003e"
            ]
          },
          "metadata": {
            "needs_background": "light"
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "# display pre-processed image\n",
        "plt.figure(figsize=(24,32))\n",
        "plt.imshow(image_np[0])\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FTHsFjR6HNwb"
      },
      "source": [
        "## Doing the inference\n",
        "\n",
        "To do the inference we just need to call our TF Hub loaded model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Gb_siXKcnnGC",
        "outputId": "1f521cea-3020-43f6-ae79-dec5b601fe1a"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "dict_keys(['detection_boxes', 'detection_classes', 'detection_masks', 'detection_scores', 'image_info', 'num_detections'])\n"
          ]
        }
      ],
      "source": [
        "# running inference\n",
        "result = runner(inputs=image_np)\n",
        "print(result.keys())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IZ5VYaBoeeFM"
      },
      "source": [
        "## Visualizing the results\n",
        "\n",
        "Here is where we will need the TensorFlow Object Detection API to show the squares from the inference step (and the keypoints when available).\n",
        "\n",
        "the full documentation of this method can be seen [here](https://github.com/tensorflow/models/blob/master/research/object_detection/utils/visualization_utils.py)\n",
        "\n",
        "Here you can, for example, set `min_score_thresh` to other values (between 0 and 1) to allow more detections in or to filter out more detections."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "PMzURFjxxqF7"
      },
      "outputs": [],
      "source": [
        "# selecting parameters for visualization\n",
        "label_id_offset = 0\n",
        "min_score_thresh =0.9\n",
        "use_normalized_coordinates=True\n",
        "\n",
        "if use_normalized_coordinates:\n",
        "  # Normalizing detection boxes\n",
        "  result['detection_boxes'][0][:,[0,2]] /= height\n",
        "  result['detection_boxes'][0][:,[1,3]] /= width"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 721
        },
        "id": "FILNrrDy0kUg",
        "outputId": "95575a29-e498-4889-f2e0-e82fb918602d"
      },
      "outputs": [
        {
          "data": {
            "image/png": "[encoded data, 1841216 characters]\n",
            "text/plain": [
              "\u003cFigure size 1728x2304 with 1 Axes\u003e"
            ]
          },
          "metadata": {
            "needs_background": "light"
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "# Visualize detection and masks\n",
        "if 'detection_masks' in result:\n",
        "  # we need to convert np.arrays to tensors\n",
        "  detection_masks = tf.convert_to_tensor(result['detection_masks'][0])\n",
        "  detection_boxes = tf.convert_to_tensor(result['detection_boxes'][0])\n",
        "\n",
        "  detection_masks_reframed = utils_ops.reframe_box_masks_to_image_masks(\n",
        "            detection_masks, detection_boxes,\n",
        "              image_np.shape[1], image_np.shape[2])\n",
        "  detection_masks_reframed = tf.cast(detection_masks_reframed \u003e 0.5,\n",
        "                                      np.uint8)\n",
        "\n",
        "  result['detection_masks_reframed'] = detection_masks_reframed.numpy()\n",
        "viz_utils.visualize_boxes_and_labels_on_image_array(\n",
        "      image_np_cp,\n",
        "      result['detection_boxes'][0],\n",
        "      (result['detection_classes'][0] + label_id_offset).astype(int),\n",
        "      result['detection_scores'][0],\n",
        "      category_index=category_index,\n",
        "      use_normalized_coordinates=use_normalized_coordinates,\n",
        "      max_boxes_to_draw=200,\n",
        "      min_score_thresh=min_score_thresh,\n",
        "      agnostic_mode=False,\n",
        "      instance_masks=result.get('detection_masks_reframed', None),\n",
        "      line_thickness=2)\n",
        "\n",
        "plt.figure(figsize=(24,32))\n",
        "plt.imshow(image_np_cp)\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c75cSAeJ5JAQ"
      },
      "source": [
        "## Visualizing the masks only"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 738
        },
        "id": "tt7RxYqhLpn9",
        "outputId": "c1e79bc1-ed27-45de-d4c5-a5ea43117e98"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Total number of objects found are: 26\n"
          ]
        },
        {
          "data": {
            "image/png": "[encoded data, 85280 characters]\n",
            "text/plain": [
              "\u003cFigure size 1728x2304 with 1 Axes\u003e"
            ]
          },
          "metadata": {
            "needs_background": "light"
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "# collecting all masks and saving\n",
        "\n",
        "mask_count = np.sum(result['detection_scores'][0] \u003e= min_score_thresh)\n",
        "print('Total number of objects found are:', mask_count)\n",
        "mask = np.zeros_like(detection_masks_reframed[0])\n",
        "for i in range(mask_count):\n",
        "  if result['detection_scores'][0][i] \u003e= min_score_thresh:\n",
        "    mask += detection_masks_reframed[i]\n",
        "\n",
        "mask = tf.clip_by_value(mask, 0,1)\n",
        "plt.figure(figsize=(24,32))\n",
        "plt.imshow(mask,cmap='gray')\n",
        "plt.show()"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "name": "tflite_model_inference.ipynb",
      "provenance": []
    },
    "gpuClass": "standard",
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
