{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rOvvWAVTkMR7"
      },
      "source": [
        "# Welcome to the Waste Identification Colab\n",
        "\n",
        "Welcome to the Instance Segmentation Colab! This notebook will take you through the steps of running an \"out-of-the-box\" Mask RCNN Instance Segmentation model on images."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HVTXSC07QwfG"
      },
      "source": [
        "Given 3 different Mask RCNN models for the material type, material form type and plastic type, your goal is to perform inference with any of the models and visualize the results. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AQUsAE0TRkmh"
      },
      "source": [
        "To finish this task, a proper path for the saved models and a single image needs to be provided. The path to the labels on which the models are trained is in the  waste_identification_ml directory inside the Tensorflow Model Garden repository. The label files are inferred automatically once you select the ML model by which you want to do the inference."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vPs64QA1Zdov"
      },
      "source": [
        "## Imports and Setup\n",
        "\n",
        "Let's start with the base imports."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "yn5_uV1HLvaz"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "import pathlib\n",
        "import cv2\n",
        "import logging\n",
        "logging.disable(logging.WARNING)\n",
        "\n",
        "import matplotlib\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "import numpy as np\n",
        "from six import BytesIO\n",
        "from PIL import Image\n",
        "from six.moves.urllib.request import urlopen\n",
        "\n",
        "import tensorflow as tf\n",
        "import tensorflow_hub as hub\n",
        "\n",
        "tf.get_logger().setLevel('ERROR')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "14bNk1gzh0TN"
      },
      "source": [
        "## Visualization tools\n",
        "\n",
        "To visualize the images with the proper detected boxes and segmentation masks, we will use the TensorFlow Object Detection API. To install it we will clone the repo."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "oi28cqGGFWnY",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c2b27919-e013-4d47-be4a-4f62291b8ac6"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Cloning into 'models'...\n",
            "remote: Enumerating objects: 3451, done.\u001b[K\n",
            "remote: Counting objects: 100% (3451/3451), done.\u001b[K\n",
            "remote: Compressing objects: 100% (2891/2891), done.\u001b[K\n",
            "remote: Total 3451 (delta 896), reused 1476 (delta 503), pack-reused 0\u001b[K\n",
            "Receiving objects: 100% (3451/3451), 46.92 MiB | 15.79 MiB/s, done.\n",
            "Resolving deltas: 100% (896/896), done.\n",
            "Checking out files: 100% (3125/3125), done.\n"
          ]
        }
      ],
      "source": [
        "# Clone the tensorflow models repository\n",
        "!git clone --depth 1 https://github.com/tensorflow/models"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yX3pb_pXDjYA"
      },
      "source": [
        "Intalling the Object Detection API"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NwdsBdGhFanc",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c507dccb-4658-4964-fcce-baed5a817db9"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Reading package lists...\n",
            "Building dependency tree...\n",
            "Reading state information...\n",
            "protobuf-compiler is already the newest version (3.0.0-9.1ubuntu1).\n",
            "The following package was automatically installed and is no longer required:\n",
            "  libnvidia-common-460\n",
            "Use 'sudo apt autoremove' to remove it.\n",
            "0 upgraded, 0 newly installed, 0 to remove and 19 not upgraded.\n",
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Processing /content/models/research\n",
            "Collecting avro-python3\n",
            "  Downloading avro-python3-1.10.2.tar.gz (38 kB)\n",
            "Collecting apache-beam\n",
            "  Downloading apache_beam-2.40.0-cp37-cp37m-manylinux2010_x86_64.whl (10.9 MB)\n",
            "Requirement already satisfied: pillow in /usr/local/lib/python3.7/dist-packages (from object-detection==0.1) (7.1.2)\n",
            "Requirement already satisfied: lxml in /usr/local/lib/python3.7/dist-packages (from object-detection==0.1) (4.9.1)\n",
            "Requirement already satisfied: matplotlib in /usr/local/lib/python3.7/dist-packages (from object-detection==0.1) (3.2.2)\n",
            "Requirement already satisfied: Cython in /usr/local/lib/python3.7/dist-packages (from object-detection==0.1) (0.29.32)\n",
            "Requirement already satisfied: contextlib2 in /usr/local/lib/python3.7/dist-packages (from object-detection==0.1) (0.5.5)\n",
            "Collecting tf-slim\n",
            "  Downloading tf_slim-1.1.0-py2.py3-none-any.whl (352 kB)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from object-detection==0.1) (1.15.0)\n",
            "Requirement already satisfied: pycocotools in /usr/local/lib/python3.7/dist-packages (from object-detection==0.1) (2.0.4)\n",
            "Collecting lvis\n",
            "  Downloading lvis-0.5.3-py3-none-any.whl (14 kB)\n",
            "Requirement already satisfied: scipy in /usr/local/lib/python3.7/dist-packages (from object-detection==0.1) (1.7.3)\n",
            "Requirement already satisfied: pandas in /usr/local/lib/python3.7/dist-packages (from object-detection==0.1) (1.3.5)\n",
            "Collecting tf-models-official>=2.5.1\n",
            "  Downloading tf_models_official-2.9.2-py2.py3-none-any.whl (2.1 MB)\n",
            "Collecting tensorflow_io\n",
            "  Downloading tensorflow_io-0.26.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (25.9 MB)\n",
            "Requirement already satisfied: keras in /usr/local/lib/python3.7/dist-packages (from object-detection==0.1) (2.8.0)\n",
            "Collecting pyparsing==2.4.7\n",
            "  Downloading pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)\n",
            "Requirement already satisfied: oauth2client in /usr/local/lib/python3.7/dist-packages (from tf-models-official>=2.5.1->object-detection==0.1) (4.1.3)\n",
            "Requirement already satisfied: opencv-python-headless in /usr/local/lib/python3.7/dist-packages (from tf-models-official>=2.5.1->object-detection==0.1) (4.6.0.66)\n",
            "Collecting sacrebleu\n",
            "  Downloading sacrebleu-2.2.0-py3-none-any.whl (116 kB)\n",
            "Requirement already satisfied: tensorflow-datasets in /usr/local/lib/python3.7/dist-packages (from tf-models-official>=2.5.1->object-detection==0.1) (4.6.0)\n",
            "Collecting py-cpuinfo>=3.3.0\n",
            "  Downloading py-cpuinfo-8.0.0.tar.gz (99 kB)\n",
            "Requirement already satisfied: numpy>=1.20 in /usr/local/lib/python3.7/dist-packages (from tf-models-official>=2.5.1->object-detection==0.1) (1.21.6)\n",
            "Requirement already satisfied: gin-config in /usr/local/lib/python3.7/dist-packages (from tf-models-official>=2.5.1->object-detection==0.1) (0.5.0)\n",
            "Collecting pyyaml<6.0,>=5.1\n",
            "  Downloading PyYAML-5.4.1-cp37-cp37m-manylinux1_x86_64.whl (636 kB)\n",
            "Requirement already satisfied: tensorflow-hub>=0.6.0 in /usr/local/lib/python3.7/dist-packages (from tf-models-official>=2.5.1->object-detection==0.1) (0.12.0)\n",
            "Collecting seqeval\n",
            "  Downloading seqeval-1.2.2.tar.gz (43 kB)\n",
            "Collecting tensorflow~=2.9.0\n",
            "  Downloading tensorflow-2.9.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (511.7 MB)\n",
            "Collecting tensorflow-text~=2.9.0\n",
            "  Downloading tensorflow_text-2.9.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.6 MB)\n",
            "Requirement already satisfied: psutil>=5.4.3 in /usr/local/lib/python3.7/dist-packages (from tf-models-official>=2.5.1->object-detection==0.1) (5.4.8)\n",
            "Requirement already satisfied: google-api-python-client>=1.6.7 in /usr/local/lib/python3.7/dist-packages (from tf-models-official>=2.5.1->object-detection==0.1) (1.12.11)\n",
            "Collecting sentencepiece\n",
            "  Downloading sentencepiece-0.1.97-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)\n",
            "Collecting tensorflow-addons\n",
            "  Downloading tensorflow_addons-0.17.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.1 MB)\n",
            "Collecting tensorflow-model-optimization>=0.4.1\n",
            "  Downloading tensorflow_model_optimization-0.7.3-py2.py3-none-any.whl (238 kB)\n",
            "Requirement already satisfied: kaggle>=1.3.9 in /usr/local/lib/python3.7/dist-packages (from tf-models-official>=2.5.1->object-detection==0.1) (1.5.12)\n",
            "Requirement already satisfied: google-auth<3dev,>=1.16.0 in /usr/local/lib/python3.7/dist-packages (from google-api-python-client>=1.6.7->tf-models-official>=2.5.1->object-detection==0.1) (1.35.0)\n",
            "Requirement already satisfied: google-auth-httplib2>=0.0.3 in /usr/local/lib/python3.7/dist-packages (from google-api-python-client>=1.6.7->tf-models-official>=2.5.1->object-detection==0.1) (0.0.4)\n",
            "Requirement already satisfied: httplib2<1dev,>=0.15.0 in /usr/local/lib/python3.7/dist-packages (from google-api-python-client>=1.6.7->tf-models-official>=2.5.1->object-detection==0.1) (0.17.4)\n",
            "Requirement already satisfied: google-api-core<3dev,>=1.21.0 in /usr/local/lib/python3.7/dist-packages (from google-api-python-client>=1.6.7->tf-models-official>=2.5.1->object-detection==0.1) (1.31.6)\n",
            "Requirement already satisfied: uritemplate<4dev,>=3.0.0 in /usr/local/lib/python3.7/dist-packages (from google-api-python-client>=1.6.7->tf-models-official>=2.5.1->object-detection==0.1) (3.0.1)\n",
            "Requirement already satisfied: pytz in /usr/local/lib/python3.7/dist-packages (from google-api-core<3dev,>=1.21.0->google-api-python-client>=1.6.7->tf-models-official>=2.5.1->object-detection==0.1) (2022.1)\n",
            "Requirement already satisfied: requests<3.0.0dev,>=2.18.0 in /usr/local/lib/python3.7/dist-packages (from google-api-core<3dev,>=1.21.0->google-api-python-client>=1.6.7->tf-models-official>=2.5.1->object-detection==0.1) (2.23.0)\n",
            "Requirement already satisfied: setuptools>=40.3.0 in /usr/local/lib/python3.7/dist-packages (from google-api-core<3dev,>=1.21.0->google-api-python-client>=1.6.7->tf-models-official>=2.5.1->object-detection==0.1) (57.4.0)\n",
            "Requirement already satisfied: packaging>=14.3 in /usr/local/lib/python3.7/dist-packages (from google-api-core<3dev,>=1.21.0->google-api-python-client>=1.6.7->tf-models-official>=2.5.1->object-detection==0.1) (21.3)\n",
            "Requirement already satisfied: googleapis-common-protos<2.0dev,>=1.6.0 in /usr/local/lib/python3.7/dist-packages (from google-api-core<3dev,>=1.21.0->google-api-python-client>=1.6.7->tf-models-official>=2.5.1->object-detection==0.1) (1.56.4)\n",
            "Requirement already satisfied: protobuf<4.0.0dev,>=3.12.0 in /usr/local/lib/python3.7/dist-packages (from google-api-core<3dev,>=1.21.0->google-api-python-client>=1.6.7->tf-models-official>=2.5.1->object-detection==0.1) (3.17.3)\n",
            "Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.7/dist-packages (from google-auth<3dev,>=1.16.0->google-api-python-client>=1.6.7->tf-models-official>=2.5.1->object-detection==0.1) (0.2.8)\n",
            "Requirement already satisfied: cachetools<5.0,>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from google-auth<3dev,>=1.16.0->google-api-python-client>=1.6.7->tf-models-official>=2.5.1->object-detection==0.1) (4.2.4)\n",
            "Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.7/dist-packages (from google-auth<3dev,>=1.16.0->google-api-python-client>=1.6.7->tf-models-official>=2.5.1->object-detection==0.1) (4.9)\n",
            "Requirement already satisfied: certifi in /usr/local/lib/python3.7/dist-packages (from kaggle>=1.3.9->tf-models-official>=2.5.1->object-detection==0.1) (2022.6.15)\n",
            "Requirement already satisfied: python-slugify in /usr/local/lib/python3.7/dist-packages (from kaggle>=1.3.9->tf-models-official>=2.5.1->object-detection==0.1) (6.1.2)\n",
            "Requirement already satisfied: python-dateutil in /usr/local/lib/python3.7/dist-packages (from kaggle>=1.3.9->tf-models-official>=2.5.1->object-detection==0.1) (2.8.2)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.7/dist-packages (from kaggle>=1.3.9->tf-models-official>=2.5.1->object-detection==0.1) (4.64.0)\n",
            "Requirement already satisfied: urllib3 in /usr/local/lib/python3.7/dist-packages (from kaggle>=1.3.9->tf-models-official>=2.5.1->object-detection==0.1) (1.24.3)\n",
            "Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /usr/local/lib/python3.7/dist-packages (from pyasn1-modules>=0.2.1->google-auth<3dev,>=1.16.0->google-api-python-client>=1.6.7->tf-models-official>=2.5.1->object-detection==0.1) (0.4.8)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0dev,>=2.18.0->google-api-core<3dev,>=1.21.0->google-api-python-client>=1.6.7->tf-models-official>=2.5.1->object-detection==0.1) (3.0.4)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0dev,>=2.18.0->google-api-core<3dev,>=1.21.0->google-api-python-client>=1.6.7->tf-models-official>=2.5.1->object-detection==0.1) (2.10)\n",
            "Requirement already satisfied: tensorflow-io-gcs-filesystem>=0.23.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow~=2.9.0->tf-models-official>=2.5.1->object-detection==0.1) (0.26.0)\n",
            "Requirement already satisfied: h5py>=2.9.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow~=2.9.0->tf-models-official>=2.5.1->object-detection==0.1) (3.1.0)\n",
            "Collecting gast<=0.4.0,>=0.2.1\n",
            "  Downloading gast-0.4.0-py3-none-any.whl (9.8 kB)\n",
            "Requirement already satisfied: google-pasta>=0.1.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow~=2.9.0->tf-models-official>=2.5.1->object-detection==0.1) (0.2.0)\n",
            "Requirement already satisfied: opt-einsum>=2.3.2 in /usr/local/lib/python3.7/dist-packages (from tensorflow~=2.9.0->tf-models-official>=2.5.1->object-detection==0.1) (3.3.0)\n",
            "Collecting keras\n",
            "  Downloading keras-2.9.0-py2.py3-none-any.whl (1.6 MB)\n",
            "Requirement already satisfied: grpcio<2.0,>=1.24.3 in /usr/local/lib/python3.7/dist-packages (from tensorflow~=2.9.0->tf-models-official>=2.5.1->object-detection==0.1) (1.47.0)\n",
            "Collecting tensorboard<2.10,>=2.9\n",
            "  Downloading tensorboard-2.9.1-py3-none-any.whl (5.8 MB)\n",
            "Collecting tensorflow-estimator<2.10.0,>=2.9.0rc0\n",
            "  Downloading tensorflow_estimator-2.9.0-py2.py3-none-any.whl (438 kB)\n",
            "Requirement already satisfied: keras-preprocessing>=1.1.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow~=2.9.0->tf-models-official>=2.5.1->object-detection==0.1) (1.1.2)\n",
            "Collecting flatbuffers<2,>=1.12\n",
            "  Downloading flatbuffers-1.12-py2.py3-none-any.whl (15 kB)\n",
            "Requirement already satisfied: typing-extensions>=3.6.6 in /usr/local/lib/python3.7/dist-packages (from tensorflow~=2.9.0->tf-models-official>=2.5.1->object-detection==0.1) (4.1.1)\n",
            "Requirement already satisfied: libclang>=13.0.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow~=2.9.0->tf-models-official>=2.5.1->object-detection==0.1) (14.0.6)\n",
            "Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow~=2.9.0->tf-models-official>=2.5.1->object-detection==0.1) (1.1.0)\n",
            "Requirement already satisfied: absl-py>=1.0.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow~=2.9.0->tf-models-official>=2.5.1->object-detection==0.1) (1.2.0)\n",
            "Requirement already satisfied: wrapt>=1.11.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow~=2.9.0->tf-models-official>=2.5.1->object-detection==0.1) (1.14.1)\n",
            "Requirement already satisfied: astunparse>=1.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow~=2.9.0->tf-models-official>=2.5.1->object-detection==0.1) (1.6.3)\n",
            "Requirement already satisfied: wheel<1.0,>=0.23.0 in /usr/local/lib/python3.7/dist-packages (from astunparse>=1.6.0->tensorflow~=2.9.0->tf-models-official>=2.5.1->object-detection==0.1) (0.37.1)\n",
            "Requirement already satisfied: cached-property in /usr/local/lib/python3.7/dist-packages (from h5py>=2.9.0->tensorflow~=2.9.0->tf-models-official>=2.5.1->object-detection==0.1) (1.5.2)\n",
            "Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.10,>=2.9->tensorflow~=2.9.0->tf-models-official>=2.5.1->object-detection==0.1) (0.4.6)\n",
            "Requirement already satisfied: werkzeug>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.10,>=2.9->tensorflow~=2.9.0->tf-models-official>=2.5.1->object-detection==0.1) (1.0.1)\n",
            "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.10,>=2.9->tensorflow~=2.9.0->tf-models-official>=2.5.1->object-detection==0.1) (3.4.1)\n",
            "Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.10,>=2.9->tensorflow~=2.9.0->tf-models-official>=2.5.1->object-detection==0.1) (1.8.1)\n",
            "Requirement already satisfied: tensorboard-data-server<0.7.0,>=0.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.10,>=2.9->tensorflow~=2.9.0->tf-models-official>=2.5.1->object-detection==0.1) (0.6.1)\n",
            "Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.7/dist-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.10,>=2.9->tensorflow~=2.9.0->tf-models-official>=2.5.1->object-detection==0.1) (1.3.1)\n",
            "Requirement already satisfied: importlib-metadata>=4.4 in /usr/local/lib/python3.7/dist-packages (from markdown>=2.6.8->tensorboard<2.10,>=2.9->tensorflow~=2.9.0->tf-models-official>=2.5.1->object-detection==0.1) (4.12.0)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=4.4->markdown>=2.6.8->tensorboard<2.10,>=2.9->tensorflow~=2.9.0->tf-models-official>=2.5.1->object-detection==0.1) (3.8.1)\n",
            "Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.7/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.10,>=2.9->tensorflow~=2.9.0->tf-models-official>=2.5.1->object-detection==0.1) (3.2.0)\n",
            "Requirement already satisfied: dm-tree~=0.1.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow-model-optimization>=0.4.1->tf-models-official>=2.5.1->object-detection==0.1) (0.1.7)\n",
            "Collecting cloudpickle<3,>=2.1.0\n",
            "  Downloading cloudpickle-2.1.0-py3-none-any.whl (25 kB)\n",
            "Collecting fastavro<2,>=0.23.6\n",
            "  Downloading fastavro-1.5.4-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.4 MB)\n",
            "Collecting pymongo<4.0.0,>=3.8.0\n",
            "  Downloading pymongo-3.12.3-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (508 kB)\n",
            "Collecting proto-plus<2,>=1.7.1\n",
            "  Downloading proto_plus-1.22.0-py3-none-any.whl (47 kB)\n",
            "Collecting hdfs<3.0.0,>=2.1.0\n",
            "  Downloading hdfs-2.7.0-py3-none-any.whl (34 kB)\n",
            "Requirement already satisfied: pyarrow<8.0.0,>=0.15.1 in /usr/local/lib/python3.7/dist-packages (from apache-beam->object-detection==0.1) (6.0.1)\n",
            "Requirement already satisfied: crcmod<2.0,>=1.7 in /usr/local/lib/python3.7/dist-packages (from apache-beam->object-detection==0.1) (1.7)\n",
            "Requirement already satisfied: pydot<2,>=1.2.0 in /usr/local/lib/python3.7/dist-packages (from apache-beam->object-detection==0.1) (1.3.0)\n",
            "Collecting requests<3.0.0dev,>=2.18.0\n",
            "  Downloading requests-2.28.1-py3-none-any.whl (62 kB)\n",
            "Collecting orjson<4.0\n",
            "  Downloading orjson-3.7.11-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (275 kB)\n",
            "Collecting dill<0.3.2,>=0.3.1.1\n",
            "  Downloading dill-0.3.1.1.tar.gz (151 kB)\n",
            "Collecting docopt\n",
            "  Downloading docopt-0.6.2.tar.gz (25 kB)\n",
            "Collecting protobuf<4.0.0dev,>=3.12.0\n",
            "  Downloading protobuf-3.19.4-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.1 MB)\n",
            "Requirement already satisfied: charset-normalizer<3,>=2 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0dev,>=2.18.0->google-api-core<3dev,>=1.21.0->google-api-python-client>=1.6.7->tf-models-official>=2.5.1->object-detection==0.1) (2.1.0)\n",
            "Requirement already satisfied: cycler>=0.10.0 in /usr/local/lib/python3.7/dist-packages (from lvis->object-detection==0.1) (0.11.0)\n",
            "Requirement already satisfied: opencv-python>=4.1.0.25 in /usr/local/lib/python3.7/dist-packages (from lvis->object-detection==0.1) (4.6.0.66)\n",
            "Requirement already satisfied: kiwisolver>=1.1.0 in /usr/local/lib/python3.7/dist-packages (from lvis->object-detection==0.1) (1.4.4)\n",
            "Requirement already satisfied: text-unidecode>=1.3 in /usr/local/lib/python3.7/dist-packages (from python-slugify->kaggle>=1.3.9->tf-models-official>=2.5.1->object-detection==0.1) (1.3)\n",
            "Collecting portalocker\n",
            "  Downloading portalocker-2.5.1-py2.py3-none-any.whl (15 kB)\n",
            "Requirement already satisfied: tabulate>=0.8.9 in /usr/local/lib/python3.7/dist-packages (from sacrebleu->tf-models-official>=2.5.1->object-detection==0.1) (0.8.10)\n",
            "Collecting colorama\n",
            "  Downloading colorama-0.4.5-py2.py3-none-any.whl (16 kB)\n",
            "Requirement already satisfied: regex in /usr/local/lib/python3.7/dist-packages (from sacrebleu->tf-models-official>=2.5.1->object-detection==0.1) (2022.6.2)\n",
            "Requirement already satisfied: scikit-learn>=0.21.3 in /usr/local/lib/python3.7/dist-packages (from seqeval->tf-models-official>=2.5.1->object-detection==0.1) (1.0.2)\n",
            "Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.7/dist-packages (from scikit-learn>=0.21.3->seqeval->tf-models-official>=2.5.1->object-detection==0.1) (1.1.0)\n",
            "Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from scikit-learn>=0.21.3->seqeval->tf-models-official>=2.5.1->object-detection==0.1) (3.1.0)\n",
            "Requirement already satisfied: typeguard>=2.7 in /usr/local/lib/python3.7/dist-packages (from tensorflow-addons->tf-models-official>=2.5.1->object-detection==0.1) (2.7.1)\n",
            "Requirement already satisfied: tensorflow-metadata in /usr/local/lib/python3.7/dist-packages (from tensorflow-datasets->tf-models-official>=2.5.1->object-detection==0.1) (1.9.0)\n",
            "Requirement already satisfied: importlib-resources in /usr/local/lib/python3.7/dist-packages (from tensorflow-datasets->tf-models-official>=2.5.1->object-detection==0.1) (5.9.0)\n",
            "Requirement already satisfied: promise in /usr/local/lib/python3.7/dist-packages (from tensorflow-datasets->tf-models-official>=2.5.1->object-detection==0.1) (2.3)\n",
            "Requirement already satisfied: toml in /usr/local/lib/python3.7/dist-packages (from tensorflow-datasets->tf-models-official>=2.5.1->object-detection==0.1) (0.10.2)\n",
            "Requirement already satisfied: etils[epath] in /usr/local/lib/python3.7/dist-packages (from tensorflow-datasets->tf-models-official>=2.5.1->object-detection==0.1) (0.7.1)\n",
            "Building wheels for collected packages: object-detection, py-cpuinfo, dill, avro-python3, docopt, seqeval\n",
            "  Building wheel for object-detection (setup.py): started\n",
            "  Building wheel for object-detection (setup.py): finished with status 'done'\n",
            "  Created wheel for object-detection: filename=object_detection-0.1-py3-none-any.whl size=1694955 sha256=5b07d9d3f9b50a4e043579877792dcc5f911757c859a40646bfe44b1411c2b0b\n",
            "  Stored in directory: /tmp/pip-ephem-wheel-cache-_e25017f/wheels/fa/a4/d2/e9a5057e414fd46c8e543d2706cd836d64e1fcd9eccceb2329\n",
            "  Building wheel for py-cpuinfo (setup.py): started\n",
            "  Building wheel for py-cpuinfo (setup.py): finished with status 'done'\n",
            "  Created wheel for py-cpuinfo: filename=py_cpuinfo-8.0.0-py3-none-any.whl size=22257 sha256=16043574dd7ea707666388babd039116f8de2bbe60cc7c7bcc35fd39aa978651\n",
            "  Stored in directory: /root/.cache/pip/wheels/d2/f1/1f/041add21dc9c4220157f1bd2bd6afe1f1a49524c3396b94401\n",
            "  Building wheel for dill (setup.py): started\n",
            "  Building wheel for dill (setup.py): finished with status 'done'\n",
            "  Created wheel for dill: filename=dill-0.3.1.1-py3-none-any.whl size=78544 sha256=ed98e6a3a08f7b25f291c3ded9aa4b85fccbf64a0e52816961d5579e44e7594b\n",
            "  Stored in directory: /root/.cache/pip/wheels/a4/61/fd/c57e374e580aa78a45ed78d5859b3a44436af17e22ca53284f\n",
            "  Building wheel for avro-python3 (setup.py): started\n",
            "  Building wheel for avro-python3 (setup.py): finished with status 'done'\n",
            "  Created wheel for avro-python3: filename=avro_python3-1.10.2-py3-none-any.whl size=44010 sha256=6a76a37f9b0865055470da37fca662fff8934ac3a2776a1f01d09fac2703eb34\n",
            "  Stored in directory: /root/.cache/pip/wheels/d6/e5/b1/6b151d9b535ee50aaa6ab27d145a0104b6df02e5636f0376da\n",
            "  Building wheel for docopt (setup.py): started\n",
            "  Building wheel for docopt (setup.py): finished with status 'done'\n",
            "  Created wheel for docopt: filename=docopt-0.6.2-py2.py3-none-any.whl size=13723 sha256=c1fd090fa0616d3d71b524dc465344600ef9cd2cfcff4f9558571a397f0fd7ad\n",
            "  Stored in directory: /root/.cache/pip/wheels/72/b0/3f/1d95f96ff986c7dfffe46ce2be4062f38ebd04b506c77c81b9\n",
            "  Building wheel for seqeval (setup.py): started\n",
            "  Building wheel for seqeval (setup.py): finished with status 'done'\n",
            "  Created wheel for seqeval: filename=seqeval-1.2.2-py3-none-any.whl size=16180 sha256=6a692bf82ea13a55bc0bb10a37d5b444a675ed7f77d72e893fa02a97246e9204\n",
            "  Stored in directory: /root/.cache/pip/wheels/05/96/ee/7cac4e74f3b19e3158dce26a20a1c86b3533c43ec72a549fd7\n",
            "Successfully built object-detection py-cpuinfo dill avro-python3 docopt seqeval\n",
            "Installing collected packages: requests, pyparsing, protobuf, tensorflow-estimator, tensorboard, keras, gast, flatbuffers, tensorflow, portalocker, docopt, dill, colorama, tf-slim, tensorflow-text, tensorflow-model-optimization, tensorflow-addons, seqeval, sentencepiece, sacrebleu, pyyaml, pymongo, py-cpuinfo, proto-plus, orjson, hdfs, fastavro, cloudpickle, tf-models-official, tensorflow-io, lvis, avro-python3, apache-beam, object-detection\n",
            "  Attempting uninstall: requests\n",
            "    Found existing installation: requests 2.23.0\n",
            "    Uninstalling requests-2.23.0:\n",
            "      Successfully uninstalled requests-2.23.0\n",
            "  Attempting uninstall: pyparsing\n",
            "    Found existing installation: pyparsing 3.0.9\n",
            "    Uninstalling pyparsing-3.0.9:\n",
            "      Successfully uninstalled pyparsing-3.0.9\n",
            "  Attempting uninstall: protobuf\n",
            "    Found existing installation: protobuf 3.17.3\n",
            "    Uninstalling protobuf-3.17.3:\n",
            "      Successfully uninstalled protobuf-3.17.3\n",
            "  Attempting uninstall: tensorflow-estimator\n",
            "    Found existing installation: tensorflow-estimator 2.8.0\n",
            "    Uninstalling tensorflow-estimator-2.8.0:\n",
            "      Successfully uninstalled tensorflow-estimator-2.8.0\n",
            "  Attempting uninstall: tensorboard\n",
            "    Found existing installation: tensorboard 2.8.0\n",
            "    Uninstalling tensorboard-2.8.0:\n",
            "      Successfully uninstalled tensorboard-2.8.0\n",
            "  Attempting uninstall: keras\n",
            "    Found existing installation: keras 2.8.0\n",
            "    Uninstalling keras-2.8.0:\n",
            "      Successfully uninstalled keras-2.8.0\n",
            "  Attempting uninstall: gast\n",
            "    Found existing installation: gast 0.5.3\n",
            "    Uninstalling gast-0.5.3:\n",
            "      Successfully uninstalled gast-0.5.3\n",
            "  Attempting uninstall: flatbuffers\n",
            "    Found existing installation: flatbuffers 2.0\n",
            "    Uninstalling flatbuffers-2.0:\n",
            "      Successfully uninstalled flatbuffers-2.0\n",
            "  Attempting uninstall: tensorflow\n",
            "    Found existing installation: tensorflow 2.8.2+zzzcolab20220719082949\n",
            "    Uninstalling tensorflow-2.8.2+zzzcolab20220719082949:\n",
            "      Successfully uninstalled tensorflow-2.8.2+zzzcolab20220719082949\n",
            "  Attempting uninstall: dill\n",
            "    Found existing installation: dill 0.3.5.1\n",
            "    Uninstalling dill-0.3.5.1:\n",
            "      Successfully uninstalled dill-0.3.5.1\n",
            "  Attempting uninstall: pyyaml\n",
            "    Found existing installation: PyYAML 3.13\n",
            "    Uninstalling PyYAML-3.13:\n",
            "      Successfully uninstalled PyYAML-3.13\n",
            "  Attempting uninstall: pymongo\n",
            "    Found existing installation: pymongo 4.2.0\n",
            "    Uninstalling pymongo-4.2.0:\n",
            "      Successfully uninstalled pymongo-4.2.0\n",
            "  Attempting uninstall: cloudpickle\n",
            "    Found existing installation: cloudpickle 1.3.0\n",
            "    Uninstalling cloudpickle-1.3.0:\n",
            "      Successfully uninstalled cloudpickle-1.3.0\n",
            "Successfully installed apache-beam-2.40.0 avro-python3-1.10.2 cloudpickle-2.1.0 colorama-0.4.5 dill-0.3.1.1 docopt-0.6.2 fastavro-1.5.4 flatbuffers-1.12 gast-0.4.0 hdfs-2.7.0 keras-2.9.0 lvis-0.5.3 object-detection-0.1 orjson-3.7.11 portalocker-2.5.1 proto-plus-1.22.0 protobuf-3.19.4 py-cpuinfo-8.0.0 pymongo-3.12.3 pyparsing-2.4.7 pyyaml-5.4.1 requests-2.28.1 sacrebleu-2.2.0 sentencepiece-0.1.97 seqeval-1.2.2 tensorboard-2.9.1 tensorflow-2.9.1 tensorflow-addons-0.17.1 tensorflow-estimator-2.9.0 tensorflow-io-0.26.0 tensorflow-model-optimization-0.7.3 tensorflow-text-2.9.0 tf-models-official-2.9.2 tf-slim-1.1.0\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "\n",
            "WARNING: apt does not have a stable CLI interface. Use with caution in scripts.\n",
            "\n",
            "  DEPRECATION: A future pip version will change local packages to be built in-place without first copying to a temporary directory. We recommend you use --use-feature=in-tree-build to test your packages with this new behavior before it becomes the default.\n",
            "   pip 21.3 will remove support for this functionality. You can find discussion regarding this at https://github.com/pypa/pip/issues/7555.\n",
            "ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
            "gym 0.17.3 requires cloudpickle<1.7.0,>=1.2.0, but you have cloudpickle 2.1.0 which is incompatible.\n"
          ]
        }
      ],
      "source": [
        "%%bash\n",
        "sudo apt install -y protobuf-compiler\n",
        "cd models/research/\n",
        "protoc object_detection/protos/*.proto --python_out=.\n",
        "cp object_detection/packages/tf2/setup.py .\n",
        "python -m pip install ."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3yDNgIx-kV7X"
      },
      "source": [
        "Now we can import the dependencies we will need later"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2JCeQU3fkayh"
      },
      "outputs": [],
      "source": [
        "from object_detection.utils import label_map_util\n",
        "from object_detection.utils import visualization_utils as viz_utils\n",
        "from object_detection.utils import ops as utils_ops\n",
        "\n",
        "%matplotlib inline"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XRUr9Aiwuho7"
      },
      "source": [
        "## Import pre-trained models from the Waste Identification project"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!wget https://storage.googleapis.com/tf_model_garden/vision/waste_identification_ml/material_model.zip \n",
        "!wget https://storage.googleapis.com/tf_model_garden/vision/waste_identification_ml/material_form_model.zip \n",
        "!wget https://storage.googleapis.com/tf_model_garden/vision/waste_identification_ml/plastic_types_model.zip "
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "7AnetxTI6BdR",
        "outputId": "dfef214d-9c6f-4464-95fa-5f0ccc9a17ba"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--2022-08-12 22:53:17--  https://storage.googleapis.com/tf_model_garden/vision/waste_identification_ml/material_model.zip\n",
            "Resolving storage.googleapis.com (storage.googleapis.com)... 142.250.103.128, 142.250.159.128, 142.251.120.128, ...\n",
            "Connecting to storage.googleapis.com (storage.googleapis.com)|142.250.103.128|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 521320844 (497M) [application/zip]\n",
            "Saving to: ‘material_model.zip’\n",
            "\n",
            "material_model.zip  100%[===================>] 497.17M  97.0MB/s    in 5.4s    \n",
            "\n",
            "2022-08-12 22:53:23 (92.2 MB/s) - ‘material_model.zip’ saved [521320844/521320844]\n",
            "\n",
            "--2022-08-12 22:53:23--  https://storage.googleapis.com/tf_model_garden/vision/waste_identification_ml/material_form_model.zip\n",
            "Resolving storage.googleapis.com (storage.googleapis.com)... 142.250.103.128, 142.250.159.128, 142.251.120.128, ...\n",
            "Connecting to storage.googleapis.com (storage.googleapis.com)|142.250.103.128|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 523568744 (499M) [application/zip]\n",
            "Saving to: ‘material_form_model.zip’\n",
            "\n",
            "material_form_model 100%[===================>] 499.31M   147MB/s    in 3.4s    \n",
            "\n",
            "2022-08-12 22:53:27 (146 MB/s) - ‘material_form_model.zip’ saved [523568744/523568744]\n",
            "\n",
            "--2022-08-12 22:53:27--  https://storage.googleapis.com/tf_model_garden/vision/waste_identification_ml/plastic_types_model.zip\n",
            "Resolving storage.googleapis.com (storage.googleapis.com)... 142.250.103.128, 142.250.159.128, 142.251.120.128, ...\n",
            "Connecting to storage.googleapis.com (storage.googleapis.com)|142.250.103.128|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 521268394 (497M) [application/zip]\n",
            "Saving to: ‘plastic_types_model.zip’\n",
            "\n",
            "plastic_types_model 100%[===================>] 497.12M   137MB/s    in 4.0s    \n",
            "\n",
            "2022-08-12 22:53:31 (124 MB/s) - ‘plastic_types_model.zip’ saved [521268394/521268394]\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "%%bash\n",
        "mkdir material material_form plastic_type\n",
        "unzip material_model.zip -d material/\n",
        "unzip material_form_model.zip -d material_form/\n",
        "unzip plastic_types_model.zip -d plastic_type/"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "SFFo0fdQtg26",
        "outputId": "8a8843e2-b251-4d2a-84ed-34bae99e22cb"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Archive:  material_model.zip\n",
            "   creating: material/saved_model/\n",
            "  inflating: material/saved_model/params.yaml  \n",
            "   creating: material/saved_model/saved_model/\n",
            "  inflating: material/saved_model/saved_model/saved_model.pb  \n",
            "   creating: material/saved_model/saved_model/variables/\n",
            "  inflating: material/saved_model/saved_model/variables/variables.data-00000-of-00001  \n",
            "  inflating: material/saved_model/saved_model/variables/variables.index  \n",
            "   creating: material/saved_model/checkpoint/\n",
            "  inflating: material/saved_model/checkpoint/ckpt-1.data-00000-of-00001  \n",
            "  inflating: material/saved_model/checkpoint/checkpoint  \n",
            "  inflating: material/saved_model/checkpoint/ckpt-1.index  \n",
            "   creating: material/tflite_model/\n",
            "  inflating: material/tflite_model/model.tflite  \n",
            "Archive:  material_form_model.zip\n",
            "   creating: material_form/saved_model/\n",
            "  inflating: material_form/saved_model/params.yaml  \n",
            "   creating: material_form/saved_model/saved_model/\n",
            "  inflating: material_form/saved_model/saved_model/saved_model.pb  \n",
            "   creating: material_form/saved_model/saved_model/variables/\n",
            "  inflating: material_form/saved_model/saved_model/variables/variables.data-00000-of-00001  \n",
            "  inflating: material_form/saved_model/saved_model/variables/variables.index  \n",
            "   creating: material_form/saved_model/checkpoint/\n",
            "  inflating: material_form/saved_model/checkpoint/ckpt-1.data-00000-of-00001  \n",
            "  inflating: material_form/saved_model/checkpoint/checkpoint  \n",
            "  inflating: material_form/saved_model/checkpoint/ckpt-1.index  \n",
            "   creating: material_form/tflite_model/\n",
            "  inflating: material_form/tflite_model/model.tflite  \n",
            "Archive:  plastic_types_model.zip\n",
            "   creating: plastic_type/saved_model/\n",
            "  inflating: plastic_type/saved_model/params.yaml  \n",
            "   creating: plastic_type/saved_model/saved_model/\n",
            "  inflating: plastic_type/saved_model/saved_model/saved_model.pb  \n",
            "   creating: plastic_type/saved_model/saved_model/variables/\n",
            "  inflating: plastic_type/saved_model/saved_model/variables/variables.data-00000-of-00001  \n",
            "  inflating: plastic_type/saved_model/saved_model/variables/variables.index  \n",
            "   creating: plastic_type/saved_model/checkpoint/\n",
            "  inflating: plastic_type/saved_model/checkpoint/ckpt-1.data-00000-of-00001  \n",
            "  inflating: plastic_type/saved_model/checkpoint/checkpoint  \n",
            "  inflating: plastic_type/saved_model/checkpoint/ckpt-1.index  \n",
            "   creating: plastic_type/tflite_model/\n",
            "  inflating: plastic_type/tflite_model/model.tflite  \n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Ey-8Ij2sKjkD"
      },
      "outputs": [],
      "source": [
        "ALL_MODELS = {\n",
        "'material_model' : 'material/saved_model/saved_model/',\n",
        "'material_form_model' : 'material_form/saved_model/saved_model/',\n",
        "'plastic_model' : 'plastic_type/saved_model/saved_model/'\n",
        "}\n",
        "\n",
        "# path to an image\n",
        "IMAGES_FOR_TEST = {\n",
        "  'Image1' : 'models/official/projects/waste_identification_ml/pre_processing/config/sample_images/image_2.png'\n",
        "}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IogyryF2lFBL"
      },
      "source": [
        "## Utilities\n",
        "\n",
        "Run the following cell to create some utils that will be needed later:\n",
        "\n",
        "- Helper method to load an image"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9XXfEdD9PMKn"
      },
      "outputs": [],
      "source": [
        "# Inputs to preprocess functions\n",
        "\n",
        "def normalize_image(image,\n",
        "                    offset=(0.485, 0.456, 0.406),\n",
        "                    scale=(0.229, 0.224, 0.225)):\n",
        "  \"\"\"Normalizes the image to zero mean and unit variance.\"\"\"\n",
        "  with tf.name_scope('normalize_image'):\n",
        "    image = tf.image.convert_image_dtype(image, dtype=tf.float32)\n",
        "    offset = tf.constant(offset)\n",
        "    offset = tf.expand_dims(offset, axis=0)\n",
        "    offset = tf.expand_dims(offset, axis=0)\n",
        "    image -= offset\n",
        "\n",
        "    scale = tf.constant(scale)\n",
        "    scale = tf.expand_dims(scale, axis=0)\n",
        "    scale = tf.expand_dims(scale, axis=0)\n",
        "    image /= scale\n",
        "    return image\n",
        "\n",
        "    \n",
        "def load_image_into_numpy_array(path):\n",
        "  \"\"\"Load an image from file into a numpy array.\n",
        "\n",
        "  Puts image into numpy array to feed into tensorflow graph.\n",
        "  Note that by convention we put it into a numpy array with shape\n",
        "  (height, width, channels), where channels=3 for RGB.\n",
        "\n",
        "  Args:\n",
        "    path: the file path to the image\n",
        "\n",
        "  Returns:\n",
        "    uint8 numpy array with shape (1, h, w, 3)\n",
        "  \"\"\"\n",
        "  image = None\n",
        "  if(path.startswith('http')):\n",
        "    response = urlopen(path)\n",
        "    image_data = response.read()\n",
        "    image_data = BytesIO(image_data)\n",
        "    image = Image.open(image_data)\n",
        "  else:\n",
        "    image_data = tf.io.gfile.GFile(path, 'rb').read()\n",
        "    image = Image.open(BytesIO(image_data))\n",
        "\n",
        "  (im_width, im_height) = image.size\n",
        "  return np.array(image.getdata()).reshape(\n",
        "      (1, im_height, im_width, 3)).astype(np.uint8)\n",
        "\n",
        "\n",
        "def build_inputs_for_segmentation(image):\n",
        "  \"\"\"Builds segmentation model inputs for serving.\"\"\"\n",
        "  # Normalizes image with mean and std pixel values.\n",
        "  image = normalize_image(image)\n",
        "  return image"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6917xnUSlp9x"
      },
      "source": [
        "## Build a instance segmentation model and load pre-trained model weights\n",
        "\n",
        "Here we will choose which Instance Segmentation model we will use.\n",
        "If you want to change the model to try other architectures later, just change the next cell and execute following ones. \n",
        "3 models are available.\n",
        "1. **material_model** : Identify the highest level of the category of the material like plastic, metal, etc.\n",
        "2. **material_form_model** : Identify the product or form in which the object is like plate, bottle, etc.\n",
        "3. **plastic_model** : Identify the types of the plastics like HDPE, PETE, etc."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "HtwrSqvakTNn",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "0a6d2e8b-30a3-4c0d-955a-6a410bdb0b0b"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Selected model:material_form_model\n",
            "Model Handle at TensorFlow Hub: material_form/saved_model/saved_model/\n"
          ]
        }
      ],
      "source": [
        "# @title Model Selection { display-mode: \"form\", run: \"auto\" }\n",
        "model_display_name = 'material_form_model' # @param ['material_model','material_form_model','plastic_model']\n",
        "model_handle = ALL_MODELS[model_display_name]\n",
        "\n",
        "print('Selected model:'+ model_display_name)\n",
        "print('Model Handle at TensorFlow Hub: {}'.format(model_handle))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NKtD0IeclbL5"
      },
      "source": [
        "### Load label map data (for plotting).\n",
        "\n",
        "Label maps correspond index numbers to category names, so that when our convolution network predicts `2`, we know that this corresponds to `Bottle`.  Here we use internal utility functions, but anything that returns a dictionary mapping integers to appropriate string labels would be fine.\n",
        "\n",
        "We are going, for simplicity, to load from the repository that we loaded the Object Detection API code"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3Kwqa0T1NTUf",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "54b701ab-c936-46b0-c76e-d0cc48e4b87c"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Labels selected for material_form_model\n",
            "\n",
            "\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{1: {'id': 1, 'name': 'Flexibles'},\n",
              " 2: {'id': 2, 'name': 'Bottle'},\n",
              " 3: {'id': 3, 'name': 'Jar'},\n",
              " 4: {'id': 4, 'name': 'Carton'},\n",
              " 5: {'id': 5, 'name': 'Sachets-&-Pouch'},\n",
              " 6: {'id': 6, 'name': 'Blister-pack'},\n",
              " 7: {'id': 7, 'name': 'Tray'},\n",
              " 8: {'id': 8, 'name': 'Tube'},\n",
              " 9: {'id': 9, 'name': 'Can'},\n",
              " 10: {'id': 10, 'name': 'Tub'},\n",
              " 11: {'id': 11, 'name': 'Cosmetic'},\n",
              " 12: {'id': 12, 'name': 'Box'},\n",
              " 13: {'id': 13, 'name': 'Clothes'},\n",
              " 14: {'id': 14, 'name': 'Bulb'},\n",
              " 15: {'id': 15, 'name': 'Cup-&-glass'},\n",
              " 16: {'id': 16, 'name': 'Book-&-magazine'},\n",
              " 17: {'id': 17, 'name': 'Bag'},\n",
              " 18: {'id': 18, 'name': 'Lid'},\n",
              " 19: {'id': 19, 'name': 'Clamshell'},\n",
              " 20: {'id': 20, 'name': 'Mirror'},\n",
              " 21: {'id': 21, 'name': 'Tangler'},\n",
              " 22: {'id': 22, 'name': 'Cutlery'},\n",
              " 23: {'id': 23, 'name': 'Cassette-&-tape'},\n",
              " 24: {'id': 24, 'name': 'Electronic-devices'},\n",
              " 25: {'id': 25, 'name': 'Battery'},\n",
              " 26: {'id': 26, 'name': 'Pen-&-pencil'},\n",
              " 27: {'id': 27, 'name': 'Paper-products'},\n",
              " 28: {'id': 28, 'name': 'Foot-wear'},\n",
              " 29: {'id': 29, 'name': 'Scissor'},\n",
              " 30: {'id': 30, 'name': 'Toys'},\n",
              " 31: {'id': 31, 'name': 'Brush'},\n",
              " 32: {'id': 32, 'name': 'Pipe'},\n",
              " 33: {'id': 33, 'name': 'Foil'},\n",
              " 34: {'id': 34, 'name': 'Hangers'}}"
            ]
          },
          "metadata": {},
          "execution_count": 11
        }
      ],
      "source": [
        "# @title Labels for the above model { display-mode: \"form\", run: \"auto\" }\n",
        "\n",
        "if model_display_name == 'material_model':\n",
        "  PATH_TO_LABELS = './models/official/projects/waste_identification_ml/pre_processing/config/data/material_labels.pbtxt'\n",
        "elif model_display_name == 'material_form_model':\n",
        "  PATH_TO_LABELS = './models/official/projects/waste_identification_ml/pre_processing/config/data/material_form_labels.pbtxt'\n",
        "elif model_display_name == 'plastic_model':\n",
        "  PATH_TO_LABELS = './models/official/projects/waste_identification_ml/pre_processing/config/data/plastic_type_labels.pbtxt'\n",
        "\n",
        "print('Labels selected for',model_display_name)\n",
        "print('\\n')\n",
        "category_index = label_map_util.create_category_index_from_labelmap(PATH_TO_LABELS, use_display_name=True)\n",
        "category_index"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "muhUt-wWL582"
      },
      "source": [
        "## Loading the selected model from TensorFlow Hub\n",
        "\n",
        "Here we just need the model handle that was selected and use the Tensorflow Hub library to load it to memory.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "rBuD07fLlcEO",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "6103a711-791a-4ed5-fcf5-96fc6de76976"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "loading model...\n",
            "model loaded!\n"
          ]
        }
      ],
      "source": [
        "print('loading model...')\n",
        "hub_model = hub.load(model_handle)\n",
        "print('model loaded!')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GIawRDKPPnd4"
      },
      "source": [
        "## Loading an image\n",
        "\n",
        "Let's try the model on a simple image. \n",
        "\n",
        "Here are some simple things to try out if you are curious:\n",
        "* Try running inference on your own images, just upload them to colab and load the same way it's done in the cell below.\n",
        "* Modify some of the input images and see if detection still works.  Some simple things to try out here include flipping the image horizontally, or converting to grayscale (note that we still expect the input image to have 3 channels).\n",
        "\n",
        "**Be careful:** when using images with an alpha channel, the model expect 3 channels images and the alpha will count as a 4th.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "hX-AWUQ1wIEr",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 666
        },
        "outputId": "3c0ddf39-c51a-4f46-cad7-8d21ebcaf4f2"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "min: 0 max: 255\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1728x2304 with 1 Axes>"
            ],
            "image/png": "[encoded data, 2489020 characters]\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ],
      "source": [
        "#@title Image Selection (don't forget to execute the cell!) { display-mode: \"form\"}\n",
        "selected_image = 'Image1' # @param ['Image1']\n",
        "flip_image_horizontally = False #@param {type:\"boolean\"}\n",
        "convert_image_to_grayscale = False #@param {type:\"boolean\"}\n",
        "\n",
        "image_path = IMAGES_FOR_TEST[selected_image]\n",
        "image_np = load_image_into_numpy_array(image_path)\n",
        "\n",
        "# Flip horizontally\n",
        "if(flip_image_horizontally):\n",
        "  image_np[0] = np.fliplr(image_np[0]).copy()\n",
        "\n",
        "# Convert image to grayscale\n",
        "if(convert_image_to_grayscale):\n",
        "  image_np[0] = np.tile(\n",
        "    np.mean(image_np[0], 2, keepdims=True), (1, 1, 3)).astype(np.uint8)\n",
        "\n",
        "print('min:',np.min(image_np[0]), 'max:', np.max(image_np[0]))\n",
        "plt.figure(figsize=(24,32))\n",
        "plt.imshow(image_np[0])\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dkkBAgGcX65P"
      },
      "source": [
        "## Pre-processing an image"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "97zIaKAhX-92",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c630f46d-9bb5-4fd6-d59a-b346d547bbd5"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "(512, 1024)\n"
          ]
        }
      ],
      "source": [
        "# get an input size of images on which an Instance Segmentation model is trained\n",
        "hub_model_fn = hub_model.signatures[\"serving_default\"]\n",
        "height=hub_model_fn.structured_input_signature[1]['inputs'].shape[1]\n",
        "width = hub_model_fn.structured_input_signature[1]['inputs'].shape[2]\n",
        "input_size = (height, width)\n",
        "print(input_size)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-K0V6KWiYYpD",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "6f27f7c1-11f4-42c1-b848-71dc44f6646d"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "TensorShape([1, 512, 1024, 3])"
            ]
          },
          "metadata": {},
          "execution_count": 15
        }
      ],
      "source": [
        "# apply pre-processing functions which were applied during training the model\n",
        "image_np_cp = cv2.resize(image_np[0], input_size[::-1], interpolation = cv2.INTER_AREA)\n",
        "image_np = build_inputs_for_segmentation(image_np_cp)\n",
        "image_np = tf.expand_dims(image_np, axis=0)\n",
        "image_np.get_shape()"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "You may notice that the image gets way darker. This is because the pre-processing normalizes the original image."
      ],
      "metadata": {
        "id": "-O6fHWIh4C8r"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ga1lccBpdxpd",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 583
        },
        "outputId": "7356f5db-625b-4aa8-ac38-293e99fe14fc"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1728x2304 with 1 Axes>"
            ],
            "image/png": "[encoded data, 975348 characters]\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ],
      "source": [
        "# display pre-processed image\n",
        "plt.figure(figsize=(24,32))\n",
        "plt.imshow(image_np[0])\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FTHsFjR6HNwb"
      },
      "source": [
        "## Doing the inference\n",
        "\n",
        "To do the inference we just need to call our TF Hub loaded model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Gb_siXKcnnGC",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "3c4d3952-74d8-49c8-98c6-70966ff241e8"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "dict_keys(['image_info', 'detection_scores', 'num_detections', 'detection_masks', 'detection_boxes', 'detection_classes'])\n"
          ]
        }
      ],
      "source": [
        "# running inference\n",
        "results = hub_model_fn(image_np)\n",
        "\n",
        "# different object detection models have additional results\n",
        "# all of them are explained in the documentation\n",
        "result = {key:value.numpy() for key,value in results.items()}\n",
        "print(result.keys())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IZ5VYaBoeeFM"
      },
      "source": [
        "## Visualizing the results\n",
        "\n",
        "Here is where we will need the TensorFlow Object Detection API to show the squares from the inference step (and the keypoints when available).\n",
        "\n",
        "the full documentation of this method can be seen [here](https://github.com/tensorflow/models/blob/master/research/object_detection/utils/visualization_utils.py)\n",
        "\n",
        "Here you can, for example, set `min_score_thresh` to other values (between 0 and 1) to allow more detections in or to filter out more detections."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "PMzURFjxxqF7"
      },
      "outputs": [],
      "source": [
        "# selecting parameters for visualization\n",
        "label_id_offset = 0\n",
        "min_score_thresh =0.6\n",
        "use_normalized_coordinates=True\n",
        "\n",
        "if use_normalized_coordinates:\n",
        "  # Normalizing detection boxes\n",
        "  result['detection_boxes'][0][:,[0,2]] /= height\n",
        "  result['detection_boxes'][0][:,[1,3]] /= width"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "FILNrrDy0kUg",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 583
        },
        "outputId": "5ba415a2-ddaa-4b26-ec58-041387751498"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1728x2304 with 1 Axes>"
            ],
            "image/png": "[encoded data, 1841388 characters]\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ],
      "source": [
        "# Visualize detection and masks\n",
        "if 'detection_masks' in result:\n",
        "  # we need to convert np.arrays to tensors\n",
        "  detection_masks = tf.convert_to_tensor(result['detection_masks'][0])\n",
        "  detection_boxes = tf.convert_to_tensor(result['detection_boxes'][0])\n",
        "\n",
        "  detection_masks_reframed = utils_ops.reframe_box_masks_to_image_masks(\n",
        "            detection_masks, detection_boxes,\n",
        "              image_np.shape[1], image_np.shape[2])\n",
        "  detection_masks_reframed = tf.cast(detection_masks_reframed > 0.5,\n",
        "                                      np.uint8)\n",
        "\n",
        "  result['detection_masks_reframed'] = detection_masks_reframed.numpy()\n",
        "viz_utils.visualize_boxes_and_labels_on_image_array(\n",
        "      image_np_cp,\n",
        "      result['detection_boxes'][0],\n",
        "      (result['detection_classes'][0] + label_id_offset).astype(int),\n",
        "      result['detection_scores'][0],\n",
        "      category_index=category_index,\n",
        "      use_normalized_coordinates=use_normalized_coordinates,\n",
        "      max_boxes_to_draw=200,\n",
        "      min_score_thresh=min_score_thresh,\n",
        "      agnostic_mode=False,\n",
        "      instance_masks=result.get('detection_masks_reframed', None),\n",
        "      line_thickness=2)\n",
        "\n",
        "plt.figure(figsize=(24,32))\n",
        "plt.imshow(image_np_cp)\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c75cSAeJ5JAQ"
      },
      "source": [
        "## Visualizing the masks only"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "tt7RxYqhLpn9",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 602
        },
        "outputId": "c0b1444d-062c-4bd8-db6f-3b01d11e6890"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Total number of objects found are: 26\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1728x2304 with 1 Axes>"
            ],
            "image/png": "[encoded data, 85268 characters]\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ],
      "source": [
        "# collecting all masks and saving\n",
        "\n",
        "mask_count = np.sum(result['detection_scores'][0] >= min_score_thresh)\n",
        "print('Total number of objects found are:', mask_count)\n",
        "mask = np.zeros_like(detection_masks_reframed[0])\n",
        "for i in range(mask_count):\n",
        "  if result['detection_scores'][0][i] >= min_score_thresh:\n",
        "    mask += detection_masks_reframed[i]\n",
        "\n",
        "mask = tf.clip_by_value(mask, 0,1)\n",
        "plt.figure(figsize=(24,32))\n",
        "plt.imshow(mask,cmap='gray')\n",
        "plt.show()"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "gpuClass": "standard"
  },
  "nbformat": 4,
  "nbformat_minor": 0
}