{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bXCInrb_b96Y"
      },
      "source": [
        "# Convert Bounding Box to Masks "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6kN0IgH7cWlq"
      },
      "source": [
        "The goal is to find the mask of an object using the bounding box coordinates. Then use the mask and image to create a COCO format JSON file. It is required to create a dataset for applying an instance segmentation algorithm."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eXwFSgoLeoQL"
      },
      "source": [
        "\n",
        "To find the mask of an object inside an image, a state-of-an-art algorithm called Deep MAC will be used. Input to the [Deep MAC](https://arxiv.org/abs/2104.00613) algorithm will be the normalized bounding box coordinate and an image. Its output will be a mask. Deep MAC pre trained weights trained on a SpineNet backbone will be used to detect the masks. These weights are available in open source. Deep MAC inference script can be [found here](https://github.com/tensorflow/models/blob/master/research/object_detection/colab_tutorials/deepmac_colab.ipynb) as well but we modified it according to the our project's need. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eCPPN3JOeszb"
      },
      "source": [
        "\n",
        "The output mask and its corresponding image will be then used to create a COCO format JSON annotation file using an open source library known as Imantics."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RHjzxxIUfSgg"
      },
      "source": [
        "## Import libraries & clone the TF models directory"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "LX6NoABvFht4",
        "outputId": "97eafac3-96f2-45c7-dd6d-05828bee1b35"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting tf-models-official\n",
            "  Downloading tf_models_official-2.9.2-py2.py3-none-any.whl (2.1 MB)\n",
            "\u001b[K     |████████████████████████████████| 2.1 MB 5.1 MB/s \n",
            "\u001b[?25hRequirement already satisfied: gin-config in /usr/local/lib/python3.7/dist-packages (from tf-models-official) (0.5.0)\n",
            "Collecting py-cpuinfo>=3.3.0\n",
            "  Downloading py-cpuinfo-8.0.0.tar.gz (99 kB)\n",
            "\u001b[K     |████████████████████████████████| 99 kB 8.6 MB/s \n",
            "\u001b[?25hCollecting seqeval\n",
            "  Downloading seqeval-1.2.2.tar.gz (43 kB)\n",
            "\u001b[K     |████████████████████████████████| 43 kB 1.7 MB/s \n",
            "\u001b[?25hRequirement already satisfied: Pillow in /usr/local/lib/python3.7/dist-packages (from tf-models-official) (7.1.2)\n",
            "Requirement already satisfied: tensorflow-hub>=0.6.0 in /usr/local/lib/python3.7/dist-packages (from tf-models-official) (0.12.0)\n",
            "Requirement already satisfied: pycocotools in /usr/local/lib/python3.7/dist-packages (from tf-models-official) (2.0.4)\n",
            "Requirement already satisfied: scipy>=0.19.1 in /usr/local/lib/python3.7/dist-packages (from tf-models-official) (1.7.3)\n",
            "Requirement already satisfied: tensorflow-datasets in /usr/local/lib/python3.7/dist-packages (from tf-models-official) (4.6.0)\n",
            "Requirement already satisfied: oauth2client in /usr/local/lib/python3.7/dist-packages (from tf-models-official) (4.1.3)\n",
            "Collecting tensorflow-text~=2.9.0\n",
            "  Downloading tensorflow_text-2.9.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.6 MB)\n",
            "\u001b[K     |████████████████████████████████| 4.6 MB 56.7 MB/s \n",
            "\u001b[?25hCollecting tensorflow-addons\n",
            "  Downloading tensorflow_addons-0.17.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.1 MB)\n",
            "\u001b[K     |████████████████████████████████| 1.1 MB 41.8 MB/s \n",
            "\u001b[?25hRequirement already satisfied: Cython in /usr/local/lib/python3.7/dist-packages (from tf-models-official) (0.29.32)\n",
            "Requirement already satisfied: matplotlib in /usr/local/lib/python3.7/dist-packages (from tf-models-official) (3.2.2)\n",
            "Collecting tensorflow-model-optimization>=0.4.1\n",
            "  Downloading tensorflow_model_optimization-0.7.3-py2.py3-none-any.whl (238 kB)\n",
            "\u001b[K     |████████████████████████████████| 238 kB 8.8 MB/s \n",
            "\u001b[?25hRequirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from tf-models-official) (1.15.0)\n",
            "Requirement already satisfied: kaggle>=1.3.9 in /usr/local/lib/python3.7/dist-packages (from tf-models-official) (1.5.12)\n",
            "Requirement already satisfied: google-api-python-client>=1.6.7 in /usr/local/lib/python3.7/dist-packages (from tf-models-official) (1.12.11)\n",
            "Requirement already satisfied: numpy>=1.20 in /usr/local/lib/python3.7/dist-packages (from tf-models-official) (1.21.6)\n",
            "Collecting tensorflow~=2.9.0\n",
            "  Downloading tensorflow-2.9.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (511.7 MB)\n",
            "\u001b[K     |████████████████████████████████| 511.7 MB 6.1 kB/s \n",
            "\u001b[?25hCollecting pyyaml<6.0,>=5.1\n",
            "  Downloading PyYAML-5.4.1-cp37-cp37m-manylinux1_x86_64.whl (636 kB)\n",
            "\u001b[K     |████████████████████████████████| 636 kB 69.9 MB/s \n",
            "\u001b[?25hRequirement already satisfied: pandas>=0.22.0 in /usr/local/lib/python3.7/dist-packages (from tf-models-official) (1.3.5)\n",
            "Collecting tf-slim>=1.1.0\n",
            "  Downloading tf_slim-1.1.0-py2.py3-none-any.whl (352 kB)\n",
            "\u001b[K     |████████████████████████████████| 352 kB 60.2 MB/s \n",
            "\u001b[?25hRequirement already satisfied: opencv-python-headless in /usr/local/lib/python3.7/dist-packages (from tf-models-official) (4.6.0.66)\n",
            "Collecting sacrebleu\n",
            "  Downloading sacrebleu-2.2.0-py3-none-any.whl (116 kB)\n",
            "\u001b[K     |████████████████████████████████| 116 kB 46.9 MB/s \n",
            "\u001b[?25hRequirement already satisfied: psutil>=5.4.3 in /usr/local/lib/python3.7/dist-packages (from tf-models-official) (5.4.8)\n",
            "Collecting sentencepiece\n",
            "  Downloading sentencepiece-0.1.97-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)\n",
            "\u001b[K     |████████████████████████████████| 1.3 MB 20.9 MB/s \n",
            "\u001b[?25hRequirement already satisfied: httplib2<1dev,>=0.15.0 in /usr/local/lib/python3.7/dist-packages (from google-api-python-client>=1.6.7->tf-models-official) (0.17.4)\n",
            "Requirement already satisfied: google-auth<3dev,>=1.16.0 in /usr/local/lib/python3.7/dist-packages (from google-api-python-client>=1.6.7->tf-models-official) (1.35.0)\n",
            "Requirement already satisfied: google-api-core<3dev,>=1.21.0 in /usr/local/lib/python3.7/dist-packages (from google-api-python-client>=1.6.7->tf-models-official) (1.31.6)\n",
            "Requirement already satisfied: uritemplate<4dev,>=3.0.0 in /usr/local/lib/python3.7/dist-packages (from google-api-python-client>=1.6.7->tf-models-official) (3.0.1)\n",
            "Requirement already satisfied: google-auth-httplib2>=0.0.3 in /usr/local/lib/python3.7/dist-packages (from google-api-python-client>=1.6.7->tf-models-official) (0.0.4)\n",
            "Requirement already satisfied: protobuf<4.0.0dev,>=3.12.0 in /usr/local/lib/python3.7/dist-packages (from google-api-core<3dev,>=1.21.0->google-api-python-client>=1.6.7->tf-models-official) (3.17.3)\n",
            "Requirement already satisfied: requests<3.0.0dev,>=2.18.0 in /usr/local/lib/python3.7/dist-packages (from google-api-core<3dev,>=1.21.0->google-api-python-client>=1.6.7->tf-models-official) (2.23.0)\n",
            "Requirement already satisfied: googleapis-common-protos<2.0dev,>=1.6.0 in /usr/local/lib/python3.7/dist-packages (from google-api-core<3dev,>=1.21.0->google-api-python-client>=1.6.7->tf-models-official) (1.56.4)\n",
            "Requirement already satisfied: packaging>=14.3 in /usr/local/lib/python3.7/dist-packages (from google-api-core<3dev,>=1.21.0->google-api-python-client>=1.6.7->tf-models-official) (21.3)\n",
            "Requirement already satisfied: setuptools>=40.3.0 in /usr/local/lib/python3.7/dist-packages (from google-api-core<3dev,>=1.21.0->google-api-python-client>=1.6.7->tf-models-official) (57.4.0)\n",
            "Requirement already satisfied: pytz in /usr/local/lib/python3.7/dist-packages (from google-api-core<3dev,>=1.21.0->google-api-python-client>=1.6.7->tf-models-official) (2022.1)\n",
            "Requirement already satisfied: cachetools<5.0,>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from google-auth<3dev,>=1.16.0->google-api-python-client>=1.6.7->tf-models-official) (4.2.4)\n",
            "Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.7/dist-packages (from google-auth<3dev,>=1.16.0->google-api-python-client>=1.6.7->tf-models-official) (0.2.8)\n",
            "Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.7/dist-packages (from google-auth<3dev,>=1.16.0->google-api-python-client>=1.6.7->tf-models-official) (4.9)\n",
            "Requirement already satisfied: python-slugify in /usr/local/lib/python3.7/dist-packages (from kaggle>=1.3.9->tf-models-official) (6.1.2)\n",
            "Requirement already satisfied: python-dateutil in /usr/local/lib/python3.7/dist-packages (from kaggle>=1.3.9->tf-models-official) (2.8.2)\n",
            "Requirement already satisfied: urllib3 in /usr/local/lib/python3.7/dist-packages (from kaggle>=1.3.9->tf-models-official) (1.24.3)\n",
            "Requirement already satisfied: certifi in /usr/local/lib/python3.7/dist-packages (from kaggle>=1.3.9->tf-models-official) (2022.6.15)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.7/dist-packages (from kaggle>=1.3.9->tf-models-official) (4.64.0)\n",
            "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging>=14.3->google-api-core<3dev,>=1.21.0->google-api-python-client>=1.6.7->tf-models-official) (3.0.9)\n",
            "Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /usr/local/lib/python3.7/dist-packages (from pyasn1-modules>=0.2.1->google-auth<3dev,>=1.16.0->google-api-python-client>=1.6.7->tf-models-official) (0.4.8)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0dev,>=2.18.0->google-api-core<3dev,>=1.21.0->google-api-python-client>=1.6.7->tf-models-official) (3.0.4)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0dev,>=2.18.0->google-api-core<3dev,>=1.21.0->google-api-python-client>=1.6.7->tf-models-official) (2.10)\n",
            "Requirement already satisfied: grpcio<2.0,>=1.24.3 in /usr/local/lib/python3.7/dist-packages (from tensorflow~=2.9.0->tf-models-official) (1.47.0)\n",
            "Collecting flatbuffers<2,>=1.12\n",
            "  Downloading flatbuffers-1.12-py2.py3-none-any.whl (15 kB)\n",
            "Requirement already satisfied: wrapt>=1.11.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow~=2.9.0->tf-models-official) (1.14.1)\n",
            "Collecting gast<=0.4.0,>=0.2.1\n",
            "  Downloading gast-0.4.0-py3-none-any.whl (9.8 kB)\n",
            "Requirement already satisfied: typing-extensions>=3.6.6 in /usr/local/lib/python3.7/dist-packages (from tensorflow~=2.9.0->tf-models-official) (4.1.1)\n",
            "Collecting tensorboard<2.10,>=2.9\n",
            "  Downloading tensorboard-2.9.1-py3-none-any.whl (5.8 MB)\n",
            "\u001b[K     |████████████████████████████████| 5.8 MB 42.9 MB/s \n",
            "\u001b[?25hRequirement already satisfied: h5py>=2.9.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow~=2.9.0->tf-models-official) (3.1.0)\n",
            "Collecting tensorflow-estimator<2.10.0,>=2.9.0rc0\n",
            "  Downloading tensorflow_estimator-2.9.0-py2.py3-none-any.whl (438 kB)\n",
            "\u001b[K     |████████████████████████████████| 438 kB 56.1 MB/s \n",
            "\u001b[?25hRequirement already satisfied: tensorflow-io-gcs-filesystem>=0.23.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow~=2.9.0->tf-models-official) (0.26.0)\n",
            "Requirement already satisfied: google-pasta>=0.1.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow~=2.9.0->tf-models-official) (0.2.0)\n",
            "Requirement already satisfied: opt-einsum>=2.3.2 in /usr/local/lib/python3.7/dist-packages (from tensorflow~=2.9.0->tf-models-official) (3.3.0)\n",
            "Requirement already satisfied: keras-preprocessing>=1.1.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow~=2.9.0->tf-models-official) (1.1.2)\n",
            "Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow~=2.9.0->tf-models-official) (1.1.0)\n",
            "Requirement already satisfied: absl-py>=1.0.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow~=2.9.0->tf-models-official) (1.2.0)\n",
            "Collecting keras<2.10.0,>=2.9.0rc0\n",
            "  Downloading keras-2.9.0-py2.py3-none-any.whl (1.6 MB)\n",
            "\u001b[K     |████████████████████████████████| 1.6 MB 6.0 MB/s \n",
            "\u001b[?25hRequirement already satisfied: astunparse>=1.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow~=2.9.0->tf-models-official) (1.6.3)\n",
            "Requirement already satisfied: libclang>=13.0.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow~=2.9.0->tf-models-official) (14.0.6)\n",
            "Requirement already satisfied: wheel<1.0,>=0.23.0 in /usr/local/lib/python3.7/dist-packages (from astunparse>=1.6.0->tensorflow~=2.9.0->tf-models-official) (0.37.1)\n",
            "Requirement already satisfied: cached-property in /usr/local/lib/python3.7/dist-packages (from h5py>=2.9.0->tensorflow~=2.9.0->tf-models-official) (1.5.2)\n",
            "Requirement already satisfied: werkzeug>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.10,>=2.9->tensorflow~=2.9.0->tf-models-official) (1.0.1)\n",
            "Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.10,>=2.9->tensorflow~=2.9.0->tf-models-official) (1.8.1)\n",
            "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.10,>=2.9->tensorflow~=2.9.0->tf-models-official) (3.4.1)\n",
            "Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.10,>=2.9->tensorflow~=2.9.0->tf-models-official) (0.4.6)\n",
            "Requirement already satisfied: tensorboard-data-server<0.7.0,>=0.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.10,>=2.9->tensorflow~=2.9.0->tf-models-official) (0.6.1)\n",
            "Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.7/dist-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.10,>=2.9->tensorflow~=2.9.0->tf-models-official) (1.3.1)\n",
            "Requirement already satisfied: importlib-metadata>=4.4 in /usr/local/lib/python3.7/dist-packages (from markdown>=2.6.8->tensorboard<2.10,>=2.9->tensorflow~=2.9.0->tf-models-official) (4.12.0)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=4.4->markdown>=2.6.8->tensorboard<2.10,>=2.9->tensorflow~=2.9.0->tf-models-official) (3.8.1)\n",
            "Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.7/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.10,>=2.9->tensorflow~=2.9.0->tf-models-official) (3.2.0)\n",
            "Requirement already satisfied: dm-tree~=0.1.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow-model-optimization>=0.4.1->tf-models-official) (0.1.7)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib->tf-models-official) (0.11.0)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->tf-models-official) (1.4.4)\n",
            "Requirement already satisfied: text-unidecode>=1.3 in /usr/local/lib/python3.7/dist-packages (from python-slugify->kaggle>=1.3.9->tf-models-official) (1.3)\n",
            "Collecting colorama\n",
            "  Downloading colorama-0.4.5-py2.py3-none-any.whl (16 kB)\n",
            "Requirement already satisfied: tabulate>=0.8.9 in /usr/local/lib/python3.7/dist-packages (from sacrebleu->tf-models-official) (0.8.10)\n",
            "Collecting portalocker\n",
            "  Downloading portalocker-2.5.1-py2.py3-none-any.whl (15 kB)\n",
            "Requirement already satisfied: regex in /usr/local/lib/python3.7/dist-packages (from sacrebleu->tf-models-official) (2022.6.2)\n",
            "Requirement already satisfied: lxml in /usr/local/lib/python3.7/dist-packages (from sacrebleu->tf-models-official) (4.9.1)\n",
            "Requirement already satisfied: scikit-learn>=0.21.3 in /usr/local/lib/python3.7/dist-packages (from seqeval->tf-models-official) (1.0.2)\n",
            "Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from scikit-learn>=0.21.3->seqeval->tf-models-official) (3.1.0)\n",
            "Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.7/dist-packages (from scikit-learn>=0.21.3->seqeval->tf-models-official) (1.1.0)\n",
            "Requirement already satisfied: typeguard>=2.7 in /usr/local/lib/python3.7/dist-packages (from tensorflow-addons->tf-models-official) (2.7.1)\n",
            "Requirement already satisfied: promise in /usr/local/lib/python3.7/dist-packages (from tensorflow-datasets->tf-models-official) (2.3)\n",
            "Requirement already satisfied: tensorflow-metadata in /usr/local/lib/python3.7/dist-packages (from tensorflow-datasets->tf-models-official) (1.9.0)\n",
            "Requirement already satisfied: toml in /usr/local/lib/python3.7/dist-packages (from tensorflow-datasets->tf-models-official) (0.10.2)\n",
            "Requirement already satisfied: importlib-resources in /usr/local/lib/python3.7/dist-packages (from tensorflow-datasets->tf-models-official) (5.9.0)\n",
            "Requirement already satisfied: etils[epath] in /usr/local/lib/python3.7/dist-packages (from tensorflow-datasets->tf-models-official) (0.6.0)\n",
            "Requirement already satisfied: dill in /usr/local/lib/python3.7/dist-packages (from tensorflow-datasets->tf-models-official) (0.3.5.1)\n",
            "Building wheels for collected packages: py-cpuinfo, seqeval\n",
            "  Building wheel for py-cpuinfo (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for py-cpuinfo: filename=py_cpuinfo-8.0.0-py3-none-any.whl size=22257 sha256=9abdaf2ab73c28e554cac77b22ceb8dbb7e5da8afbd765dd97cea1b1fb9279d9\n",
            "  Stored in directory: /root/.cache/pip/wheels/d2/f1/1f/041add21dc9c4220157f1bd2bd6afe1f1a49524c3396b94401\n",
            "  Building wheel for seqeval (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for seqeval: filename=seqeval-1.2.2-py3-none-any.whl size=16180 sha256=26e42d24a45f20224fdfd75829909e94dba226915eccc43a309861a340461060\n",
            "  Stored in directory: /root/.cache/pip/wheels/05/96/ee/7cac4e74f3b19e3158dce26a20a1c86b3533c43ec72a549fd7\n",
            "Successfully built py-cpuinfo seqeval\n",
            "Installing collected packages: tensorflow-estimator, tensorboard, keras, gast, flatbuffers, tensorflow, portalocker, colorama, tf-slim, tensorflow-text, tensorflow-model-optimization, tensorflow-addons, seqeval, sentencepiece, sacrebleu, pyyaml, py-cpuinfo, tf-models-official\n",
            "  Attempting uninstall: tensorflow-estimator\n",
            "    Found existing installation: tensorflow-estimator 2.8.0\n",
            "    Uninstalling tensorflow-estimator-2.8.0:\n",
            "      Successfully uninstalled tensorflow-estimator-2.8.0\n",
            "  Attempting uninstall: tensorboard\n",
            "    Found existing installation: tensorboard 2.8.0\n",
            "    Uninstalling tensorboard-2.8.0:\n",
            "      Successfully uninstalled tensorboard-2.8.0\n",
            "  Attempting uninstall: keras\n",
            "    Found existing installation: keras 2.8.0\n",
            "    Uninstalling keras-2.8.0:\n",
            "      Successfully uninstalled keras-2.8.0\n",
            "  Attempting uninstall: gast\n",
            "    Found existing installation: gast 0.5.3\n",
            "    Uninstalling gast-0.5.3:\n",
            "      Successfully uninstalled gast-0.5.3\n",
            "  Attempting uninstall: flatbuffers\n",
            "    Found existing installation: flatbuffers 2.0\n",
            "    Uninstalling flatbuffers-2.0:\n",
            "      Successfully uninstalled flatbuffers-2.0\n",
            "  Attempting uninstall: tensorflow\n",
            "    Found existing installation: tensorflow 2.8.2+zzzcolab20220719082949\n",
            "    Uninstalling tensorflow-2.8.2+zzzcolab20220719082949:\n",
            "      Successfully uninstalled tensorflow-2.8.2+zzzcolab20220719082949\n",
            "  Attempting uninstall: pyyaml\n",
            "    Found existing installation: PyYAML 3.13\n",
            "    Uninstalling PyYAML-3.13:\n",
            "      Successfully uninstalled PyYAML-3.13\n",
            "Successfully installed colorama-0.4.5 flatbuffers-1.12 gast-0.4.0 keras-2.9.0 portalocker-2.5.1 py-cpuinfo-8.0.0 pyyaml-5.4.1 sacrebleu-2.2.0 sentencepiece-0.1.97 seqeval-1.2.2 tensorboard-2.9.1 tensorflow-2.9.1 tensorflow-addons-0.17.1 tensorflow-estimator-2.9.0 tensorflow-model-optimization-0.7.3 tensorflow-text-2.9.0 tf-models-official-2.9.2 tf-slim-1.1.0\n",
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting imantics\n",
            "  Downloading imantics-0.1.12.tar.gz (13 kB)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from imantics) (1.21.6)\n",
            "Requirement already satisfied: opencv-python>=3 in /usr/local/lib/python3.7/dist-packages (from imantics) (4.6.0.66)\n",
            "Requirement already satisfied: lxml in /usr/local/lib/python3.7/dist-packages (from imantics) (4.9.1)\n",
            "Collecting xmljson\n",
            "  Downloading xmljson-0.2.1-py2.py3-none-any.whl (10 kB)\n",
            "Building wheels for collected packages: imantics\n",
            "  Building wheel for imantics (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for imantics: filename=imantics-0.1.12-py3-none-any.whl size=16033 sha256=fe346841a0a21db0c7d8936ed1098c7d1dfd74596a23facca44fdfa9a88c263f\n",
            "  Stored in directory: /root/.cache/pip/wheels/da/7c/3e/296fe3ed4eb3bd713e91dee0d0549f12f316d49939a64bdc96\n",
            "Successfully built imantics\n",
            "Installing collected packages: xmljson, imantics\n",
            "Successfully installed imantics-0.1.12 xmljson-0.2.1\n"
          ]
        }
      ],
      "source": [
        "# install additional libraries\n",
        "!pip install tf-models-official\n",
        "!pip3 install imantics"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "CXfMBkXvHyjg"
      },
      "outputs": [],
      "source": [
        "%matplotlib inline\n",
        "\n",
        "import logging\n",
        "logging.disable(logging.WARNING)\n",
        "\n",
        "from matplotlib import pyplot as plt\n",
        "from matplotlib import patches\n",
        "from PIL import Image\n",
        "import numpy as np\n",
        "import random\n",
        "from skimage import color\n",
        "from skimage.color import rgb_colors\n",
        "from skimage import transform\n",
        "from skimage import util\n",
        "import tensorflow as tf\n",
        "import warnings\n",
        "from imantics import Mask, Category, Image as imantics_Image\n",
        "import json\n",
        "tf.compat.v1.enable_eager_execution()\n",
        "\n",
        "\n",
        "COLORS = ([rgb_colors.cyan, rgb_colors.orange, rgb_colors.pink,\n",
        "           rgb_colors.purple, rgb_colors.limegreen , rgb_colors.crimson] + \n",
        "          [(color) for (name, color) in color.color_dict.items()])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dflD6h1vWW4G"
      },
      "source": [
        "## Visualization functions"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "dOCWTzvSFnk8"
      },
      "outputs": [],
      "source": [
        "def reframe_box_masks_to_image_masks(box_masks, boxes, image_height,\n",
        "                                     image_width, resize_method='bilinear'):\n",
        "  \"\"\"Transforms the box masks back to full image masks.\n",
        "\n",
        "  Embeds masks in bounding boxes of larger masks whose shapes correspond to\n",
        "  image shape.\n",
        "\n",
        "  Args:\n",
        "    box_masks: A tensor of size [num_masks, mask_height, mask_width].\n",
        "    boxes: A tf.float32 tensor of size [num_masks, 4] containing the box\n",
        "           corners. Row i contains [ymin, xmin, ymax, xmax] of the box\n",
        "           corresponding to mask i. Note that the box corners are in\n",
        "           normalized coordinates.\n",
        "    image_height: Image height. The output mask will have the same height as\n",
        "                  the image height.\n",
        "    image_width: Image width. The output mask will have the same width as the\n",
        "                 image width.\n",
        "    resize_method: The resize method, either 'bilinear' or 'nearest'. Note that\n",
        "      'bilinear' is only respected if box_masks is a float.\n",
        "\n",
        "  Returns:\n",
        "    A tensor of size [num_masks, image_height, image_width] with the same dtype\n",
        "    as `box_masks`.\n",
        "  \"\"\"\n",
        "  resize_method = 'nearest' if box_masks.dtype == tf.uint8 else resize_method\n",
        "  def reframe_box_masks_to_image_masks_default():\n",
        "    \"\"\"The default function when there are more than 0 box masks.\"\"\"\n",
        "\n",
        "    num_boxes = tf.shape(box_masks)[0]\n",
        "    box_masks_expanded = tf.expand_dims(box_masks, axis=3)\n",
        "\n",
        "    resized_crops = tf.image.crop_and_resize(\n",
        "        image=box_masks_expanded,\n",
        "        boxes=reframe_image_corners_relative_to_boxes(boxes),\n",
        "        box_indices=tf.range(num_boxes),\n",
        "        crop_size=[image_height, image_width],\n",
        "        method=resize_method,\n",
        "        extrapolation_value=0)\n",
        "    return tf.cast(resized_crops, box_masks.dtype)\n",
        "\n",
        "  image_masks = tf.cond(\n",
        "      tf.shape(box_masks)[0] > 0,\n",
        "      reframe_box_masks_to_image_masks_default,\n",
        "      lambda: tf.zeros([0, image_height, image_width, 1], box_masks.dtype))\n",
        "  return tf.squeeze(image_masks, axis=3)\n",
        "\n",
        "def reframe_image_corners_relative_to_boxes(boxes):\n",
        "  \"\"\"Reframe the image corners ([0, 0, 1, 1]) to be relative to boxes.\n",
        "\n",
        "  The local coordinate frame of each box is assumed to be relative to\n",
        "  its own for corners.\n",
        "\n",
        "  Args:\n",
        "    boxes: A float tensor of [num_boxes, 4] of (ymin, xmin, ymax, xmax)\n",
        "      coordinates in relative coordinate space of each bounding box.\n",
        "\n",
        "  Returns:\n",
        "    reframed_boxes: Reframes boxes with same shape as input.\n",
        "  \"\"\"\n",
        "  ymin, xmin, ymax, xmax = (boxes[:, 0], boxes[:, 1], boxes[:, 2], boxes[:, 3])\n",
        "\n",
        "  height = tf.maximum(ymax - ymin, 1e-4)\n",
        "  width = tf.maximum(xmax - xmin, 1e-4)\n",
        "\n",
        "  ymin_out = (0 - ymin) / height\n",
        "  xmin_out = (0 - xmin) / width\n",
        "  ymax_out = (1 - ymin) / height\n",
        "  xmax_out = (1 - xmin) / width\n",
        "  return tf.stack([ymin_out, xmin_out, ymax_out, xmax_out], axis=1)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "G-gUJ2qffiiH"
      },
      "source": [
        "## Utility functions"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-cWctY5cyUKC"
      },
      "outputs": [],
      "source": [
        "def read_image(path):\n",
        "  \"\"\"Read an image and optionally resize it for better plotting.\"\"\"\n",
        "  with tf.io.gfile.GFile(path, 'rb') as f:\n",
        "    img = Image.open(f)\n",
        "    return np.array(img, dtype=np.uint8)\n",
        "\n",
        "def resize_for_display(image, max_height=600):\n",
        "  height, width, _ = image.shape\n",
        "  width = int(width * max_height / height)\n",
        "  with warnings.catch_warnings():\n",
        "    warnings.simplefilter(\"ignore\", UserWarning)\n",
        "    return util.img_as_ubyte(transform.resize(image, (height, width)))\n",
        "\n",
        "\n",
        "def get_mask_prediction_function(model):\n",
        "  \"\"\"Get single image mask preidction function using a model.\"\"\"\n",
        "\n",
        "  detection_fn = model.signatures['serving_default']\n",
        "\n",
        "\n",
        "  @tf.function\n",
        "  def predict_masks(image, boxes):\n",
        "    height, width, _ = image.shape.as_list()\n",
        "    batch = image[tf.newaxis]\n",
        "    boxes = boxes[tf.newaxis]\n",
        "    detections = detection_fn(images=batch, boxes=boxes)\n",
        "    masks = detections['detection_masks']\n",
        "    return reframe_box_masks_to_image_masks(masks[0], boxes[0],\n",
        "                                                height, width)\n",
        "    \n",
        "  return predict_masks\n",
        "\n",
        "\n",
        "def display(im):\n",
        "  plt.figure(figsize=(16, 12))\n",
        "  plt.imshow(im)\n",
        "  plt.show()\n",
        "\n",
        "def plot_image_annotations(image, boxes, masks=None, darken_image=0.7):\n",
        "  fig, ax = plt.subplots(figsize=(16, 12))\n",
        "  ax.set_axis_on()\n",
        "  image = (image * darken_image).astype(np.uint8)\n",
        "  ax.imshow(image)\n",
        "\n",
        "  height, width, _ = image.shape\n",
        "\n",
        "  num_colors = len(COLORS)\n",
        "  color_index = 0\n",
        "  boxes = boxes[:20]\n",
        "  \n",
        "  masks_list = masks if masks is not None else [None] * len(boxes)\n",
        "  for box, mask in zip(boxes, masks_list):\n",
        "    ymin, xmin, ymax, xmax = box\n",
        "    ymin *= height\n",
        "    ymax *= height\n",
        "    xmin *= width\n",
        "    xmax *= width\n",
        "\n",
        "    color = COLORS[color_index]\n",
        "    color = np.array(color)\n",
        "    rect = patches.Rectangle((xmin, ymin), xmax - xmin, ymax - ymin,\n",
        "                             linewidth=2.5, edgecolor=color, facecolor='none')\n",
        "    ax.add_patch(rect)\n",
        "\n",
        "    if masks is not None:\n",
        "      mask = (mask > 0.5).astype(np.float32)\n",
        "      color_image = np.ones_like(image) * color[np.newaxis, np.newaxis, :]\n",
        "      color_and_mask = np.concatenate(\n",
        "          [color_image, mask[:, :, np.newaxis]], axis=2)\n",
        "\n",
        "      ax.imshow(color_and_mask, alpha=0.5)\n",
        "\n",
        "    color_index = (color_index + 1) % num_colors\n",
        "\n",
        "  return ax"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1Dn44FimfmId"
      },
      "source": [
        "## Import pre-trained Deep MAC weights"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "A4jhuP2zADfS",
        "outputId": "ae74d5c5-54b8-48af-ea58-197fef886557"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--2022-08-11 23:49:21--  https://storage.googleapis.com/tf_model_garden/vision/deepmac_maskrcnn/deepmarc_spinenet.zip\n",
            "Resolving storage.googleapis.com (storage.googleapis.com)... 173.194.213.128, 173.194.214.128, 173.194.215.128, ...\n",
            "Connecting to storage.googleapis.com (storage.googleapis.com)|173.194.213.128|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 314902579 (300M) [application/zip]\n",
            "Saving to: ‘deepmarc_spinenet.zip’\n",
            "\n",
            "deepmarc_spinenet.z 100%[===================>] 300.31M   142MB/s    in 2.1s    \n",
            "\n",
            "2022-08-11 23:49:24 (142 MB/s) - ‘deepmarc_spinenet.zip’ saved [314902579/314902579]\n",
            "\n",
            "Archive:  deepmarc_spinenet.zip\n",
            "   creating: deepmarc_spinenet/\n",
            "   creating: deepmarc_spinenet/variables/\n",
            "  inflating: deepmarc_spinenet/variables/variables.data-00000-of-00001  \n",
            "  inflating: deepmarc_spinenet/variables/variables.index  \n",
            "   creating: deepmarc_spinenet/assets/\n",
            "  inflating: deepmarc_spinenet/saved_model.pb  \n"
          ]
        }
      ],
      "source": [
        "!wget https://storage.googleapis.com/tf_model_garden/vision/deepmac_maskrcnn/deepmarc_spinenet.zip\n",
        "!unzip deepmarc_spinenet.zip"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PLLKy18bfzEW"
      },
      "source": [
        "## Load the model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3RGiRcAc7LDU"
      },
      "outputs": [],
      "source": [
        "MODEL = '/content/deepmarc_spinenet/'\n",
        "model = tf.saved_model.load(MODEL)\n",
        "prediction_function = get_mask_prediction_function(model)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "l8suwOvRf5P5"
      },
      "source": [
        "## MUST CHANGE - Modify the path of an image according to your convenience"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "FP8ETRBUFupE",
        "outputId": "4c529603-ea75-480f-a119-90443cd665aa"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
            "                                 Dload  Upload   Total   Spent    Left  Speed\n",
            "100 1235k  100 1235k    0     0  3687k      0 --:--:-- --:--:-- --:--:-- 3676k\n"
          ]
        }
      ],
      "source": [
        "# import an image\n",
        "!curl -O https://raw.githubusercontent.com/tensorflow/models/master/official/projects/waste_identification_ml/pre_processing/config/sample_images/image_3.jpg "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "QlzhbuxxysPy"
      },
      "outputs": [],
      "source": [
        "# path to an image\n",
        "IMAGE_PATH = 'image_3.jpg' #@param {type:\"string\"}\n",
        "\n",
        "# list of bounding box coordinates in the ymin, xmin, ymax, xmax format\n",
        "BB_CORD = [175.0, 815.06625, 948.0, 1630.125] #@param {type:\"raw\"}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "cL-IU_LN0ycU",
        "outputId": "5595008a-833d-4d72-f534-6dbfedac4806"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "2048 2592\n",
            "0.08544921875 0.3144545717592592 0.462890625 0.62890625\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[0.08544922, 0.31445457, 0.46289062, 0.62890625]])"
            ]
          },
          "metadata": {},
          "execution_count": 9
        }
      ],
      "source": [
        "# get height and width of an image\n",
        "im = read_image(IMAGE_PATH)\n",
        "height, width, _ = im.shape\n",
        "print(height, width)\n",
        "\n",
        "# convert bounding box coordinates to normalized coordinates\n",
        "YMIN, XMIN, YMAX, XMAX = BB_CORD[0], BB_CORD[1], BB_CORD[2], BB_CORD[3]\n",
        "YMIN_NOR, XMIN_NOR, YMAX_NOR, XMAX_NOR = YMIN/height, XMIN/width, YMAX/height, XMAX/width\n",
        "print(YMIN_NOR, XMIN_NOR, YMAX_NOR, XMAX_NOR)\n",
        "\n",
        "# reshape the coordinates\n",
        "boxes = np.array([YMIN_NOR, XMIN_NOR, YMAX_NOR, XMAX_NOR]).reshape(1,4)\n",
        "boxes"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 723
        },
        "id": "wo1ogZ_r5w1-",
        "outputId": "cab79b62-5493-4fae-f065-6eed106ba0a7"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<matplotlib.axes._subplots.AxesSubplot at 0x7fb47b233c90>"
            ]
          },
          "metadata": {},
          "execution_count": 10
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1152x864 with 1 Axes>"
            ],
            "image/png": "[encoded data, 1023880 characters]\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ],
      "source": [
        "%matplotlib inline\n",
        "# display bounding box over an image\n",
        "plot_image_annotations(im, boxes)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lzhbOoyWgJSO"
      },
      "source": [
        "## Doing the inference and showing the results"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "m6Yyhh6irf1n"
      },
      "outputs": [],
      "source": [
        "masks = prediction_function(tf.convert_to_tensor(im),\n",
        "                            tf.convert_to_tensor(boxes, dtype=tf.float32))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 704
        },
        "id": "Uq6vbiRirlLB",
        "outputId": "e3074a1a-37d2-41e7-8b1a-a222a5e28646"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1152x864 with 1 Axes>"
            ],
            "image/png": "[encoded data, 1011580 characters]\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ],
      "source": [
        "plot_image_annotations(im, boxes, masks.numpy())\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WcCAReeIgPBt"
      },
      "source": [
        "## Get the mask"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 704
        },
        "id": "nTkvNK6xrzm5",
        "outputId": "ca31e5c4-1b7a-44be-d537-7fcfafe5b867"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 1152x864 with 1 Axes>"
            ],
            "image/png": "[encoded data, 22712 characters]\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ],
      "source": [
        "mask = masks[0].numpy().reshape(im.shape[0], im.shape[1])\n",
        "mask = np.where(mask > 0.90, 1, 0)\n",
        "mask = np.array(mask, dtype=np.uint8)\n",
        "display(mask)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gWp0pMCRgRsc"
      },
      "source": [
        "# Convert Mask & Image to COCO JSON\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "g-7HTpM3sLbs",
        "outputId": "00ffe574-644f-4c90-87b0-be89e682132b"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "64187"
            ]
          },
          "metadata": {},
          "execution_count": 14
        }
      ],
      "source": [
        "# use of imantics library\n",
        "\n",
        "# give the path of an image\n",
        "image = imantics_Image.from_path(IMAGE_PATH)\n",
        "\n",
        "# array of the mask\n",
        "mask = Mask(mask)\n",
        "\n",
        "# define the category of an object\n",
        "image.add(mask, category=Category(\"Category Name\"))  \n",
        "\n",
        "# create a dict of coco\n",
        "coco_json = image.export(style='coco')\n",
        "coco_json.keys()\n",
        "\n",
        "# write coco_json dict to coco.json\n",
        "open('coco.json', \"w\").write(json.dumps(coco_json, indent=4))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4t8BD-isKTK7",
        "outputId": "01722bbb-84e1-41f5-f0b9-1d3af20e9d6d"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[{'color': '#1fab35',\n",
              "  'id': 1,\n",
              "  'metadata': {},\n",
              "  'name': 'Category Name',\n",
              "  'supercategory': None}]"
            ]
          },
          "metadata": {},
          "execution_count": 15
        }
      ],
      "source": [
        "# display the categories\n",
        "coco_json['categories']"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "36i8AEKzKabO",
        "outputId": "5eac0496-1ec3-4a53-ee6e-717e673e49d6"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[{'coco_url': None,\n",
              "  'date_captured': None,\n",
              "  'fickr_url': None,\n",
              "  'file_name': 'image_3.jpg',\n",
              "  'height': 2048,\n",
              "  'id': 0,\n",
              "  'license': None,\n",
              "  'metadata': {},\n",
              "  'path': 'image_3.jpg',\n",
              "  'width': 2592}]"
            ]
          },
          "metadata": {},
          "execution_count": 16
        }
      ],
      "source": [
        "# display image information\n",
        "coco_json['images']"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "FzQMbZqGKfH8",
        "outputId": "d1686f0f-5053-4b73-8353-9cc590635abc"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(833, 188, 784, 747)"
            ]
          },
          "metadata": {},
          "execution_count": 17
        }
      ],
      "source": [
        "# display bounding box\n",
        "coco_json['annotations'][0]['bbox']"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "name": "bb_to_mask_to_coco.ipynb",
      "provenance": []
    },
    "gpuClass": "standard",
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}