{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0cTM_BOrBUSU"
      },
      "source": [
        "# Merge multiple COCO annotation JSON files into one file. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cwJuts2DBaaU"
      },
      "source": [
        "Given multiple COCO annotated JSON files, your goal is to merge them into one COCO annotated JSON file. \n",
        "\n",
        "A merged COCO annotated JSON file is required where all the data is in one place and it becomes easy to split it into a training and validation JSON file according to the percentage ratio. In case you already have a validated COCO annotated JSON file, then this notebook can be used to merge multiple files into one training COCO annotated JSON file. \n",
        "\n",
        "This notebook uses a third party library to accomplish this task. Recursion is used to combine multiple JSON files using a third party library. \n",
        "\n",
        "This notebook is an end to end example. When you run the notebook, it will take all the multiple JSON files and will output one JSON file.  "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aDE5tjSUm1qu"
      },
      "source": [
        "**Note** - In this example, we assume that all our data is saved on Google drive and we will also write our outputs to Google drive. We also assume that the script will be used as a Google Colab notebook. But this can be changed according to the needs of users. They can modify this in case they are working on their local workstation, remote server or any other database. This colab notebook can be changed to a regular jupyter notebook running on a local machine according to the need of the users."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BM-tYHTlWhDQ"
      },
      "source": [
        "## **MUST DO** - Install the package and restart runtime"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zegGuOmQGBOr"
      },
      "outputs": [],
      "source": [
        "# install python object detection insights library to merge multiple COCO annotation files\n",
        "!pip install pyodi\n",
        "\n",
        "# RESTART THE RUNTIME in order to use this library"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "l7eLOdQ5F33b"
      },
      "source": [
        "## Run the below command to connect to your google drive"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "h5soS-6URktT"
      },
      "outputs": [],
      "source": [
        "# import other libraries\n",
        "from google.colab import drive\n",
        "import pyodi\n",
        "import subprocess\n",
        "import sys\n",
        "import os\n",
        "import json\n",
        "import numpy as np\n",
        "import pandas as pd"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "mTXqVbFdlqxi",
        "outputId": "b12566b2-458f-4673-eb97-7cf30075e258"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Mounted at /content/gdrive\n",
            "Successful\n"
          ]
        }
      ],
      "source": [
        "# connect to google drive\n",
        "drive.mount('/content/gdrive')\n",
        "\n",
        "# making an alias for the root path\n",
        "try:\n",
        "  !ln -s /content/gdrive/My\\ Drive/ /mydrive\n",
        "  print('Successful')\n",
        "except Exception as e:\n",
        "  print(e)\n",
        "  print('Not successful')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U-80gvViT833"
      },
      "source": [
        "## Visualization function"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ipOuE69eT6Vk"
      },
      "outputs": [],
      "source": [
        "def data_creation(path: str) -\u003e pd.DataFrame:\n",
        "  \"\"\"Create a dataframe with the occurences of images and categories.\n",
        "  Args:\n",
        "    path: path to the annotated JSON file.\n",
        "  Returns:\n",
        "    dataset consisting of the counts of images and categories.\n",
        "  \"\"\"\n",
        "  # get annotation file data into a variable\n",
        "  with open(path) as json_file:\n",
        "    data = json.load(json_file)\n",
        "\n",
        "  # count the occurance of each category and an image in the annotation file\n",
        "  category_names = [i['name'] for i in data['categories']]\n",
        "  category_ids = [i['category_id'] for i in data['annotations']]\n",
        "  image_ids = [i['image_id'] for i in data['annotations']]\n",
        "\n",
        "  # create a dataframe\n",
        "  df = pd.DataFrame(\n",
        "      list(zip(category_ids, image_ids)), columns=['category_ids', 'image_ids'])\n",
        "  df = df.groupby('category_ids').agg(\n",
        "      object_count=('category_ids', 'count'),\n",
        "      image_count=('image_ids', 'nunique'))\n",
        "  df = df.reindex(range(1, len(data['categories']) + 1), fill_value=0)\n",
        "  df.index = category_names\n",
        "  return df\n",
        "\n",
        "def visualize_detailed_counts_horizontally(path: str) -\u003e None:\n",
        "  \"\"\"Plot a vertical bar graph showing the counts of images \u0026 categories.\n",
        "  Args:\n",
        "    path: path to the annotated JSON file.\n",
        "  \"\"\"\n",
        "  df = data_creation(path)\n",
        "  ax = df.plot(\n",
        "      kind='bar',\n",
        "      figsize=(40, 10),\n",
        "      xlabel='Categories',\n",
        "      ylabel='Counts',\n",
        "      width=0.8,\n",
        "      linewidth=1,\n",
        "      edgecolor='white')  # rot = 0 for horizontal labeling\n",
        "  for p in ax.patches:\n",
        "    ax.annotate(\n",
        "        text=np.round(p.get_height()),\n",
        "        xy=(p.get_x() + p.get_width() / 2., p.get_height()),\n",
        "        ha='center',\n",
        "        va='top',\n",
        "        xytext=(4, 14),\n",
        "        textcoords='offset points')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4hF5F_QE627R"
      },
      "source": [
        "## Define the paths of inputs and outputs"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "qUE6cHse3zOT"
      },
      "outputs": [],
      "source": [
        "def list_full_paths(directory):\n",
        "    '''return the files names in a directory with absolute path.\n",
        "    Args:\n",
        "      directory: path where all the files that need to merge are saved.\n",
        "    '''\n",
        "    return [os.path.join(directory, file) for file in os.listdir(directory)]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "a5UVAF--2nas"
      },
      "outputs": [],
      "source": [
        "folder_with_jsons = '/mydrive/TFHub/jsons/' #@param {type:\"string\"}\n",
        "output_merged_file = '/mydrive/TFHub/jsons/merged.json' #@param {type:\"string\"}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "80gZnQCU30gK",
        "outputId": "d3fdb152-c63a-42b5-f5bf-f4d50eb16d02"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "8"
            ]
          },
          "execution_count": 8,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# get a list of all the JSON files that need to merge with their absolute paths\n",
        "list_of_json_files = list_full_paths(folder_with_jsons)\n",
        "len(list_of_json_files)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6ot4VWOcSTWO"
      },
      "source": [
        "# Merge the files"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "hngtw3K6S7Qx"
      },
      "outputs": [],
      "source": [
        "def merge_two_files(file1, file2, output_file):\n",
        "  \"\"\"Function to merge 2 files\n",
        "\n",
        "  Args:\n",
        "    file1: path of the 1st COCO annotation json file\n",
        "    file2: path of the 2nd COCO annotation json file\n",
        "    output_file: path of the output COCO annotation json file after merge\n",
        "\n",
        "  Returns:\n",
        "    Path of the merged COCO annotation json file\n",
        "  \"\"\"\n",
        "  subprocess.run(['pyodi', 'coco', 'merge', file1, file2, output_file])\n",
        "  return output_file\n",
        "\n",
        "def merge_multiple_files(list_of_files,output_file_path):\n",
        "  \"\"\"Recursive function to merge multiple files\n",
        "\n",
        "  Args:\n",
        "    list_of_files: list of all the COCO annotation json files that need to be merged \n",
        "    output_file_path: path of the output COCO annotation json file after merge\n",
        "\n",
        "  Returns:\n",
        "    Path of the merged COCO annotation json file\n",
        "  \"\"\"\n",
        "  if len(list_of_files) == 2:\n",
        "    return merge_two_files(list_of_files[0], list_of_files[1], output_file_path)\n",
        "\n",
        "  else:\n",
        "    return merge_two_files(list_of_files[0], merge_multiple_files(list_of_files[1:], output_file_path), output_file_path)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "__Wj0rO2D-HT"
      },
      "source": [
        "The output of the below code will be a merged COCO annotation file in the same directory."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ZMKFOjchVMlH",
        "outputId": "815a83a3-43ae-478d-abaa-58181f034b94"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Total number of files to merge : 8\n",
            "Merge Done\n"
          ]
        }
      ],
      "source": [
        "# call function to merge multiple files\n",
        "print('Total number of files to merge :', len(list_of_json_files))\n",
        "merge_multiple_files(list_of_json_files, output_merged_file)\n",
        "\n",
        "print('Merge Done')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vf4ESHiDEQYF"
      },
      "source": [
        "## Visualize the results"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 429
        },
        "id": "UBwMf-W0EqG1",
        "outputId": "9bfbfb43-2ad2-4c18-8d75-05ea56a9573e"
      },
      "outputs": [
        {
          "data": {
            "image/png": "[encoded data, 67696 characters]\n",
            "text/plain": [
              "\u003cFigure size 2880x720 with 1 Axes\u003e"
            ]
          },
          "metadata": {
            "needs_background": "light"
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "# visualize the merged COCO annotated JSON file\n",
        "visualize_detailed_counts_horizontally(output_merged_file)"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
