{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "aGYwt_UjIrqp"
   },
   "source": [
    "# Instance Segmentation\n",
    "\n",
    "In this tutorial, you will learn:\n",
    "- the basic structure of Mask R-CNN.\n",
    "- to perform inference with a MMDetection detector.\n",
    "- to train a new instance segmentation model with a new dataset.\n",
    "\n",
    "Let's start!\n",
    "\n",
    "<a href=\"https://colab.research.google.com/drive/11MqMCyF_V7Rkw6b9CCShGfNTP59OJ905?usp=sharing\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "cCk6uTQrdUUn"
   },
   "source": [
    "If you are running the tutorial files on the colab platform or a new virtual environment, please run the following code first to configure the runtime environment.\n",
    "```python\n",
    "!pip install -U openmim\n",
    "!mim install \"mmengine>=0.7.0\"\n",
    "!mim install \"mmcv>=2.0.0rc4\"\n",
    "\n",
    "# Install mmdetection\n",
    "!rm -rf mmdetection\n",
    "!git clone https://github.com/open-mmlab/mmdetection.git\n",
    "%cd mmdetection\n",
    "\n",
    "!pip install -e .\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "6hD0mmMixT0p",
    "outputId": "221dad3c-5ef8-4094-e07e-289f333f7bb9"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch version: 2.0.1+cu118 cuda: True\n",
      "mmdetection: 3.1.0\n",
      "mmcv: 2.0.1\n",
      "mmengine: 0.8.4\n"
     ]
    }
   ],
   "source": [
    "# Check Pytorch installation\n",
    "import torch, torchvision\n",
    "print(\"torch version:\",torch.__version__, \"cuda:\",torch.cuda.is_available())\n",
    "\n",
    "# Check MMDetection installation\n",
    "import mmdet\n",
    "print(\"mmdetection:\",mmdet.__version__)\n",
    "\n",
    "# Check mmcv installation\n",
    "import mmcv\n",
    "print(\"mmcv:\",mmcv.__version__)\n",
    "\n",
    "# Check mmengine installation\n",
    "import mmengine\n",
    "print(\"mmengine:\",mmengine.__version__)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "gi9zw03oM4CH"
   },
   "source": [
    "## Perform Inference with An MMDetection Detector"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "3pFYLerc0we1"
   },
   "source": [
    "### A two-stage detector\n",
    "\n",
    "In this tutorial, we use Mask R-CNN, a simple two-stage detector as an example.\n",
    "\n",
    "The high-level architecture of Mask R-CNN is shown in the following picture. More details can be found in the [paper](https://arxiv.org/abs/1703.06870).\n",
    "\n",
    "<img src=\"https://user-images.githubusercontent.com/40661020/143967081-c2552bed-9af2-46c4-ae44-5b3b74e5679f.png\" alt=\"mask rcnn\" align=\"bottom\" />\n",
    "\n",
    "Mask R-CNN adds a mask branch based on the original Faster R-CNN. It also uses RoIAlign, a more precise version of RoIPooling for RoI feature extraction to improve the performance.\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "sWI-nX5yRYYQ",
    "outputId": "fd91e337-27cb-492c-a948-98adcbcfca27"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "processing mask-rcnn_r50-caffe_fpn_ms-poly-3x_coco...\n",
      "\u001b[2Kdownloading \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m169.6/169.6 MiB\u001b[0m \u001b[31m9.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h\u001b[32mSuccessfully downloaded mask_rcnn_r50_caffe_fpn_mstrain-poly_3x_coco_bbox_mAP-0.408__segm_mAP-0.37_20200504_163245-42aa3d00.pth to /content/mmdetection/checkpoints\u001b[0m\n",
      "\u001b[32mSuccessfully dumped mask-rcnn_r50-caffe_fpn_ms-poly-3x_coco.py to /content/mmdetection/checkpoints\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!mim download mmdet --config mask-rcnn_r50-caffe_fpn_ms-poly-3x_coco --dest ./checkpoints"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "8M5KUnX7Np3h",
    "outputId": "71de79c0-9f7e-4cae-f810-5c0a20fe9be8"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loads checkpoint by local backend from path: checkpoints/mask_rcnn_r50_caffe_fpn_mstrain-poly_3x_coco_bbox_mAP-0.408__segm_mAP-0.37_20200504_163245-42aa3d00.pth\n"
     ]
    }
   ],
   "source": [
    "import mmcv\n",
    "import mmengine\n",
    "from mmdet.apis import init_detector, inference_detector\n",
    "from mmdet.utils import register_all_modules\n",
    "# Choose to use a config and initialize the detector\n",
    "config_file = 'configs/mask_rcnn/mask-rcnn_r50-caffe_fpn_ms-poly-3x_coco.py'\n",
    "# Setup a checkpoint file to load\n",
    "checkpoint_file = 'checkpoints/mask_rcnn_r50_caffe_fpn_mstrain-poly_3x_coco_bbox_mAP-0.408__segm_mAP-0.37_20200504_163245-42aa3d00.pth'\n",
    "\n",
    "# register all modules in mmdet into the registries\n",
    "register_all_modules()\n",
    "\n",
    "# build the model from a config file and a checkpoint file\n",
    "model = init_detector(config_file, checkpoint_file, device='cuda:0')  # or device='cuda:0'\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "pVqDQAOiKkJK"
   },
   "source": [
    "From the printed model, we will find that the model does consist of the components that we described earlier. It uses ResNet as its CNN backbone, and has a RPN head and RoI Head.\n",
    "The RoI Head includes box head and mask head. In addition, the model has a neural network module, named neck, directly after the CNN backbone. It is a [feature pyramid network (FPN)](https://arxiv.org/abs/1612.03144) for enhancing the multi-scale features.\n",
    "\n",
    "\n",
    "### Inference with the detector\n",
    "\n",
    "The model is successfully created and loaded, let's see how good it is. We use the high-level API `inference_detector` implemented in the MMDetection. This API is created to ease the inference process. The details of the codes can be found [here](https://github.com/open-mmlab/mmdetection/blob/master/mmdet/apis/inference.py#L15)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Wi6DRpsQPEmV",
    "outputId": "42a9dd39-edcb-49f1-e318-a3cd77f89eee"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<DetDataSample(\n",
      "\n",
      "    META INFORMATION\n",
      "    batch_input_shape: (800, 1216)\n",
      "    pad_shape: (800, 1216)\n",
      "    scale_factor: (1.8734375, 1.873536299765808)\n",
      "    img_id: 0\n",
      "    ori_shape: (427, 640)\n",
      "    img_shape: (800, 1199)\n",
      "    img_path: None\n",
      "\n",
      "    DATA FIELDS\n",
      "    gt_instances: <InstanceData(\n",
      "        \n",
      "            META INFORMATION\n",
      "        \n",
      "            DATA FIELDS\n",
      "            labels: tensor([], device='cuda:0', dtype=torch.int64)\n",
      "            bboxes: tensor([], device='cuda:0', size=(0, 4))\n",
      "            masks: BitmapMasks(num_masks=0, height=427, width=640)\n",
      "        ) at 0x79a3a9cfa0b0>\n",
      "    ignored_instances: <InstanceData(\n",
      "        \n",
      "            META INFORMATION\n",
      "        \n",
      "            DATA FIELDS\n",
      "            labels: tensor([], device='cuda:0', dtype=torch.int64)\n",
      "            bboxes: tensor([], device='cuda:0', size=(0, 4))\n",
      "            masks: BitmapMasks(num_masks=0, height=427, width=640)\n",
      "        ) at 0x79a3a9cfa0e0>\n",
      "    pred_instances: <InstanceData(\n",
      "        \n",
      "            META INFORMATION\n",
      "        \n",
      "            DATA FIELDS\n",
      "            labels: tensor([13,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2,  2, 13, 56,  2,  2,\n",
      "                         2,  0,  0,  2,  2,  2,  0,  2,  2,  2,  0,  2,  0,  2,  2,  2,  2,  2,\n",
      "                         0,  7,  7,  7,  2,  7,  7, 25,  2,  0,  2,  7,  7,  2,  2,  7],\n",
      "                       device='cuda:0')\n",
      "            bboxes: tensor([[219.8702, 177.0791, 458.3700, 382.1367],\n",
      "                        [480.5608, 110.4768, 523.1904, 130.3331],\n",
      "                        [167.2974, 108.4704, 219.3721, 139.7081],\n",
      "                        [432.7696, 104.8358, 482.8209, 132.2976],\n",
      "                        [293.9358, 116.1538, 381.3969, 150.1965],\n",
      "                        [189.7066, 107.6951, 296.3842, 154.8887],\n",
      "                        [608.6334, 111.2579, 635.6981, 136.8759],\n",
      "                        [  0.0000, 112.1323,  64.1121, 145.1977],\n",
      "                        [396.6710, 110.8235, 434.0617, 133.4602],\n",
      "                        [558.0748, 109.3840, 593.6632, 127.6792],\n",
      "                        [590.7142, 109.8773, 620.7469, 126.5424],\n",
      "                        [269.1886, 105.4429, 327.0177, 127.3188],\n",
      "                        [ 85.7339,  89.2483,  99.3820,  98.0990],\n",
      "                        [ 98.4353,  89.3797, 118.6270, 100.8068],\n",
      "                        [372.5294, 136.1234, 432.7725, 187.9528],\n",
      "                        [377.2267, 135.4557, 432.1877, 187.9861],\n",
      "                        [553.8447, 110.4809, 562.0096, 126.6809],\n",
      "                        [604.7581, 109.7503, 630.5582, 127.8616],\n",
      "                        [186.7537, 109.0132, 228.5729, 145.0153],\n",
      "                        [535.8760, 108.3876, 545.3403, 124.3354],\n",
      "                        [254.0453, 103.2393, 263.8707, 113.5147],\n",
      "                        [177.0499, 105.5779, 257.9825, 144.5236],\n",
      "                        [375.6868, 119.4991, 382.5288, 132.6492],\n",
      "                        [609.1486, 113.8924, 624.6574, 125.1476],\n",
      "                        [375.7215, 119.5242, 382.4901, 133.1054],\n",
      "                        [222.0856,  98.0710, 251.4666, 107.5862],\n",
      "                        [100.2685,  88.5259, 117.7641,  95.0929],\n",
      "                        [614.0423, 102.3213, 634.2089, 112.2706],\n",
      "                        [532.6091, 108.7414, 541.8567, 125.0906],\n",
      "                        [541.7024, 111.1429, 561.5145, 126.8495],\n",
      "                        [533.5267, 108.6170, 545.8909, 136.2182],\n",
      "                        [135.4252,  89.6805, 152.0995,  96.3841],\n",
      "                        [ 98.6447,  92.1244, 110.0710, 100.7798],\n",
      "                        [168.5464,  96.7550, 182.6953, 105.4320],\n",
      "                        [373.9429, 112.1656, 415.0832, 133.9632],\n",
      "                        [555.3735, 110.0341, 574.9896, 127.4099],\n",
      "                        [227.8013, 115.6852, 237.1723, 123.8744],\n",
      "                        [430.8856, 104.9880, 484.4790, 131.9029],\n",
      "                        [294.1423, 116.5301, 378.7433, 149.9078],\n",
      "                        [  7.9151, 101.6727, 121.0091, 142.4919],\n",
      "                        [364.8131, 108.6062, 437.1151, 133.0383],\n",
      "                        [392.4191, 111.1939, 434.2920, 133.6734],\n",
      "                        [187.0622, 108.6223, 300.6056, 155.1926],\n",
      "                        [ 13.0601,  93.1385,  25.3213, 126.1826],\n",
      "                        [ 80.9476,  89.6317,  92.6563,  97.9697],\n",
      "                        [241.6709, 116.3163, 252.0889, 125.3810],\n",
      "                        [598.0154, 103.6475, 634.9665, 124.7287],\n",
      "                        [ 82.1116,  89.2348,  99.1320,  97.9058],\n",
      "                        [ 98.8386,  89.3510, 119.2133, 101.0519],\n",
      "                        [  9.4168, 106.1699, 107.8712, 144.0217],\n",
      "                        [134.5102,  90.4127, 152.7260, 100.8205],\n",
      "                        [ 43.4677, 105.9956, 114.2341, 128.5489]], device='cuda:0')\n",
      "            masks: tensor([[[False, False, False,  ..., False, False, False],\n",
      "                         [False, False, False,  ..., False, False, False],\n",
      "                         [False, False, False,  ..., False, False, False],\n",
      "                         ...,\n",
      "                         [False, False, False,  ..., False, False, False],\n",
      "                         [False, False, False,  ..., False, False, False],\n",
      "                         [False, False, False,  ..., False, False, False]],\n",
      "                \n",
      "                        [[False, False, False,  ..., False, False, False],\n",
      "                         [False, False, False,  ..., False, False, False],\n",
      "                         [False, False, False,  ..., False, False, False],\n",
      "                         ...,\n",
      "                         [False, False, False,  ..., False, False, False],\n",
      "                         [False, False, False,  ..., False, False, False],\n",
      "                         [False, False, False,  ..., False, False, False]],\n",
      "                \n",
      "                        [[False, False, False,  ..., False, False, False],\n",
      "                         [False, False, False,  ..., False, False, False],\n",
      "                         [False, False, False,  ..., False, False, False],\n",
      "                         ...,\n",
      "                         [False, False, False,  ..., False, False, False],\n",
      "                         [False, False, False,  ..., False, False, False],\n",
      "                         [False, False, False,  ..., False, False, False]],\n",
      "                \n",
      "                        ...,\n",
      "                \n",
      "                        [[False, False, False,  ..., False, False, False],\n",
      "                         [False, False, False,  ..., False, False, False],\n",
      "                         [False, False, False,  ..., False, False, False],\n",
      "                         ...,\n",
      "                         [False, False, False,  ..., False, False, False],\n",
      "                         [False, False, False,  ..., False, False, False],\n",
      "                         [False, False, False,  ..., False, False, False]],\n",
      "                \n",
      "                        [[False, False, False,  ..., False, False, False],\n",
      "                         [False, False, False,  ..., False, False, False],\n",
      "                         [False, False, False,  ..., False, False, False],\n",
      "                         ...,\n",
      "                         [False, False, False,  ..., False, False, False],\n",
      "                         [False, False, False,  ..., False, False, False],\n",
      "                         [False, False, False,  ..., False, False, False]],\n",
      "                \n",
      "                        [[False, False, False,  ..., False, False, False],\n",
      "                         [False, False, False,  ..., False, False, False],\n",
      "                         [False, False, False,  ..., False, False, False],\n",
      "                         ...,\n",
      "                         [False, False, False,  ..., False, False, False],\n",
      "                         [False, False, False,  ..., False, False, False],\n",
      "                         [False, False, False,  ..., False, False, False]]], device='cuda:0')\n",
      "            scores: tensor([0.9990, 0.9938, 0.9878, 0.9854, 0.9836, 0.9829, 0.9781, 0.9779, 0.9731,\n",
      "                        0.9605, 0.9476, 0.9367, 0.6866, 0.6654, 0.6534, 0.5688, 0.4980, 0.3680,\n",
      "                        0.3513, 0.2965, 0.2664, 0.2557, 0.2344, 0.2336, 0.2222, 0.2082, 0.2000,\n",
      "                        0.1945, 0.1809, 0.1675, 0.1653, 0.1504, 0.1208, 0.1143, 0.1067, 0.1060,\n",
      "                        0.0810, 0.0800, 0.0776, 0.0751, 0.0732, 0.0700, 0.0686, 0.0653, 0.0638,\n",
      "                        0.0591, 0.0583, 0.0573, 0.0539, 0.0538, 0.0528, 0.0516],\n",
      "                       device='cuda:0')\n",
      "        ) at 0x79a3a9cfb550>\n",
      ") at 0x79a3c999fc10>\n"
     ]
    }
   ],
   "source": [
    "# Use the detector to do inference\n",
    "image = mmcv.imread('demo/demo.jpg',channel_order='rgb')\n",
    "result = inference_detector(model, image)\n",
    "print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4pFVhKeQRYYS"
   },
   "source": [
    "### Let's plot the result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "YinmJV1dRYYT",
    "outputId": "e6c9059f-55b3-481b-edef-b21befcbcf2e"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.10/dist-packages/mmengine/visualization/visualizer.py:196: UserWarning: Failed to add <class 'mmengine.visualization.vis_backend.LocalVisBackend'>, please provide the `save_dir` argument.\n",
      "  warnings.warn(f'Failed to add {vis_backend.__class__}, '\n"
     ]
    }
   ],
   "source": [
    "from mmdet.registry import VISUALIZERS\n",
    "# init visualizer(run the block only once in jupyter notebook)\n",
    "visualizer = VISUALIZERS.build(model.cfg.visualizer)\n",
    "# the dataset_meta is loaded from the checkpoint and\n",
    "# then pass to the model in init_detector\n",
    "visualizer.dataset_meta = model.dataset_meta"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 464
    },
    "id": "z6qT6pG1RYYT",
    "outputId": "089b652b-061f-480d-f9de-ffa06b7d385a"
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 909880 characters]",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# show the results\n",
    "visualizer.add_datasample(\n",
    "    'result',\n",
    "    image,\n",
    "    data_sample=result,\n",
    "    draw_gt = None,\n",
    "    wait_time=0,\n",
    ")\n",
    "visualizer.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7GrWIJywLV-V"
   },
   "source": [
    "## Train a Detector on A Customized Dataset\n",
    "\n",
    "To train a new detector, there are usually three things to do:\n",
    "1. Support a new dataset\n",
    "2. Modify the config\n",
    "3. Train a new detector\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "E73y5Lru-wBx"
   },
   "source": [
    "### Support a new dataset\n",
    "\n",
    "There are three ways to support a new dataset in MMDetection:\n",
    "  1. Reorganize the dataset into a COCO format\n",
    "  2. Reorganize the dataset into a middle format\n",
    "  3. Implement a new dataset\n",
    "\n",
    "We recommend the first two methods, as they are usually easier than the third.\n",
    "\n",
    "In this tutorial, we give an example that converts the data into COCO format because MMDetection **only support evaluating mask AP of dataset in COCO format for now**. Other methods and more advanced usages can be found in the [doc](https://mmdetection.readthedocs.io/en/latest/advanced_guides/customize_dataset.html).\n",
    "\n",
    "First, let's download the [the balloon dataset](https://github.com/matterport/Mask_RCNN/tree/master/samples/balloon)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "rHnw5Q_nARXq"
   },
   "outputs": [],
   "source": [
    "# download and unzip the data\n",
    "!wget -c https://github.com/matterport/Mask_RCNN/releases/download/v2.1/balloon_dataset.zip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ucSfn1U_RYYW"
   },
   "outputs": [],
   "source": [
    "!unzip balloon_dataset.zip -d ./ballondatasets/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Ro6JhfBVRYYX"
   },
   "source": [
    "# Check the directory structure of the tiny data\n",
    "\n",
    "# Install tree first in your terminal(linux)\n",
    "sudo apt-get -q install tree\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Wuwxw1oZRtVZ",
    "outputId": "2e472cfa-2e2f-41ea-ddec-5a9d49fe71cf"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/bin/bash: line 1: tree: command not found\n"
     ]
    }
   ],
   "source": [
    "!tree ballondatasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 836
    },
    "id": "YnQQqzOWzE91",
    "outputId": "ff7d3804-638c-461f-aef6-8c496a4b69c8"
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 581596 characters]",
      "text/plain": [
       "<Figure size 1500x1000 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Let's take a look at the dataset image\n",
    "import mmcv\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "img = mmcv.imread('ballondatasets/balloon/train/10464445726_6f1e3bbe6a_k.jpg')\n",
    "plt.figure(figsize=(15, 10))\n",
    "plt.imshow(mmcv.bgr2rgb(img))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "PMZvtSIl71qi"
   },
   "source": [
    "After downloading the data, we need to implement a function to convert the annotation format into the COCO format. Then we can use implemented `COCODataset` to load the data and perform training and evaluation.\n",
    "Let's take a look at the annotation json file.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "id": "n7rwalnPd6e1"
   },
   "outputs": [],
   "source": [
    "# Check the label of a single image\n",
    "import mmengine\n",
    "\n",
    "annotation = mmengine.load('./ballondatasets/balloon/train/via_region_data.json')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "keLW7uqJM54Y",
    "outputId": "8bdf087e-5ec0-4f8a-ee1d-5692986ac87d"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'fileref': '',\n",
       " 'size': 1115004,\n",
       " 'filename': '34020010494_e5cb88e1c4_k.jpg',\n",
       " 'base64_img_data': '',\n",
       " 'file_attributes': {},\n",
       " 'regions': {'0': {'shape_attributes': {'name': 'polygon',\n",
       "    'all_points_x': [1020,\n",
       "     1000,\n",
       "     994,\n",
       "     1003,\n",
       "     1023,\n",
       "     1050,\n",
       "     1089,\n",
       "     1134,\n",
       "     1190,\n",
       "     1265,\n",
       "     1321,\n",
       "     1361,\n",
       "     1403,\n",
       "     1428,\n",
       "     1442,\n",
       "     1445,\n",
       "     1441,\n",
       "     1427,\n",
       "     1400,\n",
       "     1361,\n",
       "     1316,\n",
       "     1269,\n",
       "     1228,\n",
       "     1198,\n",
       "     1207,\n",
       "     1210,\n",
       "     1190,\n",
       "     1177,\n",
       "     1172,\n",
       "     1174,\n",
       "     1170,\n",
       "     1153,\n",
       "     1127,\n",
       "     1104,\n",
       "     1061,\n",
       "     1032,\n",
       "     1020],\n",
       "    'all_points_y': [963,\n",
       "     899,\n",
       "     841,\n",
       "     787,\n",
       "     738,\n",
       "     700,\n",
       "     663,\n",
       "     638,\n",
       "     621,\n",
       "     619,\n",
       "     643,\n",
       "     672,\n",
       "     720,\n",
       "     765,\n",
       "     800,\n",
       "     860,\n",
       "     896,\n",
       "     942,\n",
       "     990,\n",
       "     1035,\n",
       "     1079,\n",
       "     1112,\n",
       "     1129,\n",
       "     1134,\n",
       "     1144,\n",
       "     1153,\n",
       "     1166,\n",
       "     1166,\n",
       "     1150,\n",
       "     1136,\n",
       "     1129,\n",
       "     1122,\n",
       "     1112,\n",
       "     1084,\n",
       "     1037,\n",
       "     989,\n",
       "     963]},\n",
       "   'region_attributes': {}}}}"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# The annotation is a dict, and its values looks like the following\n",
    "annotation['34020010494_e5cb88e1c4_k.jpg1115004']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "QA1pFg-FeO3l"
   },
   "source": [
    "According to the above observation, each single image has a corresponding annotation containing keys `filename` and `regions` that are necessary for training.\n",
    "We need to read annotations of each image and convert them into COCO format as below:\n",
    "\n",
    "```python\n",
    "{\n",
    "    \"images\": [image],\n",
    "    \"annotations\": [annotation],\n",
    "    \"categories\": [category]\n",
    "}\n",
    "\n",
    "\n",
    "image = {\n",
    "    \"id\": int,\n",
    "    \"width\": int,\n",
    "    \"height\": int,\n",
    "    \"file_name\": str,\n",
    "}\n",
    "\n",
    "annotation = {\n",
    "    \"id\": int,\n",
    "    \"image_id\": int,\n",
    "    \"category_id\": int,\n",
    "    \"segmentation\": RLE or [polygon],\n",
    "    \"area\": float,\n",
    "    \"bbox\": [x,y,width,height],\n",
    "    \"iscrowd\": 0 or 1,\n",
    "}\n",
    "\n",
    "categories = [{\n",
    "    \"id\": int,\n",
    "    \"name\": str,\n",
    "    \"supercategory\": str,\n",
    "}]\n",
    "```\n",
    "**Note**: We only list the necessary keys for training, as shown above. For a full COCO format, please see [here](https://cocodataset.org/#format-data)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "id": "GdSaB2ad0EdX"
   },
   "outputs": [],
   "source": [
    "import os.path as osp\n",
    "\n",
    "def convert_balloon_to_coco(ann_file, out_file, image_prefix):\n",
    "    data_infos = mmengine.load(ann_file)\n",
    "\n",
    "    annotations = []\n",
    "    images = []\n",
    "    obj_count = 0\n",
    "    for idx, v in enumerate(mmengine.track_iter_progress(list(data_infos.values()))):\n",
    "        filename = v['filename']\n",
    "        img_path = osp.join(image_prefix, filename)\n",
    "        height, width = mmcv.imread(img_path).shape[:2]\n",
    "\n",
    "        images.append(dict(\n",
    "            id=idx,\n",
    "            file_name=filename,\n",
    "            height=height,\n",
    "            width=width))\n",
    "\n",
    "        bboxes = []\n",
    "        labels = []\n",
    "        masks = []\n",
    "        for _, obj in v['regions'].items():\n",
    "            assert not obj['region_attributes']\n",
    "            obj = obj['shape_attributes']\n",
    "            px = obj['all_points_x']\n",
    "            py = obj['all_points_y']\n",
    "            poly = [(x + 0.5, y + 0.5) for x, y in zip(px, py)]\n",
    "            poly = [p for x in poly for p in x]\n",
    "\n",
    "            x_min, y_min, x_max, y_max = (\n",
    "                min(px), min(py), max(px), max(py))\n",
    "\n",
    "\n",
    "            data_anno = dict(\n",
    "                image_id=idx,\n",
    "                id=obj_count,\n",
    "                category_id=0,\n",
    "                bbox=[x_min, y_min, x_max - x_min, y_max - y_min],\n",
    "                area=(x_max - x_min) * (y_max - y_min),\n",
    "                segmentation=[poly],\n",
    "                iscrowd=0)\n",
    "            annotations.append(data_anno)\n",
    "            obj_count += 1\n",
    "\n",
    "    coco_format_json = dict(\n",
    "        images=images,\n",
    "        annotations=annotations,\n",
    "        categories=[{'id':0, 'name': 'balloon'}])\n",
    "    mmengine.dump(coco_format_json, out_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "G3xV5ktqlpFu",
    "outputId": "2d97137b-34e6-42e5-c8d6-0a4fe7d2c7cf"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 61/61, 19.7 task/s, elapsed: 3s, ETA:     0s\n",
      "[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 13/13, 20.1 task/s, elapsed: 1s, ETA:     0s\n"
     ]
    }
   ],
   "source": [
    "convert_balloon_to_coco(\n",
    "    './ballondatasets/balloon/train/via_region_data.json',\n",
    "    './ballondatasets/balloon/train/annotation_coco.json',\n",
    "    './ballondatasets/balloon/train/')\n",
    "convert_balloon_to_coco(\n",
    "    './ballondatasets/balloon/val/via_region_data.json',\n",
    "    './ballondatasets/balloon/val/annotation_coco.json',\n",
    "    './ballondatasets/balloon/val/')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "h85AtunjRvx4"
   },
   "source": [
    "Checking the label corresponding to the instance split ID after the data format conversion is complete"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "zaYkWbxORwZq",
    "outputId": "02ad1ff6-f138-49af-b733-1d23c51557f5"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loading annotations into memory...\n",
      "Done (t=0.01s)\n",
      "creating index...\n",
      "index created!\n",
      "Category ID: 0, Category Name: balloon\n"
     ]
    }
   ],
   "source": [
    "from pycocotools.coco import COCO\n",
    "\n",
    "# Path to load the COCO annotation file\n",
    "annotation_file = './ballondatasets/balloon/train/annotation_coco.json'\n",
    "\n",
    "# Initialise the COCO object\n",
    "coco = COCO(annotation_file)\n",
    "\n",
    "# Get all category tags and corresponding category IDs\n",
    "categories = coco.loadCats(coco.getCatIds())\n",
    "category_id_to_name = {cat['id']: cat['name'] for cat in categories}\n",
    "\n",
    "# Print all category IDs and corresponding category names\n",
    "for category_id, category_name in category_id_to_name.items():\n",
    "    print(f\"Category ID: {category_id}, Category Name: {category_name}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "PwqJOpBe-bMj"
   },
   "source": [
    "### Modify the config\n",
    "\n",
    "In the next step, we need to modify the config for the training.\n",
    "To accelerate the process, we finetune a detector using a pre-trained detector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "id": "hamZrlnH-YDD"
   },
   "outputs": [],
   "source": [
    "from mmengine import Config\n",
    "cfg = Config.fromfile('./configs/mask_rcnn/mask-rcnn_r50-caffe_fpn_ms-poly-1x_coco.py')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HntziLGq-92Z"
   },
   "source": [
    "Given a config that trains a Mask R-CNN on COCO dataset, we need to modify some values to use it for training on the balloon dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "id": "pUbwD8uV0PR8"
   },
   "outputs": [],
   "source": [
    "from mmengine.runner import set_random_seed\n",
    "\n",
    "# Modify dataset classes and color\n",
    "cfg.metainfo = {\n",
    "    'classes': ('balloon', ),\n",
    "    'palette': [\n",
    "        (220, 20, 60),\n",
    "    ]\n",
    "}\n",
    "\n",
    "# Modify dataset type and path\n",
    "cfg.data_root = './ballondatasets/balloon'\n",
    "\n",
    "cfg.train_dataloader.dataset.ann_file = 'train/annotation_coco.json'\n",
    "cfg.train_dataloader.dataset.data_root = cfg.data_root\n",
    "cfg.train_dataloader.dataset.data_prefix.img = 'train/'\n",
    "cfg.train_dataloader.dataset.metainfo = cfg.metainfo\n",
    "\n",
    "cfg.val_dataloader.dataset.ann_file = 'val/annotation_coco.json'\n",
    "cfg.val_dataloader.dataset.data_root = cfg.data_root\n",
    "cfg.val_dataloader.dataset.data_prefix.img = 'val/'\n",
    "cfg.val_dataloader.dataset.metainfo = cfg.metainfo\n",
    "\n",
    "cfg.test_dataloader = cfg.val_dataloader\n",
    "\n",
    "# Modify metric config\n",
    "cfg.val_evaluator.ann_file = cfg.data_root+'/'+'val/annotation_coco.json'\n",
    "cfg.test_evaluator = cfg.val_evaluator\n",
    "\n",
    "# Modify num classes of the model in box head and mask head\n",
    "cfg.model.roi_head.bbox_head.num_classes = 1\n",
    "cfg.model.roi_head.mask_head.num_classes = 1\n",
    "\n",
    "# We can still the pre-trained Mask RCNN model to obtain a higher performance\n",
    "cfg.load_from = 'checkpoints/mask_rcnn_r50_caffe_fpn_mstrain-poly_3x_coco_bbox_mAP-0.408__segm_mAP-0.37_20200504_163245-42aa3d00.pth'\n",
    "\n",
    "# Set up working dir to save files and logs.\n",
    "cfg.work_dir = './tutorial_exps'\n",
    "\n",
    "\n",
    "# We can set the evaluation interval to reduce the evaluation times\n",
    "cfg.train_cfg.val_interval = 3\n",
    "# We can set the checkpoint saving interval to reduce the storage cost\n",
    "cfg.default_hooks.checkpoint.interval = 3\n",
    "\n",
    "# The original learning rate (LR) is set for 8-GPU training.\n",
    "# We divide it by 8 since we only use one GPU.\n",
    "cfg.optim_wrapper.optimizer.lr = 0.02 / 8\n",
    "cfg.default_hooks.logger.interval = 10\n",
    "\n",
    "\n",
    "# Set seed thus the results are more reproducible\n",
    "# cfg.seed = 0\n",
    "set_random_seed(0, deterministic=False)\n",
    "\n",
    "# We can also use tensorboard to log the training process\n",
    "cfg.visualizer.vis_backends.append({\"type\":'TensorboardVisBackend'})\n",
    "\n",
    "#------------------------------------------------------\n",
    "config=f'./configs/mask_rcnn/mask-rcnn_r50-caffe_fpn_ms-poly-3x_balloon.py'\n",
    "with open(config, 'w') as f:\n",
    "    f.write(cfg.pretty_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "111W_oZV_3wa"
   },
   "source": [
    "### Train a new detector\n",
    "\n",
    "Finally, lets initialize the dataset and detector, then train a new detector!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "JiqDnPdAMGyg",
    "outputId": "0de25679-3541-488e-eceb-5b5400f92745"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "08/15 04:31:57 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - \n",
      "------------------------------------------------------------\n",
      "System environment:\n",
      "    sys.platform: linux\n",
      "    Python: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0]\n",
      "    CUDA available: True\n",
      "    numpy_random_seed: 1186080067\n",
      "    GPU 0: Tesla T4\n",
      "    CUDA_HOME: /usr/local/cuda\n",
      "    NVCC: Cuda compilation tools, release 11.8, V11.8.89\n",
      "    GCC: x86_64-linux-gnu-gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0\n",
      "    PyTorch: 2.0.1+cu118\n",
      "    PyTorch compiling details: PyTorch built with:\n",
      "  - GCC 9.3\n",
      "  - C++ Version: 201703\n",
      "  - Intel(R) oneAPI Math Kernel Library Version 2022.2-Product Build 20220804 for Intel(R) 64 architecture applications\n",
      "  - Intel(R) MKL-DNN v2.7.3 (Git Hash 6dbeffbae1f23cbbeae17adb7b5b13f1f37c080e)\n",
      "  - OpenMP 201511 (a.k.a. OpenMP 4.5)\n",
      "  - LAPACK is enabled (usually provided by MKL)\n",
      "  - NNPACK is enabled\n",
      "  - CPU capability usage: AVX2\n",
      "  - CUDA Runtime 11.8\n",
      "  - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90\n",
      "  - CuDNN 8.7\n",
      "  - Magma 2.6.1\n",
      "  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.8, CUDNN_VERSION=8.7.0, CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 -fabi-version=11 -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wunused-local-typedefs -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_DISABLE_GPU_ASSERTS=ON, TORCH_VERSION=2.0.1, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=1, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, \n",
      "\n",
      "    TorchVision: 0.15.2+cu118\n",
      "    OpenCV: 4.8.0\n",
      "    MMEngine: 0.8.4\n",
      "\n",
      "Runtime environment:\n",
      "    cudnn_benchmark: False\n",
      "    dist_cfg: {'backend': 'nccl'}\n",
      "    mp_cfg: {'mp_start_method': 'fork', 'opencv_num_threads': 0}\n",
      "    seed: 1186080067\n",
      "    Distributed launcher: none\n",
      "    Distributed training: False\n",
      "    GPU number: 1\n",
      "------------------------------------------------------------\n",
      "\n",
      "08/15 04:31:57 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Config:\n",
      "auto_scale_lr = dict(base_batch_size=16, enable=False)\n",
      "backend_args = None\n",
      "data_root = './ballondatasets/balloon'\n",
      "dataset_type = 'CocoDataset'\n",
      "default_hooks = dict(\n",
      "    checkpoint=dict(interval=3, type='CheckpointHook'),\n",
      "    logger=dict(interval=10, type='LoggerHook'),\n",
      "    param_scheduler=dict(type='ParamSchedulerHook'),\n",
      "    sampler_seed=dict(type='DistSamplerSeedHook'),\n",
      "    timer=dict(type='IterTimerHook'),\n",
      "    visualization=dict(type='DetVisualizationHook'))\n",
      "default_scope = 'mmdet'\n",
      "env_cfg = dict(\n",
      "    cudnn_benchmark=False,\n",
      "    dist_cfg=dict(backend='nccl'),\n",
      "    mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0))\n",
      "launcher = 'none'\n",
      "load_from = 'checkpoints/mask_rcnn_r50_caffe_fpn_mstrain-poly_3x_coco_bbox_mAP-0.408__segm_mAP-0.37_20200504_163245-42aa3d00.pth'\n",
      "log_level = 'INFO'\n",
      "log_processor = dict(by_epoch=True, type='LogProcessor', window_size=50)\n",
      "metainfo = dict(\n",
      "    classes=('balloon', ), palette=[\n",
      "        (\n",
      "            220,\n",
      "            20,\n",
      "            60,\n",
      "        ),\n",
      "    ])\n",
      "model = dict(\n",
      "    backbone=dict(\n",
      "        depth=50,\n",
      "        frozen_stages=1,\n",
      "        init_cfg=dict(\n",
      "            checkpoint='open-mmlab://detectron2/resnet50_caffe',\n",
      "            type='Pretrained'),\n",
      "        norm_cfg=dict(requires_grad=False, type='BN'),\n",
      "        norm_eval=True,\n",
      "        num_stages=4,\n",
      "        out_indices=(\n",
      "            0,\n",
      "            1,\n",
      "            2,\n",
      "            3,\n",
      "        ),\n",
      "        style='caffe',\n",
      "        type='ResNet'),\n",
      "    data_preprocessor=dict(\n",
      "        bgr_to_rgb=False,\n",
      "        mean=[\n",
      "            103.53,\n",
      "            116.28,\n",
      "            123.675,\n",
      "        ],\n",
      "        pad_mask=True,\n",
      "        pad_size_divisor=32,\n",
      "        std=[\n",
      "            1.0,\n",
      "            1.0,\n",
      "            1.0,\n",
      "        ],\n",
      "        type='DetDataPreprocessor'),\n",
      "    neck=dict(\n",
      "        in_channels=[\n",
      "            256,\n",
      "            512,\n",
      "            1024,\n",
      "            2048,\n",
      "        ],\n",
      "        num_outs=5,\n",
      "        out_channels=256,\n",
      "        type='FPN'),\n",
      "    roi_head=dict(\n",
      "        bbox_head=dict(\n",
      "            bbox_coder=dict(\n",
      "                target_means=[\n",
      "                    0.0,\n",
      "                    0.0,\n",
      "                    0.0,\n",
      "                    0.0,\n",
      "                ],\n",
      "                target_stds=[\n",
      "                    0.1,\n",
      "                    0.1,\n",
      "                    0.2,\n",
      "                    0.2,\n",
      "                ],\n",
      "                type='DeltaXYWHBBoxCoder'),\n",
      "            fc_out_channels=1024,\n",
      "            in_channels=256,\n",
      "            loss_bbox=dict(loss_weight=1.0, type='L1Loss'),\n",
      "            loss_cls=dict(\n",
      "                loss_weight=1.0, type='CrossEntropyLoss', use_sigmoid=False),\n",
      "            num_classes=1,\n",
      "            reg_class_agnostic=False,\n",
      "            roi_feat_size=7,\n",
      "            type='Shared2FCBBoxHead'),\n",
      "        bbox_roi_extractor=dict(\n",
      "            featmap_strides=[\n",
      "                4,\n",
      "                8,\n",
      "                16,\n",
      "                32,\n",
      "            ],\n",
      "            out_channels=256,\n",
      "            roi_layer=dict(output_size=7, sampling_ratio=0, type='RoIAlign'),\n",
      "            type='SingleRoIExtractor'),\n",
      "        mask_head=dict(\n",
      "            conv_out_channels=256,\n",
      "            in_channels=256,\n",
      "            loss_mask=dict(\n",
      "                loss_weight=1.0, type='CrossEntropyLoss', use_mask=True),\n",
      "            num_classes=1,\n",
      "            num_convs=4,\n",
      "            type='FCNMaskHead'),\n",
      "        mask_roi_extractor=dict(\n",
      "            featmap_strides=[\n",
      "                4,\n",
      "                8,\n",
      "                16,\n",
      "                32,\n",
      "            ],\n",
      "            out_channels=256,\n",
      "            roi_layer=dict(output_size=14, sampling_ratio=0, type='RoIAlign'),\n",
      "            type='SingleRoIExtractor'),\n",
      "        type='StandardRoIHead'),\n",
      "    rpn_head=dict(\n",
      "        anchor_generator=dict(\n",
      "            ratios=[\n",
      "                0.5,\n",
      "                1.0,\n",
      "                2.0,\n",
      "            ],\n",
      "            scales=[\n",
      "                8,\n",
      "            ],\n",
      "            strides=[\n",
      "                4,\n",
      "                8,\n",
      "                16,\n",
      "                32,\n",
      "                64,\n",
      "            ],\n",
      "            type='AnchorGenerator'),\n",
      "        bbox_coder=dict(\n",
      "            target_means=[\n",
      "                0.0,\n",
      "                0.0,\n",
      "                0.0,\n",
      "                0.0,\n",
      "            ],\n",
      "            target_stds=[\n",
      "                1.0,\n",
      "                1.0,\n",
      "                1.0,\n",
      "                1.0,\n",
      "            ],\n",
      "            type='DeltaXYWHBBoxCoder'),\n",
      "        feat_channels=256,\n",
      "        in_channels=256,\n",
      "        loss_bbox=dict(loss_weight=1.0, type='L1Loss'),\n",
      "        loss_cls=dict(\n",
      "            loss_weight=1.0, type='CrossEntropyLoss', use_sigmoid=True),\n",
      "        type='RPNHead'),\n",
      "    test_cfg=dict(\n",
      "        rcnn=dict(\n",
      "            mask_thr_binary=0.5,\n",
      "            max_per_img=100,\n",
      "            nms=dict(iou_threshold=0.5, type='nms'),\n",
      "            score_thr=0.05),\n",
      "        rpn=dict(\n",
      "            max_per_img=1000,\n",
      "            min_bbox_size=0,\n",
      "            nms=dict(iou_threshold=0.7, type='nms'),\n",
      "            nms_pre=1000)),\n",
      "    train_cfg=dict(\n",
      "        rcnn=dict(\n",
      "            assigner=dict(\n",
      "                ignore_iof_thr=-1,\n",
      "                match_low_quality=True,\n",
      "                min_pos_iou=0.5,\n",
      "                neg_iou_thr=0.5,\n",
      "                pos_iou_thr=0.5,\n",
      "                type='MaxIoUAssigner'),\n",
      "            debug=False,\n",
      "            mask_size=28,\n",
      "            pos_weight=-1,\n",
      "            sampler=dict(\n",
      "                add_gt_as_proposals=True,\n",
      "                neg_pos_ub=-1,\n",
      "                num=512,\n",
      "                pos_fraction=0.25,\n",
      "                type='RandomSampler')),\n",
      "        rpn=dict(\n",
      "            allowed_border=-1,\n",
      "            assigner=dict(\n",
      "                ignore_iof_thr=-1,\n",
      "                match_low_quality=True,\n",
      "                min_pos_iou=0.3,\n",
      "                neg_iou_thr=0.3,\n",
      "                pos_iou_thr=0.7,\n",
      "                type='MaxIoUAssigner'),\n",
      "            debug=False,\n",
      "            pos_weight=-1,\n",
      "            sampler=dict(\n",
      "                add_gt_as_proposals=False,\n",
      "                neg_pos_ub=-1,\n",
      "                num=256,\n",
      "                pos_fraction=0.5,\n",
      "                type='RandomSampler')),\n",
      "        rpn_proposal=dict(\n",
      "            max_per_img=1000,\n",
      "            min_bbox_size=0,\n",
      "            nms=dict(iou_threshold=0.7, type='nms'),\n",
      "            nms_pre=2000)),\n",
      "    type='MaskRCNN')\n",
      "optim_wrapper = dict(\n",
      "    optimizer=dict(lr=0.0025, momentum=0.9, type='SGD', weight_decay=0.0001),\n",
      "    type='OptimWrapper')\n",
      "param_scheduler = [\n",
      "    dict(\n",
      "        begin=0, by_epoch=False, end=500, start_factor=0.001, type='LinearLR'),\n",
      "    dict(\n",
      "        begin=0,\n",
      "        by_epoch=True,\n",
      "        end=12,\n",
      "        gamma=0.1,\n",
      "        milestones=[\n",
      "            8,\n",
      "            11,\n",
      "        ],\n",
      "        type='MultiStepLR'),\n",
      "]\n",
      "resume = False\n",
      "test_cfg = dict(type='TestLoop')\n",
      "test_dataloader = dict(\n",
      "    batch_size=1,\n",
      "    dataset=dict(\n",
      "        ann_file='val/annotation_coco.json',\n",
      "        backend_args=None,\n",
      "        data_prefix=dict(img='val/'),\n",
      "        data_root='./ballondatasets/balloon',\n",
      "        metainfo=dict(classes=('balloon', ), palette=[\n",
      "            (\n",
      "                220,\n",
      "                20,\n",
      "                60,\n",
      "            ),\n",
      "        ]),\n",
      "        pipeline=[\n",
      "            dict(backend_args=None, type='LoadImageFromFile'),\n",
      "            dict(keep_ratio=True, scale=(\n",
      "                1333,\n",
      "                800,\n",
      "            ), type='Resize'),\n",
      "            dict(type='LoadAnnotations', with_bbox=True, with_mask=True),\n",
      "            dict(\n",
      "                meta_keys=(\n",
      "                    'img_id',\n",
      "                    'img_path',\n",
      "                    'ori_shape',\n",
      "                    'img_shape',\n",
      "                    'scale_factor',\n",
      "                ),\n",
      "                type='PackDetInputs'),\n",
      "        ],\n",
      "        test_mode=True,\n",
      "        type='CocoDataset'),\n",
      "    drop_last=False,\n",
      "    num_workers=2,\n",
      "    persistent_workers=True,\n",
      "    sampler=dict(shuffle=False, type='DefaultSampler'))\n",
      "test_evaluator = dict(\n",
      "    ann_file='./ballondatasets/balloon/val/annotation_coco.json',\n",
      "    backend_args=None,\n",
      "    format_only=False,\n",
      "    metric=[\n",
      "        'bbox',\n",
      "        'segm',\n",
      "    ],\n",
      "    type='CocoMetric')\n",
      "test_pipeline = [\n",
      "    dict(backend_args=None, type='LoadImageFromFile'),\n",
      "    dict(keep_ratio=True, scale=(\n",
      "        1333,\n",
      "        800,\n",
      "    ), type='Resize'),\n",
      "    dict(type='LoadAnnotations', with_bbox=True, with_mask=True),\n",
      "    dict(\n",
      "        meta_keys=(\n",
      "            'img_id',\n",
      "            'img_path',\n",
      "            'ori_shape',\n",
      "            'img_shape',\n",
      "            'scale_factor',\n",
      "        ),\n",
      "        type='PackDetInputs'),\n",
      "]\n",
      "train_cfg = dict(max_epochs=12, type='EpochBasedTrainLoop', val_interval=3)\n",
      "train_dataloader = dict(\n",
      "    batch_sampler=dict(type='AspectRatioBatchSampler'),\n",
      "    batch_size=2,\n",
      "    dataset=dict(\n",
      "        ann_file='train/annotation_coco.json',\n",
      "        backend_args=None,\n",
      "        data_prefix=dict(img='train/'),\n",
      "        data_root='./ballondatasets/balloon',\n",
      "        filter_cfg=dict(filter_empty_gt=True, min_size=32),\n",
      "        metainfo=dict(classes=('balloon', ), palette=[\n",
      "            (\n",
      "                220,\n",
      "                20,\n",
      "                60,\n",
      "            ),\n",
      "        ]),\n",
      "        pipeline=[\n",
      "            dict(backend_args=None, type='LoadImageFromFile'),\n",
      "            dict(\n",
      "                poly2mask=False,\n",
      "                type='LoadAnnotations',\n",
      "                with_bbox=True,\n",
      "                with_mask=True),\n",
      "            dict(\n",
      "                keep_ratio=True,\n",
      "                scales=[\n",
      "                    (\n",
      "                        1333,\n",
      "                        640,\n",
      "                    ),\n",
      "                    (\n",
      "                        1333,\n",
      "                        672,\n",
      "                    ),\n",
      "                    (\n",
      "                        1333,\n",
      "                        704,\n",
      "                    ),\n",
      "                    (\n",
      "                        1333,\n",
      "                        736,\n",
      "                    ),\n",
      "                    (\n",
      "                        1333,\n",
      "                        768,\n",
      "                    ),\n",
      "                    (\n",
      "                        1333,\n",
      "                        800,\n",
      "                    ),\n",
      "                ],\n",
      "                type='RandomChoiceResize'),\n",
      "            dict(prob=0.5, type='RandomFlip'),\n",
      "            dict(type='PackDetInputs'),\n",
      "        ],\n",
      "        type='CocoDataset'),\n",
      "    num_workers=2,\n",
      "    persistent_workers=True,\n",
      "    sampler=dict(shuffle=True, type='DefaultSampler'))\n",
      "train_pipeline = [\n",
      "    dict(backend_args=None, type='LoadImageFromFile'),\n",
      "    dict(\n",
      "        poly2mask=False,\n",
      "        type='LoadAnnotations',\n",
      "        with_bbox=True,\n",
      "        with_mask=True),\n",
      "    dict(\n",
      "        keep_ratio=True,\n",
      "        scales=[\n",
      "            (\n",
      "                1333,\n",
      "                640,\n",
      "            ),\n",
      "            (\n",
      "                1333,\n",
      "                672,\n",
      "            ),\n",
      "            (\n",
      "                1333,\n",
      "                704,\n",
      "            ),\n",
      "            (\n",
      "                1333,\n",
      "                736,\n",
      "            ),\n",
      "            (\n",
      "                1333,\n",
      "                768,\n",
      "            ),\n",
      "            (\n",
      "                1333,\n",
      "                800,\n",
      "            ),\n",
      "        ],\n",
      "        type='RandomChoiceResize'),\n",
      "    dict(prob=0.5, type='RandomFlip'),\n",
      "    dict(type='PackDetInputs'),\n",
      "]\n",
      "val_cfg = dict(type='ValLoop')\n",
      "val_dataloader = dict(\n",
      "    batch_size=1,\n",
      "    dataset=dict(\n",
      "        ann_file='val/annotation_coco.json',\n",
      "        backend_args=None,\n",
      "        data_prefix=dict(img='val/'),\n",
      "        data_root='./ballondatasets/balloon',\n",
      "        metainfo=dict(classes=('balloon', ), palette=[\n",
      "            (\n",
      "                220,\n",
      "                20,\n",
      "                60,\n",
      "            ),\n",
      "        ]),\n",
      "        pipeline=[\n",
      "            dict(backend_args=None, type='LoadImageFromFile'),\n",
      "            dict(keep_ratio=True, scale=(\n",
      "                1333,\n",
      "                800,\n",
      "            ), type='Resize'),\n",
      "            dict(type='LoadAnnotations', with_bbox=True, with_mask=True),\n",
      "            dict(\n",
      "                meta_keys=(\n",
      "                    'img_id',\n",
      "                    'img_path',\n",
      "                    'ori_shape',\n",
      "                    'img_shape',\n",
      "                    'scale_factor',\n",
      "                ),\n",
      "                type='PackDetInputs'),\n",
      "        ],\n",
      "        test_mode=True,\n",
      "        type='CocoDataset'),\n",
      "    drop_last=False,\n",
      "    num_workers=2,\n",
      "    persistent_workers=True,\n",
      "    sampler=dict(shuffle=False, type='DefaultSampler'))\n",
      "val_evaluator = dict(\n",
      "    ann_file='./ballondatasets/balloon/val/annotation_coco.json',\n",
      "    backend_args=None,\n",
      "    format_only=False,\n",
      "    metric=[\n",
      "        'bbox',\n",
      "        'segm',\n",
      "    ],\n",
      "    type='CocoMetric')\n",
      "vis_backends = [\n",
      "    dict(type='LocalVisBackend'),\n",
      "]\n",
      "visualizer = dict(\n",
      "    name='visualizer',\n",
      "    type='DetLocalVisualizer',\n",
      "    vis_backends=[\n",
      "        dict(type='LocalVisBackend'),\n",
      "        dict(type='TensorboardVisBackend'),\n",
      "    ])\n",
      "work_dir = './tutorial_exps'\n",
      "\n",
      "2023-08-15 04:31:59.033157: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.\n",
      "To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
      "2023-08-15 04:32:00.371943: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT\n",
      "08/15 04:32:04 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Distributed training is not used, all SyncBatchNorm (SyncBN) layers in the model will be automatically reverted to BatchNormXd layers if they are used.\n",
      "08/15 04:32:04 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Hooks will be executed in the following order:\n",
      "before_run:\n",
      "(VERY_HIGH   ) RuntimeInfoHook                    \n",
      "(BELOW_NORMAL) LoggerHook                         \n",
      " -------------------- \n",
      "before_train:\n",
      "(VERY_HIGH   ) RuntimeInfoHook                    \n",
      "(NORMAL      ) IterTimerHook                      \n",
      "(VERY_LOW    ) CheckpointHook                     \n",
      " -------------------- \n",
      "before_train_epoch:\n",
      "(VERY_HIGH   ) RuntimeInfoHook                    \n",
      "(NORMAL      ) IterTimerHook                      \n",
      "(NORMAL      ) DistSamplerSeedHook                \n",
      " -------------------- \n",
      "before_train_iter:\n",
      "(VERY_HIGH   ) RuntimeInfoHook                    \n",
      "(NORMAL      ) IterTimerHook                      \n",
      " -------------------- \n",
      "after_train_iter:\n",
      "(VERY_HIGH   ) RuntimeInfoHook                    \n",
      "(NORMAL      ) IterTimerHook                      \n",
      "(BELOW_NORMAL) LoggerHook                         \n",
      "(LOW         ) ParamSchedulerHook                 \n",
      "(VERY_LOW    ) CheckpointHook                     \n",
      " -------------------- \n",
      "after_train_epoch:\n",
      "(NORMAL      ) IterTimerHook                      \n",
      "(LOW         ) ParamSchedulerHook                 \n",
      "(VERY_LOW    ) CheckpointHook                     \n",
      " -------------------- \n",
      "before_val:\n",
      "(VERY_HIGH   ) RuntimeInfoHook                    \n",
      " -------------------- \n",
      "before_val_epoch:\n",
      "(NORMAL      ) IterTimerHook                      \n",
      " -------------------- \n",
      "before_val_iter:\n",
      "(NORMAL      ) IterTimerHook                      \n",
      " -------------------- \n",
      "after_val_iter:\n",
      "(NORMAL      ) IterTimerHook                      \n",
      "(NORMAL      ) DetVisualizationHook               \n",
      "(BELOW_NORMAL) LoggerHook                         \n",
      " -------------------- \n",
      "after_val_epoch:\n",
      "(VERY_HIGH   ) RuntimeInfoHook                    \n",
      "(NORMAL      ) IterTimerHook                      \n",
      "(BELOW_NORMAL) LoggerHook                         \n",
      "(LOW         ) ParamSchedulerHook                 \n",
      "(VERY_LOW    ) CheckpointHook                     \n",
      " -------------------- \n",
      "after_val:\n",
      "(VERY_HIGH   ) RuntimeInfoHook                    \n",
      " -------------------- \n",
      "after_train:\n",
      "(VERY_HIGH   ) RuntimeInfoHook                    \n",
      "(VERY_LOW    ) CheckpointHook                     \n",
      " -------------------- \n",
      "before_test:\n",
      "(VERY_HIGH   ) RuntimeInfoHook                    \n",
      " -------------------- \n",
      "before_test_epoch:\n",
      "(NORMAL      ) IterTimerHook                      \n",
      " -------------------- \n",
      "before_test_iter:\n",
      "(NORMAL      ) IterTimerHook                      \n",
      " -------------------- \n",
      "after_test_iter:\n",
      "(NORMAL      ) IterTimerHook                      \n",
      "(NORMAL      ) DetVisualizationHook               \n",
      "(BELOW_NORMAL) LoggerHook                         \n",
      " -------------------- \n",
      "after_test_epoch:\n",
      "(VERY_HIGH   ) RuntimeInfoHook                    \n",
      "(NORMAL      ) IterTimerHook                      \n",
      "(BELOW_NORMAL) LoggerHook                         \n",
      " -------------------- \n",
      "after_test:\n",
      "(VERY_HIGH   ) RuntimeInfoHook                    \n",
      " -------------------- \n",
      "after_run:\n",
      "(BELOW_NORMAL) LoggerHook                         \n",
      " -------------------- \n",
      "loading annotations into memory...\n",
      "Done (t=0.01s)\n",
      "creating index...\n",
      "index created!\n",
      "loading annotations into memory...\n",
      "Done (t=0.00s)\n",
      "creating index...\n",
      "index created!\n",
      "loading annotations into memory...\n",
      "Done (t=0.00s)\n",
      "creating index...\n",
      "index created!\n",
      "08/15 04:32:05 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - load model from: open-mmlab://detectron2/resnet50_caffe\n",
      "08/15 04:32:05 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Loads checkpoint by openmmlab backend from path: open-mmlab://detectron2/resnet50_caffe\n",
      "Downloading: \"https://download.openmmlab.com/pretrain/third_party/resnet50_msra-5891d200.pth\" to /root/.cache/torch/hub/checkpoints/resnet50_msra-5891d200.pth\n",
      "100% 89.9M/89.9M [00:12<00:00, 7.43MB/s]\n",
      "08/15 04:32:19 - mmengine - \u001b[5m\u001b[4m\u001b[33mWARNING\u001b[0m - The model and loaded state dict do not match exactly\n",
      "\n",
      "unexpected key in source state_dict: conv1.bias\n",
      "\n",
      "Loads checkpoint by local backend from path: checkpoints/mask_rcnn_r50_caffe_fpn_mstrain-poly_3x_coco_bbox_mAP-0.408__segm_mAP-0.37_20200504_163245-42aa3d00.pth\n",
      "The model and loaded state dict do not match exactly\n",
      "\n",
      "size mismatch for roi_head.bbox_head.fc_cls.weight: copying a param with shape torch.Size([81, 1024]) from checkpoint, the shape in current model is torch.Size([2, 1024]).\n",
      "size mismatch for roi_head.bbox_head.fc_cls.bias: copying a param with shape torch.Size([81]) from checkpoint, the shape in current model is torch.Size([2]).\n",
      "size mismatch for roi_head.bbox_head.fc_reg.weight: copying a param with shape torch.Size([320, 1024]) from checkpoint, the shape in current model is torch.Size([4, 1024]).\n",
      "size mismatch for roi_head.bbox_head.fc_reg.bias: copying a param with shape torch.Size([320]) from checkpoint, the shape in current model is torch.Size([4]).\n",
      "size mismatch for roi_head.mask_head.conv_logits.weight: copying a param with shape torch.Size([80, 256, 1, 1]) from checkpoint, the shape in current model is torch.Size([1, 256, 1, 1]).\n",
      "size mismatch for roi_head.mask_head.conv_logits.bias: copying a param with shape torch.Size([80]) from checkpoint, the shape in current model is torch.Size([1]).\n",
      "08/15 04:32:19 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Load checkpoint from checkpoints/mask_rcnn_r50_caffe_fpn_mstrain-poly_3x_coco_bbox_mAP-0.408__segm_mAP-0.37_20200504_163245-42aa3d00.pth\n",
      "08/15 04:32:19 - mmengine - \u001b[5m\u001b[4m\u001b[33mWARNING\u001b[0m - \"FileClient\" will be deprecated in future. Please use io functions in https://mmengine.readthedocs.io/en/latest/api/fileio.html#file-io\n",
      "08/15 04:32:19 - mmengine - \u001b[5m\u001b[4m\u001b[33mWARNING\u001b[0m - \"HardDiskBackend\" is the alias of \"LocalBackend\" and the former will be deprecated in future.\n",
      "08/15 04:32:19 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Checkpoints will be saved to /content/mmdetection/tutorial_exps.\n",
      "08/15 04:32:25 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train)  [1][10/31]  lr: 4.7545e-05  eta: 0:03:33  time: 0.5898  data_time: 0.0359  memory: 3283  loss: 16.7169  loss_rpn_cls: 0.1373  loss_rpn_bbox: 0.0201  loss_cls: 0.6155  acc: 83.3984  loss_bbox: 0.3727  loss_mask: 15.5713\n",
      "08/15 04:32:30 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train)  [1][20/31]  lr: 9.7595e-05  eta: 0:03:05  time: 0.5271  data_time: 0.0214  memory: 3283  loss: 10.8499  loss_rpn_cls: 0.0836  loss_rpn_bbox: 0.0148  loss_cls: 0.5401  acc: 84.6680  loss_bbox: 0.2949  loss_mask: 9.9164\n",
      "08/15 04:32:35 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train)  [1][30/31]  lr: 1.4765e-04  eta: 0:03:02  time: 0.5348  data_time: 0.0180  memory: 3283  loss: 7.7485  loss_rpn_cls: 0.0823  loss_rpn_bbox: 0.0183  loss_cls: 0.4752  acc: 95.7031  loss_bbox: 0.2886  loss_mask: 6.8839\n",
      "08/15 04:32:35 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Exp name: mask-rcnn_r50-caffe_fpn_ms-poly-3x_balloon_20230815_043154\n",
      "08/15 04:32:41 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train)  [2][10/31]  lr: 2.0270e-04  eta: 0:02:53  time: 0.5254  data_time: 0.0198  memory: 3449  loss: 5.9555  loss_rpn_cls: 0.0705  loss_rpn_bbox: 0.0176  loss_cls: 0.4270  acc: 93.2617  loss_bbox: 0.3093  loss_mask: 5.1312\n",
      "08/15 04:32:46 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train)  [2][20/31]  lr: 2.5275e-04  eta: 0:02:49  time: 0.5077  data_time: 0.0126  memory: 3282  loss: 4.6321  loss_rpn_cls: 0.0661  loss_rpn_bbox: 0.0176  loss_cls: 0.3947  acc: 84.6680  loss_bbox: 0.3221  loss_mask: 3.8315\n",
      "08/15 04:32:52 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train)  [2][30/31]  lr: 3.0280e-04  eta: 0:02:45  time: 0.5237  data_time: 0.0137  memory: 3283  loss: 1.8118  loss_rpn_cls: 0.0470  loss_rpn_bbox: 0.0193  loss_cls: 0.3295  acc: 85.7422  loss_bbox: 0.3360  loss_mask: 1.0801\n",
      "08/15 04:32:52 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Exp name: mask-rcnn_r50-caffe_fpn_ms-poly-3x_balloon_20230815_043154\n",
      "08/15 04:32:57 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train)  [3][10/31]  lr: 3.5786e-04  eta: 0:02:37  time: 0.5224  data_time: 0.0146  memory: 3282  loss: 1.1179  loss_rpn_cls: 0.0411  loss_rpn_bbox: 0.0203  loss_cls: 0.2903  acc: 96.0938  loss_bbox: 0.3861  loss_mask: 0.3801\n",
      "08/15 04:33:02 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train)  [3][20/31]  lr: 4.0791e-04  eta: 0:02:31  time: 0.5148  data_time: 0.0105  memory: 3283  loss: 0.9559  loss_rpn_cls: 0.0353  loss_rpn_bbox: 0.0189  loss_cls: 0.2542  acc: 95.4102  loss_bbox: 0.3753  loss_mask: 0.2723\n",
      "08/15 04:33:08 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train)  [3][30/31]  lr: 4.5796e-04  eta: 0:02:27  time: 0.5319  data_time: 0.0118  memory: 3283  loss: 0.9498  loss_rpn_cls: 0.0316  loss_rpn_bbox: 0.0194  loss_cls: 0.2459  acc: 86.4258  loss_bbox: 0.4030  loss_mask: 0.2500\n",
      "08/15 04:33:08 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Exp name: mask-rcnn_r50-caffe_fpn_ms-poly-3x_balloon_20230815_043154\n",
      "08/15 04:33:08 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Saving checkpoint at 3 epochs\n",
      "08/15 04:33:20 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val)  [3][10/13]    eta: 0:00:03  time: 1.0153  data_time: 0.0759  memory: 2785  \n",
      "08/15 04:33:21 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Evaluating bbox...\n",
      "Loading and preparing results...\n",
      "DONE (t=0.00s)\n",
      "creating index...\n",
      "index created!\n",
      "Running per image evaluation...\n",
      "Evaluate annotation type *bbox*\n",
      "DONE (t=0.16s).\n",
      "Accumulating evaluation results...\n",
      "DONE (t=0.06s).\n",
      " Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.515\n",
      " Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=1000 ] = 0.736\n",
      " Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=1000 ] = 0.597\n",
      " Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.000\n",
      " Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.266\n",
      " Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.621\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.644\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 ] = 0.644\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=1000 ] = 0.644\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.000\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.525\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.719\n",
      "08/15 04:33:21 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - bbox_mAP_copypaste: 0.515 0.736 0.597 0.000 0.266 0.621\n",
      "08/15 04:33:21 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Evaluating segm...\n",
      "Loading and preparing results...\n",
      "DONE (t=0.04s)\n",
      "creating index...\n",
      "index created!\n",
      "Running per image evaluation...\n",
      "Evaluate annotation type *segm*\n",
      "DONE (t=0.21s).\n",
      "Accumulating evaluation results...\n",
      "DONE (t=0.06s).\n",
      " Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.622\n",
      " Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=1000 ] = 0.733\n",
      " Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=1000 ] = 0.729\n",
      " Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.001\n",
      " Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.331\n",
      " Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.734\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.764\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 ] = 0.764\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=1000 ] = 0.764\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.150\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.750\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.803\n",
      "08/15 04:33:22 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - segm_mAP_copypaste: 0.622 0.733 0.729 0.001 0.331 0.734\n",
      "08/15 04:33:22 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [3][13/13]    coco/bbox_mAP: 0.5150  coco/bbox_mAP_50: 0.7360  coco/bbox_mAP_75: 0.5970  coco/bbox_mAP_s: 0.0000  coco/bbox_mAP_m: 0.2660  coco/bbox_mAP_l: 0.6210  coco/segm_mAP: 0.6220  coco/segm_mAP_50: 0.7330  coco/segm_mAP_75: 0.7290  coco/segm_mAP_s: 0.0010  coco/segm_mAP_m: 0.3310  coco/segm_mAP_l: 0.7340  data_time: 0.0597  time: 0.8604\n",
      "08/15 04:33:27 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train)  [4][10/31]  lr: 5.1301e-04  eta: 0:02:22  time: 0.5251  data_time: 0.0119  memory: 3283  loss: 0.8807  loss_rpn_cls: 0.0273  loss_rpn_bbox: 0.0187  loss_cls: 0.2192  acc: 95.1172  loss_bbox: 0.4036  loss_mask: 0.2119\n",
      "08/15 04:33:32 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train)  [4][20/31]  lr: 5.6306e-04  eta: 0:02:16  time: 0.5211  data_time: 0.0099  memory: 3004  loss: 0.8362  loss_rpn_cls: 0.0217  loss_rpn_bbox: 0.0173  loss_cls: 0.2001  acc: 97.4609  loss_bbox: 0.4115  loss_mask: 0.1856\n",
      "08/15 04:33:38 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train)  [4][30/31]  lr: 6.1311e-04  eta: 0:02:11  time: 0.5334  data_time: 0.0108  memory: 3284  loss: 0.7356  loss_rpn_cls: 0.0198  loss_rpn_bbox: 0.0148  loss_cls: 0.1705  acc: 99.1211  loss_bbox: 0.3685  loss_mask: 0.1621\n",
      "08/15 04:33:38 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Exp name: mask-rcnn_r50-caffe_fpn_ms-poly-3x_balloon_20230815_043154\n",
      "08/15 04:33:44 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train)  [5][10/31]  lr: 6.6817e-04  eta: 0:02:05  time: 0.5305  data_time: 0.0137  memory: 3283  loss: 0.6874  loss_rpn_cls: 0.0174  loss_rpn_bbox: 0.0127  loss_cls: 0.1545  acc: 95.1172  loss_bbox: 0.3607  loss_mask: 0.1421\n",
      "08/15 04:33:49 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train)  [5][20/31]  lr: 7.1822e-04  eta: 0:01:59  time: 0.5216  data_time: 0.0115  memory: 3283  loss: 0.6122  loss_rpn_cls: 0.0162  loss_rpn_bbox: 0.0123  loss_cls: 0.1331  acc: 93.5547  loss_bbox: 0.3217  loss_mask: 0.1289\n",
      "08/15 04:33:54 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train)  [5][30/31]  lr: 7.6827e-04  eta: 0:01:55  time: 0.5289  data_time: 0.0125  memory: 3283  loss: 0.5432  loss_rpn_cls: 0.0145  loss_rpn_bbox: 0.0122  loss_cls: 0.1175  acc: 93.7500  loss_bbox: 0.2771  loss_mask: 0.1219\n",
      "08/15 04:33:55 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Exp name: mask-rcnn_r50-caffe_fpn_ms-poly-3x_balloon_20230815_043154\n",
      "08/15 04:34:00 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train)  [6][10/31]  lr: 8.2332e-04  eta: 0:01:49  time: 0.5335  data_time: 0.0137  memory: 3284  loss: 0.4541  loss_rpn_cls: 0.0139  loss_rpn_bbox: 0.0115  loss_cls: 0.0988  acc: 94.2383  loss_bbox: 0.2158  loss_mask: 0.1141\n",
      "08/15 04:34:06 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train)  [6][20/31]  lr: 8.7337e-04  eta: 0:01:44  time: 0.5318  data_time: 0.0095  memory: 3283  loss: 0.4245  loss_rpn_cls: 0.0130  loss_rpn_bbox: 0.0122  loss_cls: 0.0944  acc: 99.4141  loss_bbox: 0.1922  loss_mask: 0.1127\n",
      "08/15 04:34:11 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train)  [6][30/31]  lr: 9.2342e-04  eta: 0:01:39  time: 0.5465  data_time: 0.0102  memory: 3497  loss: 0.3810  loss_rpn_cls: 0.0133  loss_rpn_bbox: 0.0126  loss_cls: 0.0883  acc: 98.1445  loss_bbox: 0.1617  loss_mask: 0.1052\n",
      "08/15 04:34:12 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Exp name: mask-rcnn_r50-caffe_fpn_ms-poly-3x_balloon_20230815_043154\n",
      "08/15 04:34:12 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Saving checkpoint at 6 epochs\n",
      "08/15 04:34:17 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val)  [6][10/13]    eta: 0:00:01  time: 0.6382  data_time: 0.0735  memory: 1810  \n",
      "08/15 04:34:18 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Evaluating bbox...\n",
      "Loading and preparing results...\n",
      "DONE (t=0.00s)\n",
      "creating index...\n",
      "index created!\n",
      "Running per image evaluation...\n",
      "Evaluate annotation type *bbox*\n",
      "DONE (t=0.03s).\n",
      "Accumulating evaluation results...\n",
      "DONE (t=0.02s).\n",
      " Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.709\n",
      " Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=1000 ] = 0.854\n",
      " Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=1000 ] = 0.799\n",
      " Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.000\n",
      " Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.474\n",
      " Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.795\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.776\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 ] = 0.776\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=1000 ] = 0.776\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.000\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.733\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.833\n",
      "08/15 04:34:18 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - bbox_mAP_copypaste: 0.709 0.854 0.799 0.000 0.474 0.795\n",
      "08/15 04:34:18 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Evaluating segm...\n",
      "Loading and preparing results...\n",
      "DONE (t=0.00s)\n",
      "creating index...\n",
      "index created!\n",
      "Running per image evaluation...\n",
      "Evaluate annotation type *segm*\n",
      "DONE (t=0.03s).\n",
      "Accumulating evaluation results...\n",
      "DONE (t=0.01s).\n",
      " Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.762\n",
      " Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=1000 ] = 0.834\n",
      " Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=1000 ] = 0.834\n",
      " Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.000\n",
      " Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.453\n",
      " Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.860\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.822\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 ] = 0.822\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=1000 ] = 0.822\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.000\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.758\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.889\n",
      "08/15 04:34:18 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - segm_mAP_copypaste: 0.762 0.834 0.834 0.000 0.453 0.860\n",
      "08/15 04:34:18 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [6][13/13]    coco/bbox_mAP: 0.7090  coco/bbox_mAP_50: 0.8540  coco/bbox_mAP_75: 0.7990  coco/bbox_mAP_s: 0.0000  coco/bbox_mAP_m: 0.4740  coco/bbox_mAP_l: 0.7950  coco/segm_mAP: 0.7620  coco/segm_mAP_50: 0.8340  coco/segm_mAP_75: 0.8340  coco/segm_mAP_s: 0.0000  coco/segm_mAP_m: 0.4530  coco/segm_mAP_l: 0.8600  data_time: 0.0661  time: 0.3035\n",
      "08/15 04:34:23 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train)  [7][10/31]  lr: 9.7848e-04  eta: 0:01:33  time: 0.5453  data_time: 0.0117  memory: 3283  loss: 0.3427  loss_rpn_cls: 0.0123  loss_rpn_bbox: 0.0119  loss_cls: 0.0819  acc: 98.7305  loss_bbox: 0.1376  loss_mask: 0.0989\n",
      "08/15 04:34:29 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train)  [7][20/31]  lr: 1.0285e-03  eta: 0:01:28  time: 0.5473  data_time: 0.0103  memory: 3282  loss: 0.2958  loss_rpn_cls: 0.0098  loss_rpn_bbox: 0.0095  loss_cls: 0.0710  acc: 97.1680  loss_bbox: 0.1132  loss_mask: 0.0922\n",
      "08/15 04:34:35 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train)  [7][30/31]  lr: 1.0786e-03  eta: 0:01:23  time: 0.5543  data_time: 0.0105  memory: 3448  loss: 0.2984  loss_rpn_cls: 0.0092  loss_rpn_bbox: 0.0104  loss_cls: 0.0723  acc: 91.1133  loss_bbox: 0.1114  loss_mask: 0.0952\n",
      "08/15 04:34:35 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Exp name: mask-rcnn_r50-caffe_fpn_ms-poly-3x_balloon_20230815_043154\n",
      "08/15 04:34:41 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train)  [8][10/31]  lr: 1.1336e-03  eta: 0:01:17  time: 0.5499  data_time: 0.0125  memory: 3283  loss: 0.2648  loss_rpn_cls: 0.0083  loss_rpn_bbox: 0.0094  loss_cls: 0.0634  acc: 97.7539  loss_bbox: 0.0944  loss_mask: 0.0894\n",
      "08/15 04:34:47 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train)  [8][20/31]  lr: 1.1837e-03  eta: 0:01:12  time: 0.5652  data_time: 0.0119  memory: 3283  loss: 0.2537  loss_rpn_cls: 0.0073  loss_rpn_bbox: 0.0086  loss_cls: 0.0605  acc: 99.2188  loss_bbox: 0.0873  loss_mask: 0.0900\n",
      "08/15 04:34:52 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train)  [8][30/31]  lr: 1.2337e-03  eta: 0:01:07  time: 0.5620  data_time: 0.0117  memory: 3283  loss: 0.2575  loss_rpn_cls: 0.0073  loss_rpn_bbox: 0.0099  loss_cls: 0.0642  acc: 95.4102  loss_bbox: 0.0891  loss_mask: 0.0870\n",
      "08/15 04:34:52 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Exp name: mask-rcnn_r50-caffe_fpn_ms-poly-3x_balloon_20230815_043154\n",
      "08/15 04:34:58 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train)  [9][10/31]  lr: 1.2888e-04  eta: 0:01:01  time: 0.5621  data_time: 0.0160  memory: 3283  loss: 0.2680  loss_rpn_cls: 0.0079  loss_rpn_bbox: 0.0108  loss_cls: 0.0709  acc: 95.8008  loss_bbox: 0.0934  loss_mask: 0.0851\n",
      "08/15 04:35:04 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train)  [9][20/31]  lr: 1.3388e-04  eta: 0:00:56  time: 0.5631  data_time: 0.0150  memory: 3283  loss: 0.2295  loss_rpn_cls: 0.0063  loss_rpn_bbox: 0.0082  loss_cls: 0.0603  acc: 99.1211  loss_bbox: 0.0787  loss_mask: 0.0760\n",
      "08/15 04:35:09 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train)  [9][30/31]  lr: 1.3889e-04  eta: 0:00:50  time: 0.5553  data_time: 0.0133  memory: 3091  loss: 0.2444  loss_rpn_cls: 0.0064  loss_rpn_bbox: 0.0094  loss_cls: 0.0650  acc: 96.5820  loss_bbox: 0.0847  loss_mask: 0.0789\n",
      "08/15 04:35:09 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Exp name: mask-rcnn_r50-caffe_fpn_ms-poly-3x_balloon_20230815_043154\n",
      "08/15 04:35:09 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Saving checkpoint at 9 epochs\n",
      "08/15 04:35:16 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val)  [9][10/13]    eta: 0:00:01  time: 0.5437  data_time: 0.0917  memory: 1693  \n",
      "08/15 04:35:16 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Evaluating bbox...\n",
      "Loading and preparing results...\n",
      "DONE (t=0.00s)\n",
      "creating index...\n",
      "index created!\n",
      "Running per image evaluation...\n",
      "Evaluate annotation type *bbox*\n",
      "DONE (t=0.04s).\n",
      "Accumulating evaluation results...\n",
      "DONE (t=0.02s).\n",
      " Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.741\n",
      " Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=1000 ] = 0.869\n",
      " Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=1000 ] = 0.807\n",
      " Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.000\n",
      " Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.473\n",
      " Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.833\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.794\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 ] = 0.794\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=1000 ] = 0.794\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.000\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.717\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.864\n",
      "08/15 04:35:16 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - bbox_mAP_copypaste: 0.741 0.869 0.807 0.000 0.473 0.833\n",
      "08/15 04:35:16 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Evaluating segm...\n",
      "Loading and preparing results...\n",
      "DONE (t=0.00s)\n",
      "creating index...\n",
      "index created!\n",
      "Running per image evaluation...\n",
      "Evaluate annotation type *segm*\n",
      "DONE (t=0.04s).\n",
      "Accumulating evaluation results...\n",
      "DONE (t=0.02s).\n",
      " Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.779\n",
      " Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=1000 ] = 0.847\n",
      " Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=1000 ] = 0.847\n",
      " Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.000\n",
      " Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.476\n",
      " Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.877\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.830\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 ] = 0.830\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=1000 ] = 0.830\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.000\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.767\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.897\n",
      "08/15 04:35:16 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - segm_mAP_copypaste: 0.779 0.847 0.847 0.000 0.476 0.877\n",
      "08/15 04:35:16 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [9][13/13]    coco/bbox_mAP: 0.7410  coco/bbox_mAP_50: 0.8690  coco/bbox_mAP_75: 0.8070  coco/bbox_mAP_s: 0.0000  coco/bbox_mAP_m: 0.4730  coco/bbox_mAP_l: 0.8330  coco/segm_mAP: 0.7790  coco/segm_mAP_50: 0.8470  coco/segm_mAP_75: 0.8470  coco/segm_mAP_s: 0.0000  coco/segm_mAP_m: 0.4760  coco/segm_mAP_l: 0.8770  data_time: 0.1157  time: 0.3601\n",
      "08/15 04:35:22 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [10][10/31]  lr: 1.4439e-04  eta: 0:00:44  time: 0.5374  data_time: 0.0169  memory: 3284  loss: 0.2415  loss_rpn_cls: 0.0063  loss_rpn_bbox: 0.0093  loss_cls: 0.0644  acc: 98.3398  loss_bbox: 0.0832  loss_mask: 0.0784\n",
      "08/15 04:35:27 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [10][20/31]  lr: 1.4940e-04  eta: 0:00:39  time: 0.5342  data_time: 0.0130  memory: 3283  loss: 0.2161  loss_rpn_cls: 0.0048  loss_rpn_bbox: 0.0080  loss_cls: 0.0573  acc: 99.4141  loss_bbox: 0.0721  loss_mask: 0.0738\n",
      "08/15 04:35:33 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [10][30/31]  lr: 1.5440e-04  eta: 0:00:33  time: 0.5374  data_time: 0.0129  memory: 3428  loss: 0.2210  loss_rpn_cls: 0.0054  loss_rpn_bbox: 0.0086  loss_cls: 0.0584  acc: 99.8047  loss_bbox: 0.0737  loss_mask: 0.0749\n",
      "08/15 04:35:33 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Exp name: mask-rcnn_r50-caffe_fpn_ms-poly-3x_balloon_20230815_043154\n",
      "08/15 04:35:38 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [11][10/31]  lr: 1.5991e-04  eta: 0:00:27  time: 0.5344  data_time: 0.0149  memory: 3067  loss: 0.2405  loss_rpn_cls: 0.0052  loss_rpn_bbox: 0.0099  loss_cls: 0.0639  acc: 98.8281  loss_bbox: 0.0807  loss_mask: 0.0808\n",
      "08/15 04:35:44 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [11][20/31]  lr: 1.6491e-04  eta: 0:00:22  time: 0.5326  data_time: 0.0105  memory: 3283  loss: 0.2297  loss_rpn_cls: 0.0047  loss_rpn_bbox: 0.0094  loss_cls: 0.0606  acc: 96.7773  loss_bbox: 0.0768  loss_mask: 0.0781\n",
      "08/15 04:35:49 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [11][30/31]  lr: 1.6992e-04  eta: 0:00:17  time: 0.5409  data_time: 0.0115  memory: 3283  loss: 0.2221  loss_rpn_cls: 0.0048  loss_rpn_bbox: 0.0092  loss_cls: 0.0595  acc: 95.3125  loss_bbox: 0.0753  loss_mask: 0.0733\n",
      "08/15 04:35:50 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Exp name: mask-rcnn_r50-caffe_fpn_ms-poly-3x_balloon_20230815_043154\n",
      "08/15 04:35:55 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [12][10/31]  lr: 1.7543e-05  eta: 0:00:11  time: 0.5311  data_time: 0.0129  memory: 3200  loss: 0.2094  loss_rpn_cls: 0.0054  loss_rpn_bbox: 0.0084  loss_cls: 0.0555  acc: 96.9727  loss_bbox: 0.0689  loss_mask: 0.0711\n",
      "08/15 04:36:00 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [12][20/31]  lr: 1.8043e-05  eta: 0:00:05  time: 0.5275  data_time: 0.0098  memory: 3448  loss: 0.2108  loss_rpn_cls: 0.0047  loss_rpn_bbox: 0.0089  loss_cls: 0.0560  acc: 99.5117  loss_bbox: 0.0698  loss_mask: 0.0713\n",
      "08/15 04:36:06 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(train) [12][30/31]  lr: 1.8544e-05  eta: 0:00:00  time: 0.5425  data_time: 0.0111  memory: 3282  loss: 0.2048  loss_rpn_cls: 0.0043  loss_rpn_bbox: 0.0080  loss_cls: 0.0540  acc: 98.2422  loss_bbox: 0.0695  loss_mask: 0.0690\n",
      "08/15 04:36:06 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Exp name: mask-rcnn_r50-caffe_fpn_ms-poly-3x_balloon_20230815_043154\n",
      "08/15 04:36:06 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Saving checkpoint at 12 epochs\n",
      "08/15 04:36:12 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [12][10/13]    eta: 0:00:01  time: 0.4801  data_time: 0.0931  memory: 1589  \n",
      "08/15 04:36:12 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Evaluating bbox...\n",
      "Loading and preparing results...\n",
      "DONE (t=0.00s)\n",
      "creating index...\n",
      "index created!\n",
      "Running per image evaluation...\n",
      "Evaluate annotation type *bbox*\n",
      "DONE (t=0.02s).\n",
      "Accumulating evaluation results...\n",
      "DONE (t=0.01s).\n",
      " Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.740\n",
      " Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=1000 ] = 0.868\n",
      " Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=1000 ] = 0.804\n",
      " Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.000\n",
      " Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.506\n",
      " Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.828\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.798\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 ] = 0.798\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=1000 ] = 0.798\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.000\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.742\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.861\n",
      "08/15 04:36:12 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - bbox_mAP_copypaste: 0.740 0.868 0.804 0.000 0.506 0.828\n",
      "08/15 04:36:12 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Evaluating segm...\n",
      "Loading and preparing results...\n",
      "DONE (t=0.00s)\n",
      "creating index...\n",
      "index created!\n",
      "Running per image evaluation...\n",
      "Evaluate annotation type *segm*\n",
      "DONE (t=0.02s).\n",
      "Accumulating evaluation results...\n",
      "DONE (t=0.01s).\n",
      " Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.776\n",
      " Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=1000 ] = 0.845\n",
      " Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=1000 ] = 0.845\n",
      " Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.000\n",
      " Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.481\n",
      " Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.868\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.832\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=300 ] = 0.832\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=1000 ] = 0.832\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.000\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.775\n",
      " Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.897\n",
      "08/15 04:36:12 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - segm_mAP_copypaste: 0.776 0.845 0.845 0.000 0.481 0.868\n",
      "08/15 04:36:12 - mmengine - \u001b[4m\u001b[97mINFO\u001b[0m - Epoch(val) [12][13/13]    coco/bbox_mAP: 0.7400  coco/bbox_mAP_50: 0.8680  coco/bbox_mAP_75: 0.8040  coco/bbox_mAP_s: 0.0000  coco/bbox_mAP_m: 0.5060  coco/bbox_mAP_l: 0.8280  coco/segm_mAP: 0.7760  coco/segm_mAP_50: 0.8450  coco/segm_mAP_75: 0.8450  coco/segm_mAP_s: 0.0000  coco/segm_mAP_m: 0.4810  coco/segm_mAP_l: 0.8680  data_time: 0.0903  time: 0.2892\n"
     ]
    }
   ],
   "source": [
    "!python tools/train.py {config}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_vYQF5K2NqqI"
   },
   "source": [
    "### Understand the log\n",
    "From the log, we can have a basic understanding on the training process and know how well the detector is trained.\n",
    "\n",
    "First, since the dataset we are using is small, we loaded a Mask R-CNN model and finetune it for detection. Because the original Mask R-CNN is trained on COCO dataset that contains 80 classes but KITTI Tiny dataset only have 3 classes. Therefore, the last FC layers of the pre-trained Mask R-CNN for classification and regression have different weight shape and are not used. The pre-trained weights of mask prediction layer `mask_head.conv_logits` also does not matches the current model and is not used due to similar reason.\n",
    "\n",
    "Third, after training, the detector is evaluated by the default COCO-style evaluation. The results show that the detector achieves 79.6 bbox AP and 81.5 mask AP on the val dataset, not bad!\n",
    "\n",
    " We can also check the tensorboard to see the curves."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "gbLNlJR-RYYd"
   },
   "outputs": [],
   "source": [
    "%pip install tensorboard  -i https://mirrors.ustc.edu.cn/pypi/web/simple"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "id": "PW2NAam_7irv"
   },
   "outputs": [],
   "source": [
    "# load tensorboard in jupyter notebook\n",
    "%load_ext tensorboard"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "4G9MCbL2RYYd"
   },
   "outputs": [],
   "source": [
    "# see curves in tensorboard\n",
    "# if you see <IPython.core.display.HTML object> please run it again\n",
    "%tensorboard --logdir tutorial_exps/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "MfQ-yspZLuuI"
   },
   "source": [
    "## Test the Trained Detector\n",
    "\n",
    "After finetuning the detector, let's visualize the prediction results!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "_MuZurfGLq0p",
    "outputId": "4b25759c-8e22-405e-a061-3abc44e38043"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loads checkpoint by local backend from path: tutorial_exps/epoch_12.pth\n",
      "<DetDataSample(\n",
      "\n",
      "    META INFORMATION\n",
      "    batch_input_shape: (800, 1216)\n",
      "    pad_shape: (800, 1216)\n",
      "    scale_factor: (0.58837890625, 0.5882352941176471)\n",
      "    img_id: 0\n",
      "    ori_shape: (1360, 2048)\n",
      "    img_shape: (800, 1205)\n",
      "    img_path: None\n",
      "\n",
      "    DATA FIELDS\n",
      "    gt_instances: <InstanceData(\n",
      "        \n",
      "            META INFORMATION\n",
      "        \n",
      "            DATA FIELDS\n",
      "            labels: tensor([], dtype=torch.int64)\n",
      "            bboxes: tensor([], size=(0, 4))\n",
      "            masks: BitmapMasks(num_masks=0, height=1360, width=2048)\n",
      "        ) at 0x79a3a96e32b0>\n",
      "    ignored_instances: <InstanceData(\n",
      "        \n",
      "            META INFORMATION\n",
      "        \n",
      "            DATA FIELDS\n",
      "            labels: tensor([], dtype=torch.int64)\n",
      "            bboxes: tensor([], size=(0, 4))\n",
      "            masks: BitmapMasks(num_masks=0, height=1360, width=2048)\n",
      "        ) at 0x79a3a96e2e90>\n",
      "    pred_instances: <InstanceData(\n",
      "        \n",
      "            META INFORMATION\n",
      "        \n",
      "            DATA FIELDS\n",
      "            labels: tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])\n",
      "            bboxes: tensor([[ 669.8222,  242.3854, 1406.3090,  999.6241],\n",
      "                        [1284.7911,  415.3238, 1697.2852,  925.6313],\n",
      "                        [ 589.4117,  419.8082,  729.0321,  839.9910],\n",
      "                        [1034.3956,    8.1391, 1430.1029,  165.9097],\n",
      "                        [1030.5734, 1166.0953, 1149.0190, 1229.6295],\n",
      "                        [1017.3596, 1311.2710, 1193.4612, 1359.9869],\n",
      "                        [ 724.5475, 1260.3674,  835.1842, 1339.4836],\n",
      "                        [ 491.2525,  405.2336,  684.7927,  529.9492],\n",
      "                        [ 980.2676,   15.9298, 1496.0459,  413.1231],\n",
      "                        [ 635.1721,  399.2910,  786.5412,  847.4308],\n",
      "                        [ 994.9816,    3.6709, 1188.3881,  169.8953],\n",
      "                        [1786.0100,  203.3728, 1813.8008,  230.7004],\n",
      "                        [ 416.1780,  117.8048,  559.7723,  383.0736],\n",
      "                        [ 183.7183,  849.5600,  332.8814,  895.9375],\n",
      "                        [ 310.0583,  784.4355,  378.9562,  896.0558],\n",
      "                        [1136.9861,  912.4040, 1391.0818, 1344.7209],\n",
      "                        [1300.2130, 1108.3888, 1587.3857, 1348.6622]])\n",
      "            masks: tensor([[[False, False, False,  ..., False, False, False],\n",
      "                         [False, False, False,  ..., False, False, False],\n",
      "                         [False, False, False,  ..., False, False, False],\n",
      "                         ...,\n",
      "                         [False, False, False,  ..., False, False, False],\n",
      "                         [False, False, False,  ..., False, False, False],\n",
      "                         [False, False, False,  ..., False, False, False]],\n",
      "                \n",
      "                        [[False, False, False,  ..., False, False, False],\n",
      "                         [False, False, False,  ..., False, False, False],\n",
      "                         [False, False, False,  ..., False, False, False],\n",
      "                         ...,\n",
      "                         [False, False, False,  ..., False, False, False],\n",
      "                         [False, False, False,  ..., False, False, False],\n",
      "                         [False, False, False,  ..., False, False, False]],\n",
      "                \n",
      "                        [[False, False, False,  ..., False, False, False],\n",
      "                         [False, False, False,  ..., False, False, False],\n",
      "                         [False, False, False,  ..., False, False, False],\n",
      "                         ...,\n",
      "                         [False, False, False,  ..., False, False, False],\n",
      "                         [False, False, False,  ..., False, False, False],\n",
      "                         [False, False, False,  ..., False, False, False]],\n",
      "                \n",
      "                        ...,\n",
      "                \n",
      "                        [[False, False, False,  ..., False, False, False],\n",
      "                         [False, False, False,  ..., False, False, False],\n",
      "                         [False, False, False,  ..., False, False, False],\n",
      "                         ...,\n",
      "                         [False, False, False,  ..., False, False, False],\n",
      "                         [False, False, False,  ..., False, False, False],\n",
      "                         [False, False, False,  ..., False, False, False]],\n",
      "                \n",
      "                        [[False, False, False,  ..., False, False, False],\n",
      "                         [False, False, False,  ..., False, False, False],\n",
      "                         [False, False, False,  ..., False, False, False],\n",
      "                         ...,\n",
      "                         [False, False, False,  ..., False, False, False],\n",
      "                         [False, False, False,  ..., False, False, False],\n",
      "                         [False, False, False,  ..., False, False, False]],\n",
      "                \n",
      "                        [[False, False, False,  ..., False, False, False],\n",
      "                         [False, False, False,  ..., False, False, False],\n",
      "                         [False, False, False,  ..., False, False, False],\n",
      "                         ...,\n",
      "                         [False, False, False,  ..., False, False, False],\n",
      "                         [False, False, False,  ..., False, False, False],\n",
      "                         [False, False, False,  ..., False, False, False]]])\n",
      "            scores: tensor([0.9977, 0.9940, 0.9682, 0.8142, 0.6556, 0.3539, 0.3009, 0.2788, 0.1958,\n",
      "                        0.1624, 0.1309, 0.1179, 0.1106, 0.0928, 0.0888, 0.0737, 0.0584])\n",
      "        ) at 0x79a3a96cb070>\n",
      ") at 0x79a3a96e3130>\n"
     ]
    }
   ],
   "source": [
    "import mmcv\n",
    "from mmdet.apis import init_detector, inference_detector\n",
    "img = mmcv.imread('./ballondatasets/balloon/train/7178882742_f090f3ce56_k.jpg',channel_order='rgb')\n",
    "checkpoint_file = 'tutorial_exps/epoch_12.pth'\n",
    "model = init_detector(cfg, checkpoint_file, device='cpu')\n",
    "new_result = inference_detector(model, img)\n",
    "print(new_result)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 461
    },
    "id": "7SSTauCURYYe",
    "outputId": "3becb5ea-cb4e-44f6-d93d-c10194a2263b"
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 604840 characters]",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from mmengine.visualization import Visualizer\n",
    "# get built visualizer\n",
    "visualizer_now = Visualizer.get_current_instance()\n",
    "# the dataset_meta is loaded from the checkpoint and\n",
    "# then pass to the model in init_detector\n",
    "visualizer_now.dataset_meta = model.dataset_meta\n",
    "# show the results\n",
    "visualizer_now.add_datasample(\n",
    "    'new_result',\n",
    "    img,\n",
    "    data_sample=new_result,\n",
    "    draw_gt=False,\n",
    "    wait_time=0,\n",
    "    out_file=None,\n",
    "    pred_score_thr=0.5\n",
    ")\n",
    "visualizer_now.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6rzruCwFgPXm"
   },
   "source": [
    "## What to Do Next?\n",
    "\n",
    "So far, we have learnt how to test and train Mask R-CNN. To further explore the segmentation task, you could do several other things as shown below:\n",
    "\n",
    "- Try cascade methods, e.g., [Cascade Mask R-CNN](https://github.com/open-mmlab/mmdetection/tree/master/configs/cascade_rcnn) and [HTC](https://github.com/open-mmlab/mmdetection/tree/master/configs/htc) in [MMDetection model zoo](https://github.com/open-mmlab/mmdetection/blob/master/docs/en/model_zoo.md). They are powerful detectors that are ranked high in many benchmarks, e.g., COCO dataset.\n",
    "- Try single-stage methods, e.g., [K-Net](https://github.com/ZwwWayne/K-Net) and [Dense-RepPoints](https://github.com/justimyhxu/Dense-RepPoints). These two algorithms are based on MMDetection. Box-free instance segmentation is a new trend in the instance segmentation community.\n",
    "- Try semantic segmentation. Semantic segmentation is also a popular task with wide applications. You can explore [MMSegmentation](https://github.com/open-mmlab/mmsegmentation/); we also provide a [colab tutorial](https://github.com/open-mmlab/mmsegmentation/blob/master/demo/MMSegmentation_Tutorial.ipynb) for semantic segmentation using MMSegmentation.\n"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.15"
  },
  "vscode": {
   "interpreter": {
    "hash": "8868640c17582ff5a3e06365ba2fb344ce697cf42d4745ae8b85a9738303c037"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
