{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# OCR Form Recognition Practice\n",
    "\n",
    "This section will introduce how to use PaddleOCR to complete the training and operation of the table recognition algorithm, including:\n",
    "\n",
    "1. Understand the principle of table recognition algorithm\n",
    "3. Master the training and prediction process of PaddleOCR form recognition code\n",
    "\n",
    "\n",
    "## 1 Quick Experience\n",
    "To quickly demonstrate the PP-Structure prediction, first download the PaddleOCR code and install the dependency package"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple\n",
      "Requirement already satisfied: pip in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (21.3.1)\n",
      "Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple\n",
      "Collecting layoutparser==0.0.0\n",
      "  Using cached https://paddleocr.bj.bcebos.com/whl/layoutparser-0.0.0-py3-none-any.whl (19.1 MB)\n",
      "Requirement already satisfied: tqdm in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from layoutparser==0.0.0) (4.27.0)\n",
      "Requirement already satisfied: opencv-python in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from layoutparser==0.0.0) (4.1.1.26)\n",
      "Requirement already satisfied: pillow in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from layoutparser==0.0.0) (7.1.2)\n",
      "Requirement already satisfied: iopath in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from layoutparser==0.0.0) (0.1.9)\n",
      "Requirement already satisfied: numpy in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from layoutparser==0.0.0) (1.21.5)\n",
      "Requirement already satisfied: pandas in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from layoutparser==0.0.0) (1.1.5)\n",
      "Requirement already satisfied: pyyaml>=5.1 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from layoutparser==0.0.0) (5.1.2)\n",
      "Requirement already satisfied: portalocker in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from iopath->layoutparser==0.0.0) (2.3.2)\n",
      "Requirement already satisfied: python-dateutil>=2.7.3 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from pandas->layoutparser==0.0.0) (2.8.0)\n",
      "Requirement already satisfied: pytz>=2017.2 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from pandas->layoutparser==0.0.0) (2019.3)\n",
      "Requirement already satisfied: six>=1.5 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from python-dateutil>=2.7.3->pandas->layoutparser==0.0.0) (1.15.0)\n",
      "Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple\n",
      "Requirement already satisfied: shapely in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from -r PaddleOCR/requirements.txt (line 1)) (1.8.0)\n",
      "Requirement already satisfied: scikit-image in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from -r PaddleOCR/requirements.txt (line 2)) (0.19.1)\n",
      "Requirement already satisfied: imgaug==0.4.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from -r PaddleOCR/requirements.txt (line 3)) (0.4.0)\n",
      "Requirement already satisfied: pyclipper in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from -r PaddleOCR/requirements.txt (line 4)) (1.3.0.post2)\n",
      "Requirement already satisfied: lmdb in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from -r PaddleOCR/requirements.txt (line 5)) (1.2.1)\n",
      "Requirement already satisfied: tqdm in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from -r PaddleOCR/requirements.txt (line 6)) (4.27.0)\n",
      "Requirement already satisfied: numpy in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from -r PaddleOCR/requirements.txt (line 7)) (1.21.5)\n",
      "Requirement already satisfied: visualdl in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from -r PaddleOCR/requirements.txt (line 8)) (2.2.0)\n",
      "Requirement already satisfied: python-Levenshtein in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from -r PaddleOCR/requirements.txt (line 9)) (0.12.2)\n",
      "Requirement already satisfied: opencv-contrib-python==4.4.0.46 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from -r PaddleOCR/requirements.txt (line 10)) (4.4.0.46)\n",
      "Requirement already satisfied: cython in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from -r PaddleOCR/requirements.txt (line 11)) (0.29)\n",
      "Requirement already satisfied: lxml in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from -r PaddleOCR/requirements.txt (line 12)) (4.7.1)\n",
      "Requirement already satisfied: premailer in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from -r PaddleOCR/requirements.txt (line 13)) (3.10.0)\n",
      "Requirement already satisfied: openpyxl in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from -r PaddleOCR/requirements.txt (line 14)) (3.0.5)\n",
      "Requirement already satisfied: fasttext==0.9.1 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from -r PaddleOCR/requirements.txt (line 15)) (0.9.1)\n",
      "Requirement already satisfied: scipy in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from imgaug==0.4.0->-r PaddleOCR/requirements.txt (line 3)) (1.7.3)\n",
      "Requirement already satisfied: Pillow in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from imgaug==0.4.0->-r PaddleOCR/requirements.txt (line 3)) (7.1.2)\n",
      "Requirement already satisfied: matplotlib in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from imgaug==0.4.0->-r PaddleOCR/requirements.txt (line 3)) (2.2.3)\n",
      "Requirement already satisfied: imageio in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from imgaug==0.4.0->-r PaddleOCR/requirements.txt (line 3)) (2.6.1)\n",
      "Requirement already satisfied: opencv-python in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from imgaug==0.4.0->-r PaddleOCR/requirements.txt (line 3)) (4.1.1.26)\n",
      "Requirement already satisfied: six in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from imgaug==0.4.0->-r PaddleOCR/requirements.txt (line 3)) (1.15.0)\n",
      "Requirement already satisfied: pybind11>=2.2 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from fasttext==0.9.1->-r PaddleOCR/requirements.txt (line 15)) (2.8.1)\n",
      "Requirement already satisfied: setuptools>=0.7.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from fasttext==0.9.1->-r PaddleOCR/requirements.txt (line 15)) (41.4.0)\n",
      "Requirement already satisfied: packaging>=20.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from scikit-image->-r PaddleOCR/requirements.txt (line 2)) (20.9)\n",
      "Requirement already satisfied: networkx>=2.2 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from scikit-image->-r PaddleOCR/requirements.txt (line 2)) (2.4)\n",
      "Requirement already satisfied: PyWavelets>=1.1.1 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from scikit-image->-r PaddleOCR/requirements.txt (line 2)) (1.2.0)\n",
      "Requirement already satisfied: tifffile>=2019.7.26 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from scikit-image->-r PaddleOCR/requirements.txt (line 2)) (2021.11.2)\n",
      "Requirement already satisfied: pre-commit in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from visualdl->-r PaddleOCR/requirements.txt (line 8)) (1.21.0)\n",
      "Requirement already satisfied: bce-python-sdk in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from visualdl->-r PaddleOCR/requirements.txt (line 8)) (0.8.53)\n",
      "Requirement already satisfied: shellcheck-py in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from visualdl->-r PaddleOCR/requirements.txt (line 8)) (0.7.1.1)\n",
      "Requirement already satisfied: requests in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from visualdl->-r PaddleOCR/requirements.txt (line 8)) (2.22.0)\n",
      "Requirement already satisfied: protobuf>=3.11.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from visualdl->-r PaddleOCR/requirements.txt (line 8)) (3.14.0)\n",
      "Requirement already satisfied: Flask-Babel>=1.0.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from visualdl->-r PaddleOCR/requirements.txt (line 8)) (1.0.0)\n",
      "Requirement already satisfied: pandas in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from visualdl->-r PaddleOCR/requirements.txt (line 8)) (1.1.5)\n",
      "Requirement already satisfied: flask>=1.1.1 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from visualdl->-r PaddleOCR/requirements.txt (line 8)) (1.1.1)\n",
      "Requirement already satisfied: flake8>=3.7.9 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from visualdl->-r PaddleOCR/requirements.txt (line 8)) (3.8.2)\n",
      "Requirement already satisfied: cachetools in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from premailer->-r PaddleOCR/requirements.txt (line 13)) (4.0.0)\n",
      "Requirement already satisfied: cssselect in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from premailer->-r PaddleOCR/requirements.txt (line 13)) (1.1.0)\n",
      "Requirement already satisfied: cssutils in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from premailer->-r PaddleOCR/requirements.txt (line 13)) (2.3.0)\n",
      "Requirement already satisfied: jdcal in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from openpyxl->-r PaddleOCR/requirements.txt (line 14)) (1.4.1)\n",
      "Requirement already satisfied: et-xmlfile in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from openpyxl->-r PaddleOCR/requirements.txt (line 14)) (1.0.1)\n",
      "Requirement already satisfied: pycodestyle<2.7.0,>=2.6.0a1 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from flake8>=3.7.9->visualdl->-r PaddleOCR/requirements.txt (line 8)) (2.6.0)\n",
      "Requirement already satisfied: importlib-metadata in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from flake8>=3.7.9->visualdl->-r PaddleOCR/requirements.txt (line 8)) (0.23)\n",
      "Requirement already satisfied: pyflakes<2.3.0,>=2.2.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from flake8>=3.7.9->visualdl->-r PaddleOCR/requirements.txt (line 8)) (2.2.0)\n",
      "Requirement already satisfied: mccabe<0.7.0,>=0.6.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from flake8>=3.7.9->visualdl->-r PaddleOCR/requirements.txt (line 8)) (0.6.1)\n",
      "Requirement already satisfied: Jinja2>=2.10.1 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from flask>=1.1.1->visualdl->-r PaddleOCR/requirements.txt (line 8)) (2.11.0)\n",
      "Requirement already satisfied: itsdangerous>=0.24 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from flask>=1.1.1->visualdl->-r PaddleOCR/requirements.txt (line 8)) (1.1.0)\n",
      "Requirement already satisfied: Werkzeug>=0.15 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from flask>=1.1.1->visualdl->-r PaddleOCR/requirements.txt (line 8)) (0.16.0)\n",
      "Requirement already satisfied: click>=5.1 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from flask>=1.1.1->visualdl->-r PaddleOCR/requirements.txt (line 8)) (7.0)\n",
      "Requirement already satisfied: pytz in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from Flask-Babel>=1.0.0->visualdl->-r PaddleOCR/requirements.txt (line 8)) (2019.3)\n",
      "Requirement already satisfied: Babel>=2.3 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from Flask-Babel>=1.0.0->visualdl->-r PaddleOCR/requirements.txt (line 8)) (2.8.0)\n",
      "Requirement already satisfied: decorator>=4.3.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from networkx>=2.2->scikit-image->-r PaddleOCR/requirements.txt (line 2)) (4.4.2)\n",
      "Requirement already satisfied: pyparsing>=2.0.2 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from packaging>=20.0->scikit-image->-r PaddleOCR/requirements.txt (line 2)) (2.4.2)\n",
      "Requirement already satisfied: pycryptodome>=3.8.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from bce-python-sdk->visualdl->-r PaddleOCR/requirements.txt (line 8)) (3.9.9)\n",
      "Requirement already satisfied: future>=0.6.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from bce-python-sdk->visualdl->-r PaddleOCR/requirements.txt (line 8)) (0.18.0)\n",
      "Requirement already satisfied: python-dateutil>=2.1 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from matplotlib->imgaug==0.4.0->-r PaddleOCR/requirements.txt (line 3)) (2.8.0)\n",
      "Requirement already satisfied: cycler>=0.10 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from matplotlib->imgaug==0.4.0->-r PaddleOCR/requirements.txt (line 3)) (0.10.0)\n",
      "Requirement already satisfied: kiwisolver>=1.0.1 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from matplotlib->imgaug==0.4.0->-r PaddleOCR/requirements.txt (line 3)) (1.1.0)\n",
      "Requirement already satisfied: aspy.yaml in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from pre-commit->visualdl->-r PaddleOCR/requirements.txt (line 8)) (1.3.0)\n",
      "Requirement already satisfied: virtualenv>=15.2 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from pre-commit->visualdl->-r PaddleOCR/requirements.txt (line 8)) (16.7.9)\n",
      "Requirement already satisfied: pyyaml in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from pre-commit->visualdl->-r PaddleOCR/requirements.txt (line 8)) (5.1.2)\n",
      "Requirement already satisfied: nodeenv>=0.11.1 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from pre-commit->visualdl->-r PaddleOCR/requirements.txt (line 8)) (1.3.4)\n",
      "Requirement already satisfied: identify>=1.0.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from pre-commit->visualdl->-r PaddleOCR/requirements.txt (line 8)) (1.4.10)\n",
      "Requirement already satisfied: toml in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from pre-commit->visualdl->-r PaddleOCR/requirements.txt (line 8)) (0.10.0)\n",
      "Requirement already satisfied: cfgv>=2.0.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from pre-commit->visualdl->-r PaddleOCR/requirements.txt (line 8)) (2.0.1)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from requests->visualdl->-r PaddleOCR/requirements.txt (line 8)) (2019.9.11)\n",
      "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from requests->visualdl->-r PaddleOCR/requirements.txt (line 8)) (1.25.6)\n",
      "Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from requests->visualdl->-r PaddleOCR/requirements.txt (line 8)) (3.0.4)\n",
      "Requirement already satisfied: idna<2.9,>=2.5 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from requests->visualdl->-r PaddleOCR/requirements.txt (line 8)) (2.8)\n",
      "Requirement already satisfied: MarkupSafe>=0.23 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from Jinja2>=2.10.1->flask>=1.1.1->visualdl->-r PaddleOCR/requirements.txt (line 8)) (1.1.1)\n",
      "Requirement already satisfied: zipp>=0.5 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from importlib-metadata->flake8>=3.7.9->visualdl->-r PaddleOCR/requirements.txt (line 8)) (3.6.0)\n",
      "Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple\n",
      "Requirement already satisfied: pandas in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (1.1.5)\n",
      "Requirement already satisfied: python-dateutil>=2.7.3 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from pandas) (2.8.0)\n",
      "Requirement already satisfied: pytz>=2017.2 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from pandas) (2019.3)\n",
      "Requirement already satisfied: numpy>=1.15.4 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from pandas) (1.21.5)\n",
      "Requirement already satisfied: six>=1.5 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from python-dateutil>=2.7.3->pandas) (1.15.0)\n"
     ]
    }
   ],
   "source": [
    "# clone PaddleOCR code\n",
    "# ! git  clone https://github.com/PaddlePaddle/PaddleOCR\n",
    "\n",
    "# Install dependencies\n",
    "! pip install -U pip\n",
    "! pip install -U https://paddleocr.bj.bcebos.com/whl/layoutparser-0.0.0-py3-none-any.whl\n",
    "! pip install -r PaddleOCR/requirements.txt\n",
    "! pip install pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After the installation is complete, you can quickly complete the identification of the form through the following command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "# Switch to working directory\n",
    "import os\n",
    "os.chdir('/home/aistudio/PaddleOCR/ppstructure')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mkdir: cannot create directory ‘inference’: File exists\n",
      "--2021-12-25 20:46:49--  https://paddleocr.bj.bcebos.com/PP-OCRv2/chinese/ch_PP-OCRv2_det_infer.tar\n",
      "Resolving paddleocr.bj.bcebos.com (paddleocr.bj.bcebos.com)... 182.61.200.229, 182.61.200.195, 2409:8c04:1001:1002:0:ff:b001:368a\n",
      "Connecting to paddleocr.bj.bcebos.com (paddleocr.bj.bcebos.com)|182.61.200.229|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 3190272 (3.0M) [application/x-tar]\n",
      "Saving to: ‘./inference/ch_PP-OCRv2_det_infer.tar.2’\n",
      "\n",
      "ch_PP-OCRv2_det_inf 100%[===================>]   3.04M  6.77MB/s    in 0.4s    \n",
      "\n",
      "2021-12-25 20:46:49 (6.77 MB/s) - ‘./inference/ch_PP-OCRv2_det_infer.tar.2’ saved [3190272/3190272]\n",
      "\n",
      "--2021-12-25 20:46:50--  https://paddleocr.bj.bcebos.com/PP-OCRv2/chinese/ch_PP-OCRv2_rec_infer.tar\n",
      "Resolving paddleocr.bj.bcebos.com (paddleocr.bj.bcebos.com)... 182.61.200.229, 182.61.200.195, 2409:8c04:1001:1002:0:ff:b001:368a\n",
      "Connecting to paddleocr.bj.bcebos.com (paddleocr.bj.bcebos.com)|182.61.200.229|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 8875520 (8.5M) [application/x-tar]\n",
      "Saving to: ‘./inference/ch_PP-OCRv2_rec_infer.tar.2’\n",
      "\n",
      "ch_PP-OCRv2_rec_inf 100%[===================>]   8.46M  12.7MB/s    in 0.7s    \n",
      "\n",
      "2021-12-25 20:46:50 (12.7 MB/s) - ‘./inference/ch_PP-OCRv2_rec_infer.tar.2’ saved [8875520/8875520]\n",
      "\n",
      "--2021-12-25 20:46:51--  https://paddleocr.bj.bcebos.com/dygraph_v2.0/table/en_ppocr_mobile_v2.0_table_structure_infer.tar\n",
      "Resolving paddleocr.bj.bcebos.com (paddleocr.bj.bcebos.com)... 182.61.200.229, 182.61.200.195, 2409:8c04:1001:1002:0:ff:b001:368a\n",
      "Connecting to paddleocr.bj.bcebos.com (paddleocr.bj.bcebos.com)|182.61.200.229|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 19667456 (19M) [application/x-tar]\n",
      "Saving to: ‘./inference/en_ppocr_mobile_v2.0_table_structure_infer.tar.2’\n",
      "\n",
      "en_ppocr_mobile_v2. 100%[===================>]  18.76M  21.0MB/s    in 0.9s    \n",
      "\n",
      "2021-12-25 20:46:52 (21.0 MB/s) - ‘./inference/en_ppocr_mobile_v2.0_table_structure_infer.tar.2’ saved [19667456/19667456]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Download model\n",
    "! mkdir inference && cd inference\n",
    "# Download the detection model of the ultra-lightweight table English OCR model and unzip it\n",
    "! wget -P ./inference/ https://paddleocr.bj.bcebos.com/PP-OCRv2/chinese/ch_PP-OCRv2_det_infer.tar && cd inference && tar xf ch_PP-OCRv2_det_infer.tar && cd ..\n",
    "# Download the recognition model of the ultra-lightweight table English OCR model and unzip it\n",
    "! wget -P ./inference/ https://paddleocr.bj.bcebos.com/PP-OCRv2/chinese/ch_PP-OCRv2_rec_infer.tar && cd inference && tar xf ch_PP-OCRv2_rec_infer.tar && cd ..\n",
    "# Download the ultra-lightweight English table inch model and unzip it\n",
    "! wget -P ./inference/ https://paddleocr.bj.bcebos.com/dygraph_v2.0/table/en_ppocr_mobile_v2.0_table_structure_infer.tar && cd inference && tar xf en_ppocr_mobile_v2.0_table_structure_infer.tar && cd .."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x7fab5d10c150>"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 113748 characters]",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# First enter the image\n",
    "\n",
    "import cv2\n",
    "from matplotlib import pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "# Read the table image and display\n",
    "img = cv2.imread('/home/aistudio/1.jpg')\n",
    "plt.imshow(img)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2021/12/26 19:55:37] root DEBUG: dt_boxes num : 69, elapse : 2.900609254837036\n",
      "[2021/12/26 19:55:43] root DEBUG: rec_res num  : 69, elapse : 5.992196321487427\n",
      "<html><body><table><thead><tr><td>代号</td><td>项目</td><td>结果</td><td>参考值</td><td>单位</td></tr></thead><tbody><tr><td>ALT</td><td>谷丙转氨酶</td><td>25.6</td><td>0--40</td><td>U/L</td></tr><tr><td>TBIL</td><td>总胆红素</td><td>11.2</td><td><20</td><td>UMOL/L</td></tr><tr><td>DBIL</td><td>直接胆红素</td><td>3.3</td><td>0--7</td><td>UMOL/L</td></tr><tr><td>IBIL</td><td>间接胆红素</td><td>7.9</td><td>1.5--15</td><td>UMOL/L</td></tr><tr><td>TP</td><td>总蛋白</td><td>58.9J</td><td>60--80</td><td>g/L</td></tr><tr><td>ALB</td><td>白蛋白</td><td>35.1</td><td>33--55</td><td>g/L</td></tr><tr><td>GLO</td><td>球蛋白</td><td>23.8</td><td>20--30</td><td>8/L</td></tr><tr><td>A/G</td><td>白球比</td><td>1.5</td><td>1.5--2.5</td><td></td></tr><tr><td>ALP</td><td>碱性磷酸酶</td><td>93</td><td>15--112</td><td>HUSL</td></tr><tr><td>GGT</td><td>谷氨酰转肽酶</td><td>14.3</td><td><50</td><td>U/L</td></tr><tr><td>AST</td><td>谷草转氨酶</td><td>16.3</td><td>8--40</td><td>W/L</td></tr><tr><td>LDH</td><td>乳酸脱氢酶</td><td>167</td><td>114--240</td><td>U/L</td></tr><tr><td>ADA</td><td>腺甘脱氨酶</td><td>12.6</td><td>4--24</td><td>U/L</td></tr></table></body></html>\n"
     ]
    }
   ],
   "source": [
    "# https://github.com/PaddlePaddle/PaddleOCR/blob/dygraph/ppstructure/table/predict_table.py#L55\n",
    "\n",
    "from table.predict_table import TableSystem,to_excel\n",
    "from utility import init_args\n",
    "# Initialization parameters\n",
    "args = init_args().parse_args(args=[])\n",
    "args.det_model_dir='inference/ch_PP-OCRv2_det_infer'\n",
    "args.rec_model_dir='inference/ch_PP-OCRv2_rec_infer'\n",
    "args.table_model_dir='inference/en_ppocr_mobile_v2.0_table_structure_infer'\n",
    "args.image_dir='/home/aistudio/1.jpg'\n",
    "args.rec_char_dict_path='../ppocr/utils/ppocr_keys_v1.txt'\n",
    "args.table_char_dict_path='../ppocr/utils/dict/table_structure_dict.txt'\n",
    "args.det_limit_side_len=736\n",
    "args.det_limit_type='min'\n",
    "args.output='../output/table'\n",
    "args.use_gpu=False\n",
    "\n",
    "# Initialize the form recognition system\n",
    "table_sys = TableSystem(args)\n",
    "img = cv2.imread('/home/aistudio/1.jpg')\n",
    "# Perform form recognition\n",
    "pred_html = table_sys(img)\n",
    "# Store the result to an excel file\n",
    "to_excel(pred_html,'1.xlsx')\n",
    "print(pred_html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "      代号      项目     结果       参考值      单位\n",
      "0    ALT   谷丙转氨酶   25.6     0--40     U/L\n",
      "1   TBIL    总胆红素   11.2       <20  UMOL/L\n",
      "2   DBIL   直接胆红素    3.3      0--7  UMOL/L\n",
      "3   IBIL   间接胆红素    7.9   1.5--15  UMOL/L\n",
      "4     TP     总蛋白  58.9J    60--80     g/L\n",
      "5    ALB     白蛋白   35.1    33--55     g/L\n",
      "6    GLO     球蛋白   23.8    20--30     8/L\n",
      "7    A/G     白球比    1.5  1.5--2.5        \n",
      "8    ALP   碱性磷酸酶     93   15--112    HUSL\n",
      "9    GGT  谷氨酰转肽酶   14.3       <50     U/L\n",
      "10   AST   谷草转氨酶   16.3     8--40     W/L\n",
      "11   LDH   乳酸脱氢酶    167  114--240     U/L\n",
      "12   ADA   腺甘脱氨酶   12.6     4--24     U/L\n"
     ]
    }
   ],
   "source": [
    "# Read excel and display\n",
    "import pandas as pd\n",
    "df = pd.read_excel('1.xlsx').fillna('')\n",
    "print(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2 Detailed Explanation of Prediction Principle:\n",
    "\n",
    "### 2.1 Introduction to The Overall Pipeline\n",
    "\n",
    "PP-Structure's table recognition model algorithm is based on an end-to-end method.\n",
    "\n",
    "The table recognition algorithm consists of three models:\n",
    "1. Text detection model: used to detect the text in the table\n",
    "2. Text recognition model: used to recognize the detected text\n",
    "3. Table cell prediction and table structure prediction model: HTML information and table cell coordinates used to predict table structure\n",
    "\n",
    "The series process of the three models is shown in the figure below:\n",
    "<center class=\"img\">\n",
    "<img src=\"https://ai-studio-static-online.cdn.bcebos.com/07fad4f0bc6a473f9258d913a9afc380c3cd582cc44f4d0fa4cdbade934e07b5\" width=\"1300\"/></center>\n",
    "<center>Figure 1: Form recognition pipeline</center>\n",
    "\n",
    "\n",
    "The specific process is:\n",
    "1. Use the text detection model to detect the text in the table\n",
    "2. Use the text recognition model to recognize the detected text. At this step, we get the text box and text information\n",
    "3. Use table cell prediction and table structure prediction models for cell coordinate prediction and table structure HTML information prediction\n",
    "4. Aggregate the text box in 2 and the cell coordinates in 3, as shown in the figure below, according to the difference between the red text detection box and the blue cell coordinate detection box. IOU</font> to determine whether aggregation is needed.\n",
    "5. After completing the text box aggregation, sort the text boxes from top to bottom and from left to right. According to the index of the sorted text box, you can get the corresponding text information, and then make a text information as a <font color =\"#dd0000\">String splicing</font> to get the text content of the final cell.\n",
    "<center class=\"img\">\n",
    "<img src=\"https://ai-studio-static-online.cdn.bcebos.com/32a7368a59f142dcb735247fa7537ae1681c5541f92444388bd916a942fcdfa5\" width=\"1300\"/></center>\n",
    "<center>Figure 2: Schematic diagram of text box and cell coordinate aggregation</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 Introduction to Table Structure Prediction Model\n",
    "\n",
    "Form recognition requires three models: text detection, text recognition and table structure recognition models. The text detection and recognition model has been introduced in the previous course, and here is the focus on the table structure prediction model.\n",
    "\n",
    "The table structure prediction model completes the prediction of table structure and the detection of table cell coordinates. The table structure model is modified from the RARE algorithm, mainly in the following aspects:\n",
    "\n",
    "#### 2.2.1 Input Data\n",
    "\n",
    "For the text recognition model, each character marked in the data set is independent, but in the table structure prediction model, the predicted category is not a single character. The following is a dictionary comparison between RARE and the table structure prediction model:\n",
    "\n",
    "|model| dictionary|\n",
    "|---|---|\n",
    "|RARE|`'<', 's', 'u', 'p', '>', '<', '/', 's', 'u', 'b', '>', '<', 'b', '>', '<', '/', 'b', '>', '<', 'i', '>', '<', '/', 'i', '>’`|\n",
    "|Table structure prediction model|`'sos', '<thead>', '<tr>', '<td>', '</td>', '</tr>', '</thead>', '<tbody>', '</tbody>', '<td', ' colspan=\"5\"', '>', ' colspan=\"2\"', ' colspan=\"3\"', ' rowspan=\"2\"', ' colspan=\"4\"', ' colspan=\"6\"', ' rowspan=\"3\"', ' colspan=\"9\"', ' colspan=\"10\"', ' colspan=\"7\"', ' rowspan=\"4\"', ' rowspan=\"5\"', ' rowspan=\"9\"', ' colspan=\"8\"', ' rowspan=\"8\"', ' rowspan=\"6\"', ' rowspan=\"7\"', ' rowspan=\"10\"', 'eos'`|\n",
    "\n",
    "In the table structure prediction model, <font color=\"#dd0000\"> treats a string like `<thead>` as one character for recognition</font>.\n",
    "\n",
    "#### 2.2.2 Model\n",
    "\n",
    "The comparison chart of the table structure recognition model and EARE is as follows:\n",
    "<center class=\"img\">\n",
    "<img src=\"https://ai-studio-static-online.cdn.bcebos.com/6f08c91824954cb0aba30d816e2d493c58193463acd34a12a366129cc5f89458\" width=\"1300\"/></center>\n",
    "<center>Figure 3: Schematic diagram of table structure recognition model</center>\n",
    "\n",
    "\n",
    "The RARE model is composed of TPS+CNN+RNN+AttentionHead, and the main functions of each part are as follows:\n",
    "1. TPS: Correct the curved text to restore the image to a level\n",
    "2. CNN: feature extraction of images\n",
    "3. RNN: further enhance the extracted features and extract semantic features\n",
    "3. AttentionHead: output\n",
    "\n",
    "In the table structure recognition model, the input image is the entire image, so the TPS module is removed. In addition, it is proved through experiments that RNN has little effect on the result. Therefore, the RNN module is removed. The final table structure recognition model is CNN+AttentionHead.\n",
    "\n",
    "In order to output the cell coordinates, we tried to detect the cell coordinates in the detection model. On the basis of the DB model, we tried the following schemes 2 and 3.\n",
    "\n",
    "|Solution | Results|\n",
    "|---|---|\n",
    "|1. Single-line text detection|<center class=\"img\"><img src=\"https://ai-studio-static-online.cdn.bcebos.com/cd8b4dd7df4d411086ba6dd455afcb7b6ab130639aad4607b05293876c64c419\" width=\"1300\"/></center>|\n",
    "|2. Use a model to check text and cells|<center class=\"img\"><img src=\"https://ai-studio-static-online.cdn.bcebos.com/d93353bacff74545b663be245102d03b716535221f15465284591987144d9fb8\" width=\"1300\"/></center>|\n",
    "|3. Text and cell detection with two models|<center class=\"img\"><img src=\"https://ai-studio-static-online.cdn.bcebos.com/a84881137cdd49d593abad35bb91f18d31cd49c9e2f44292aa33d7948e43d977\" width=\"1300\"/></center>|\n",
    "\n",
    "It can be seen that the completion of text and cell detection in the segmentation model will lead to the peculiarities of GT: Is the GT of the background between each line in the cell text or background?\n",
    "\n",
    "Among the three models of the entire table recognition pipeline, only the text detection and table structure recognition models can obtain the information of the entire image. Therefore, an additional regression-based branch is added to the AttentionHead of the table structure recognition model to complete the cell Coordinate (x0, y0, x1, y1) detection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3 Forward Analysis of Table Structure Prediction Model\n",
    "\n",
    "Model forward analysis analyzes the output shape changes in each module between image input from preprocessing to network output to better understand table cell prediction and table structure prediction models. The modules involved are as follows:\n",
    " \n",
    "  |Type|Module Name|\n",
    "  |---|---|\n",
    "  |Data Processing|ResizeTableImage|\n",
    "  |Data Processing|PaddingTableImage|\n",
    "  |Backbone|MobileNetV3|\n",
    "  |Head|TableAttentionHead|\n",
    "\n",
    "#### 2.3.1 Input Data Processing\n",
    "In this example, the input image and the output of the data processing module are visualized as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 461528 characters]",
      "text/plain": [
       "<Figure size 1728x576 with 3 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Switch to the PaddleOCR directory\n",
    "os.chdir('/home/aistudio/PaddleOCR')\n",
    "from ppocr.data import create_operators, transform\n",
    "plt.figure(figsize=(24,8))\n",
    "\n",
    "# Read input image\n",
    "img = cv2.imread('/home/aistudio/1.jpg')\n",
    "\n",
    "# Display input image\n",
    "plt.subplot(1,3,1)\n",
    "plt.title('src, shape:{}'.format(img.shape))\n",
    "plt.imshow(img)\n",
    "\n",
    "# implement ResizeTableImage\n",
    "# https://github.com/PaddlePaddle/PaddleOCR/blob/dygraph/ppocr/data/imaug/gen_table_mask.py#L182\n",
    "\n",
    "pre_process_list = [{'ResizeTableImage': {'max_len': args.table_max_len }}]  # Scale the long side of the picture to the specified length, and scale the short side in equal proportions\n",
    "preprocess_op = create_operators(pre_process_list)\n",
    "data = {'image': img}\n",
    "data = transform(data, preprocess_op)\n",
    "\n",
    "# Image after displaying ResizeTableImage\n",
    "plt.subplot(1,3,2)\n",
    "plt.title('ResizeTableImage, shape:{}'.format(data['image'].shape))\n",
    "plt.imshow(data['image'])\n",
    "\n",
    "# implement PaddingTableImage\n",
    "# https://github.com/PaddlePaddle/PaddleOCR/blob/dygraph/ppocr/data/imaug/gen_table_mask.py#L232\n",
    "\n",
    "pre_process_list = [{'PaddingTableImage': None}]\n",
    "preprocess_op = create_operators(pre_process_list)\n",
    "\n",
    "data = transform(data, preprocess_op)\n",
    "\n",
    "# Image after displaying PaddingTableImage\n",
    "plt.subplot(1,3,3)\n",
    "plt.title('PaddingTableImage, shape:{}'.format(data['image'].shape))\n",
    "plt.imshow(data['image']/255)\n",
    "plt.show()\n",
    "\n",
    "# Define a complete list of processing ops\n",
    "pre_process_list = [\n",
    "    {'ResizeTableImage': {'max_len': args.table_max_len }},\n",
    "    {'NormalizeImage':{'scale':1./255., 'mean': [0.485, 0.456, 0.406],'std': [0.229, 0.224, 0.225], 'order': 'hwc'}},\n",
    "    {'PaddingTableImage': None},\n",
    "    {'ToCHWImage': None}\n",
    "] \n",
    "# Create op list\n",
    "preprocess_op = create_operators(pre_process_list)\n",
    "# Execute op list\n",
    "data = {'image': img}\n",
    "data = transform(data, preprocess_op)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2021-12-26 19:56:07--  https://paddleocr.bj.bcebos.com/dygraph_v2.1/table/en_ppocr_mobile_v2.0_table_structure_train.tar\n",
      "Resolving paddleocr.bj.bcebos.com (paddleocr.bj.bcebos.com)... 182.61.200.229, 182.61.200.195, 2409:8c04:1001:1002:0:ff:b001:368a\n",
      "Connecting to paddleocr.bj.bcebos.com (paddleocr.bj.bcebos.com)|182.61.200.229|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 76103680 (73M) [application/x-tar]\n",
      "Saving to: ‘./pre_train/en_ppocr_mobile_v2.0_table_structure_train.tar.3’\n",
      "\n",
      "en_ppocr_mobile_v2. 100%[===================>]  72.58M  36.6MB/s    in 2.0s    \n",
      "\n",
      "2021-12-26 19:56:09 (36.6 MB/s) - ‘./pre_train/en_ppocr_mobile_v2.0_table_structure_train.tar.3’ saved [76103680/76103680]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Download the pre-trained model\n",
    "! wget -P ./pre_train/ https://paddleocr.bj.bcebos.com/dygraph_v2.1/table/en_ppocr_mobile_v2.0_table_structure_train.tar && cd pre_train && tar xf en_ppocr_mobile_v2.0_table_structure_train.tar && cd ..\n",
    "# Downloaded pre-trained model\n",
    "import paddle\n",
    "\n",
    "# Read pre-training parameters and divide them into backbone parameters and head parameters\n",
    "pretrain_params = paddle.load('/home/aistudio/PaddleOCR/pre_train/en_ppocr_mobile_v2.0_table_structure_train/best_accuracy.pdparams')\n",
    "def filter_params(pretrain_params,prefix):\n",
    "    new_dict = {}\n",
    "    for k,v in pretrain_params.items():\n",
    "        if k.startswith(prefix):\n",
    "            new_dict[k.replace(prefix+'.','')] = v\n",
    "    return new_dict\n",
    "# Extract parameters\n",
    "backbone_dict = filter_params(pretrain_params,'backbone')\n",
    "head_dict = filter_params(pretrain_params,'head')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.3.2 Backbone\n",
    "\n",
    "The backbone is the same as the detected backbone, and both output four feature maps with sizes 1/4, 1/8, 1/16 and 1/32 of the input image. Relevant backbones have been introduced in the text detection chapter, and will not be repeated here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "# https://github.com/PaddlePaddle/PaddleOCR/blob/dygraph/ppocr/modeling/backbones/det_mobilenet_v3.py\n",
    "\n",
    "from ppocr.modeling.backbones import build_backbone\n",
    "# Initialize the backbone\n",
    "backbone = build_backbone(dict(name='MobileNetV3',scale=1.0,model_name='large'),model_type='table')\n",
    "backbone.eval()\n",
    "# Load backbone parameters\n",
    "backbone.set_state_dict(backbone_dict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1, 24, 122, 122]\n",
      "[1, 40, 61, 61]\n",
      "[1, 112, 31, 31]\n",
      "[1, 960, 16, 16]\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "x = np.expand_dims(data['image'],axis=0)\n",
    "x = paddle.to_tensor(x)\n",
    "backbone_out = backbone(x)\n",
    "for item in backbone_out:\n",
    "    print(item.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.3.3 Head\n",
    "\n",
    "The input of the Head is the four feature maps output by the backbone, and the output is the prediction result of the table structure and cell coordinates\n",
    "\n",
    "The meaning of the input parameters is:\n",
    "|Parameter|Meaning|\n",
    "|---|---|\n",
    "|in_channels|The number of channels of the input feature map|\n",
    "|hidden_size|The hidden layer unit of the RNN module in Attention|\n",
    "|max_elem_length|Maximum number of predicted characters|\n",
    "|in_max_len| The size of the input image|\n",
    "|loc_type|Input of the output cell coordinate branch<br>1: Only the hidden layer after Attention is used <br>2: Fusion CNN part + Attention part|\n",
    "\n",
    "\n",
    "The code is as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "# https://github.com/PaddlePaddle/PaddleOCR/blob/dygraph/ppocr/modeling/heads/table_att_head.py\n",
    "\n",
    "from paddle import nn\n",
    "import paddle.nn.functional as F\n",
    "from ppocr.modeling.heads.table_att_head import AttentionGRUCell\n",
    "\n",
    "class TableAttentionHead(nn.Layer):\n",
    "    def __init__(self,\n",
    "                 in_channels,\n",
    "                 hidden_size,\n",
    "                 loc_type=2,\n",
    "                 in_max_len=488,  # Enter the size of the image\n",
    "                 max_elem_length=800, # Maximum number of labels output\n",
    "                 **kwargs):\n",
    "        super(TableAttentionHead, self).__init__()\n",
    "        self.input_size = in_channels[-1]\n",
    "        self.hidden_size = hidden_size\n",
    "        self.elem_num = 30\n",
    "        self.max_elem_length = max_elem_length\n",
    "\n",
    "        self.structure_attention_cell = AttentionGRUCell(\n",
    "            self.input_size, hidden_size, self.elem_num, use_gru=False)\n",
    "        self.structure_generator = nn.Linear(hidden_size, self.elem_num)\n",
    "        self.loc_type = loc_type\n",
    "        self.in_max_len = in_max_len\n",
    "        \n",
    "        # Coordinate box regression branch\n",
    "        if self.loc_type == 1:\n",
    "            self.loc_generator = nn.Linear(hidden_size, 4)\n",
    "        else:\n",
    "            if self.in_max_len == 640:\n",
    "                # The last feature map after 640 passes through the backbone is 20*20, so the size of the feature map entered here is 400\n",
    "                self.loc_fea_trans = nn.Linear(400, self.max_elem_length + 1)\n",
    "            elif self.in_max_len == 800:\n",
    "                # 800 After the backbone, the last feature map is 23*25, so the size of the feature map entered here is 625\n",
    "                self.loc_fea_trans = nn.Linear(625, self.max_elem_length + 1)\n",
    "            elif self.in_max_len == 488:\n",
    "                # 800 After the backbone, the last feature map is 16*16, so the size of the feature map entered here is 256\n",
    "                self.loc_fea_trans = nn.Linear(256, self.max_elem_length + 1)\n",
    "            self.loc_generator = nn.Linear(self.input_size + hidden_size, 4)\n",
    "\n",
    "    def _char_to_onehot(self, input_char, onehot_dim):\n",
    "        input_ont_hot = F.one_hot(input_char, onehot_dim)\n",
    "        return input_ont_hot\n",
    "\n",
    "    def forward(self, inputs, targets=None):\n",
    "        # Take out the smallest map output by the backbone\n",
    "        fea = inputs[-1]\n",
    "        if len(fea.shape) == 3:\n",
    "            pass\n",
    "        else:\n",
    "            # B,C,H,W reshape as B,C,H*W\n",
    "            last_shape = int(np.prod(fea.shape[2:])) \n",
    "            fea = paddle.reshape(fea, [fea.shape[0], fea.shape[1], last_shape])\n",
    "            # B,C,W as B,W,C\n",
    "            fea = fea.transpose([0, 2, 1])\n",
    "        batch_size = fea.shape[0]\n",
    "\n",
    "        hidden = paddle.zeros((batch_size, self.hidden_size))\n",
    "        output_hiddens = []\n",
    "        if self.training and targets is not None:\n",
    "            structure = targets[0]\n",
    "            for i in range(self.max_elem_length + 1):\n",
    "                elem_onehots = self._char_to_onehot(\n",
    "                    structure[:, i], onehot_dim=self.elem_num)\n",
    "                (outputs, hidden), alpha = self.structure_attention_cell(\n",
    "                    hidden, fea, elem_onehots)\n",
    "                output_hiddens.append(paddle.unsqueeze(outputs, axis=1))\n",
    "            output = paddle.concat(output_hiddens, axis=1)\n",
    "            structure_probs = self.structure_generator(output)\n",
    "            if self.loc_type == 1:\n",
    "                loc_preds = self.loc_generator(output)\n",
    "                loc_preds = F.sigmoid(loc_preds)\n",
    "            else:\n",
    "                loc_fea = fea.transpose([0, 2, 1])\n",
    "                loc_fea = self.loc_fea_trans(loc_fea)\n",
    "                loc_fea = loc_fea.transpose([0, 2, 1])\n",
    "                loc_concat = paddle.concat([output, loc_fea], axis=2)\n",
    "                loc_preds = self.loc_generator(loc_concat)\n",
    "                loc_preds = F.sigmoid(loc_preds)\n",
    "        else:\n",
    "            temp_elem = paddle.zeros(shape=[batch_size], dtype=\"int32\")\n",
    "            structure_probs = None\n",
    "            loc_preds = None\n",
    "            elem_onehots = None\n",
    "            outputs = None\n",
    "            alpha = None\n",
    "            max_elem_length = paddle.to_tensor(self.max_elem_length)\n",
    "            i = 0\n",
    "            # Attention forward\n",
    "            while i < max_elem_length + 1:\n",
    "                elem_onehots = self._char_to_onehot(\n",
    "                    temp_elem, onehot_dim=self.elem_num)\n",
    "                (outputs, hidden), alpha = self.structure_attention_cell(\n",
    "                    hidden, fea, elem_onehots)\n",
    "                output_hiddens.append(paddle.unsqueeze(outputs, axis=1))\n",
    "                structure_probs_step = self.structure_generator(outputs)\n",
    "                temp_elem = structure_probs_step.argmax(axis=1, dtype=\"int32\")\n",
    "                i += 1\n",
    "\n",
    "            output = paddle.concat(output_hiddens, axis=1)\n",
    "            print('Attention output shape',output.shape)\n",
    "            # Table structure branch\n",
    "            structure_probs = self.structure_generator(output)\n",
    "            structure_probs = F.softmax(structure_probs)\n",
    "\n",
    "            # Cell coordinate branch\n",
    "            if self.loc_type == 1:\n",
    "                loc_preds = self.loc_generator(output)\n",
    "                loc_preds = F.sigmoid(loc_preds)\n",
    "            else:\n",
    "                # B,W,C 改为 B,C,W\n",
    "                loc_fea = fea.transpose([0, 2, 1])\n",
    "\n",
    "                loc_fea = self.loc_fea_trans(loc_fea)\n",
    "                loc_fea = loc_fea.transpose([0, 2, 1])\n",
    "                loc_concat = paddle.concat([output, loc_fea], axis=2)\n",
    "                loc_preds = self.loc_generator(loc_concat)\n",
    "                loc_preds = F.sigmoid(loc_preds)\n",
    "        return {'structure_probs': structure_probs, 'loc_preds': loc_preds}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "********** head forward shape **********\n",
      "Attention output shape [1, 801, 256]\n",
      "********** head out shape **********\n",
      "structure_probs [1, 801, 30]\n",
      "loc_preds [1, 801, 4]\n"
     ]
    }
   ],
   "source": [
    "# Initialize head\n",
    "head = TableAttentionHead(in_channels=backbone.out_channels,hidden_size=256,loc_type=2)\n",
    "head.eval()\n",
    "# Load the head parameter\n",
    "head.set_state_dict(head_dict)\n",
    "\n",
    "# Execute head\n",
    "print('*'*10,'head forward shape','*'*10)\n",
    "head_out = head(backbone_out)\n",
    "print('*'*10,'head out shape','*'*10)\n",
    "\n",
    "# Print head output and corresponding shape\n",
    "for key in head_out:\n",
    "    print(key,head_out[key].shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Post-processing**\n",
    "\n",
    "The dictionary file for post-processing is ppocr/utils/dict/table_structure_dict.txt\n",
    "\n",
    "Post-processing decoding ideas:\n",
    "\n",
    "1. Perform CTC decoding on structure_probs: Don’t use background characters sos and eos, take only one of the consecutively repeated characters\n",
    "2. The output coordinates are normalized to a value of 0-1, multiply the coordinates by the width and height of the picture, and decode to the image space"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['<html>', '<body>', '<table>', '<thead>', '<tr>', '<td>', '</td>', '<td>', '</td>', '<td>', '</td>', '<td>', '</td>', '<td>', '</td>', '</tr>', '</thead>', '<tbody>', '<tr>', '<td>', '</td>', '<td>', '</td>', '<td>', '</td>', '<td>', '</td>', '<td>', '</td>', '</tr>', '<tr>', '<td>', '</td>', '<td>', '</td>', '<td>', '</td>', '<td>', '</td>', '<td>', '</td>', '</tr>', '<tr>', '<td>', '</td>', '<td>', '</td>', '<td>', '</td>', '<td>', '</td>', '<td>', '</td>', '</tr>', '<tr>', '<td>', '</td>', '<td>', '</td>', '<td>', '</td>', '<td>', '</td>', '<td>', '</td>', '</tr>', '<tr>', '<td>', '</td>', '<td>', '</td>', '<td>', '</td>', '<td>', '</td>', '<td>', '</td>', '</tr>', '<tr>', '<td>', '</td>', '<td>', '</td>', '<td>', '</td>', '<td>', '</td>', '<td>', '</td>', '</tr>', '<tr>', '<td>', '</td>', '<td>', '</td>', '<td>', '</td>', '<td>', '</td>', '<td>', '</td>', '</tr>', '<tr>', '<td>', '</td>', '<td>', '</td>', '<td>', '</td>', '<td>', '</td>', '<td>', '</td>', '</tr>', '<tr>', '<td>', '</td>', '<td>', '</td>', '<td>', '</td>', '<td>', '</td>', '<td>', '</td>', '</tr>', '<tr>', '<td>', '</td>', '<td>', '</td>', '<td>', '</td>', '<td>', '</td>', '<td>', '</td>', '</tr>', '<tr>', '<td>', '</td>', '<td>', '</td>', '<td>', '</td>', '<td>', '</td>', '<td>', '</td>', '</tr>', '<tr>', '<td>', '</td>', '<td>', '</td>', '<td>', '</td>', '<td>', '</td>', '<td>', '</td>', '</tr>', '<tr>', '<td>', '</td>', '<td>', '</td>', '<td>', '</td>', '<td>', '</td>', '<td>', '</td>', '</tr>', '</tbody>', '</table>', '</body>', '</html>']\n",
      "[[32, 9, 104, 40], [232, 8, 307, 41], [429, 7, 500, 44], [559, 8, 656, 44], [715, 7, 780, 44], [37, 45, 99, 73], [190, 44, 342, 74], [432, 45, 502, 74], [565, 44, 655, 73], [712, 46, 777, 74], [30, 81, 101, 109], [202, 80, 337, 110], [433, 81, 503, 111], [578, 83, 638, 110], [698, 82, 790, 110], [31, 119, 104, 148], [197, 116, 347, 147], [443, 117, 492, 148], [572, 118, 643, 147], [698, 118, 797, 147], [35, 154, 101, 183], [199, 152, 342, 184], [436, 154, 501, 184], [558, 155, 670, 184], [701, 153, 801, 183], [40, 188, 93, 217], [217, 187, 314, 219], [417, 187, 516, 218], [556, 187, 667, 217], [716, 188, 772, 216], [48, 227, 98, 255], [223, 224, 313, 256], [429, 226, 500, 256], [558, 226, 667, 256], [722, 225, 772, 254], [47, 262, 99, 291], [217, 260, 313, 293], [439, 261, 506, 293], [557, 260, 678, 292], [722, 261, 777, 290], [36, 295, 95, 324], [210, 296, 317, 326], [443, 296, 499, 326], [547, 296, 681, 326], [701, 300, 767, 328], [42, 332, 99, 361], [191, 330, 350, 360], [451, 331, 493, 361], [557, 331, 683, 361], [717, 335, 785, 362], [45, 369, 97, 396], [186, 367, 355, 400], [444, 369, 504, 398], [581, 369, 640, 397], [723, 368, 773, 396], [37, 404, 95, 431], [192, 404, 351, 433], [438, 404, 511, 432], [560, 405, 658, 432], [723, 404, 775, 431], [46, 444, 104, 469], [188, 441, 346, 470], [444, 444, 496, 471], [544, 445, 681, 471], [721, 444, 773, 470], [35, 478, 104, 503], [190, 475, 345, 505], [436, 477, 504, 504], [559, 476, 662, 505], [712, 477, 778, 504]]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x7faad993b2d0>"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 356904 characters]",
      "text/plain": [
       "<Figure size 1728x576 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# https://github.com/PaddlePaddle/PaddleOCR/blob/dygraph/ppocr/postprocess/rec_postprocess.py#L441\n",
    "\n",
    "from ppocr.postprocess.rec_postprocess import TableLabelDecode\n",
    "\n",
    "def post_process(out):\n",
    "    character_dict_path = '/home/aistudio/PaddleOCR/ppocr/utils/dict/table_structure_dict.txt'\n",
    "    # Processing op after initialization\n",
    "    post_op = TableLabelDecode(character_dict_path)\n",
    "\n",
    "    post_result = post_op(out)\n",
    "    \n",
    "    structure_str_list = post_result['structure_str_list']\n",
    "\n",
    "    # The normalized coordinates are restored to the original image size\n",
    "    res_loc = post_result['res_loc']\n",
    "    imgh, imgw = img.shape[0:2]\n",
    "    res_loc_final = []\n",
    "    for rno in range(len(res_loc[0])):\n",
    "        x0, y0, x1, y1 = res_loc[0][rno]\n",
    "        left = max(int(imgw * x0), 0)\n",
    "        top = max(int(imgh * y0), 0)\n",
    "        right = min(int(imgw * x1), imgw - 1)\n",
    "        bottom = min(int(imgh * y1), imgh - 1)\n",
    "        res_loc_final.append([left, top, right, bottom])\n",
    "    \n",
    "    # Structure information processing\n",
    "    structure_str_list = structure_str_list[0]\n",
    "    structure_str_list = ['<html>', '<body>', '<table>'] + structure_str_list + ['</table>', '</body>', '</html>']\n",
    "    return structure_str_list,res_loc_final\n",
    "structure_str_list,res_loc_final = post_process(head_out)\n",
    "\n",
    "print(structure_str_list)\n",
    "print(res_loc_final)\n",
    "\n",
    "# Visual prediction box\n",
    "plt.figure(figsize=(24,8))\n",
    "img_show = img.copy()\n",
    "for box in res_loc_final:\n",
    "    cv2.rectangle(img_show, (box[0], box[1]), (box[2], box[3]), (0, 255, 0), 2)\n",
    "plt.imshow(img_show)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3 Training\n",
    "\n",
    "To train table recognition, three models need to be trained, namely text detection, text recognition, and table structure model. For the training in text detection and recognition, you can refer to the previous courses. Only the training process of the table structure model is introduced here.\n",
    "\n",
    "This section uses the pubtabnet dataset and MobileNetV3 as the table structure model model of the backbone network to introduce how to complete the training, evaluation and testing of the table structure model.\n",
    "\n",
    "### 3.1 Data Preparation\n",
    "\n",
    "This experiment selects the PubTabNet data set as our demonstration data set. The sample diagram of the PubTabNet data set is shown in the following figure:\n",
    "<center class=\"img\">\n",
    "<img src=\"https://ai-studio-static-online.cdn.bcebos.com/9732a3b97aff4a4194c5aec210400a8b0031c4f1887548d78f92f6941db0a6bd\" width=\"1300\"/></center>\n",
    "<center>Figure 4: Schematic diagram of PubTabNet data set</center>\n",
    "\n",
    "Part of the data set of PubTabNet has been downloaded in the project and stored in /home/aistudio/data/data119702. You can run the following command to complete the decompression of the data set, or download it from [https://github.com/ibm-aur-nlp/PubTabNet ](https://github.com/ibm-aur-nlp/PubTabNet) download it by yourself."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/aistudio/PaddleOCR\n",
      "PubTabNet_2.0.0_val.jsonl  pubtabnet_val.tar  val\n"
     ]
    }
   ],
   "source": [
    "# Unzip the data set\n",
    "! cd /home/aistudio/data/data119702 && tar -xf pubtabnet_val.tar  && cd -\n",
    "! ls /home/aistudio/data/data119702"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After running the above command `/home/aistudio/data/data119702` There is a folder and a file, respectively：\n",
    "```bash\n",
    "/home/aistudio/data/data119702\n",
    "  └─ val/        \t\t \tPicture storage folder\n",
    "  └─ PubTabNet_2.0.0_val.jsonl/         Label information\n",
    "```\n",
    "\n",
    "The label format of this dataset is\n",
    "\n",
    "```json\n",
    "{\n",
    "   'filename': PMC5755158_010_01.png,\t\t\t\t\t\t\t# Image name\n",
    "   'split': ’train‘, \t\t\t\t\t\t\t\t\t# Does the image belong to the training set or the validation set?\n",
    "   'imgid': 0,\t\t\t\t\t\t\t\t \t\t# The index of the image\n",
    "   'html': {\n",
    "     'structure': {'tokens': ['<thead>', '<tr>', '<td>', ...]}, \t\t\t#HTML string of the table\n",
    "     'cell': [\n",
    "       {\n",
    "         'tokens': ['P', 'a', 'd', 'd', 'l', 'e', 'P', 'a', 'd', 'd', 'l', 'e'], \t# Single text in the table\n",
    "         'bbox': [x0, y0, x1, y1]  \t\t\t\t\t\t\t# The coordinates of a single text in the table\n",
    "       }\n",
    "     ]\n",
    "   }\n",
    "}\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2 Data Preprocessing\n",
    "\n",
    "There are certain requirements for the format and size of the input pictures during practice. Therefore, before the data is input into the model, the data needs to be preprocessed to make the pictures and labels meet the needs of network training and prediction.\n",
    "\n",
    "The data preprocessing of the table structure model mainly includes the following:\n",
    "\n",
    "* DecodeImage, convert the image to Numpy format\n",
    "* ResizeTableImage, resize the picture, resize the long side to the specified size, and scale the short side equally\n",
    "* TableLabelEncode, parse the label information in the label file and save it in a unified format\n",
    "* NormalizeImage, through the normalization method, the input value distribution of any neuron in each layer of the neural network is changed to a standard normal distribution with a mean value of 0 and a variance of 1, so that the optimization process of the optimal solution will obviously become smooth and the training process Easier to converge;\n",
    "* PaddingTableImage, pad the short side of the image and pad it to the same size as the long side\n",
    "* ToCHWImage, the image data format is [H, W, C] (that is, height, width, and number of channels), and the training data format used by the neural network is [C, H, W], so the image data needs to be rearranged , For example [224, 224, 3] becomes [3, 224, 224];\n",
    "* KeepKeys, dict filtering\n",
    "\n",
    "**TableLabelEncode**\n",
    "\n",
    "Analyze the label information in the label file, first load the label data and take out a label"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "# Load a piece of data in the data set\n",
    "import json\n",
    "from pprint import pprint\n",
    "with open('/home/aistudio/data/data119702/PubTabNet_2.0.0_val.jsonl', \"rb\") as f:\n",
    "    data_lines = f.readlines()\n",
    "    for line in data_lines:\n",
    "        data_line = line.decode('utf-8').strip(\"\\n\")\n",
    "        info = json.loads(data_line)\n",
    "        break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the following code to observe the comparison before and after the TableLabelEncode class encoding label."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The cells and structure before decode\n",
      "cells:  [{'tokens': []}, {'tokens': ['<b>', 'W', 'e', 'a', 'n', 'i', 'n', 'g', '</b>'], 'bbox': [66, 4, 96, 13]}, {'tokens': ['<b>', 'W', 'e', 'e', 'k', ' ', '1', '5', '</b>'], 'bbox': [131, 4, 160, 13]}, {'tokens': ['<b>', 'O', 'f', 'f', '-', 't', 'e', 's', 't', '</b>'], 'bbox': [201, 4, 226, 13]}, {'tokens': ['W', 'e', 'a', 'n', 'i', 'n', 'g'], 'bbox': [1, 17, 31, 26]}, {'tokens': ['–'], 'bbox': [66, 21, 72, 25]}, {'tokens': ['–'], 'bbox': [131, 21, 137, 25]}, {'tokens': ['–'], 'bbox': [201, 21, 207, 25]}, {'tokens': ['W', 'e', 'e', 'k', ' ', '1', '5'], 'bbox': [1, 31, 30, 40]}, {'tokens': ['–'], 'bbox': [66, 35, 72, 39]}, {'tokens': ['0', '.', '1', '7', ' ', '±', ' ', '0', '.', '0', '8'], 'bbox': [131, 31, 166, 40]}, {'tokens': ['0', '.', '1', '6', ' ', '±', ' ', '0', '.', '0', '3'], 'bbox': [201, 31, 236, 40]}, {'tokens': ['O', 'f', 'f', '-', 't', 'e', 's', 't'], 'bbox': [1, 45, 26, 54]}, {'tokens': ['–'], 'bbox': [66, 49, 72, 53]}, {'tokens': ['0', '.', '8', '0', ' ', '±', ' ', '0', '.', '2', '4'], 'bbox': [131, 45, 166, 54]}, {'tokens': ['0', '.', '1', '9', ' ', '±', ' ', '0', '.', '0', '9'], 'bbox': [201, 45, 236, 54]}]\n",
      "structure:  {'tokens': ['<thead>', '<tr>', '<td>', '</td>', '<td>', '</td>', '<td>', '</td>', '<td>', '</td>', '</tr>', '</thead>', '<tbody>', '<tr>', '<td>', '</td>', '<td>', '</td>', '<td>', '</td>', '<td>', '</td>', '</tr>', '<tr>', '<td>', '</td>', '<td>', '</td>', '<td>', '</td>', '<td>', '</td>', '</tr>', '<tr>', '<td>', '</td>', '<td>', '</td>', '<td>', '</td>', '<td>', '</td>', '</tr>', '</tbody>']}\n",
      "The bbox_list and structure after decode\n",
      "bbox_list: [[0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.27731093764305115, 0.06779661029577255, 0.40336135029792786, 0.22033898532390594], [0.0, 0.0, 0.0, 0.0], [0.5504201650619507, 0.06779661029577255, 0.6722689270973206, 0.22033898532390594], [0.0, 0.0, 0.0, 0.0], [0.8445377945899963, 0.06779661029577255, 0.9495798349380493, 0.22033898532390594], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.004201680887490511, 0.2881355881690979, 0.13025210797786713, 0.4406779706478119], [0.0, 0.0, 0.0, 0.0], [0.27731093764305115, 0.35593220591545105, 0.3025210201740265, 0.4237288236618042], [0.0, 0.0, 0.0, 0.0], [0.5504201650619507, 0.35593220591545105, 0.575630247592926, 0.4237288236618042], [0.0, 0.0, 0.0, 0.0], [0.8445377945899963, 0.35593220591545105, 0.8697478771209717, 0.4237288236618042], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.004201680887490511, 0.5254237055778503, 0.1260504275560379, 0.6779661178588867], [0.0, 0.0, 0.0, 0.0], [0.27731093764305115, 0.5932203531265259, 0.3025210201740265, 0.6610169410705566], [0.0, 0.0, 0.0, 0.0], [0.5504201650619507, 0.5254237055778503, 0.6974790096282959, 0.6779661178588867], [0.0, 0.0, 0.0, 0.0], [0.8445377945899963, 0.5254237055778503, 0.9915966391563416, 0.6779661178588867], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.004201680887490511, 0.7627118825912476, 0.10924369841814041, 0.9152542352676392], [0.0, 0.0, 0.0, 0.0], [0.27731093764305115, 0.8305084705352783, 0.3025210201740265, 0.8983050584793091], [0.0, 0.0, 0.0, 0.0], [0.5504201650619507, 0.7627118825912476, 0.6974790096282959, 0.9152542352676392], [0.0, 0.0, 0.0, 0.0], [0.8445377945899963, 0.7627118825912476, 0.9915966391563416, 0.9152542352676392], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0]]\n",
      "structure: [0, 1, 2, 3, 4, 3, 4, 3, 4, 3, 4, 5, 6, 7, 2, 3, 4, 3, 4, 3, 4, 3, 4, 5, 2, 3, 4, 3, 4, 3, 4, 3, 4, 5, 2, 3, 4, 3, 4, 3, 4, 3, 4, 5, 8, 29, 0, 0, 0, 0, 0, 0]\n"
     ]
    }
   ],
   "source": [
    "from ppocr.data.imaug import TableLabelEncode\n",
    "# Initialize the label encoder\n",
    "label_eocoder_op = TableLabelEncode(max_text_length=100,# Unused\n",
    "                                    max_elem_length=50, # How many cels can be predicted at most for each picture\n",
    "                                    max_cell_num=500, # unused\n",
    "                                    character_dict_path='ppocr/utils/dict/table_structure_dict.txt')\n",
    "# Construct input data\n",
    "cells = info['html']['cells']\n",
    "structure = info['html']['structure']\n",
    "# 2. Print the label before decoding\n",
    "print(\"The cells and structure before decode\")\n",
    "print(\"cells: \", cells)\n",
    "print(\"structure: \", structure)\n",
    "\n",
    "image = cv2.imread(os.path.join('/home/aistudio/data/data119702/val', info['filename']))\n",
    "data = {'image':image,'cells': cells, 'structure':structure}\n",
    "# Execute label encoder\n",
    "data = label_eocoder_op(data)\n",
    "# Print the encoded information\n",
    "print(\"The bbox_list and structure after decode\")\n",
    "print(\"bbox_list:\",data['bbox_list'].tolist())\n",
    "print(\"structure:\", data['structure'].tolist())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.3 Loss Function Definition\n",
    "\n",
    "The loss of the model is divided into two parts:\n",
    "1. structure loss: structure loss uses the common CrossEntropyLoss classification\n",
    "2. loc loss: loc loss uses MSELoss\n",
    "\n",
    "The two losses are fused by weighting, and the code summary structure_weight and loc_weight are 100 and 10000 respectively\n",
    "```python\n",
    "total_loss = structure_loss * structure_weight + loc_loss * loc_weight\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.4 Model Training\n",
    "\n",
    "Once the data processing and loss function definitions are complete, the model can be trained. \n",
    "\n",
    "Training is based on PaddleOCR training, in the form of parameter configuration, parameter file reference [https://github.com/PaddlePaddle/PaddleOCR/blob/dygraph/configs/table/table_mv3.yml](https://github.com/PaddlePaddle/ PaddleOCR/blob/dygraph/configs/table/table_mv3.yml), the network structure parameters are as follows\n",
    "\n",
    "```YAML\n",
    "Architecture:\n",
    "  model_type: table\n",
    "  algorithm: TableAttn\n",
    "  Backbone:\n",
    "    name: MobileNetV3\n",
    "    scale: 1.0\n",
    "    model_name: large\n",
    "  Head:\n",
    "    name: TableAttentionHead\n",
    "    hidden_size: 256\n",
    "    loc_type: 2\n",
    "    max_text_length: 100\n",
    "    max_elem_length: 800\n",
    "    max_cell_num: 500\n",
    "```\n",
    "\n",
    "The loss function parameters are as follows:\n",
    "```YAML\n",
    "Loss:\n",
    "  name: TableAttentionLoss\n",
    "  structure_weight: 100.0\n",
    "  loc_weight: 10000.0\n",
    "```\n",
    "After the configuration is complete, the training can be started by the following command"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ln: failed to create symbolic link 'PubTabNet_2.0.0_train.jsonl': File exists\n"
     ]
    }
   ],
   "source": [
    "# Configuration data set\n",
    "# !mkdir -p train_data/table/pubtabnet\n",
    "!cd train_data/table/pubtabnet && ln -s /home/aistudio/data/data119702/PubTabNet_2.0.0_val.jsonl PubTabNet_2.0.0_train.jsonl  \\\n",
    "&& ln -s /home/aistudio/data/data119702/PubTabNet_2.0.0_val.jsonl PubTabNet_2.0.0_val.jsonl \\\n",
    "&& ln -s /home/aistudio/data/data119702/val train \\\n",
    "&& ln -s /home/aistudio/data/data119702/val val"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/setuptools/depends.py:2: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses\n",
      "  import imp\n",
      "[2021/12/26 19:57:29] root INFO: Architecture : \n",
      "[2021/12/26 19:57:29] root INFO:     Backbone : \n",
      "[2021/12/26 19:57:29] root INFO:         model_name : large\n",
      "[2021/12/26 19:57:29] root INFO:         name : MobileNetV3\n",
      "[2021/12/26 19:57:29] root INFO:         scale : 1.0\n",
      "[2021/12/26 19:57:29] root INFO:     Head : \n",
      "[2021/12/26 19:57:29] root INFO:         hidden_size : 256\n",
      "[2021/12/26 19:57:29] root INFO:         l2_decay : 1e-05\n",
      "[2021/12/26 19:57:29] root INFO:         loc_type : 2\n",
      "[2021/12/26 19:57:29] root INFO:         max_cell_num : 500\n",
      "[2021/12/26 19:57:29] root INFO:         max_elem_length : 800\n",
      "[2021/12/26 19:57:29] root INFO:         max_text_length : 100\n",
      "[2021/12/26 19:57:29] root INFO:         name : TableAttentionHead\n",
      "[2021/12/26 19:57:29] root INFO:     algorithm : TableAttn\n",
      "[2021/12/26 19:57:29] root INFO:     model_type : table\n",
      "[2021/12/26 19:57:29] root INFO: Eval : \n",
      "[2021/12/26 19:57:29] root INFO:     dataset : \n",
      "[2021/12/26 19:57:29] root INFO:         data_dir : train_data/table/pubtabnet/val/\n",
      "[2021/12/26 19:57:29] root INFO:         label_file_path : train_data/table/pubtabnet/PubTabNet_2.0.0_val.jsonl\n",
      "[2021/12/26 19:57:29] root INFO:         name : PubTabDataSet\n",
      "[2021/12/26 19:57:29] root INFO:         transforms : \n",
      "[2021/12/26 19:57:29] root INFO:             DecodeImage : \n",
      "[2021/12/26 19:57:29] root INFO:                 channel_first : False\n",
      "[2021/12/26 19:57:29] root INFO:                 img_mode : BGR\n",
      "[2021/12/26 19:57:29] root INFO:             ResizeTableImage : \n",
      "[2021/12/26 19:57:29] root INFO:                 max_len : 488\n",
      "[2021/12/26 19:57:29] root INFO:             TableLabelEncode : None\n",
      "[2021/12/26 19:57:29] root INFO:             NormalizeImage : \n",
      "[2021/12/26 19:57:29] root INFO:                 mean : [0.485, 0.456, 0.406]\n",
      "[2021/12/26 19:57:29] root INFO:                 order : hwc\n",
      "[2021/12/26 19:57:29] root INFO:                 scale : 1./255.\n",
      "[2021/12/26 19:57:29] root INFO:                 std : [0.229, 0.224, 0.225]\n",
      "[2021/12/26 19:57:29] root INFO:             PaddingTableImage : None\n",
      "[2021/12/26 19:57:29] root INFO:             ToCHWImage : None\n",
      "[2021/12/26 19:57:29] root INFO:             KeepKeys : \n",
      "[2021/12/26 19:57:29] root INFO:                 keep_keys : ['image', 'structure', 'bbox_list', 'sp_tokens', 'bbox_list_mask']\n",
      "[2021/12/26 19:57:29] root INFO:     loader : \n",
      "[2021/12/26 19:57:29] root INFO:         batch_size_per_card : 1\n",
      "[2021/12/26 19:57:29] root INFO:         drop_last : False\n",
      "[2021/12/26 19:57:29] root INFO:         num_workers : 1\n",
      "[2021/12/26 19:57:29] root INFO:         shuffle : False\n",
      "[2021/12/26 19:57:29] root INFO: Global : \n",
      "[2021/12/26 19:57:29] root INFO:     cal_metric_during_train : True\n",
      "[2021/12/26 19:57:29] root INFO:     character_dict_path : ppocr/utils/dict/table_structure_dict.txt\n",
      "[2021/12/26 19:57:29] root INFO:     character_type : en\n",
      "[2021/12/26 19:57:29] root INFO:     checkpoints : None\n",
      "[2021/12/26 19:57:29] root INFO:     debug : False\n",
      "[2021/12/26 19:57:29] root INFO:     distributed : False\n",
      "[2021/12/26 19:57:29] root INFO:     epoch_num : 400\n",
      "[2021/12/26 19:57:29] root INFO:     eval_batch_step : [0, 400]\n",
      "[2021/12/26 19:57:29] root INFO:     infer_img : doc/table/table.jpg\n",
      "[2021/12/26 19:57:29] root INFO:     infer_mode : False\n",
      "[2021/12/26 19:57:29] root INFO:     log_smooth_window : 20\n",
      "[2021/12/26 19:57:29] root INFO:     max_cell_num : 500\n",
      "[2021/12/26 19:57:29] root INFO:     max_elem_length : 800\n",
      "[2021/12/26 19:57:29] root INFO:     max_text_length : 100\n",
      "[2021/12/26 19:57:29] root INFO:     pretrained_model : None\n",
      "[2021/12/26 19:57:29] root INFO:     print_batch_step : 1\n",
      "[2021/12/26 19:57:29] root INFO:     process_cut_num : 0\n",
      "[2021/12/26 19:57:29] root INFO:     process_total_num : 0\n",
      "[2021/12/26 19:57:29] root INFO:     save_epoch_step : 3\n",
      "[2021/12/26 19:57:29] root INFO:     save_inference_dir : None\n",
      "[2021/12/26 19:57:29] root INFO:     save_model_dir : ./output/table_mv3/\n",
      "[2021/12/26 19:57:29] root INFO:     use_gpu : False\n",
      "[2021/12/26 19:57:29] root INFO:     use_visualdl : False\n",
      "[2021/12/26 19:57:29] root INFO: Loss : \n",
      "[2021/12/26 19:57:29] root INFO:     loc_weight : 10000.0\n",
      "[2021/12/26 19:57:29] root INFO:     name : TableAttentionLoss\n",
      "[2021/12/26 19:57:29] root INFO:     structure_weight : 100.0\n",
      "[2021/12/26 19:57:29] root INFO: Metric : \n",
      "[2021/12/26 19:57:29] root INFO:     main_indicator : acc\n",
      "[2021/12/26 19:57:29] root INFO:     name : TableMetric\n",
      "[2021/12/26 19:57:29] root INFO: Optimizer : \n",
      "[2021/12/26 19:57:29] root INFO:     beta1 : 0.9\n",
      "[2021/12/26 19:57:29] root INFO:     beta2 : 0.999\n",
      "[2021/12/26 19:57:29] root INFO:     clip_norm : 5.0\n",
      "[2021/12/26 19:57:29] root INFO:     lr : \n",
      "[2021/12/26 19:57:29] root INFO:         learning_rate : 0.001\n",
      "[2021/12/26 19:57:29] root INFO:     name : Adam\n",
      "[2021/12/26 19:57:29] root INFO:     regularizer : \n",
      "[2021/12/26 19:57:29] root INFO:         factor : 0.0\n",
      "[2021/12/26 19:57:29] root INFO:         name : L2\n",
      "[2021/12/26 19:57:29] root INFO: PostProcess : \n",
      "[2021/12/26 19:57:29] root INFO:     name : TableLabelDecode\n",
      "[2021/12/26 19:57:29] root INFO: Train : \n",
      "[2021/12/26 19:57:29] root INFO:     dataset : \n",
      "[2021/12/26 19:57:29] root INFO:         data_dir : train_data/table/pubtabnet/train/\n",
      "[2021/12/26 19:57:29] root INFO:         label_file_path : train_data/table/pubtabnet/PubTabNet_2.0.0_train.jsonl\n",
      "[2021/12/26 19:57:29] root INFO:         name : PubTabDataSet\n",
      "[2021/12/26 19:57:29] root INFO:         transforms : \n",
      "[2021/12/26 19:57:29] root INFO:             DecodeImage : \n",
      "[2021/12/26 19:57:29] root INFO:                 channel_first : False\n",
      "[2021/12/26 19:57:29] root INFO:                 img_mode : BGR\n",
      "[2021/12/26 19:57:29] root INFO:             ResizeTableImage : \n",
      "[2021/12/26 19:57:29] root INFO:                 max_len : 488\n",
      "[2021/12/26 19:57:29] root INFO:             TableLabelEncode : None\n",
      "[2021/12/26 19:57:29] root INFO:             NormalizeImage : \n",
      "[2021/12/26 19:57:29] root INFO:                 mean : [0.485, 0.456, 0.406]\n",
      "[2021/12/26 19:57:29] root INFO:                 order : hwc\n",
      "[2021/12/26 19:57:29] root INFO:                 scale : 1./255.\n",
      "[2021/12/26 19:57:29] root INFO:                 std : [0.229, 0.224, 0.225]\n",
      "[2021/12/26 19:57:29] root INFO:             PaddingTableImage : None\n",
      "[2021/12/26 19:57:29] root INFO:             ToCHWImage : None\n",
      "[2021/12/26 19:57:29] root INFO:             KeepKeys : \n",
      "[2021/12/26 19:57:29] root INFO:                 keep_keys : ['image', 'structure', 'bbox_list', 'sp_tokens', 'bbox_list_mask']\n",
      "[2021/12/26 19:57:29] root INFO:     loader : \n",
      "[2021/12/26 19:57:29] root INFO:         batch_size_per_card : 1\n",
      "[2021/12/26 19:57:29] root INFO:         drop_last : True\n",
      "[2021/12/26 19:57:29] root INFO:         num_workers : 1\n",
      "[2021/12/26 19:57:29] root INFO:         shuffle : True\n",
      "[2021/12/26 19:57:29] root INFO: profiler_options : None\n",
      "[2021/12/26 19:57:29] root INFO: train with paddle 2.2.1 and device CPUPlace\n",
      "[2021/12/26 19:57:29] root INFO: Initialize indexs of datasets:train_data/table/pubtabnet/PubTabNet_2.0.0_train.jsonl\n",
      "[2021/12/26 19:57:29] root INFO: Initialize indexs of datasets:train_data/table/pubtabnet/PubTabNet_2.0.0_val.jsonl\n",
      "[2021/12/26 19:57:29] root INFO: train from scratch\n",
      "[2021/12/26 19:57:29] root INFO: train dataloader has 9115 iters\n",
      "[2021/12/26 19:57:29] root INFO: valid dataloader has 9115 iters\n",
      "[2021/12/26 19:57:29] root INFO: During the training process, after the 0th iteration, an evaluation is run every 400 iterations\n",
      "[2021/12/26 19:57:29] root INFO: Initialize indexs of datasets:train_data/table/pubtabnet/PubTabNet_2.0.0_train.jsonl\n",
      "[2021/12/26 19:57:47] root INFO: epoch: [1/400], iter: 1, lr: 0.001000, loss: 358.711182, structure_loss: 277.904785, loc_loss: 80.806374, acc: 0.000000, reader_cost: 0.05254 s, batch_cost: 17.39120 s, samples: 2, ips: 0.11500\n",
      "[2021/12/26 19:57:55] root INFO: epoch: [1/400], iter: 2, lr: 0.001000, loss: 353.381165, structure_loss: 208.200623, loc_loss: 137.825607, acc: 0.000000, reader_cost: 0.00041 s, batch_cost: 8.65134 s, samples: 1, ips: 0.11559\n",
      "^C\n",
      "main proc 431 exit, kill process group 417\n",
      "main proc 417 exit, kill process group 417\n"
     ]
    }
   ],
   "source": [
    "! python tools/train.py -c configs/table/table_mv3.yml -o Global.use_gpu=False Global.print_batch_step=1 Train.loader.batch_size_per_card=1 Eval.loader.batch_size_per_card=1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.5 Model Evaluation\n",
    "\n",
    "During the training process, two models are saved by default, one is the latest trained model named latest, and the other is the model named best_accuracy with the highest accuracy. Next use the saved model parameters to evaluate the accuracy of the test set:\n",
    "\n",
    "The accuracy evaluation code of the table structure model is located in [PaddleOCR/ppocr/metrics/table_metric.py](https://github.com/PaddlePaddle/PaddleOCR/blob/dygraph/ppocr/metrics/table_metric.py), call tools/eval .py can be used to evaluate the accuracy of the trained model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/setuptools/depends.py:2: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses\n",
      "  import imp\n",
      "[2021/12/26 20:00:08] root INFO: Architecture : \n",
      "[2021/12/26 20:00:08] root INFO:     Backbone : \n",
      "[2021/12/26 20:00:08] root INFO:         model_name : large\n",
      "[2021/12/26 20:00:08] root INFO:         name : MobileNetV3\n",
      "[2021/12/26 20:00:08] root INFO:         scale : 1.0\n",
      "[2021/12/26 20:00:08] root INFO:     Head : \n",
      "[2021/12/26 20:00:08] root INFO:         hidden_size : 256\n",
      "[2021/12/26 20:00:08] root INFO:         l2_decay : 1e-05\n",
      "[2021/12/26 20:00:08] root INFO:         loc_type : 2\n",
      "[2021/12/26 20:00:08] root INFO:         max_cell_num : 500\n",
      "[2021/12/26 20:00:08] root INFO:         max_elem_length : 800\n",
      "[2021/12/26 20:00:08] root INFO:         max_text_length : 100\n",
      "[2021/12/26 20:00:08] root INFO:         name : TableAttentionHead\n",
      "[2021/12/26 20:00:08] root INFO:     algorithm : TableAttn\n",
      "[2021/12/26 20:00:08] root INFO:     model_type : table\n",
      "[2021/12/26 20:00:08] root INFO: Eval : \n",
      "[2021/12/26 20:00:08] root INFO:     dataset : \n",
      "[2021/12/26 20:00:08] root INFO:         data_dir : train_data/table/pubtabnet/val/\n",
      "[2021/12/26 20:00:08] root INFO:         label_file_path : train_data/table/pubtabnet/PubTabNet_2.0.0_val.jsonl\n",
      "[2021/12/26 20:00:08] root INFO:         name : PubTabDataSet\n",
      "[2021/12/26 20:00:08] root INFO:         transforms : \n",
      "[2021/12/26 20:00:08] root INFO:             DecodeImage : \n",
      "[2021/12/26 20:00:08] root INFO:                 channel_first : False\n",
      "[2021/12/26 20:00:08] root INFO:                 img_mode : BGR\n",
      "[2021/12/26 20:00:08] root INFO:             ResizeTableImage : \n",
      "[2021/12/26 20:00:08] root INFO:                 max_len : 488\n",
      "[2021/12/26 20:00:08] root INFO:             TableLabelEncode : None\n",
      "[2021/12/26 20:00:08] root INFO:             NormalizeImage : \n",
      "[2021/12/26 20:00:08] root INFO:                 mean : [0.485, 0.456, 0.406]\n",
      "[2021/12/26 20:00:08] root INFO:                 order : hwc\n",
      "[2021/12/26 20:00:08] root INFO:                 scale : 1./255.\n",
      "[2021/12/26 20:00:08] root INFO:                 std : [0.229, 0.224, 0.225]\n",
      "[2021/12/26 20:00:08] root INFO:             PaddingTableImage : None\n",
      "[2021/12/26 20:00:08] root INFO:             ToCHWImage : None\n",
      "[2021/12/26 20:00:08] root INFO:             KeepKeys : \n",
      "[2021/12/26 20:00:08] root INFO:                 keep_keys : ['image', 'structure', 'bbox_list', 'sp_tokens', 'bbox_list_mask']\n",
      "[2021/12/26 20:00:08] root INFO:     loader : \n",
      "[2021/12/26 20:00:08] root INFO:         batch_size_per_card : 1\n",
      "[2021/12/26 20:00:08] root INFO:         drop_last : False\n",
      "[2021/12/26 20:00:08] root INFO:         num_workers : 1\n",
      "[2021/12/26 20:00:08] root INFO:         shuffle : False\n",
      "[2021/12/26 20:00:08] root INFO: Global : \n",
      "[2021/12/26 20:00:08] root INFO:     cal_metric_during_train : True\n",
      "[2021/12/26 20:00:08] root INFO:     character_dict_path : ppocr/utils/dict/table_structure_dict.txt\n",
      "[2021/12/26 20:00:08] root INFO:     character_type : en\n",
      "[2021/12/26 20:00:08] root INFO:     checkpoints : /home/aistudio/PaddleOCR/pre_train/en_ppocr_mobile_v2.0_table_structure_train/best_accuracy\n",
      "[2021/12/26 20:00:08] root INFO:     debug : False\n",
      "[2021/12/26 20:00:08] root INFO:     distributed : False\n",
      "[2021/12/26 20:00:08] root INFO:     epoch_num : 400\n",
      "[2021/12/26 20:00:08] root INFO:     eval_batch_step : [0, 400]\n",
      "[2021/12/26 20:00:08] root INFO:     infer_img : doc/table/table.jpg\n",
      "[2021/12/26 20:00:08] root INFO:     infer_mode : False\n",
      "[2021/12/26 20:00:08] root INFO:     log_smooth_window : 20\n",
      "[2021/12/26 20:00:08] root INFO:     max_cell_num : 500\n",
      "[2021/12/26 20:00:08] root INFO:     max_elem_length : 800\n",
      "[2021/12/26 20:00:08] root INFO:     max_text_length : 100\n",
      "[2021/12/26 20:00:08] root INFO:     pretrained_model : None\n",
      "[2021/12/26 20:00:08] root INFO:     print_batch_step : 5\n",
      "[2021/12/26 20:00:08] root INFO:     process_cut_num : 0\n",
      "[2021/12/26 20:00:08] root INFO:     process_total_num : 0\n",
      "[2021/12/26 20:00:08] root INFO:     save_epoch_step : 3\n",
      "[2021/12/26 20:00:08] root INFO:     save_inference_dir : None\n",
      "[2021/12/26 20:00:08] root INFO:     save_model_dir : ./output/table_mv3/\n",
      "[2021/12/26 20:00:08] root INFO:     use_gpu : False\n",
      "[2021/12/26 20:00:08] root INFO:     use_visualdl : False\n",
      "[2021/12/26 20:00:08] root INFO: Loss : \n",
      "[2021/12/26 20:00:08] root INFO:     loc_weight : 10000.0\n",
      "[2021/12/26 20:00:08] root INFO:     name : TableAttentionLoss\n",
      "[2021/12/26 20:00:08] root INFO:     structure_weight : 100.0\n",
      "[2021/12/26 20:00:08] root INFO: Metric : \n",
      "[2021/12/26 20:00:08] root INFO:     main_indicator : acc\n",
      "[2021/12/26 20:00:08] root INFO:     name : TableMetric\n",
      "[2021/12/26 20:00:08] root INFO: Optimizer : \n",
      "[2021/12/26 20:00:08] root INFO:     beta1 : 0.9\n",
      "[2021/12/26 20:00:08] root INFO:     beta2 : 0.999\n",
      "[2021/12/26 20:00:08] root INFO:     clip_norm : 5.0\n",
      "[2021/12/26 20:00:08] root INFO:     lr : \n",
      "[2021/12/26 20:00:08] root INFO:         learning_rate : 0.001\n",
      "[2021/12/26 20:00:08] root INFO:     name : Adam\n",
      "[2021/12/26 20:00:08] root INFO:     regularizer : \n",
      "[2021/12/26 20:00:08] root INFO:         factor : 0.0\n",
      "[2021/12/26 20:00:08] root INFO:         name : L2\n",
      "[2021/12/26 20:00:08] root INFO: PostProcess : \n",
      "[2021/12/26 20:00:08] root INFO:     name : TableLabelDecode\n",
      "[2021/12/26 20:00:08] root INFO: Train : \n",
      "[2021/12/26 20:00:08] root INFO:     dataset : \n",
      "[2021/12/26 20:00:08] root INFO:         data_dir : train_data/table/pubtabnet/train/\n",
      "[2021/12/26 20:00:08] root INFO:         label_file_path : train_data/table/pubtabnet/PubTabNet_2.0.0_train.jsonl\n",
      "[2021/12/26 20:00:08] root INFO:         name : PubTabDataSet\n",
      "[2021/12/26 20:00:08] root INFO:         transforms : \n",
      "[2021/12/26 20:00:08] root INFO:             DecodeImage : \n",
      "[2021/12/26 20:00:08] root INFO:                 channel_first : False\n",
      "[2021/12/26 20:00:08] root INFO:                 img_mode : BGR\n",
      "[2021/12/26 20:00:08] root INFO:             ResizeTableImage : \n",
      "[2021/12/26 20:00:08] root INFO:                 max_len : 488\n",
      "[2021/12/26 20:00:08] root INFO:             TableLabelEncode : None\n",
      "[2021/12/26 20:00:08] root INFO:             NormalizeImage : \n",
      "[2021/12/26 20:00:08] root INFO:                 mean : [0.485, 0.456, 0.406]\n",
      "[2021/12/26 20:00:08] root INFO:                 order : hwc\n",
      "[2021/12/26 20:00:08] root INFO:                 scale : 1./255.\n",
      "[2021/12/26 20:00:08] root INFO:                 std : [0.229, 0.224, 0.225]\n",
      "[2021/12/26 20:00:08] root INFO:             PaddingTableImage : None\n",
      "[2021/12/26 20:00:08] root INFO:             ToCHWImage : None\n",
      "[2021/12/26 20:00:08] root INFO:             KeepKeys : \n",
      "[2021/12/26 20:00:08] root INFO:                 keep_keys : ['image', 'structure', 'bbox_list', 'sp_tokens', 'bbox_list_mask']\n",
      "[2021/12/26 20:00:08] root INFO:     loader : \n",
      "[2021/12/26 20:00:08] root INFO:         batch_size_per_card : 32\n",
      "[2021/12/26 20:00:08] root INFO:         drop_last : True\n",
      "[2021/12/26 20:00:08] root INFO:         num_workers : 1\n",
      "[2021/12/26 20:00:08] root INFO:         shuffle : True\n",
      "[2021/12/26 20:00:08] root INFO: profiler_options : None\n",
      "[2021/12/26 20:00:08] root INFO: train with paddle 2.2.1 and device CPUPlace\n",
      "[2021/12/26 20:00:08] root INFO: Initialize indexs of datasets:train_data/table/pubtabnet/PubTabNet_2.0.0_val.jsonl\n",
      "[2021/12/26 20:00:08] root INFO: resume from /home/aistudio/PaddleOCR/pre_train/en_ppocr_mobile_v2.0_table_structure_train/best_accuracy\n",
      "[2021/12/26 20:00:08] root INFO: metric in ckpt ***************\n",
      "[2021/12/26 20:00:08] root INFO: acc:0.7380142622051563\n",
      "[2021/12/26 20:00:08] root INFO: fps:8.360272547972942\n",
      "[2021/12/26 20:00:08] root INFO: best_epoch:7\n",
      "[2021/12/26 20:00:08] root INFO: start_epoch:8\n",
      "eval model::   0%|                           | 2/9115 [00:07<8:55:26,  3.53s/it]^C\n",
      "main proc 602 exit, kill process group 576\n",
      "main proc 576 exit, kill process group 576\n"
     ]
    }
   ],
   "source": [
    "!python tools/eval.py -c configs/table/table_mv3.yml -o Global.checkpoints=/home/aistudio/PaddleOCR/pre_train/en_ppocr_mobile_v2.0_table_structure_train/best_accuracy Global.use_gpu=False Eval.loader.batch_size_per_card=1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.6 Model Prediction\n",
    "\n",
    "After training the model, you can also use the saved model to perform model inference on a single picture or an image in a folder, and observe the prediction effect of the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/setuptools/depends.py:2: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses\n",
      "  import imp\n",
      "[2021/12/26 20:00:22] root INFO: Architecture : \n",
      "[2021/12/26 20:00:22] root INFO:     Backbone : \n",
      "[2021/12/26 20:00:22] root INFO:         model_name : large\n",
      "[2021/12/26 20:00:22] root INFO:         name : MobileNetV3\n",
      "[2021/12/26 20:00:22] root INFO:         scale : 1.0\n",
      "[2021/12/26 20:00:22] root INFO:     Head : \n",
      "[2021/12/26 20:00:22] root INFO:         hidden_size : 256\n",
      "[2021/12/26 20:00:22] root INFO:         l2_decay : 1e-05\n",
      "[2021/12/26 20:00:22] root INFO:         loc_type : 2\n",
      "[2021/12/26 20:00:22] root INFO:         max_cell_num : 500\n",
      "[2021/12/26 20:00:22] root INFO:         max_elem_length : 800\n",
      "[2021/12/26 20:00:22] root INFO:         max_text_length : 100\n",
      "[2021/12/26 20:00:22] root INFO:         name : TableAttentionHead\n",
      "[2021/12/26 20:00:22] root INFO:     algorithm : TableAttn\n",
      "[2021/12/26 20:00:22] root INFO:     model_type : table\n",
      "[2021/12/26 20:00:22] root INFO: Eval : \n",
      "[2021/12/26 20:00:22] root INFO:     dataset : \n",
      "[2021/12/26 20:00:22] root INFO:         data_dir : train_data/table/pubtabnet/val/\n",
      "[2021/12/26 20:00:22] root INFO:         label_file_path : train_data/table/pubtabnet/PubTabNet_2.0.0_val.jsonl\n",
      "[2021/12/26 20:00:22] root INFO:         name : PubTabDataSet\n",
      "[2021/12/26 20:00:22] root INFO:         transforms : \n",
      "[2021/12/26 20:00:22] root INFO:             DecodeImage : \n",
      "[2021/12/26 20:00:22] root INFO:                 channel_first : False\n",
      "[2021/12/26 20:00:22] root INFO:                 img_mode : BGR\n",
      "[2021/12/26 20:00:22] root INFO:             ResizeTableImage : \n",
      "[2021/12/26 20:00:22] root INFO:                 max_len : 488\n",
      "[2021/12/26 20:00:22] root INFO:             TableLabelEncode : None\n",
      "[2021/12/26 20:00:22] root INFO:             NormalizeImage : \n",
      "[2021/12/26 20:00:22] root INFO:                 mean : [0.485, 0.456, 0.406]\n",
      "[2021/12/26 20:00:22] root INFO:                 order : hwc\n",
      "[2021/12/26 20:00:22] root INFO:                 scale : 1./255.\n",
      "[2021/12/26 20:00:22] root INFO:                 std : [0.229, 0.224, 0.225]\n",
      "[2021/12/26 20:00:22] root INFO:             PaddingTableImage : None\n",
      "[2021/12/26 20:00:22] root INFO:             ToCHWImage : None\n",
      "[2021/12/26 20:00:22] root INFO:             KeepKeys : \n",
      "[2021/12/26 20:00:22] root INFO:                 keep_keys : ['image', 'structure', 'bbox_list', 'sp_tokens', 'bbox_list_mask']\n",
      "[2021/12/26 20:00:22] root INFO:     loader : \n",
      "[2021/12/26 20:00:22] root INFO:         batch_size_per_card : 16\n",
      "[2021/12/26 20:00:22] root INFO:         drop_last : False\n",
      "[2021/12/26 20:00:22] root INFO:         num_workers : 1\n",
      "[2021/12/26 20:00:22] root INFO:         shuffle : False\n",
      "[2021/12/26 20:00:22] root INFO: Global : \n",
      "[2021/12/26 20:00:22] root INFO:     cal_metric_during_train : True\n",
      "[2021/12/26 20:00:22] root INFO:     character_dict_path : ppocr/utils/dict/table_structure_dict.txt\n",
      "[2021/12/26 20:00:22] root INFO:     character_type : en\n",
      "[2021/12/26 20:00:22] root INFO:     checkpoints : /home/aistudio/PaddleOCR/pre_train/en_ppocr_mobile_v2.0_table_structure_train/best_accuracy\n",
      "[2021/12/26 20:00:22] root INFO:     debug : False\n",
      "[2021/12/26 20:00:22] root INFO:     distributed : False\n",
      "[2021/12/26 20:00:22] root INFO:     epoch_num : 400\n",
      "[2021/12/26 20:00:22] root INFO:     eval_batch_step : [0, 400]\n",
      "[2021/12/26 20:00:22] root INFO:     infer_img : /home/aistudio/1.jpg\n",
      "[2021/12/26 20:00:22] root INFO:     infer_mode : False\n",
      "[2021/12/26 20:00:22] root INFO:     log_smooth_window : 20\n",
      "[2021/12/26 20:00:22] root INFO:     max_cell_num : 500\n",
      "[2021/12/26 20:00:22] root INFO:     max_elem_length : 800\n",
      "[2021/12/26 20:00:22] root INFO:     max_text_length : 100\n",
      "[2021/12/26 20:00:22] root INFO:     pretrained_model : None\n",
      "[2021/12/26 20:00:22] root INFO:     print_batch_step : 5\n",
      "[2021/12/26 20:00:22] root INFO:     process_cut_num : 0\n",
      "[2021/12/26 20:00:22] root INFO:     process_total_num : 0\n",
      "[2021/12/26 20:00:22] root INFO:     save_epoch_step : 3\n",
      "[2021/12/26 20:00:22] root INFO:     save_inference_dir : None\n",
      "[2021/12/26 20:00:22] root INFO:     save_model_dir : ./output/table_mv3/\n",
      "[2021/12/26 20:00:22] root INFO:     use_gpu : False\n",
      "[2021/12/26 20:00:22] root INFO:     use_visualdl : False\n",
      "[2021/12/26 20:00:22] root INFO: Loss : \n",
      "[2021/12/26 20:00:22] root INFO:     loc_weight : 10000.0\n",
      "[2021/12/26 20:00:22] root INFO:     name : TableAttentionLoss\n",
      "[2021/12/26 20:00:22] root INFO:     structure_weight : 100.0\n",
      "[2021/12/26 20:00:22] root INFO: Metric : \n",
      "[2021/12/26 20:00:22] root INFO:     main_indicator : acc\n",
      "[2021/12/26 20:00:22] root INFO:     name : TableMetric\n",
      "[2021/12/26 20:00:22] root INFO: Optimizer : \n",
      "[2021/12/26 20:00:22] root INFO:     beta1 : 0.9\n",
      "[2021/12/26 20:00:22] root INFO:     beta2 : 0.999\n",
      "[2021/12/26 20:00:22] root INFO:     clip_norm : 5.0\n",
      "[2021/12/26 20:00:22] root INFO:     lr : \n",
      "[2021/12/26 20:00:22] root INFO:         learning_rate : 0.001\n",
      "[2021/12/26 20:00:22] root INFO:     name : Adam\n",
      "[2021/12/26 20:00:22] root INFO:     regularizer : \n",
      "[2021/12/26 20:00:22] root INFO:         factor : 0.0\n",
      "[2021/12/26 20:00:22] root INFO:         name : L2\n",
      "[2021/12/26 20:00:22] root INFO: PostProcess : \n",
      "[2021/12/26 20:00:22] root INFO:     name : TableLabelDecode\n",
      "[2021/12/26 20:00:22] root INFO: Train : \n",
      "[2021/12/26 20:00:22] root INFO:     dataset : \n",
      "[2021/12/26 20:00:22] root INFO:         data_dir : train_data/table/pubtabnet/train/\n",
      "[2021/12/26 20:00:22] root INFO:         label_file_path : train_data/table/pubtabnet/PubTabNet_2.0.0_train.jsonl\n",
      "[2021/12/26 20:00:22] root INFO:         name : PubTabDataSet\n",
      "[2021/12/26 20:00:22] root INFO:         transforms : \n",
      "[2021/12/26 20:00:22] root INFO:             DecodeImage : \n",
      "[2021/12/26 20:00:22] root INFO:                 channel_first : False\n",
      "[2021/12/26 20:00:22] root INFO:                 img_mode : BGR\n",
      "[2021/12/26 20:00:22] root INFO:             ResizeTableImage : \n",
      "[2021/12/26 20:00:22] root INFO:                 max_len : 488\n",
      "[2021/12/26 20:00:22] root INFO:             TableLabelEncode : None\n",
      "[2021/12/26 20:00:22] root INFO:             NormalizeImage : \n",
      "[2021/12/26 20:00:22] root INFO:                 mean : [0.485, 0.456, 0.406]\n",
      "[2021/12/26 20:00:22] root INFO:                 order : hwc\n",
      "[2021/12/26 20:00:22] root INFO:                 scale : 1./255.\n",
      "[2021/12/26 20:00:22] root INFO:                 std : [0.229, 0.224, 0.225]\n",
      "[2021/12/26 20:00:22] root INFO:             PaddingTableImage : None\n",
      "[2021/12/26 20:00:22] root INFO:             ToCHWImage : None\n",
      "[2021/12/26 20:00:22] root INFO:             KeepKeys : \n",
      "[2021/12/26 20:00:22] root INFO:                 keep_keys : ['image', 'structure', 'bbox_list', 'sp_tokens', 'bbox_list_mask']\n",
      "[2021/12/26 20:00:22] root INFO:     loader : \n",
      "[2021/12/26 20:00:22] root INFO:         batch_size_per_card : 32\n",
      "[2021/12/26 20:00:22] root INFO:         drop_last : True\n",
      "[2021/12/26 20:00:22] root INFO:         num_workers : 1\n",
      "[2021/12/26 20:00:22] root INFO:         shuffle : True\n",
      "[2021/12/26 20:00:22] root INFO: profiler_options : None\n",
      "[2021/12/26 20:00:22] root INFO: train with paddle 2.2.1 and device CPUPlace\n",
      "[2021/12/26 20:00:22] root INFO: resume from /home/aistudio/PaddleOCR/pre_train/en_ppocr_mobile_v2.0_table_structure_train/best_accuracy\n",
      "[2021/12/26 20:00:22] root INFO: infer_img: /home/aistudio/1.jpg\n",
      "[2021/12/26 20:00:26] root INFO: result: ['<thead><tr><td></td><td></td><td></td><td></td><td></td></tr></thead><tbody><tr><td></td><td></td><td></td><td></td><td></td></tr><tr><td></td><td></td><td></td><td></td><td></td></tr><tr><td></td><td></td><td></td><td></td><td></td></tr><tr><td></td><td></td><td></td><td></td><td></td></tr><tr><td></td><td></td><td></td><td></td><td></td></tr><tr><td></td><td></td><td></td><td></td><td></td></tr><tr><td></td><td></td><td></td><td></td><td></td></tr><tr><td></td><td></td><td></td><td></td><td></td></tr><tr><td></td><td></td><td></td><td></td><td></td></tr><tr><td></td><td></td><td></td><td></td><td></td></tr><tr><td></td><td></td><td></td><td></td><td></td></tr><tr><td></td><td></td><td></td><td></td><td></td></tr><tr><td></td><td></td><td></td><td></td><td></td></tr></tbody>'], [[32, 9, 104, 40], [232, 8, 307, 41], [429, 7, 500, 44], [559, 8, 656, 44], [715, 7, 780, 44], [37, 45, 99, 73], [190, 44, 342, 74], [432, 45, 502, 74], [565, 44, 655, 73], [712, 46, 777, 74], [30, 81, 101, 109], [202, 80, 337, 110], [433, 81, 503, 111], [578, 83, 638, 110], [698, 82, 790, 110], [31, 119, 104, 148], [197, 116, 347, 147], [443, 117, 492, 148], [572, 118, 643, 147], [698, 118, 797, 147], [35, 154, 101, 183], [199, 152, 342, 184], [436, 154, 501, 184], [558, 155, 670, 184], [701, 153, 801, 183], [40, 188, 93, 217], [217, 187, 314, 219], [417, 187, 516, 218], [556, 187, 667, 217], [716, 188, 772, 216], [48, 227, 98, 255], [223, 224, 313, 256], [429, 226, 500, 256], [558, 226, 667, 256], [722, 225, 772, 254], [47, 262, 99, 291], [217, 260, 313, 293], [439, 261, 506, 293], [557, 260, 678, 292], [722, 261, 777, 290], [36, 295, 95, 324], [210, 296, 317, 326], [443, 296, 499, 326], [547, 296, 681, 326], [701, 300, 767, 328], [42, 332, 99, 361], [191, 330, 350, 360], [451, 331, 493, 361], [557, 331, 683, 361], [717, 335, 785, 362], [45, 369, 97, 396], [186, 367, 355, 400], [444, 369, 504, 398], [581, 369, 640, 397], [723, 368, 773, 396], [37, 404, 95, 431], [192, 404, 351, 433], [438, 404, 511, 432], [560, 405, 658, 432], [723, 404, 775, 431], [46, 444, 104, 469], [188, 441, 346, 470], [444, 444, 496, 471], [544, 445, 681, 471], [721, 444, 773, 470], [35, 478, 104, 503], [190, 475, 345, 505], [436, 477, 504, 504], [559, 476, 662, 505], [712, 477, 778, 504]]\n",
      "[2021/12/26 20:00:26] root INFO: success!\n"
     ]
    }
   ],
   "source": [
    "! python tools/infer_table.py -c configs/table/table_mv3.yml -o Global.checkpoints=/home/aistudio/PaddleOCR/pre_train/en_ppocr_mobile_v2.0_table_structure_train/best_accuracy Global.infer_img=/home/aistudio/1.jpg Global.use_gpu=False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4 Summary\n",
    "\n",
    "This section introduces the principle of PP-Structure table recognition algorithm in PaddleOCR, and introduces the process of table structure model from data processing to completion of training."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5 Assignment\n",
    "\n",
    "[https://aistudio.baidu.com/aistudio/education/objective/28711](https://aistudio.baidu.com/aistudio/education/objective/28711)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "py35-paddle1.2.0"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
