"vscode:/vscode.git/clone" did not exist on "ed740a2504f18663f4cb0b326c1c48fbc54c2cbd"
README.md 6.71 KB
Newer Older
MissPenguin's avatar
MissPenguin committed
1
# Table Recognition
WenmuZhou's avatar
WenmuZhou committed
2
3

## 1. pipeline
MissPenguin's avatar
MissPenguin committed
4
The table recognition mainly contains three models
WenmuZhou's avatar
WenmuZhou committed
5
6
7
8
1. Single line text detection-DB
2. Single line text recognition-CRNN
3. Table structure and cell coordinate prediction-RARE

MissPenguin's avatar
MissPenguin committed
9
The table recognition flow chart is as follows
WenmuZhou's avatar
WenmuZhou committed
10

WenmuZhou's avatar
WenmuZhou committed
11
![tableocr_pipeline](../../doc/table/tableocr_pipeline_en.jpg)
WenmuZhou's avatar
WenmuZhou committed
12
13
14
15
16
17
18
19

1. The coordinates of single-line text is detected by DB model, and then sends it to the recognition model to get the recognition result.
2. The table structure and cell coordinates is predicted by RARE model.
3. The recognition result of the cell is combined by the coordinates, recognition result of the single line and the coordinates of the cell.
4. The cell recognition result and the table structure together construct the html string of the table.

## 2. How to use

WenmuZhou's avatar
WenmuZhou committed
20
### 2.1 quick start
WenmuZhou's avatar
WenmuZhou committed
21

WenmuZhou's avatar
WenmuZhou committed
22
23
24
25
26
```python
cd PaddleOCR/ppstructure

# download model
mkdir inference && cd inference
27
28
29
30
31
# Download the detection model of the ultra-lightweight table English OCR model and unzip it
wget https://paddleocr.bj.bcebos.com/dygraph_v2.0/table/en_ppocr_mobile_v2.0_table_det_infer.tar && tar xf en_ppocr_mobile_v2.0_table_det_infer.tar
# Download the recognition model of the ultra-lightweight table English OCR model and unzip it
wget https://paddleocr.bj.bcebos.com/dygraph_v2.0/table/en_ppocr_mobile_v2.0_table_rec_infer.tar && tar xf en_ppocr_mobile_v2.0_table_rec_infer.tar
# Download the ultra-lightweight English table inch model and unzip it
WenmuZhou's avatar
WenmuZhou committed
32
33
wget https://paddleocr.bj.bcebos.com/dygraph_v2.0/table/en_ppocr_mobile_v2.0_table_structure_infer.tar && tar xf en_ppocr_mobile_v2.0_table_structure_infer.tar
cd ..
34
35
# run
python3 table/predict_table.py --det_model_dir=inference/en_ppocr_mobile_v2.0_table_det_infer --rec_model_dir=inference/en_ppocr_mobile_v2.0_table_rec_infer --table_model_dir=inference/en_ppocr_mobile_v2.0_table_structure_infer --image_dir=../doc/table/table.jpg --rec_char_dict_path=../ppocr/utils/ppocr_keys_v1.txt --table_char_dict_path=../ppocr/utils/dict/table_structure_dict.txt --rec_char_type=ch --det_limit_side_len=736 --det_limit_type=min --output ../output/table
WenmuZhou's avatar
WenmuZhou committed
36
```
37
38
Note: The above model is trained on the PubLayNet dataset and only supports English scanning scenarios. If you need to identify other scenarios, you need to train the model yourself and replace the three fields `det_model_dir`, `rec_model_dir`, `table_model_dir`.

WenmuZhou's avatar
WenmuZhou committed
39
40
41
After running, the excel sheet of each picture will be saved in the directory specified by the output field

### 2.2 Train
WenmuZhou's avatar
WenmuZhou committed
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69

In this chapter, we only introduce the training of the table structure model, For model training of [text detection](../../doc/doc_en/detection_en.md) and [text recognition](../../doc/doc_en/recognition_en.md), please refer to the corresponding documents

#### data preparation  
The training data uses public data set [PubTabNet](https://arxiv.org/abs/1911.10683 ), Can be downloaded from the official [website](https://github.com/ibm-aur-nlp/PubTabNet) 。The PubTabNet data set contains about 500,000 images, as well as annotations in html format。

#### Start training  
*If you are installing the cpu version of paddle, please modify the `use_gpu` field in the configuration file to false*
```shell
# single GPU training
python3 tools/train.py -c configs/table/table_mv3.yml
# multi-GPU training
# Set the GPU ID used by the '--gpus' parameter.
python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/table/table_mv3.yml
```

In the above instruction, use `-c` to select the training to use the `configs/table/table_mv3.yml` configuration file.
For a detailed explanation of the configuration file, please refer to [config](../../doc/doc_en/config_en.md).

#### load trained model and continue training

If you expect to load trained model and continue the training again, you can specify the parameter `Global.checkpoints` as the model path to be loaded.

```shell
python3 tools/train.py -c configs/table/table_mv3.yml -o Global.checkpoints=./your/trained/model
```

**Note**: The priority of `Global.checkpoints` is higher than that of `Global.pretrain_weights`, that is, when two parameters are specified at the same time, the model specified by `Global.checkpoints` will be loaded first. If the model path specified by `Global.checkpoints` is wrong, the one specified by `Global.pretrain_weights` will be loaded.
WenmuZhou's avatar
WenmuZhou committed
70

WenmuZhou's avatar
WenmuZhou committed
71
### 2.3 Eval
WenmuZhou's avatar
WenmuZhou committed
72

WenmuZhou's avatar
WenmuZhou committed
73
The table uses [TEDS(Tree-Edit-Distance-based Similarity)](https://github.com/ibm-aur-nlp/PubTabNet/tree/master/src)) as the evaluation metric of the model. Before the model evaluation, the three models in the pipeline need to be exported as inference models (we have provided them), and the gt for evaluation needs to be prepared. Examples of gt are as follows:
WenmuZhou's avatar
WenmuZhou committed
74
```json
WenmuZhou's avatar
WenmuZhou committed
75
76
77
78
79
{"PMC4289340_004_00.png": [
  ["<html>", "<body>", "<table>", "<thead>", "<tr>", "<td>", "</td>", "<td>", "</td>", "<td>", "</td>", "</tr>", "</thead>", "<tbody>", "<tr>", "<td>", "</td>", "<td>", "</td>", "<td>", "</td>", "</tr>",  "</tbody>", "</table>", "</body>", "</html>"], 
  [[1, 4, 29, 13], [137, 4, 161, 13], [215, 4, 236, 13], [1, 17, 30, 27], [137, 17, 147, 27], [215, 17, 225, 27]], 
  [["<b>", "F", "e", "a", "t", "u", "r", "e", "</b>"], ["<b>", "G", "b", "3", " ", "+", "</b>"], ["<b>", "G", "b", "3", " ", "-", "</b>"], ["<b>", "P", "a", "t", "i", "e", "n", "t", "s", "</b>"], ["6", "2"], ["4", "5"]]
]}
WenmuZhou's avatar
WenmuZhou committed
80
81
82
83
84
85
86
87
```
In gt json, the key is the image name, the value is the corresponding gt, and gt is a list composed of four items, and each item is
1. HTML string list of table structure
2. The coordinates of each cell (not including the empty text in the cell)
3. The text information in each cell (not including the empty text in the cell)

Use the following command to evaluate. After the evaluation is completed, the teds indicator will be output.
```python
WenmuZhou's avatar
WenmuZhou committed
88
cd PaddleOCR/ppstructure
WenmuZhou's avatar
WenmuZhou committed
89
python3 table/eval_table.py --det_model_dir=path/to/det_model_dir --rec_model_dir=path/to/rec_model_dir --table_model_dir=path/to/table_model_dir --image_dir=../doc/table/1.png --rec_char_dict_path=../ppocr/utils/dict/table_dict.txt --table_char_dict_path=../ppocr/utils/dict/table_structure_dict.txt --rec_char_type=EN --det_limit_side_len=736 --det_limit_type=min --gt_path=path/to/gt.json
WenmuZhou's avatar
WenmuZhou committed
90
91
```

WenmuZhou's avatar
WenmuZhou committed
92
93
94
95
If the PubLatNet eval dataset is used, it will be output
```bash
teds: 94.85
```
WenmuZhou's avatar
WenmuZhou committed
96

WenmuZhou's avatar
WenmuZhou committed
97
### 2.4 Inference
WenmuZhou's avatar
WenmuZhou committed
98
99

```python
WenmuZhou's avatar
WenmuZhou committed
100
cd PaddleOCR/ppstructure
WenmuZhou's avatar
WenmuZhou committed
101
python3 table/predict_table.py --det_model_dir=path/to/det_model_dir --rec_model_dir=path/to/rec_model_dir --table_model_dir=path/to/table_model_dir --image_dir=../doc/table/1.png --rec_char_dict_path=../ppocr/utils/dict/table_dict.txt --table_char_dict_path=../ppocr/utils/dict/table_structure_dict.txt --rec_char_type=EN --det_limit_side_len=736 --det_limit_type=min --output ../output/table
WenmuZhou's avatar
WenmuZhou committed
102
```
MissPenguin's avatar
MissPenguin committed
103
After running, the excel sheet of each picture will be saved in the directory specified by the output field