README_en.md 14.8 KB
Newer Older
1
2
English | [简体中文](README.md)

3
4
5
6
## Introduction
PaddleOCR aims to create a rich, leading, and practical OCR tools that help users train better models and apply them into practice.

**Recent updates**
7
- 2020.6.8 Add [dataset](./doc/doc_en/datasets_en.md) and keep updating
8
9
10
11
- 2020.6.5 Support exporting `attention` model to `inference_model`
- 2020.6.5 Support separate prediction and recognition, output result score
- 2020.5.30 Provide ultra-lightweight Chinese OCR online experience
- 2020.5.30 Model prediction and training supported on Windows system
12
- [more](./doc/doc_en/update_en.md)
xxxpsyduck's avatar
xxxpsyduck committed
13

14
15
16
17
18
19
## Features
- Ultra-lightweight Chinese OCR model, total model size is only 8.6M
    - Single model supports Chinese and English numbers combination recognition, vertical text recognition, long text recognition
    - Detection model DB (4.1M) + recognition model CRNN (4.5M)
- Various text detection algorithms: EAST, DB
- Various text recognition algorithms: Rosetta, CRNN, STAR-Net, RARE
xxxpsyduck's avatar
xxxpsyduck committed
20

21
### Supported Chinese models list:
xxxpsyduck's avatar
xxxpsyduck committed
22

23
|Model Name|Description |Detection Model link|Recognition Model link|
xxxpsyduck's avatar
xxxpsyduck committed
24
|-|-|-|-|
25
26
|chinese_db_crnn_mobile|Ultra-lightweight Chinese OCR model|[inference model](https://paddleocr.bj.bcebos.com/ch_models/ch_det_mv3_db_infer.tar) & [pre-trained model](https://paddleocr.bj.bcebos.com/ch_models/ch_det_mv3_db.tar)|[inference model](https://paddleocr.bj.bcebos.com/ch_models/ch_rec_mv3_crnn_infer.tar) & [pre-trained model](https://paddleocr.bj.bcebos.com/ch_models/ch_rec_mv3_crnn.tar)|
|chinese_db_crnn_server|General Chinese OCR model|[inference model](https://paddleocr.bj.bcebos.com/ch_models/ch_det_r50_vd_db_infer.tar) & [pre-trained model](https://paddleocr.bj.bcebos.com/ch_models/ch_det_r50_vd_db.tar)|[inference model](https://paddleocr.bj.bcebos.com/ch_models/ch_rec_r34_vd_crnn_infer.tar) & [pre-trained model](https://paddleocr.bj.bcebos.com/ch_models/ch_rec_r34_vd_crnn.tar)|
xxxpsyduck's avatar
xxxpsyduck committed
27
28


29
For testing our Chinese OCR online:https://www.paddlepaddle.org.cn/hub/scene/ocr
xxxpsyduck's avatar
xxxpsyduck committed
30

31
**You can also quickly experience the Ultra-lightweight Chinese OCR and General Chinese OCR models as follows:**
xxxpsyduck's avatar
xxxpsyduck committed
32

33
## **Ultra-lightweight Chinese OCR and General Chinese OCR inference**
xxxpsyduck's avatar
xxxpsyduck committed
34

35
![](doc/imgs_results/11.jpg)
xxxpsyduck's avatar
xxxpsyduck committed
36

37
The picture above is the result of our Ultra-lightweight Chinese OCR model. For more testing results, please see the end of the article [Ultra-lightweight Chinese OCR results](#Ultra-lightweight-Chinese-OCR-results) and [General Chinese OCR results](#General-Chinese-OCR-results).
xxxpsyduck's avatar
xxxpsyduck committed
38

39
#### 1. Environment configuration
xxxpsyduck's avatar
xxxpsyduck committed
40

41
Please see [Quick installation](./doc/doc_en/installation_en.md)
xxxpsyduck's avatar
xxxpsyduck committed
42

43
#### 2. Download inference models
xxxpsyduck's avatar
xxxpsyduck committed
44

45
46
#### (1) Download Ultra-lightweight Chinese OCR models
*If wget is not installed in the windows system, you can copy the link to the browser to download the model. After model downloaded, unzip it and place it in the corresponding directory*
xxxpsyduck's avatar
xxxpsyduck committed
47
48
49

```
mkdir inference && cd inference
50
# Download the detection part of the Ultra-lightweight Chinese OCR and decompress it
xxxpsyduck's avatar
xxxpsyduck committed
51
wget https://paddleocr.bj.bcebos.com/ch_models/ch_det_mv3_db_infer.tar && tar xf ch_det_mv3_db_infer.tar
52
# Download the recognition part of the Ultra-lightweight Chinese OCR and decompress it
xxxpsyduck's avatar
xxxpsyduck committed
53
54
55
wget https://paddleocr.bj.bcebos.com/ch_models/ch_rec_mv3_crnn_infer.tar && tar xf ch_rec_mv3_crnn_infer.tar
cd ..
```
56
#### (2) Download General Chinese OCR models
xxxpsyduck's avatar
xxxpsyduck committed
57
58
```
mkdir inference && cd inference
59
# Download the detection part of the general Chinese OCR model and decompress it
xxxpsyduck's avatar
xxxpsyduck committed
60
wget https://paddleocr.bj.bcebos.com/ch_models/ch_det_r50_vd_db_infer.tar && tar xf ch_det_r50_vd_db_infer.tar
61
# Download the recognition part of the generic Chinese OCR model and decompress it
xxxpsyduck's avatar
xxxpsyduck committed
62
63
64
65
wget https://paddleocr.bj.bcebos.com/ch_models/ch_rec_r34_vd_crnn_infer.tar && tar xf ch_rec_r34_vd_crnn_infer.tar
cd ..
```

66
#### 3. Single image and batch image prediction
xxxpsyduck's avatar
xxxpsyduck committed
67

68
The following code implements text detection and recognition inference tandemly. When performing prediction, you need to specify the path of a single image or image folder through the parameter `image_dir`, the parameter `det_model_dir` specifies the path to detection model, and the parameter `rec_model_dir` specifies the path to the recognition model. The visual prediction results are saved to the `./inference_results` folder by default.
xxxpsyduck's avatar
xxxpsyduck committed
69

70
```bash
tink2123's avatar
revert  
tink2123 committed
71

72
# Prediction on a single image by specifying image path to image_dir
xxxpsyduck's avatar
xxxpsyduck committed
73
74
python3 tools/infer/predict_system.py --image_dir="./doc/imgs/11.jpg" --det_model_dir="./inference/ch_det_mv3_db/"  --rec_model_dir="./inference/ch_rec_mv3_crnn/"

75
# Prediction on a batch of images by specifying image folder path to image_dir
xxxpsyduck's avatar
xxxpsyduck committed
76
77
python3 tools/infer/predict_system.py --image_dir="./doc/imgs/" --det_model_dir="./inference/ch_det_mv3_db/"  --rec_model_dir="./inference/ch_rec_mv3_crnn/"

78
# If you want to use CPU for prediction, you need to set the use_gpu parameter to False
xxxpsyduck's avatar
xxxpsyduck committed
79
80
81
python3 tools/infer/predict_system.py --image_dir="./doc/imgs/11.jpg" --det_model_dir="./inference/ch_det_mv3_db/"  --rec_model_dir="./inference/ch_rec_mv3_crnn/" --use_gpu=False
```

82
To run inference of the Generic Chinese OCR model, follow these steps above to download the corresponding models and update the relevant parameters. Examples are as follows:
xxxpsyduck's avatar
xxxpsyduck committed
83
```
84
# Prediction on a single image by specifying image path to image_dir
xxxpsyduck's avatar
xxxpsyduck committed
85
86
87
python3 tools/infer/predict_system.py --image_dir="./doc/imgs/11.jpg" --det_model_dir="./inference/ch_det_r50_vd_db/"  --rec_model_dir="./inference/ch_rec_r34_vd_crnn/"
```

88
For more text detection and recognition models, please refer to the document [Inference](./doc/doc_en/inference_en.md)
xxxpsyduck's avatar
xxxpsyduck committed
89

LDOUBLEV's avatar
LDOUBLEV committed
90
## Documentation
91
92
93
94
95
- [Quick installation](./doc/doc_en/installation_en.md)
- [Text detection model training/evaluation/prediction](./doc/doc_en/detection_en.md)
- [Text recognition model training/evaluation/prediction](./doc/doc_en/recognition_en.md)
- [Inference](./doc/doc_en/inference_en.md)
- [Dataset](./doc/doc_en/datasets_en.md)
xxxpsyduck's avatar
xxxpsyduck committed
96

97
## Text detection algorithm
xxxpsyduck's avatar
xxxpsyduck committed
98

99
PaddleOCR open source text detection algorithms list:
xxxpsyduck's avatar
xxxpsyduck committed
100
101
- [x]  EAST([paper](https://arxiv.org/abs/1704.03155))
- [x]  DB([paper](https://arxiv.org/abs/1911.08947))
102
- [ ]  SAST([paper](https://arxiv.org/abs/1908.05498))(Baidu Self-Research, comming soon)
xxxpsyduck's avatar
xxxpsyduck committed
103

104
On the ICDAR2015 dataset, the text detection result is as follows:
xxxpsyduck's avatar
xxxpsyduck committed
105

106
|Model|Backbone|precision|recall|Hmean|Download link|
xxxpsyduck's avatar
xxxpsyduck committed
107
|-|-|-|-|-|-|
108
109
110
111
|EAST|ResNet50_vd|88.18%|85.51%|86.82%|[Download link](https://paddleocr.bj.bcebos.com/det_r50_vd_east.tar)|
|EAST|MobileNetV3|81.67%|79.83%|80.74%|[Download link](https://paddleocr.bj.bcebos.com/det_mv3_east.tar)|
|DB|ResNet50_vd|83.79%|80.65%|82.19%|[Download link](https://paddleocr.bj.bcebos.com/det_r50_vd_db.tar)|
|DB|MobileNetV3|75.92%|73.18%|74.53%|[Download link](https://paddleocr.bj.bcebos.com/det_mv3_db.tar)|
xxxpsyduck's avatar
xxxpsyduck committed
112

MissPenguin's avatar
MissPenguin committed
113
For use of [LSVT](https://github.com/PaddlePaddle/PaddleOCR/blob/develop/doc/doc_en/datasets_en.md#1-icdar2019-lsvt) street view dataset with a total of 3w training data,the related configuration and pre-trained models for Chinese detection task are as follows:
114
|Model|Backbone|Configuration file|Pre-trained model|
xxxpsyduck's avatar
xxxpsyduck committed
115
|-|-|-|-|
116
117
|Ultra-lightweight Chinese model|MobileNetV3|det_mv3_db.yml|[Download link](https://paddleocr.bj.bcebos.com/ch_models/ch_det_mv3_db.tar)|
|General Chinese OCR model|ResNet50_vd|det_r50_vd_db.yml|[Download link](https://paddleocr.bj.bcebos.com/ch_models/ch_det_r50_vd_db.tar)|
xxxpsyduck's avatar
xxxpsyduck committed
118

119
* Note: For the training and evaluation of the above DB model, post-processing parameters box_thresh=0.6 and unclip_ratio=1.5 need to be set. If using different datasets and different models for training, these two parameters can be adjusted for better result.
xxxpsyduck's avatar
xxxpsyduck committed
120

MissPenguin's avatar
MissPenguin committed
121
For the training guide and use of PaddleOCR text detection algorithms, please refer to the document [Text detection model training/evaluation/prediction](./doc/doc_en/detection_en.md)
xxxpsyduck's avatar
xxxpsyduck committed
122

123
## Text recognition algorithm
xxxpsyduck's avatar
xxxpsyduck committed
124

125
PaddleOCR open-source text recognition algorithms list:
xxxpsyduck's avatar
xxxpsyduck committed
126
127
128
129
- [x]  CRNN([paper](https://arxiv.org/abs/1507.05717))
- [x]  Rosetta([paper](https://arxiv.org/abs/1910.05085))
- [x]  STAR-Net([paper](http://www.bmva.org/bmvc/2016/papers/paper043/index.html))
- [x]  RARE([paper](https://arxiv.org/abs/1603.03915v1))
130
- [ ]  SRN([paper](https://arxiv.org/abs/2003.12294))(Baidu Self-Research, comming soon)
xxxpsyduck's avatar
xxxpsyduck committed
131

132
Refer to [DTRB](https://arxiv.org/abs/1904.01906), the training and evaluation result of these above text recognition (using MJSynth and SynthText for training, evaluate on IIIT, SVT, IC03, IC13, IC15, SVTP, CUTE) is as follow:
xxxpsyduck's avatar
xxxpsyduck committed
133

134
|Model|Backbone|Avg Accuracy|Module combination|Download link|
xxxpsyduck's avatar
xxxpsyduck committed
135
|-|-|-|-|-|
136
137
138
139
140
141
142
143
144
|Rosetta|Resnet34_vd|80.24%|rec_r34_vd_none_none_ctc|[Download link](https://paddleocr.bj.bcebos.com/rec_r34_vd_none_none_ctc.tar)|
|Rosetta|MobileNetV3|78.16%|rec_mv3_none_none_ctc|[Download link](https://paddleocr.bj.bcebos.com/rec_mv3_none_none_ctc.tar)|
|CRNN|Resnet34_vd|82.20%|rec_r34_vd_none_bilstm_ctc|[Download link](https://paddleocr.bj.bcebos.com/rec_r34_vd_none_bilstm_ctc.tar)|
|CRNN|MobileNetV3|79.37%|rec_mv3_none_bilstm_ctc|[Download link](https://paddleocr.bj.bcebos.com/rec_mv3_none_bilstm_ctc.tar)|
|STAR-Net|Resnet34_vd|83.93%|rec_r34_vd_tps_bilstm_ctc|[Download link](https://paddleocr.bj.bcebos.com/rec_r34_vd_tps_bilstm_ctc.tar)|
|STAR-Net|MobileNetV3|81.56%|rec_mv3_tps_bilstm_ctc|[Download link](https://paddleocr.bj.bcebos.com/rec_mv3_tps_bilstm_ctc.tar)|
|RARE|Resnet34_vd|84.90%|rec_r34_vd_tps_bilstm_attn|[Download link](https://paddleocr.bj.bcebos.com/rec_r34_vd_tps_bilstm_attn.tar)|
|RARE|MobileNetV3|83.32%|rec_mv3_tps_bilstm_attn|[Download link](https://paddleocr.bj.bcebos.com/rec_mv3_tps_bilstm_attn.tar)|

MissPenguin's avatar
MissPenguin committed
145
We use [LSVT](https://github.com/PaddlePaddle/PaddleOCR/blob/develop/doc/doc_en/datasets_en.md#1-icdar2019-lsvt) dataset and cropout 30w  traning data from original photos by using position groundtruth and make some calibration needed. In addition, based on the LSVT corpus, 500w synthetic data is generated to train the Chinese model. The related configuration and pre-trained models are as follows:
146
|Model|Backbone|Configuration file|Pre-trained model|
xxxpsyduck's avatar
xxxpsyduck committed
147
|-|-|-|-|
148
149
|Ultra-lightweight Chinese model|MobileNetV3|rec_chinese_lite_train.yml|[Download link](https://paddleocr.bj.bcebos.com/ch_models/ch_rec_mv3_crnn.tar)|
|General Chinese OCR model|Resnet34_vd|rec_chinese_common_train.yml|[Download link](https://paddleocr.bj.bcebos.com/ch_models/ch_rec_r34_vd_crnn.tar)|
xxxpsyduck's avatar
xxxpsyduck committed
150

MissPenguin's avatar
MissPenguin committed
151
Please refer to the document for training guide and use of PaddleOCR text recognition algorithms [Text recognition model training/evaluation/prediction](./doc/doc_en/recognition_en.md)
xxxpsyduck's avatar
xxxpsyduck committed
152

153
154
## End-to-end OCR algorithm
- [ ]  [End2End-PSL](https://arxiv.org/abs/1909.07808)(Baidu Self-Research, comming soon)
xxxpsyduck's avatar
xxxpsyduck committed
155

156
157
<a name="Ultra-lightweight Chinese OCR results"></a>
## Ultra-lightweight Chinese OCR results
xxxpsyduck's avatar
xxxpsyduck committed
158
159
160
161
162
163
164
165
166
![](doc/imgs_results/1.jpg)
![](doc/imgs_results/7.jpg)
![](doc/imgs_results/12.jpg)
![](doc/imgs_results/4.jpg)
![](doc/imgs_results/6.jpg)
![](doc/imgs_results/9.jpg)
![](doc/imgs_results/16.png)
![](doc/imgs_results/22.jpg)

167
168
<a name="General Chinese OCR results"></a>
## General Chinese OCR results
xxxpsyduck's avatar
xxxpsyduck committed
169
170
171
172
173
![](doc/imgs_results/chinese_db_crnn_server/11.jpg)
![](doc/imgs_results/chinese_db_crnn_server/2.jpg)
![](doc/imgs_results/chinese_db_crnn_server/8.jpg)

## FAQ
tink2123's avatar
tink2123 committed
174
1. Error when using attention-based recognition model: KeyError: 'predict'
xxxpsyduck's avatar
xxxpsyduck committed
175

176
    The inference of recognition model based on attention loss is still being debugged. For Chinese text recognition, it is recommended to choose the recognition model based on CTC loss first. In practice, it is also found that the recognition model based on attention loss is not as effective as the one based on CTC loss.
LDOUBLEV's avatar
LDOUBLEV committed
177

tink2123's avatar
tink2123 committed
178
2. About inference speed
xxxpsyduck's avatar
xxxpsyduck committed
179

180
    When there are a lot of texts in the picture, the prediction time will increase. You can use `--rec_batch_num` to set a smaller prediction batch size. The default value is 30, which can be changed to 10 or other values.
xxxpsyduck's avatar
xxxpsyduck committed
181

tink2123's avatar
tink2123 committed
182
3. Service deployment and mobile deployment
xxxpsyduck's avatar
xxxpsyduck committed
183

184
    It is expected that the service deployment based on Serving and the mobile deployment based on Paddle Lite will be released successively in mid-to-late June. Stay tuned for more updates.
LDOUBLEV's avatar
LDOUBLEV committed
185

tink2123's avatar
tink2123 committed
186
4. Release time of self-developed algorithm
187
188

    Baidu Self-developed algorithms such as SAST, SRN and end2end PSL will be released in June or July. Please be patient.
LDOUBLEV's avatar
LDOUBLEV committed
189

190
[more](./doc/doc_en/FAQ_en.md)
xxxpsyduck's avatar
xxxpsyduck committed
191

192
## Welcome to the PaddleOCR technical exchange group
xxxpsyduck's avatar
xxxpsyduck committed
193
WeChat: paddlehelp, note OCR, our assistant will get you into the group~
194

xxxpsyduck's avatar
xxxpsyduck committed
195
<img src="./doc/paddlehelp.jpg"  width = "200" height = "200" />
xxxpsyduck's avatar
xxxpsyduck committed
196

197
## References
xxxpsyduck's avatar
xxxpsyduck committed
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
```
1. EAST:
@inproceedings{zhou2017east,
  title={EAST: an efficient and accurate scene text detector},
  author={Zhou, Xinyu and Yao, Cong and Wen, He and Wang, Yuzhi and Zhou, Shuchang and He, Weiran and Liang, Jiajun},
  booktitle={Proceedings of the IEEE conference on Computer Vision and Pattern Recognition},
  pages={5551--5560},
  year={2017}
}

2. DB:
@article{liao2019real,
  title={Real-time Scene Text Detection with Differentiable Binarization},
  author={Liao, Minghui and Wan, Zhaoyi and Yao, Cong and Chen, Kai and Bai, Xiang},
  journal={arXiv preprint arXiv:1911.08947},
  year={2019}
}

3. DTRB:
@inproceedings{baek2019wrong,
  title={What is wrong with scene text recognition model comparisons? dataset and model analysis},
  author={Baek, Jeonghun and Kim, Geewook and Lee, Junyeop and Park, Sungrae and Han, Dongyoon and Yun, Sangdoo and Oh, Seong Joon and Lee, Hwalsuk},
  booktitle={Proceedings of the IEEE International Conference on Computer Vision},
  pages={4715--4723},
  year={2019}
}

4. SAST:
@inproceedings{wang2019single,
  title={A Single-Shot Arbitrarily-Shaped Text Detector based on Context Attended Multi-Task Learning},
  author={Wang, Pengfei and Zhang, Chengquan and Qi, Fei and Huang, Zuming and En, Mengyi and Han, Junyu and Liu, Jingtuo and Ding, Errui and Shi, Guangming},
  booktitle={Proceedings of the 27th ACM International Conference on Multimedia},
  pages={1277--1285},
  year={2019}
}

5. SRN:
@article{yu2020towards,
  title={Towards Accurate Scene Text Recognition with Semantic Reasoning Networks},
  author={Yu, Deli and Li, Xuan and Zhang, Chengquan and Han, Junyu and Liu, Jingtuo and Ding, Errui},
  journal={arXiv preprint arXiv:2003.12294},
  year={2020}
}

6. end2end-psl:
@inproceedings{sun2019chinese,
  title={Chinese Street View Text: Large-scale Chinese Text Reading with Partially Supervised Learning},
  author={Sun, Yipeng and Liu, Jiaming and Liu, Wei and Han, Junyu and Ding, Errui and Liu, Jingtuo},
  booktitle={Proceedings of the IEEE International Conference on Computer Vision},
  pages={9086--9095},
  year={2019}
}
```

252
253
## License
This project is released under <a href="https://github.com/PaddlePaddle/PaddleOCR/blob/master/LICENSE">Apache 2.0 license</a>
xxxpsyduck's avatar
xxxpsyduck committed
254

255
256
## Contribution
We welcome all the contributions to PaddleOCR and appreciate for your feedback very much.
tink2123's avatar
tink2123 committed
257
258

- Many thanks to [Khanh Tran](https://github.com/xxxpsyduck) for contributing the English documentation.
LDOUBLEV's avatar
LDOUBLEV committed
259
- Many thanks to [zhangxin](https://github.com/ZhangXinNan) for contributing the new visualize function、add .gitgnore and discard set PYTHONPATH manually.