Merge pull request #3 from PaddlePaddle/develop

update2020-7-20

doc/doc_en/serving_en.md

+# Service deployment
+
+PaddleOCR provides 2 service deployment methods:：
+- Based on **HubServing**：Has been integrated into PaddleOCR ([code](https://github.com/PaddlePaddle/PaddleOCR/tree/develop/deploy/hubserving)). Please follow this tutorial.
+- Based on **PaddleServing**：See PaddleServing official website for details ([demo](https://github.com/PaddlePaddle/Serving/tree/develop/python/examples/ocr)). Follow-up will also be integrated into PaddleOCR.  
+
+The service deployment directory includes three service packages: detection, recognition, and two-stage series connection. Select the corresponding service package to install and start service according to your needs. The directory is as follows:  
+```
+deploy/hubserving/
+  └─  ocr_det     detection module service package
+  └─  ocr_rec     recognition module service package
+  └─  ocr_system  two-stage series connection service package
+```
+
+Each service pack contains 3 files. Take the 2-stage series connection service package as an example, the directory is as follows:  
+```
+deploy/hubserving/ocr_system/
+  └─  __init__.py    Empty file, required
+  └─  config.json    Configuration file, optional, passed in as a parameter when using configuration to start the service
+  └─  module.py      Main module file, required, contains the complete logic of the service
+  └─  params.py      Parameter file, required, including parameters such as model path, pre- and post-processing parameters
+```
+
+## Quick start service
+The following steps take the 2-stage series service as an example. If only the detection service or recognition service is needed, replace the corresponding file path.
+
+### 1. Prepare the environment
+```shell
+# Install paddlehub  
+pip3 install paddlehub --upgrade -i https://pypi.tuna.tsinghua.edu.cn/simple
+
+# Set environment variables  
+export PYTHONPATH=.
+```   
+
+### 2. Install Service Module
+PaddleOCR provides 3 kinds of service modules, install the required modules according to your needs. Such as:  
+
+Install the detection service module:  
+```shell
+hub install deploy/hubserving/ocr_det/
+```  
+Or, install the recognition service module:  
+```shell
+hub install deploy/hubserving/ocr_rec/
+```  
+Or, install the 2-stage series service module:  
+```shell
+hub install deploy/hubserving/ocr_system/
+```  
+
+### 3. Start service
+#### Way 1. Start with command line parameters (CPU only)
+
+**start command：**  
+```shell
+$ hub serving start --modules [Module1==Version1, Module2==Version2, ...] \
+                    --port XXXX \
+                    --use_multiprocess \
+                    --workers \
+```  
+**parameters：**  
+
+|parameters|usage|  
+|-|-|  
+|--modules/-m|PaddleHub Serving pre-installed model, listed in the form of multiple Module==Version key-value pairs<br>*`When Version is not specified, the latest version is selected by default`*|
+|--port/-p|Service port, default is 8866|  
+|--use_multiprocess|Enable concurrent mode, the default is single-process mode, this mode is recommended for multi-core CPU machines<br>*`Windows operating system only supports single-process mode`*|
+|--workers|The number of concurrent tasks specified in concurrent mode, the default is `2*cpu_count-1`, where `cpu_count` is the number of CPU cores|  
+
+For example, start the 2-stage series service:  
+```shell
+hub serving start -m ocr_system
+```  
+
+This completes the deployment of a service API, using the default port number 8866.  
+
+#### Way 2. Start with configuration file（CPU、GPU）
+**start command：**  
+```shell
+hub serving start --config/-c config.json
+```  
+Wherein, the format of `config.json` is as follows:
+```python
+{
+    "modules_info": {
+        "ocr_system": {
+            "init_args": {
+                "version": "1.0.0",
+                "use_gpu": true
+            },
+            "predict_args": {
+            }
+        }
+    },
+    "port": 8868,
+    "use_multiprocess": false,
+    "workers": 2
+}
+```
+- The configurable parameters in `init_args` are consistent with the `_initialize` function interface in `module.py`. Among them, **when `use_gpu` is `true`, it means that the GPU is used to start the service**.
+- The configurable parameters in `predict_args` are consistent with the `predict` function interface in `module.py`.
+
+**Note:**  
+- When using the configuration file to start the service, other parameters will be ignored.
+- If you use GPU prediction (that is, `use_gpu` is set to `true`), you need to set the environment variable CUDA_VISIBLE_DEVICES before starting the service, such as: ```export CUDA_VISIBLE_DEVICES=0```, otherwise you do not need to set it.
+- **`use_gpu` and `use_multiprocess` cannot be `true` at the same time.**  
+
+For example, use GPU card No. 3 to start the 2-stage series service:
+```shell
+export CUDA_VISIBLE_DEVICES=3
+hub serving start -c deploy/hubserving/ocr_system/config.json
+```  
+
+## Send prediction requests
+After the service starts, you can use the following command to send a prediction request to obtain the prediction result:  
+```shell
+python tools/test_hubserving.py server_url image_path
+```  
+
+Two parameters need to be passed to the script:
+- **server_url**：service address，format of which is 
+`http://[ip_address]:[port]/predict/[module_name]`  
+For example, if the detection, recognition and 2-stage serial services are started with provided configuration files, the respective `server_url` would be:  
+`http://127.0.0.1:8866/predict/ocr_det`  
+`http://127.0.0.1:8867/predict/ocr_rec`  
+`http://127.0.0.1:8868/predict/ocr_system`  
+- **image_path**：Test image path, can be a single image path or an image directory path
+
+**Eg.**
+```shell
+python tools/test_hubserving.py http://127.0.0.1:8868/predict/ocr_system ./doc/imgs/
+```
+
+## Returned result format
+The returned result is a list. Each item in the list is a dict. The dict may contain three fields. The information is as follows:
+
+|field name|data type|description| 
+|-|-|-|
+|text|str|text content|
+|confidence|float|text recognition confidence|
+|text_region|list|text location coordinates|
+
+The fields returned by different modules are different. For example, the results returned by the text recognition service module do not contain `text_region`. The details are as follows:
+
+|field name/module name|ocr_det|ocr_rec|ocr_system|
+|-|-|-|-|  
+|text||✔|✔| 
+|confidence||✔|✔| 
+|text_region|✔||✔| 
+
+**Note：** If you need to add, delete or modify the returned fields, you can modify the file `module.py` of the corresponding module. For the complete process, refer to the user-defined modification service module in the next section.
+
+## User defined service module modification
+If you need to modify the service logic, the following steps are generally required (take the modification of `ocr_system` for example):
+
+- 1. Stop service
+```shell
+hub serving stop --port/-p XXXX
+```
+- 2. Modify the code in the corresponding files, like `module.py` and `params.py`, according to the actual needs.  
+For example, if you need to replace the model used by the deployed service, you need to modify model path parameters `det_model_dir` and `rec_model_dir` in `params.py`. Of course, other related parameters may need to be modified at the same time. Please modify and debug according to the actual situation. It is suggested to run `module.py` directly for debugging after modification before starting the service test.  
+- 3. Uninstall old service module
+```shell
+hub uninstall ocr_system
+```
+- 4. Install modified service module
+```shell
+hub install deploy/hubserving/ocr_system/
+```
+- 5. Restart service
+```shell
+hub serving start -m ocr_system
+```

doc/doc_en/tricks_en.md

+## Tricks
+Here we have sorted out some Chinese OCR training and prediction tricks, which are being updated continuously. You are welcome to contribute more OCR tricks ~
+
+- [Replace Backbone Network](#ReplaceBackboneNetwork)
+- [Long Chinese Text Recognition](#LongChineseTextRecognition)
+- [Space Recognition](#SpaceRecognition)
+
+<a name="ReplaceBackboneNetwork"></a>
+#### 1、Replace Backbone Network
+- **Problem Description**
+
+  At present, ResNet_vd series and MobileNetV3 series are the backbone networks used in PaddleOCR, whether replacing the other backbone networks will help to improve the accuracy? What should be paid attention to when replacing?
+
+- **Tips**
+  - Whether text detection or text recognition, the choice of backbone network is a trade-off between prediction effect and prediction efficiency. Generally, a larger backbone network is selected, e.g. ResNet101_vd, then the performance of the detection or recognition is more accurate, but the time cost will increase accordingly. And a smaller backbone network is selected, e.g. MobileNetV3_small_x0_35, the prediction speed is faster, but the accuracy of detection or recognition will be reduced. Fortunately, the detection or recognition effect of different backbone networks is positively correlated with the performance of ImageNet 1000 classification task. [**PaddleClas**](https://github.com/PaddlePaddle/PaddleClas/blob/master/README_en.md) have sorted out the 23 series of classification network structures, such as ResNet_vd、Res2Net、HRNet、MobileNetV3、GhostNet. It provides the top1 accuracy of classification, the time cost of GPU(V100 and T4) and CPU(SD 855), and the 117 pretrained models [**download addresses**](https://paddleclas-en.readthedocs.io/en/latest/models/models_intro_en.html).
+  
+  - Similar as the 4 stages of ResNet, the replacement of text detection backbone network is to determine those four stages to facilitate the integration of FPN like the object detection heads. In addition, for the text detection problem, the pre trained model in ImageNet1000 can accelerate the convergence and improve the accuracy.
+  
+  - In order to replace the backbone network of text recognition, we need to pay attention to the descending position of network width and height stride. Since the ratio between width and height is large in chinese text recognition, the frequency of height decrease is less and the frequency of width decrease is more. You can refer the [modifies of MobileNetV3](https://github.com/PaddlePaddle/PaddleOCR/blob/develop/ppocr/modeling/backbones/rec_mobilenet_v3.py) in PaddleOCR.
+
+<a name="LongChineseTextRecognition"></a>
+#### 2、Long Chinese Text Recognition
+- **Problem Description** 
+  The maximum resolution of Chinese recognition model during training is [3,32,320], if the text image to be recognized is too long, as shown in the figure below, how to adapt?
+  
+  <div align="center">
+    <img src="../tricks/long_text_examples.jpg" width="600">
+  </div>
+  
+- **Tips**
+
+  During the training, the training samples are not directly resized to [3,32,320]. At first, the height of samples are resized to 32 and keep the ratio between the width and the height. When the width is less than 320, the excess parts are padding 0. Besides, when the ratio between the width and the height of the samples is larger than 10, these samples will be ignored. When the prediction for one image, do as above, but do not limit the max ratio between the width and the height. When the prediction for an images batch, do as training, but the resized target width is the longest width of the images in the batch. [Code as following](https://github.com/PaddlePaddle/PaddleOCR/blob/develop/tools/infer/predict_rec.py)：
+  
+  ```
+    def resize_norm_img(self, img, max_wh_ratio):
+        imgC, imgH, imgW = self.rec_image_shape
+        assert imgC == img.shape[2]
+        if self.character_type == "ch":
+            imgW = int((32 * max_wh_ratio))
+        h, w = img.shape[:2]
+        ratio = w / float(h)
+        if math.ceil(imgH * ratio) > imgW:
+            resized_w = imgW
+        else:
+            resized_w = int(math.ceil(imgH * ratio))
+        resized_image = cv2.resize(img, (resized_w, imgH))
+        resized_image = resized_image.astype('float32')
+        resized_image = resized_image.transpose((2, 0, 1)) / 255
+        resized_image -= 0.5
+        resized_image /= 0.5
+        padding_im = np.zeros((imgC, imgH, imgW), dtype=np.float32)
+        padding_im[:, :, 0:resized_w] = resized_image
+        return padding_im  
+  ```
+
+<a name="SpaceRecognition"></a>
+#### 3、Space Recognition
+- **Problem Description**
+
+  As shown in the figure below, for Chinese and English mixed scenes, in order to facilitate reading and using the recognition results, it is often necessary to recognize the spaces between words. How can this situation be adapted?
+   
+  <div align="center">
+    <img src="../imgs_results/chinese_db_crnn_server/en_paper.jpg" width="600">
+  </div>
+  
+- **Tips**
+  
+  There are two possible methods for space recognition. (1) Optimize the text detection. For spliting the text at the space in detection results, it needs to divide the text line with space into many segments when label the data for detection. (2) Optimize the text recognition. The space character is introduced into the recognition dictionary. Label the blank line in the training data for text recognition. In addition, we can also concat multiple word lines to synthesize the training data with spaces. PaddleOCR currently uses the second method.

doc/doc_en/update_en.md

 # RECENT UPDATES
 
+- 2020.7.15, Add mobile App demo , support both iOS and  Android  ( based on easyedge and Paddle Lite)
+- 2020.7.15, Improve the  deployment ability, add the C + +  inference , serving deployment. In addtion, the benchmarks of the ultra-lightweight Chinese OCR model are provided.
+- 2020.7.15, Add several related datasets, data annotation and synthesis tools.
+- 2020.7.9 Add a new model to support recognize the  character "space".
+- 2020.7.9 Add the data augument and learning rate decay strategies during training.
+- 2020.6.8 Add [datasets](./doc/doc_en/datasets_en.md) and keep updating
+- 2020.6.5 Support exporting `attention` model to `inference_model`
+- 2020.6.5 Support separate prediction and recognition, output result score
 - 2020.6.5 Support exporting `attention` model to `inference_model`
 - 2020.6.5 Support separate prediction and recognition, output result score
 - 2020.5.30 Provide Lightweight Chinese OCR online experience

doc/doc_en/vertical_and_multilingual_datasets_en.md

+# Vertical multi-language OCR dataset
+Here we have sorted out the commonly used vertical multi-language OCR dataset datasets, which are being updated continuously. We welcome you to contribute datasets ～
+- [Chinese urban license plate dataset](#Chinese urban license plate dataset)
+- [Bank credit card dataset](#Bank credit card dataset)
+- [Captcha dataset-Captcha](#Captcha dataset-Captcha)
+- [multi-language dataset](#multi-language dataset)
+
+
+<a name="Chinese urban license plate dataset"></a>
+## Chinese urban license plate dataset
+
+- **Data source**：[https://github.com/detectRecog/CCPD](https://github.com/detectRecog/CCPD)
+
+- **Data introduction**: It contains more than 250000 vehicle license plate images and vehicle license plate detection and recognition information labeling. It contains the following license plate image information in different scenes.
+    * CCPD-Base: General license plate picture
+    * CCPD-DB: The brightness of license plate area is bright, dark or uneven
+    * CCPD-FN: The license plate is farther or closer to the camera location
+    * CCPD-Rotate: License plate includes rotation (horizontal 20\~50 degrees, vertical-10\~10 degrees)
+    * CCPD-Tilt: License plate includes rotation (horizontal 15\~45 degrees, vertical 15\~45 degrees)
+    * CCPD-Blur: The license plate contains blurring due to camera lens jitter
+    * CCPD-Weather: The license plate is photographed on rainy, snowy or foggy days
+    * CCPD-Challenge: So far, some of the most challenging images in license plate detection and recognition tasks
+    * CCPD-NP: Pictures of new cars without license plates.
+
+    ![](../datasets/ccpd_demo.png)
+
+
+- **Download address**
+    * Baidu cloud download address (extracted code is hm0U): [https://pan.baidu.com/s/1i5AOjAbtkwb17Zy-NQGqkw](https://pan.baidu.com/s/1i5AOjAbtkwb17Zy-NQGqkw)
+    * Google drive download address:[https://drive.google.com/file/d/1rdEsCUcIUaYOVRkx5IMTRNA7PcGMmSgc/view](https://drive.google.com/file/d/1rdEsCUcIUaYOVRkx5IMTRNA7PcGMmSgc/view)
+
+
+<a name="Bank credit card dataset"></a>
+## Bank credit card dataset
+
+- **Data source**: [https://www.kesci.com/home/dataset/5954cf1372ead054a5e25870](https://www.kesci.com/home/dataset/5954cf1372ead054a5e25870)
+
+- **Data introduction**: There are three types of training data
+    * 1.Sample card data of China Merchants Bank: including card image data and annotation data, a total of 618 pictures
+    * 2.Single character data: including pictures and annotation data, 37 pictures in total.
+    * 3.There are only other bank cards, no more detailed information, a total of 50 pictures.
+
+    * The demo image is shown as follows. The annotation information is stored in excel, and the demo image below is marked as
+        * Top 8 card number: 62257583
+        * Card type: card of our bank
+        * End of validity: 07/41
+        * Chinese phonetic alphabet of card users: MICHAEL
+
+    ![](../datasets/cmb_demo.jpg)
+
+- **Download address**: [https://cdn.kesci.com/cmb2017-2.zip](https://cdn.kesci.com/cmb2017-2.zip)
+
+
+
+<a name="Captcha dataset-Captcha"></a>
+## Captcha dataset-Captcha
+
+- **Data source**: [https://github.com/lepture/captcha](https://github.com/lepture/captcha)
+
+- **Data introduction**: This is a toolkit for data synthesis. You can output captcha images according to the input text. Use the toolkit to generate several demo images as follows.
+
+    ![](../datasets/captcha_demo.png)
+
+- **Download address**: The dataset is generated and has no download address.
+
+
+
+<a name="multi-language dataset"></a>
+## multi-language dataset(Multi-lingual scene text detection and recognition)
+
+- **Data source**: [https://rrc.cvc.uab.es/?ch=15&com=downloads](https://rrc.cvc.uab.es/?ch=15&com=downloads)
+
+- **Data introduction**: Multi language detection dataset MLT contains both language recognition and detection tasks.
+    * In the detection task, the training set contains 10000 images in 10 languages, and each language contains 1000 training images. The test set contains 10000 images.
+    * In the recognition task, the training set contains 111998 samples.
+
+
+- **Download address**: The training set is large and can be downloaded in two parts. It can only be downloaded after registering on the website:
+[https://rrc.cvc.uab.es/?ch=15&com=downloads](https://rrc.cvc.uab.es/?ch=15&com=downloads)

doc/doc_en/visualization_en.md

+# Visualization
+
+- [Chinese/English OCR Visualization (Space_support )](#Space_support)
+- [Ultra-lightweight Chinese/English OCR Visualization](#Ultra-lightweight)
+- [General Chinese/English OCR Visualization](#General)
+
+<a name="Space_support"></a>
+
+## Chinese/English OCR Visualization (Space_support )
+
+###  Ultra-lightweight Model
+<div align="center">
+    <img src="../imgs_results/img_11.jpg" width="800">
+</div>
+
+### General OCR Model
+<div align="center">
+    <img src="../imgs_results/chinese_db_crnn_server/en_paper.jpg" width="800">
+</div>
+
+<a name="Ultra-lightweight"></a>
+## Ultra-lightweight Chinese/English OCR Visualization
+
+<div align="center">
+    <img src="../imgs_results/1.jpg" width="800">
+</div>
+
+<div align="center">
+    <img src="../imgs_results/7.jpg" width="800">
+</div>
+
+<div align="center">
+    <img src="../imgs_results/12.jpg" width="800">
+</div>
+
+<div align="center">
+    <img src="../imgs_results/4.jpg" width="800">
+</div>
+
+<div align="center">
+    <img src="../imgs_results/6.jpg" width="800">
+</div>
+
+<div align="center">
+    <img src="../imgs_results/9.jpg" width="800">
+</div>
+
+<div align="center">
+    <img src="../imgs_results/16.png" width="800">
+</div>
+
+<div align="center">
+    <img src="../imgs_results/22.jpg" width="800">
+</div>
+
+<a name="General"></a>
+## General Chinese/English OCR Visualization
+
+<div align="center">
+    <img src="../imgs_results/chinese_db_crnn_server/11.jpg" width="800">
+</div>
+
+<div align="center">
+    <img src="../imgs_results/chinese_db_crnn_server/2.jpg" width="800">
+</div>
+
+<div align="center">
+    <img src="../imgs_results/chinese_db_crnn_server/8.jpg" width="800">
+</div>
+
+