Merge branch 'dygraph' of https://github.com/PaddlePaddle/PaddleOCR into test_v11

ppstructure/docs/model_list.md

+# Model List
+
+## 1. LayoutParser 模型
+
+|模型名称|模型简介|下载地址|
+| --- | --- | --- |
+| ppyolov2_r50vd_dcn_365e_publaynet | PubLayNet 数据集训练的版面分析模型，可以划分**文字、标题、表格、图片以及列表**5类区域 | [PubLayNet](https://paddle-model-ecology.bj.bcebos.com/model/layout-parser/ppyolov2_r50vd_dcn_365e_publaynet.tar) |
+| ppyolov2_r50vd_dcn_365e_tableBank_word | TableBank Word 数据集训练的版面分析模型，只能检测表格 | [TableBank Word](https://paddle-model-ecology.bj.bcebos.com/model/layout-parser/ppyolov2_r50vd_dcn_365e_tableBank_word.tar) |
+| ppyolov2_r50vd_dcn_365e_tableBank_latex | TableBank Latex 数据集训练的版面分析模型，只能检测表格 | [TableBank Latex](https://paddle-model-ecology.bj.bcebos.com/model/layout-parser/ppyolov2_r50vd_dcn_365e_tableBank_latex.tar) |
+
+## 2. OCR和表格识别模型
+
+|模型名称|模型简介|推理模型大小|下载地址|
+| --- | --- | --- | --- |
+|ch_ppocr_mobile_slim_v2.0_det|slim裁剪版超轻量模型，支持中英文、多语种文本检测|2.6M|[推理模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/slim/ch_ppocr_mobile_v2.0_det_prune_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/slim/ch_ppocr_mobile_v2.0_det_prune_infer.tar) |
+|ch_ppocr_mobile_slim_v2.0_rec|slim裁剪量化版超轻量模型，支持中英文、数字识别|6M|[推理模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_rec_slim_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_rec_slim_train.tar) |
+|en_ppocr_mobile_v2.0_table_det|PubLayNet数据集训练的英文表格场景的文字检测|4.7M|[推理模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/table/en_ppocr_mobile_v2.0_table_det_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.1/table/en_ppocr_mobile_v2.0_table_det_train.tar) |
+|en_ppocr_mobile_v2.0_table_rec|PubLayNet数据集训练的英文表格场景的文字识别|6.9M|[推理模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/table/en_ppocr_mobile_v2.0_table_rec_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.1/table/en_ppocr_mobile_v2.0_table_rec_train.tar) |
+|en_ppocr_mobile_v2.0_table_structure|PubLayNet数据集训练的英文表格场景的表格结构预测|18.6M|[推理模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/table/en_ppocr_mobile_v2.0_table_structure_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.1/table/en_ppocr_mobile_v2.0_table_structure_train.tar) |
+
+如需要使用其他OCR模型，可以在 [model_list](../../doc/doc_ch/models_list.md) 下载模型或者使用自己训练好的模型配置到`det_model_dir`,`rec_model_dir`两个字段即可。
+
+## 3. VQA模型
+
+|模型名称|模型简介|推理模型大小|下载地址|
+| --- | --- | --- | --- |
+|PP-Layout_v1.0_ser_pretrained|基于LayoutXLM在xfun中文数据集上训练的SER模型|1.4G|[推理模型 coming soon]() / [训练模型](https://paddleocr.bj.bcebos.com/pplayout/PP-Layout_v1.0_ser_pretrained.tar) |
+|PP-Layout_v1.0_re_pretrained|基于LayoutXLM在xfun中文数据集上训练的RE模型|1.4G|[推理模型 coming soon]() / [训练模型](https://paddleocr.bj.bcebos.com/pplayout/PP-Layout_v1.0_re_pretrained.tar) |

ppstructure/docs/quickstart.md

+# PP-Structure 快速开始
+
+* [1. 安装PaddleOCR whl包](#1)
+* [2. 便捷使用](#2)
+    + [2.1 命令行使用](#21)
+    + [2.2 Python脚本使用](#22)
+    + [2.3 返回结果说明](#23)
+    + [2.4 参数说明](#24)
+* [3. Python脚本使用](#3)
+
+
+<a name="1"></a>
+
+## 1. 安装依赖包
+
+```bash
+pip install "paddleocr>=2.3.0.2" # 推荐使用2.3.0.2+版本
+pip3 install -U https://paddleocr.bj.bcebos.com/whl/layoutparser-0.0.0-py3-none-any.whl
+
+# 安装 PaddleNLP
+git clone https://github.com/PaddlePaddle/PaddleNLP -b develop
+cd PaddleNLP
+pip3 install -e .
+
+```
+
+<a name="2"></a>
+
+## 2. 便捷使用
+
+<a name="21"></a>
+
+### 2.1 命令行使用
+
+* 版面分析+表格识别
+```bash
+paddleocr --image_dir=../doc/table/1.png --type=structure
+```
+
+* VQA
+
+coming soon
+
+<a name="22"></a>
+
+### 2.2 Python脚本使用
+
+* 版面分析+表格识别
+```python
+import os
+import cv2
+from paddleocr import PPStructure,draw_structure_result,save_structure_res
+
+table_engine = PPStructure(show_log=True)
+
+save_folder = './output/table'
+img_path = '../doc/table/1.png'
+img = cv2.imread(img_path)
+result = table_engine(img)
+save_structure_res(result, save_folder,os.path.basename(img_path).split('.')[0])
+
+for line in result:
+    line.pop('img')
+    print(line)
+
+from PIL import Image
+
+font_path = '../doc/fonts/simfang.ttf' # PaddleOCR下提供字体包
+image = Image.open(img_path).convert('RGB')
+im_show = draw_structure_result(image, result,font_path=font_path)
+im_show = Image.fromarray(im_show)
+im_show.save('result.jpg')
+```
+
+* VQA
+
+comming soon
+
+<a name="23"></a>
+
+### 2.3 返回结果说明
+PP-Structure的返回结果为一个dict组成的list，示例如下
+
+* 版面分析+表格识别
+```shell
+[
+  {   'type': 'Text',
+      'bbox': [34, 432, 345, 462],
+      'res': ([[36.0, 437.0, 341.0, 437.0, 341.0, 446.0, 36.0, 447.0], [41.0, 454.0, 125.0, 453.0, 125.0, 459.0, 41.0, 460.0]],
+                [('Tigure-6. The performance of CNN and IPT models using difforen', 0.90060663), ('Tent  ', 0.465441)])
+  }
+]
+```
+dict 里各个字段说明如下
+
+| 字段            | 说明           |
+| --------------- | -------------|
+|type|图片区域的类型|
+|bbox|图片区域的在原图的坐标，分别[左上角x，左上角y，右下角x，右下角y]|
+|res|图片区域的OCR或表格识别结果。<br> 表格: 表格的HTML字符串; <br> OCR: 一个包含各个单行文字的检测坐标和识别结果的元组|
+
+* VQA
+
+comming soon
+
+<a name="24"></a>
+
+### 2.4 参数说明
+
+| 字段            | 说明                                     | 默认值                                      |
+| --------------- | ---------------------------------------- | ------------------------------------------- |
+| output          | excel和识别结果保存的地址                | ./output/table                              |
+| table_max_len   | 表格结构模型预测时，图像的长边resize尺度 | 488                                         |
+| table_model_dir | 表格结构模型 inference 模型地址          | None                                        |
+| table_char_type | 表格结构模型所用字典地址                 | ../ppocr/utils/dict/table_structure_dict.txt |
+| model_name_or_path | VQA SER模型地址                | None |
+| max_seq_length | VQA SER模型最大支持token长度              | 512 |
+| label_map_path | VQA SER 标签文件地址              | ./vqa/labels/labels_ser.txt |
+| mode | pipeline预测模式，structure: 版面分析+表格识别; vqa: ser文档信息抽取              | structure |
+
+大部分参数和paddleocr whl包保持一致，见 [whl包文档](../doc/doc_ch/whl.md)
+
+运行完成后，每张图片会在`output`字段指定的目录下有一个同名目录，图片里的每个表格会存储为一个excel，图片区域会被裁剪之后保存下来，excel文件和图片名名为表格在图片里的坐标。
+
+<a name="3"></a>
+
+## 3. Python脚本使用
+
+* 版面分析+表格识别
+
+```bash
+cd ppstructure
+
+# 下载模型
+mkdir inference && cd inference
+# 下载超轻量级中文OCR模型的检测模型并解压
+wget https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_det_infer.tar && tar xf ch_ppocr_mobile_v2.0_det_infer.tar
+# 下载超轻量级中文OCR模型的识别模型并解压
+wget https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_rec_infer.tar && tar xf ch_ppocr_mobile_v2.0_rec_infer.tar
+# 下载超轻量级英文表格英寸模型并解压
+wget https://paddleocr.bj.bcebos.com/dygraph_v2.0/table/en_ppocr_mobile_v2.0_table_structure_infer.tar && tar xf en_ppocr_mobile_v2.0_table_structure_infer.tar
+cd ..
+
+python3 predict_system.py --det_model_dir=inference/ch_ppocr_mobile_v2.0_det_infer \
+                          --rec_model_dir=inference/ch_ppocr_mobile_v2.0_rec_infer \
+                          --table_model_dir=inference/en_ppocr_mobile_v2.0_table_structure_infer \
+                          --image_dir=../doc/table/1.png \
+                          --rec_char_dict_path=../ppocr/utils/ppocr_keys_v1.txt \
+                          --table_char_dict_path=../ppocr/utils/dict/table_structure_dict.txt \
+                          --output=../output/table \
+                          --vis_font_path=../doc/fonts/simfang.ttf
+```
+运行完成后，每张图片会在`output`字段指定的目录下的`talbe`目录下有一个同名目录，图片里的每个表格会存储为一个excel，图片区域会被裁剪之后保存下来，excel文件和图片名名为表格在图片里的坐标。
+
+* VQA
+
+```bash
+cd ppstructure
+
+# 下载模型
+mkdir inference && cd inference
+# 下载SER xfun 模型并解压
+wget https://paddleocr.bj.bcebos.com/pplayout/PP-Layout_v1.0_ser_pretrained.tar && tar xf PP-Layout_v1.0_ser_pretrained.tar
+cd ..
+
+python3 predict_system.py --model_name_or_path=vqa/PP-Layout_v1.0_ser_pretrained/ \
+                          --mode=vqa \
+                          --image_dir=vqa/images/input/zh_val_0.jpg  \
+                          --vis_font_path=../doc/fonts/simfang.ttf
+```
+运行完成后，每张图片会在`output`字段指定的目录下的`vqa`目录下存放可视化之后的图片，图片名和输入图片名一致。

ppstructure/predict_system.py

@@ -30,6 +30,7 @@ from ppocr.utils.utility import get_image_file_list, check_and_read_gif
 from ppocr.utils.logging import get_logger
 from tools.infer.predict_system import TextSystem
 from ppstructure.table.predict_table import TableSystem, to_excel
+from ppstructure.vqa.infer_ser_e2e import SerPredictor, draw_ser_results
 from ppstructure.utility import parse_args, draw_structure_result
 
 logger = get_logger()
@@ -37,53 +38,75 @@ logger = get_logger()
 
 class OCRSystem(object):
     def __init__(self, args):
-        import layoutparser as lp
-        # args.det_limit_type = 'resize_long'
-        args.drop_score = 0
-        if not args.show_log:
-            logger.setLevel(logging.INFO)
-        self.text_system = TextSystem(args)
-        self.table_system = TableSystem(args, self.text_system.text_detector, self.text_system.text_recognizer)
-
-        config_path = None
-        model_path = None
-        if os.path.isdir(args.layout_path_model):
-            model_path = args.layout_path_model
-        else:
-            config_path = args.layout_path_model
-        self.table_layout = lp.PaddleDetectionLayoutModel(config_path=config_path,
-                                                          model_path=model_path,
-                                                          threshold=0.5, enable_mkldnn=args.enable_mkldnn,
-                                                          enforce_cpu=not args.use_gpu, thread_num=args.cpu_threads)
-        self.use_angle_cls = args.use_angle_cls
-        self.drop_score = args.drop_score
+        self.mode = args.mode
+        if self.mode == 'structure':
+            import layoutparser as lp
+            # args.det_limit_type = 'resize_long'
+            args.drop_score = 0
+            if not args.show_log:
+                logger.setLevel(logging.INFO)
+            self.text_system = TextSystem(args)
+            self.table_system = TableSystem(args,
+                                            self.text_system.text_detector,
+                                            self.text_system.text_recognizer)
+
+            config_path = None
+            model_path = None
+            if os.path.isdir(args.layout_path_model):
+                model_path = args.layout_path_model
+            else:
+                config_path = args.layout_path_model
+            self.table_layout = lp.PaddleDetectionLayoutModel(
+                config_path=config_path,
+                model_path=model_path,
+                threshold=0.5,
+                enable_mkldnn=args.enable_mkldnn,
+                enforce_cpu=not args.use_gpu,
+                thread_num=args.cpu_threads)
+            self.use_angle_cls = args.use_angle_cls
+            self.drop_score = args.drop_score
+        elif self.mode == 'vqa':
+            self.vqa_engine = SerPredictor(args)
 
     def __call__(self, img):
-        ori_im = img.copy()
-        layout_res = self.table_layout.detect(img[..., ::-1])
-        res_list = []
-        for region in layout_res:
-            x1, y1, x2, y2 = region.coordinates
-            x1, y1, x2, y2 = int(x1), int(y1), int(x2), int(y2)
-            roi_img = ori_im[y1:y2, x1:x2, :]
-            if region.type == 'Table':
-                res = self.table_system(roi_img)
-            else:
-                filter_boxes, filter_rec_res = self.text_system(roi_img)
-                filter_boxes = [x + [x1, y1] for x in filter_boxes]
-                filter_boxes = [x.reshape(-1).tolist() for x in filter_boxes]
-                # remove style char
-                style_token = ['<strike>', '<strike>', '<sup>', '</sub>', '<b>', '</b>', '<sub>', '</sup>',
-                               '<overline>', '</overline>', '<underline>', '</underline>', '<i>', '</i>']
-                filter_rec_res_tmp = []
-                for rec_res in filter_rec_res:
-                    rec_str, rec_conf = rec_res
-                    for token in style_token:
-                        if token in rec_str:
-                            rec_str = rec_str.replace(token, '')
-                    filter_rec_res_tmp.append((rec_str, rec_conf))
-                res = (filter_boxes, filter_rec_res_tmp)
-            res_list.append({'type': region.type, 'bbox': [x1, y1, x2, y2], 'img': roi_img, 'res': res})
+        if self.mode == 'structure':
+            ori_im = img.copy()
+            layout_res = self.table_layout.detect(img[..., ::-1])
+            res_list = []
+            for region in layout_res:
+                x1, y1, x2, y2 = region.coordinates
+                x1, y1, x2, y2 = int(x1), int(y1), int(x2), int(y2)
+                roi_img = ori_im[y1:y2, x1:x2, :]
+                if region.type == 'Table':
+                    res = self.table_system(roi_img)
+                else:
+                    filter_boxes, filter_rec_res = self.text_system(roi_img)
+                    filter_boxes = [x + [x1, y1] for x in filter_boxes]
+                    filter_boxes = [
+                        x.reshape(-1).tolist() for x in filter_boxes
+                    ]
+                    # remove style char
+                    style_token = [
+                        '<strike>', '<strike>', '<sup>', '</sub>', '<b>',
+                        '</b>', '<sub>', '</sup>', '<overline>', '</overline>',
+                        '<underline>', '</underline>', '<i>', '</i>'
+                    ]
+                    filter_rec_res_tmp = []
+                    for rec_res in filter_rec_res:
+                        rec_str, rec_conf = rec_res
+                        for token in style_token:
+                            if token in rec_str:
+                                rec_str = rec_str.replace(token, '')
+                        filter_rec_res_tmp.append((rec_str, rec_conf))
+                    res = (filter_boxes, filter_rec_res_tmp)
+                res_list.append({
+                    'type': region.type,
+                    'bbox': [x1, y1, x2, y2],
+                    'img': roi_img,
+                    'res': res
+                })
+        elif self.mode == 'vqa':
+            res_list, _ = self.vqa_engine(img)
         return res_list
 
 
@@ -91,29 +114,35 @@ def save_structure_res(res, save_folder, img_name):
     excel_save_folder = os.path.join(save_folder, img_name)
     os.makedirs(excel_save_folder, exist_ok=True)
     # save res
-    with open(os.path.join(excel_save_folder, 'res.txt'), 'w', encoding='utf8') as f:
+    with open(
+            os.path.join(excel_save_folder, 'res.txt'), 'w',
+            encoding='utf8') as f:
         for region in res:
             if region['type'] == 'Table':
-                excel_path = os.path.join(excel_save_folder, '{}.xlsx'.format(region['bbox']))
+                excel_path = os.path.join(excel_save_folder,
+                                          '{}.xlsx'.format(region['bbox']))
                 to_excel(region['res'], excel_path)
             if region['type'] == 'Figure':
                 roi_img = region['img']
-                img_path = os.path.join(excel_save_folder, '{}.jpg'.format(region['bbox']))
+                img_path = os.path.join(excel_save_folder,
+                                        '{}.jpg'.format(region['bbox']))
                 cv2.imwrite(img_path, roi_img)
             else:
                 for box, rec_res in zip(region['res'][0], region['res'][1]):
-                    f.write('{}\t{}\n'.format(np.array(box).reshape(-1).tolist(), rec_res))
+                    f.write('{}\t{}\n'.format(
+                        np.array(box).reshape(-1).tolist(), rec_res))
 
 
 def main(args):
     image_file_list = get_image_file_list(args.image_dir)
     image_file_list = image_file_list
     image_file_list = image_file_list[args.process_id::args.total_process_num]
-    save_folder = args.output
-    os.makedirs(save_folder, exist_ok=True)
 
     structure_sys = OCRSystem(args)
     img_num = len(image_file_list)
+    save_folder = os.path.join(args.output, structure_sys.mode)
+    os.makedirs(save_folder, exist_ok=True)
+
     for i, image_file in enumerate(image_file_list):
         logger.info("[{}/{}] {}".format(i, img_num, image_file))
         img, flag = check_and_read_gif(image_file)
@@ -126,10 +155,16 @@ def main(args):
             continue
         starttime = time.time()
         res = structure_sys(img)
-        save_structure_res(res, save_folder, img_name)
-        draw_img = draw_structure_result(img, res, args.vis_font_path)
-        cv2.imwrite(os.path.join(save_folder, img_name, 'show.jpg'), draw_img)
-        logger.info('result save to {}'.format(os.path.join(save_folder, img_name)))
+
+        if structure_sys.mode == 'structure':
+            save_structure_res(res, save_folder, img_name)
+            draw_img = draw_structure_result(img, res, args.vis_font_path)
+            img_save_path = os.path.join(save_folder, img_name, 'show.jpg')
+        elif structure_sys.mode == 'vqa':
+            draw_img = draw_ser_results(img, res, args.vis_font_path)
+            img_save_path = os.path.join(save_folder, img_name + '.jpg')
+        cv2.imwrite(img_save_path, draw_img)
+        logger.info('result save to {}'.format(img_save_path))
         elapse = time.time() - starttime
         logger.info("Predict time : {:.3f}s".format(elapse))
 

ppstructure/table/README.md

@@ -20,9 +20,9 @@ We evaluated the algorithm on the PubTabNet<sup>[1]</sup> eval dataset, and the
 
 
 |Method|[TEDS(Tree-Edit-Distance-based Similarity)](https://github.com/ibm-aur-nlp/PubTabNet/tree/master/src)|
-| --- | --- | 
-| EDD<sup>[2]</sup> | 88.3 | 
-| Ours | 93.32 | 
+| --- | --- |
+| EDD<sup>[2]</sup> | 88.3 |
+| Ours | 93.32 |
 
 ## 3. How to use
 
@@ -41,7 +41,7 @@ wget https://paddleocr.bj.bcebos.com/dygraph_v2.0/table/en_ppocr_mobile_v2.0_tab
 wget https://paddleocr.bj.bcebos.com/dygraph_v2.0/table/en_ppocr_mobile_v2.0_table_structure_infer.tar && tar xf en_ppocr_mobile_v2.0_table_structure_infer.tar
 cd ..
 # run
-python3 table/predict_table.py --det_model_dir=inference/en_ppocr_mobile_v2.0_table_det_infer --rec_model_dir=inference/en_ppocr_mobile_v2.0_table_rec_infer --table_model_dir=inference/en_ppocr_mobile_v2.0_table_structure_infer --image_dir=../doc/table/table.jpg --rec_char_dict_path=../ppocr/utils/dict/table_dict.txt --table_char_dict_path=../ppocr/utils/dict/table_structure_dict.txt --rec_char_type=EN --det_limit_side_len=736 --det_limit_type=min --output ../output/table
+python3 table/predict_table.py --det_model_dir=inference/en_ppocr_mobile_v2.0_table_det_infer --rec_model_dir=inference/en_ppocr_mobile_v2.0_table_rec_infer --table_model_dir=inference/en_ppocr_mobile_v2.0_table_structure_infer --image_dir=../doc/table/table.jpg --rec_char_dict_path=../ppocr/utils/dict/table_dict.txt --table_char_dict_path=../ppocr/utils/dict/table_structure_dict.txt --rec_char_dict_path=../ppocr/utils/dict/en_dict.txt --det_limit_side_len=736 --det_limit_type=min --output ../output/table
 ```
 Note: The above model is trained on the PubLayNet dataset and only supports English scanning scenarios. If you need to identify other scenarios, you need to train the model yourself and replace the three fields `det_model_dir`, `rec_model_dir`, `table_model_dir`.
 
@@ -82,8 +82,8 @@ python3 tools/train.py -c configs/table/table_mv3.yml -o Global.checkpoints=./yo
 The table uses [TEDS(Tree-Edit-Distance-based Similarity)](https://github.com/ibm-aur-nlp/PubTabNet/tree/master/src) as the evaluation metric of the model. Before the model evaluation, the three models in the pipeline need to be exported as inference models (we have provided them), and the gt for evaluation needs to be prepared. Examples of gt are as follows:
 ```json
 {"PMC4289340_004_00.png": [
-  ["<html>", "<body>", "<table>", "<thead>", "<tr>", "<td>", "</td>", "<td>", "</td>", "<td>", "</td>", "</tr>", "</thead>", "<tbody>", "<tr>", "<td>", "</td>", "<td>", "</td>", "<td>", "</td>", "</tr>",  "</tbody>", "</table>", "</body>", "</html>"], 
-  [[1, 4, 29, 13], [137, 4, 161, 13], [215, 4, 236, 13], [1, 17, 30, 27], [137, 17, 147, 27], [215, 17, 225, 27]], 
+  ["<html>", "<body>", "<table>", "<thead>", "<tr>", "<td>", "</td>", "<td>", "</td>", "<td>", "</td>", "</tr>", "</thead>", "<tbody>", "<tr>", "<td>", "</td>", "<td>", "</td>", "<td>", "</td>", "</tr>",  "</tbody>", "</table>", "</body>", "</html>"],
+  [[1, 4, 29, 13], [137, 4, 161, 13], [215, 4, 236, 13], [1, 17, 30, 27], [137, 17, 147, 27], [215, 17, 225, 27]],
   [["<b>", "F", "e", "a", "t", "u", "r", "e", "</b>"], ["<b>", "G", "b", "3", " ", "+", "</b>"], ["<b>", "G", "b", "3", " ", "-", "</b>"], ["<b>", "P", "a", "t", "i", "e", "n", "t", "s", "</b>"], ["6", "2"], ["4", "5"]]
 ]}
 ```
@@ -95,7 +95,7 @@ In gt json, the key is the image name, the value is the corresponding gt, and gt
 Use the following command to evaluate. After the evaluation is completed, the teds indicator will be output.
 ```python
 cd PaddleOCR/ppstructure
-python3 table/eval_table.py --det_model_dir=path/to/det_model_dir --rec_model_dir=path/to/rec_model_dir --table_model_dir=path/to/table_model_dir --image_dir=../doc/table/1.png --rec_char_dict_path=../ppocr/utils/dict/table_dict.txt --table_char_dict_path=../ppocr/utils/dict/table_structure_dict.txt --rec_char_type=EN --det_limit_side_len=736 --det_limit_type=min --gt_path=path/to/gt.json
+python3 table/eval_table.py --det_model_dir=path/to/det_model_dir --rec_model_dir=path/to/rec_model_dir --table_model_dir=path/to/table_model_dir --image_dir=../doc/table/1.png --rec_char_dict_path=../ppocr/utils/dict/table_dict.txt --table_char_dict_path=../ppocr/utils/dict/table_structure_dict.txt --det_limit_side_len=736 --det_limit_type=min --gt_path=path/to/gt.json
 ```
 
 If the PubLatNet eval dataset is used, it will be output
@@ -113,4 +113,4 @@ After running, the excel sheet of each picture will be saved in the directory sp
 
 Reference
 1. https://github.com/ibm-aur-nlp/PubTabNet
-2. https://arxiv.org/pdf/1911.10683
+2. https://arxiv.org/pdf/1911.10683
\ No newline at end of file

ppstructure/table/README_ch.md

@@ -34,9 +34,9 @@
 
 
 |算法|[TEDS(Tree-Edit-Distance-based Similarity)](https://github.com/ibm-aur-nlp/PubTabNet/tree/master/src)|
-| --- | --- | 
-| EDD<sup>[2]</sup> | 88.3 | 
-| Ours | 93.32 | 
+| --- | --- |
+| EDD<sup>[2]</sup> | 88.3 |
+| Ours | 93.32 |
 
 <a name="3"></a>
 ## 3. 使用
@@ -56,7 +56,7 @@ wget https://paddleocr.bj.bcebos.com/dygraph_v2.0/table/en_ppocr_mobile_v2.0_tab
 wget https://paddleocr.bj.bcebos.com/dygraph_v2.0/table/en_ppocr_mobile_v2.0_table_structure_infer.tar && tar xf en_ppocr_mobile_v2.0_table_structure_infer.tar
 cd ..
 # 执行预测
-python3 table/predict_table.py --det_model_dir=inference/en_ppocr_mobile_v2.0_table_det_infer --rec_model_dir=inference/en_ppocr_mobile_v2.0_table_rec_infer --table_model_dir=inference/en_ppocr_mobile_v2.0_table_structure_infer --image_dir=../doc/table/table.jpg --rec_char_dict_path=../ppocr/utils/dict/table_dict.txt --table_char_dict_path=../ppocr/utils/dict/table_structure_dict.txt --rec_char_type=EN --det_limit_side_len=736 --det_limit_type=min --output ../output/table
+python3 table/predict_table.py --det_model_dir=inference/en_ppocr_mobile_v2.0_table_det_infer --rec_model_dir=inference/en_ppocr_mobile_v2.0_table_rec_infer --table_model_dir=inference/en_ppocr_mobile_v2.0_table_structure_infer --image_dir=../doc/table/table.jpg --rec_char_dict_path=../ppocr/utils/dict/table_dict.txt --table_char_dict_path=../ppocr/utils/dict/table_structure_dict.txt --rec_char_dict_path=../ppocr/utils/dict/en_dict.txt --det_limit_side_len=736 --det_limit_type=min --output ../output/table
 ```
 运行完成后，每张图片的excel表格会保存到output字段指定的目录下
 
@@ -94,8 +94,8 @@ python3 tools/train.py -c configs/table/table_mv3.yml -o Global.checkpoints=./yo
 表格使用 [TEDS(Tree-Edit-Distance-based Similarity)](https://github.com/ibm-aur-nlp/PubTabNet/tree/master/src) 作为模型的评估指标。在进行模型评估之前，需要将pipeline中的三个模型分别导出为inference模型(我们已经提供好)，还需要准备评估的gt， gt示例如下:
 ```json
 {"PMC4289340_004_00.png": [
-  ["<html>", "<body>", "<table>", "<thead>", "<tr>", "<td>", "</td>", "<td>", "</td>", "<td>", "</td>", "</tr>", "</thead>", "<tbody>", "<tr>", "<td>", "</td>", "<td>", "</td>", "<td>", "</td>", "</tr>",  "</tbody>", "</table>", "</body>", "</html>"], 
-  [[1, 4, 29, 13], [137, 4, 161, 13], [215, 4, 236, 13], [1, 17, 30, 27], [137, 17, 147, 27], [215, 17, 225, 27]], 
+  ["<html>", "<body>", "<table>", "<thead>", "<tr>", "<td>", "</td>", "<td>", "</td>", "<td>", "</td>", "</tr>", "</thead>", "<tbody>", "<tr>", "<td>", "</td>", "<td>", "</td>", "<td>", "</td>", "</tr>",  "</tbody>", "</table>", "</body>", "</html>"],
+  [[1, 4, 29, 13], [137, 4, 161, 13], [215, 4, 236, 13], [1, 17, 30, 27], [137, 17, 147, 27], [215, 17, 225, 27]],
   [["<b>", "F", "e", "a", "t", "u", "r", "e", "</b>"], ["<b>", "G", "b", "3", " ", "+", "</b>"], ["<b>", "G", "b", "3", " ", "-", "</b>"], ["<b>", "P", "a", "t", "i", "e", "n", "t", "s", "</b>"], ["6", "2"], ["4", "5"]]
 ]}
 ```
@@ -107,7 +107,7 @@ json 中，key为图片名，value为对应的gt，gt是一个由三个item组
 准备完成后使用如下命令进行评估，评估完成后会输出teds指标。
 ```python
 cd PaddleOCR/ppstructure
-python3 table/eval_table.py --det_model_dir=path/to/det_model_dir --rec_model_dir=path/to/rec_model_dir --table_model_dir=path/to/table_model_dir --image_dir=../doc/table/1.png --rec_char_dict_path=../ppocr/utils/dict/table_dict.txt --table_char_dict_path=../ppocr/utils/dict/table_structure_dict.txt --rec_char_type=EN --det_limit_side_len=736 --det_limit_type=min --gt_path=path/to/gt.json
+python3 table/eval_table.py --det_model_dir=path/to/det_model_dir --rec_model_dir=path/to/rec_model_dir --table_model_dir=path/to/table_model_dir --image_dir=../doc/table/1.png --rec_char_dict_path=../ppocr/utils/dict/table_dict.txt --table_char_dict_path=../ppocr/utils/dict/table_structure_dict.txt --det_limit_side_len=736 --det_limit_type=min --gt_path=path/to/gt.json
 ```
 如使用PubLatNet评估数据集，将会输出
 ```bash
@@ -123,4 +123,4 @@ python3 table/predict_table.py --det_model_dir=path/to/det_model_dir --rec_model
 
 Reference
 1. https://github.com/ibm-aur-nlp/PubTabNet
-2. https://arxiv.org/pdf/1911.10683
+2. https://arxiv.org/pdf/1911.10683
\ No newline at end of file

ppstructure/utility.py

@@ -21,13 +21,31 @@ def init_args():
     parser = infer_args()
 
     # params for output
-    parser.add_argument("--output", type=str, default='./output/table')
+    parser.add_argument("--output", type=str, default='./output')
     # params for table structure
     parser.add_argument("--table_max_len", type=int, default=488)
     parser.add_argument("--table_model_dir", type=str)
     parser.add_argument("--table_char_type", type=str, default='en')
-    parser.add_argument("--table_char_dict_path", type=str, default="../ppocr/utils/dict/table_structure_dict.txt")
-    parser.add_argument("--layout_path_model", type=str, default="lp://PubLayNet/ppyolov2_r50vd_dcn_365e_publaynet/config")
+    parser.add_argument(
+        "--table_char_dict_path",
+        type=str,
+        default="../ppocr/utils/dict/table_structure_dict.txt")
+    parser.add_argument(
+        "--layout_path_model",
+        type=str,
+        default="lp://PubLayNet/ppyolov2_r50vd_dcn_365e_publaynet/config")
+
+    # params for ser
+    parser.add_argument("--model_name_or_path", type=str)
+    parser.add_argument("--max_seq_length", type=int, default=512)
+    parser.add_argument(
+        "--label_map_path", type=str, default='./vqa/labels/labels_ser.txt')
+
+    parser.add_argument(
+        "--mode",
+        type=str,
+        default='structure',
+        help='structure and vqa is supported')
     return parser
 
 
@@ -48,5 +66,6 @@ def draw_structure_result(image, result, font_path):
                 boxes.append(np.array(box).reshape(-1, 2))
                 txts.append(rec_res[0])
                 scores.append(rec_res[1])
-    im_show = draw_ocr_box_txt(image, boxes, txts, scores, font_path=font_path,drop_score=0)
-    return im_show
+    im_show = draw_ocr_box_txt(
+        image, boxes, txts, scores, font_path=font_path, drop_score=0)
\ No newline at end of file
+    return im_show

ppstructure/vqa/README.md

+# 文档视觉问答（DOC-VQA）
+
+VQA指视觉问答，主要针对图像内容进行提问和回答,DOC-VQA是VQA任务中的一种，DOC-VQA主要针对文本图像的文字内容提出问题。
+
+PP-Structure 里的 DOC-VQA算法基于PaddleNLP自然语言处理算法库进行开发。
+
+主要特性如下：
+
+- 集成[LayoutXLM](https://arxiv.org/pdf/2104.08836.pdf)模型以及PP-OCR预测引擎。
+- 支持基于多模态方法的语义实体识别 (Semantic Entity Recognition, SER) 以及关系抽取 (Relation Extraction, RE) 任务。基于 SER 任务，可以完成对图像中的文本识别与分类；基于 RE 任务，可以完成对图象中的文本内容的关系提取，如判断问题对(pair)。
+- 支持SER任务和RE任务的自定义训练。
+- 支持OCR+SER的端到端系统预测与评估。
+- 支持OCR+SER+RE的端到端系统预测。
+
+
+本项目是 [LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich Document Understanding](https://arxiv.org/pdf/2104.08836.pdf) 在 Paddle 2.2上的开源实现，
+包含了在 [XFUND数据集](https://github.com/doc-analysis/XFUND) 上的微调代码。
+
+## 1 性能
+
+我们在 [XFUN](https://github.com/doc-analysis/XFUND) 评估数据集上对算法进行了评估，性能如下
+
+|任务|    f1 | 模型下载地址|
+|:---:|:---:| :---:|
+|SER|0.9056| [链接](https://paddleocr.bj.bcebos.com/pplayout/PP-Layout_v1.0_ser_pretrained.tar)|
+|RE|0.7113| [链接](https://paddleocr.bj.bcebos.com/pplayout/PP-Layout_v1.0_re_pretrained.tar)|
+
+
+
+## 2. 效果演示
+
+**注意：** 测试图片来源于XFUN数据集。
+
+### 2.1 SER
+
+![](./images/result_ser/zh_val_0_ser.jpg) | ![](./images/result_ser/zh_val_42_ser.jpg)
+---|---
+
+图中不同颜色的框表示不同的类别，对于XFUN数据集，有`QUESTION`, `ANSWER`, `HEADER` 3种类别
+
+* 深紫色：HEADER
+* 浅紫色：QUESTION
+* 军绿色：ANSWER
+
+在OCR检测框的左上方也标出了对应的类别和OCR识别结果。
+
+
+### 2.2 RE
+
+![](./images/result_re/zh_val_21_re.jpg) | ![](./images/result_re/zh_val_40_re.jpg)
+---|---
+
+
+图中红色框表示问题，蓝色框表示答案，问题和答案之间使用绿色线连接。在OCR检测框的左上方也标出了对应的类别和OCR识别结果。
+
+
+## 3. 安装
+
+### 3.1 安装依赖
+
+- **（1) 安装PaddlePaddle**
+
+```bash
+pip3 install --upgrade pip
+
+# GPU安装
+python3 -m pip install paddlepaddle-gpu==2.2 -i https://mirror.baidu.com/pypi/simple
+
+# CPU安装
+python3 -m pip install paddlepaddle==2.2 -i https://mirror.baidu.com/pypi/simple
+
+```
+更多需求，请参照[安装文档](https://www.paddlepaddle.org.cn/install/quick)中的说明进行操作。
+
+
+### 3.2 安装PaddleOCR（包含 PP-OCR 和 VQA ）
+
+- **（1）pip快速安装PaddleOCR whl包（仅预测）**
+
+```bash
+pip install paddleocr
+```
+
+- **（2）下载VQA源码（预测+训练）**
+
+```bash
+【推荐】git clone https://github.com/PaddlePaddle/PaddleOCR
+
+# 如果因为网络问题无法pull成功，也可选择使用码云上的托管：
+git clone https://gitee.com/paddlepaddle/PaddleOCR
+
+# 注：码云托管代码可能无法实时同步本github项目更新，存在3~5天延时，请优先使用推荐方式。
+```
+
+- **（3）安装PaddleNLP**
+
+```bash
+# 需要使用PaddleNLP最新的代码版本进行安装
+git clone https://github.com/PaddlePaddle/PaddleNLP -b develop
+cd PaddleNLP
+pip install -e .
+```
+
+
+- **（4）安装VQA的`requirements`**
+
+```bash
+cd ppstructure/vqa
+pip install -r requirements.txt
+```
+
+## 4. 使用
+
+
+### 4.1 数据和预训练模型准备
+
+处理好的XFUN中文数据集下载地址：[https://paddleocr.bj.bcebos.com/dataset/XFUND.tar](https://paddleocr.bj.bcebos.com/dataset/XFUND.tar)。
+
+
+下载并解压该数据集，解压后将数据集放置在当前目录下。
+
+```shell
+wget https://paddleocr.bj.bcebos.com/dataset/XFUND.tar
+```
+
+如果希望转换XFUN中其他语言的数据集，可以参考[XFUN数据转换脚本](helper/trans_xfun_data.py)。
+
+如果希望直接体验预测过程，可以下载我们提供的预训练模型，跳过训练过程，直接预测即可。
+
+
+### 4.2 SER任务
+
+* 启动训练
+
+```shell
+python3.7 train_ser.py \
+    --model_name_or_path "layoutxlm-base-uncased" \
+    --train_data_dir "XFUND/zh_train/image" \
+    --train_label_path "XFUND/zh_train/xfun_normalize_train.json" \
+    --eval_data_dir "XFUND/zh_val/image" \
+    --eval_label_path "XFUND/zh_val/xfun_normalize_val.json" \
+    --num_train_epochs 200 \
+    --eval_steps 10 \
+    --save_steps 500 \
+    --output_dir "./output/ser/" \
+    --learning_rate 5e-5 \
+    --warmup_steps 50 \
+    --evaluate_during_training \
+    --seed 2048
+```
+
+最终会打印出`precision`, `recall`, `f1`等指标，模型和训练日志会保存在`./output/ser/`文件夹中。
+
+* 使用评估集合中提供的OCR识别结果进行预测
+
+```shell
+export CUDA_VISIBLE_DEVICES=0
+python3.7 infer_ser.py \
+    --model_name_or_path "./PP-Layout_v1.0_ser_pretrained/" \
+    --output_dir "output_res/" \
+    --infer_imgs "XFUND/zh_val/image/" \
+    --ocr_json_path "XFUND/zh_val/xfun_normalize_val.json"
+```
+
+最终会在`output_res`目录下保存预测结果可视化图像以及预测结果文本文件，文件名为`infer_results.txt`。
+
+* 使用`OCR引擎 + SER`串联结果
+
+```shell
+export CUDA_VISIBLE_DEVICES=0
+python3.7 infer_ser_e2e.py \
+    --model_name_or_path "./output/PP-Layout_v1.0_ser_pretrained/" \
+    --max_seq_length 512 \
+    --output_dir "output_res_e2e/" \
+    --infer_imgs "images/input/zh_val_0.jpg"
+```
+
+* 对`OCR引擎 + SER`预测系统进行端到端评估
+
+```shell
+export CUDA_VISIBLE_DEVICES=0
+python3.7 helper/eval_with_label_end2end.py --gt_json_path XFUND/zh_val/xfun_normalize_val.json  --pred_json_path output_res/infer_results.txt
+```
+
+
+### 3.3 RE任务
+
+* 启动训练
+
+```shell
+python3 train_re.py \
+    --model_name_or_path "layoutxlm-base-uncased" \
+    --train_data_dir "XFUND/zh_train/image" \
+    --train_label_path "XFUND/zh_train/xfun_normalize_train.json" \
+    --eval_data_dir "XFUND/zh_val/image" \
+    --eval_label_path "XFUND/zh_val/xfun_normalize_val.json" \
+    --label_map_path 'labels/labels_ser.txt' \
+    --num_train_epochs 2 \
+    --eval_steps 10 \
+    --save_steps 500 \
+    --output_dir "output/re/"  \
+    --learning_rate 5e-5 \
+    --warmup_steps 50 \
+    --per_gpu_train_batch_size 8 \
+    --per_gpu_eval_batch_size 8 \
+    --evaluate_during_training \
+    --seed 2048
+
+```
+
+最终会打印出`precision`, `recall`, `f1`等指标，模型和训练日志会保存在`./output/re/`文件夹中。
+
+* 使用评估集合中提供的OCR识别结果进行预测
+
+```shell
+export CUDA_VISIBLE_DEVICES=0
+python3 infer_re.py \
+    --model_name_or_path "./PP-Layout_v1.0_re_pretrained/" \
+    --max_seq_length 512 \
+    --eval_data_dir "XFUND/zh_val/image" \
+    --eval_label_path "XFUND/zh_val/xfun_normalize_val.json" \
+    --label_map_path 'labels/labels_ser.txt' \
+    --output_dir "output_res"  \
+    --per_gpu_eval_batch_size 1 \
+    --seed 2048
+```
+
+最终会在`output_res`目录下保存预测结果可视化图像以及预测结果文本文件，文件名为`infer_results.txt`。
+
+* 使用`OCR引擎 + SER + RE`串联结果
+
+```shell
+export CUDA_VISIBLE_DEVICES=0
+# python3.7 infer_ser_re_e2e.py \
+    --model_name_or_path "./PP-Layout_v1.0_ser_pretrained/" \
+    --re_model_name_or_path "./PP-Layout_v1.0_re_pretrained/" \
+    --max_seq_length 512 \
+    --output_dir "output_ser_re_e2e_train/" \
+    --infer_imgs "images/input/zh_val_21.jpg"
+```
+
+## 参考链接
+
+- LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich Document Understanding, https://arxiv.org/pdf/2104.08836.pdf
+- microsoft/unilm/layoutxlm, https://github.com/microsoft/unilm/tree/master/layoutxlm
+- XFUND dataset, https://github.com/doc-analysis/XFUND

ppstructure/vqa/data_collator.py

+# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import paddle
+import numbers
+import numpy as np
+
+
+class DataCollator:
+    """
+    data batch
+    """
+
+    def __call__(self, batch):
+        data_dict = {}
+        to_tensor_keys = []
+        for sample in batch:
+            for k, v in sample.items():
+                if k not in data_dict:
+                    data_dict[k] = []
+                if isinstance(v, (np.ndarray, paddle.Tensor, numbers.Number)):
+                    if k not in to_tensor_keys:
+                        to_tensor_keys.append(k)
+                data_dict[k].append(v)
+        for k in to_tensor_keys:
+            data_dict[k] = paddle.to_tensor(data_dict[k])
+        return data_dict

ppstructure/vqa/helper/eval_with_label_end2end.py

+# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import os
+import re
+import sys
+# import Polygon
+import shapely
+from shapely.geometry import Polygon
+import numpy as np
+from collections import defaultdict
+import operator
+import editdistance
+import argparse
+import json
+import copy
+
+
+def parse_ser_results_fp(fp, fp_type="gt", ignore_background=True):
+    # img/zh_val_0.jpg        {
+    #     "height": 3508,
+    #     "width": 2480,
+    #     "ocr_info": [
+    #         {"text": "Maribyrnong", "label": "other", "bbox": [1958, 144, 2184, 198]},
+    #         {"text": "CITYCOUNCIL", "label": "other", "bbox": [2052, 183, 2171, 214]},
+    #     ]
+    assert fp_type in ["gt", "pred"]
+    key = "label" if fp_type == "gt" else "pred"
+    res_dict = dict()
+    with open(fp, "r") as fin:
+        lines = fin.readlines()
+
+    for _, line in enumerate(lines):
+        img_path, info = line.strip().split("\t")
+        # get key
+        image_name = os.path.basename(img_path)
+        res_dict[image_name] = []
+        # get infos
+        json_info = json.loads(info)
+        for single_ocr_info in json_info["ocr_info"]:
+            label = single_ocr_info[key].upper()
+            if label in ["O", "OTHERS", "OTHER"]:
+                label = "O"
+            if ignore_background and label == "O":
+                continue
+            single_ocr_info["label"] = label
+            res_dict[image_name].append(copy.deepcopy(single_ocr_info))
+    return res_dict
+
+
+def polygon_from_str(polygon_points):
+    """
+    Create a shapely polygon object from gt or dt line.
+    """
+    polygon_points = np.array(polygon_points).reshape(4, 2)
+    polygon = Polygon(polygon_points).convex_hull
+    return polygon
+
+
+def polygon_iou(poly1, poly2):
+    """
+    Intersection over union between two shapely polygons.
+    """
+    if not poly1.intersects(
+            poly2):  # this test is fast and can accelerate calculation
+        iou = 0
+    else:
+        try:
+            inter_area = poly1.intersection(poly2).area
+            union_area = poly1.area + poly2.area - inter_area
+            iou = float(inter_area) / union_area
+        except shapely.geos.TopologicalError:
+            # except Exception as e:
+            #     print(e)
+            print('shapely.geos.TopologicalError occured, iou set to 0')
+            iou = 0
+    return iou
+
+
+def ed(args, str1, str2):
+    if args.ignore_space:
+        str1 = str1.replace(" ", "")
+        str2 = str2.replace(" ", "")
+    if args.ignore_case:
+        str1 = str1.lower()
+        str2 = str2.lower()
+    return editdistance.eval(str1, str2)
+
+
+def convert_bbox_to_polygon(bbox):
+    """
+    bbox  : [x1, y1, x2, y2]
+    output: [[x1, y1], [x2, y2], [x3, y3], [x4, y4]]
+    """
+    xmin, ymin, xmax, ymax = bbox
+    poly = [[xmin, ymin], [xmax, ymin], [xmax, ymax], [xmin, ymax]]
+    return poly
+
+
+def eval_e2e(args):
+    # gt
+    gt_results = parse_ser_results_fp(args.gt_json_path, "gt",
+                                      args.ignore_background)
+    # pred
+    dt_results = parse_ser_results_fp(args.pred_json_path, "pred",
+                                      args.ignore_background)
+    assert set(gt_results.keys()) == set(dt_results.keys())
+
+    iou_thresh = args.iou_thres
+    num_gt_chars = 0
+    gt_count = 0
+    dt_count = 0
+    hit = 0
+    ed_sum = 0
+
+    for img_name in gt_results:
+        gt_info = gt_results[img_name]
+        gt_count += len(gt_info)
+
+        dt_info = dt_results[img_name]
+        dt_count += len(dt_info)
+
+        dt_match = [False] * len(dt_info)
+        gt_match = [False] * len(gt_info)
+
+        all_ious = defaultdict(tuple)
+        # gt: {text, label, bbox or poly}
+        for index_gt, gt in enumerate(gt_info):
+            if "poly" not in gt:
+                gt["poly"] = convert_bbox_to_polygon(gt["bbox"])
+            gt_poly = polygon_from_str(gt["poly"])
+            for index_dt, dt in enumerate(dt_info):
+                if "poly" not in dt:
+                    dt["poly"] = convert_bbox_to_polygon(dt["bbox"])
+                dt_poly = polygon_from_str(dt["poly"])
+                iou = polygon_iou(dt_poly, gt_poly)
+                if iou >= iou_thresh:
+                    all_ious[(index_gt, index_dt)] = iou
+        sorted_ious = sorted(
+            all_ious.items(), key=operator.itemgetter(1), reverse=True)
+        sorted_gt_dt_pairs = [item[0] for item in sorted_ious]
+
+        # matched gt and dt
+        for gt_dt_pair in sorted_gt_dt_pairs:
+            index_gt, index_dt = gt_dt_pair
+            if gt_match[index_gt] == False and dt_match[index_dt] == False:
+                gt_match[index_gt] = True
+                dt_match[index_dt] = True
+                # ocr rec results
+                gt_text = gt_info[index_gt]["text"]
+                dt_text = dt_info[index_dt]["text"]
+
+                # ser results
+                gt_label = gt_info[index_gt]["label"]
+                dt_label = dt_info[index_dt]["pred"]
+
+                if True:  # ignore_masks[index_gt] == '0':
+                    ed_sum += ed(args, gt_text, dt_text)
+                    num_gt_chars += len(gt_text)
+                    if gt_text == dt_text:
+                        if args.ignore_ser_prediction or gt_label == dt_label:
+                            hit += 1
+
+# unmatched dt
+        for tindex, dt_match_flag in enumerate(dt_match):
+            if dt_match_flag == False:
+                dt_text = dt_info[tindex]["text"]
+                gt_text = ""
+                ed_sum += ed(args, dt_text, gt_text)
+
+# unmatched gt
+        for tindex, gt_match_flag in enumerate(gt_match):
+            if gt_match_flag == False:
+                dt_text = ""
+                gt_text = gt_info[tindex]["text"]
+                ed_sum += ed(args, gt_text, dt_text)
+                num_gt_chars += len(gt_text)
+
+    eps = 1e-9
+    print("config: ", args)
+    print('hit, dt_count, gt_count', hit, dt_count, gt_count)
+    precision = hit / (dt_count + eps)
+    recall = hit / (gt_count + eps)
+    fmeasure = 2.0 * precision * recall / (precision + recall + eps)
+    avg_edit_dist_img = ed_sum / len(gt_results)
+    avg_edit_dist_field = ed_sum / (gt_count + eps)
+    character_acc = 1 - ed_sum / (num_gt_chars + eps)
+
+    print('character_acc: %.2f' % (character_acc * 100) + "%")
+    print('avg_edit_dist_field: %.2f' % (avg_edit_dist_field))
+    print('avg_edit_dist_img: %.2f' % (avg_edit_dist_img))
+    print('precision: %.2f' % (precision * 100) + "%")
+    print('recall: %.2f' % (recall * 100) + "%")
+    print('fmeasure: %.2f' % (fmeasure * 100) + "%")
+
+    return
+
+
+def parse_args():
+    """
+    """
+
+    def str2bool(v):
+        return v.lower() in ("true", "t", "1")
+
+    parser = argparse.ArgumentParser()
+    ## Required parameters
+    parser.add_argument(
+        "--gt_json_path",
+        default=None,
+        type=str,
+        required=True, )
+    parser.add_argument(
+        "--pred_json_path",
+        default=None,
+        type=str,
+        required=True, )
+
+    parser.add_argument("--iou_thres", default=0.5, type=float)
+
+    parser.add_argument(
+        "--ignore_case",
+        default=False,
+        type=str2bool,
+        help="whether to do lower case for the strs")
+
+    parser.add_argument(
+        "--ignore_space",
+        default=True,
+        type=str2bool,
+        help="whether to ignore space")
+
+    parser.add_argument(
+        "--ignore_background",
+        default=True,
+        type=str2bool,
+        help="whether to ignore other label")
+
+    parser.add_argument(
+        "--ignore_ser_prediction",
+        default=False,
+        type=str2bool,
+        help="whether to ignore ocr pred results")
+
+    args = parser.parse_args()
+    return args
+
+
+if __name__ == '__main__':
+    args = parse_args()
+    eval_e2e(args)

ppstructure/vqa/helper/trans_xfun_data.py

+# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import json
+
+
+def transfer_xfun_data(json_path=None, output_file=None):
+    with open(json_path, "r") as fin:
+        lines = fin.readlines()
+
+    json_info = json.loads(lines[0])
+    documents = json_info["documents"]
+    label_info = {}
+    with open(output_file, "w") as fout:
+        for idx, document in enumerate(documents):
+            img_info = document["img"]
+            document = document["document"]
+            image_path = img_info["fname"]
+
+            label_info["height"] = img_info["height"]
+            label_info["width"] = img_info["width"]
+
+            label_info["ocr_info"] = []
+
+            for doc in document:
+                label_info["ocr_info"].append({
+                    "text": doc["text"],
+                    "label": doc["label"],
+                    "bbox": doc["box"],
+                    "id": doc["id"],
+                    "linking": doc["linking"],
+                    "words": doc["words"]
+                })
+
+            fout.write(image_path + "\t" + json.dumps(
+                label_info, ensure_ascii=False) + "\n")
+
+    print("===ok====")
+
+
+transfer_xfun_data("./xfun/zh.val.json", "./xfun_normalize_val.json")

ppstructure/vqa/infer_re.py

+import os
+import sys
+
+__dir__ = os.path.dirname(os.path.abspath(__file__))
+sys.path.append(__dir__)
+sys.path.append(os.path.abspath(os.path.join(__dir__, '../..')))
+
+import random
+
+import cv2
+import matplotlib.pyplot as plt
+import numpy as np
+import paddle
+
+from paddlenlp.transformers import LayoutXLMTokenizer, LayoutXLMModel, LayoutXLMForRelationExtraction
+
+from xfun import XFUNDataset
+from utils import parse_args, get_bio_label_maps, draw_re_results
+from data_collator import DataCollator
+
+from ppocr.utils.logging import get_logger
+
+
+def infer(args):
+    os.makedirs(args.output_dir, exist_ok=True)
+    logger = get_logger()
+    label2id_map, id2label_map = get_bio_label_maps(args.label_map_path)
+    pad_token_label_id = paddle.nn.CrossEntropyLoss().ignore_index
+
+    tokenizer = LayoutXLMTokenizer.from_pretrained(args.model_name_or_path)
+
+    model = LayoutXLMForRelationExtraction.from_pretrained(
+        args.model_name_or_path)
+
+    eval_dataset = XFUNDataset(
+        tokenizer,
+        data_dir=args.eval_data_dir,
+        label_path=args.eval_label_path,
+        label2id_map=label2id_map,
+        img_size=(224, 224),
+        max_seq_len=args.max_seq_length,
+        pad_token_label_id=pad_token_label_id,
+        contains_re=True,
+        add_special_ids=False,
+        return_attention_mask=True,
+        load_mode='all')
+
+    eval_dataloader = paddle.io.DataLoader(
+        eval_dataset,
+        batch_size=args.per_gpu_eval_batch_size,
+        num_workers=8,
+        shuffle=False,
+        collate_fn=DataCollator())
+
+    # 读取gt的oct数据
+    ocr_info_list = load_ocr(args.eval_data_dir, args.eval_label_path)
+
+    for idx, batch in enumerate(eval_dataloader):
+        logger.info("[Infer] process: {}/{}".format(idx, len(eval_dataloader)))
+        with paddle.no_grad():
+            outputs = model(**batch)
+        pred_relations = outputs['pred_relations']
+
+        ocr_info = ocr_info_list[idx]
+        image_path = ocr_info['image_path']
+        ocr_info = ocr_info['ocr_info']
+
+        # 根据entity里的信息，做token解码后去过滤不要的ocr_info
+        ocr_info = filter_bg_by_txt(ocr_info, batch, tokenizer)
+
+        # 进行 relations 到 ocr信息的转换
+        result = []
+        used_tail_id = []
+        for relations in pred_relations:
+            for relation in relations:
+                if relation['tail_id'] in used_tail_id:
+                    continue
+                if relation['head_id'] not in ocr_info or relation[
+                        'tail_id'] not in ocr_info:
+                    continue
+                used_tail_id.append(relation['tail_id'])
+                ocr_info_head = ocr_info[relation['head_id']]
+                ocr_info_tail = ocr_info[relation['tail_id']]
+                result.append((ocr_info_head, ocr_info_tail))
+
+        img = cv2.imread(image_path)
+        img_show = draw_re_results(img, result)
+        save_path = os.path.join(args.output_dir, os.path.basename(image_path))
+        cv2.imwrite(save_path, img_show)
+
+
+def load_ocr(img_folder, json_path):
+    import json
+    d = []
+    with open(json_path, "r") as fin:
+        lines = fin.readlines()
+        for line in lines:
+            image_name, info_str = line.split("\t")
+            info_dict = json.loads(info_str)
+            info_dict['image_path'] = os.path.join(img_folder, image_name)
+            d.append(info_dict)
+    return d
+
+
+def filter_bg_by_txt(ocr_info, batch, tokenizer):
+    entities = batch['entities'][0]
+    input_ids = batch['input_ids'][0]
+
+    new_info_dict = {}
+    for i in range(len(entities['start'])):
+        entitie_head = entities['start'][i]
+        entitie_tail = entities['end'][i]
+        word_input_ids = input_ids[entitie_head:entitie_tail].numpy().tolist()
+        txt = tokenizer.convert_ids_to_tokens(word_input_ids)
+        txt = tokenizer.convert_tokens_to_string(txt)
+
+        for i, info in enumerate(ocr_info):
+            if info['text'] == txt:
+                new_info_dict[i] = info
+    return new_info_dict
+
+
+def post_process(pred_relations, ocr_info, img):
+    result = []
+    for relations in pred_relations:
+        for relation in relations:
+            ocr_info_head = ocr_info[relation['head_id']]
+            ocr_info_tail = ocr_info[relation['tail_id']]
+            result.append((ocr_info_head, ocr_info_tail))
+    return result
+
+
+def draw_re(result, image_path, output_folder):
+    img = cv2.imread(image_path)
+
+    from matplotlib import pyplot as plt
+    for ocr_info_head, ocr_info_tail in result:
+        cv2.rectangle(
+            img,
+            tuple(ocr_info_head['bbox'][:2]),
+            tuple(ocr_info_head['bbox'][2:]), (255, 0, 0),
+            thickness=2)
+        cv2.rectangle(
+            img,
+            tuple(ocr_info_tail['bbox'][:2]),
+            tuple(ocr_info_tail['bbox'][2:]), (0, 0, 255),
+            thickness=2)
+        center_p1 = [(ocr_info_head['bbox'][0] + ocr_info_head['bbox'][2]) // 2,
+                     (ocr_info_head['bbox'][1] + ocr_info_head['bbox'][3]) // 2]
+        center_p2 = [(ocr_info_tail['bbox'][0] + ocr_info_tail['bbox'][2]) // 2,
+                     (ocr_info_tail['bbox'][1] + ocr_info_tail['bbox'][3]) // 2]
+        cv2.line(
+            img, tuple(center_p1), tuple(center_p2), (0, 255, 0), thickness=2)
+    plt.imshow(img)
+    plt.savefig(
+        os.path.join(output_folder, os.path.basename(image_path)), dpi=600)
+    # plt.show()
+
+
+if __name__ == "__main__":
+    args = parse_args()
+    infer(args)

ppstructure/vqa/infer_ser.py

+# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import os
+import sys
+import json
+import cv2
+import numpy as np
+from copy import deepcopy
+
+import paddle
+
+# relative reference
+from utils import parse_args, get_image_file_list, draw_ser_results, get_bio_label_maps
+from paddlenlp.transformers import LayoutXLMModel, LayoutXLMTokenizer, LayoutXLMForTokenClassification
+
+
+def pad_sentences(tokenizer,
+                  encoded_inputs,
+                  max_seq_len=512,
+                  pad_to_max_seq_len=True,
+                  return_attention_mask=True,
+                  return_token_type_ids=True,
+                  return_overflowing_tokens=False,
+                  return_special_tokens_mask=False):
+    # Padding with larger size, reshape is carried out
+    max_seq_len = (
+        len(encoded_inputs["input_ids"]) // max_seq_len + 1) * max_seq_len
+
+    needs_to_be_padded = pad_to_max_seq_len and \
+                         max_seq_len and len(encoded_inputs["input_ids"]) < max_seq_len
+
+    if needs_to_be_padded:
+        difference = max_seq_len - len(encoded_inputs["input_ids"])
+        if tokenizer.padding_side == 'right':
+            if return_attention_mask:
+                encoded_inputs["attention_mask"] = [1] * len(encoded_inputs[
+                    "input_ids"]) + [0] * difference
+            if return_token_type_ids:
+                encoded_inputs["token_type_ids"] = (
+                    encoded_inputs["token_type_ids"] +
+                    [tokenizer.pad_token_type_id] * difference)
+            if return_special_tokens_mask:
+                encoded_inputs["special_tokens_mask"] = encoded_inputs[
+                    "special_tokens_mask"] + [1] * difference
+            encoded_inputs["input_ids"] = encoded_inputs[
+                "input_ids"] + [tokenizer.pad_token_id] * difference
+            encoded_inputs["bbox"] = encoded_inputs["bbox"] + [[0, 0, 0, 0]
+                                                               ] * difference
+        else:
+            assert False, f"padding_side of tokenizer just supports [\"right\"] but got {tokenizer.padding_side}"
+    else:
+        if return_attention_mask:
+            encoded_inputs["attention_mask"] = [1] * len(encoded_inputs[
+                "input_ids"])
+
+    return encoded_inputs
+
+
+def split_page(encoded_inputs, max_seq_len=512):
+    """
+    truncate is often used in training process
+    """
+    for key in encoded_inputs:
+        encoded_inputs[key] = paddle.to_tensor(encoded_inputs[key])
+        if encoded_inputs[key].ndim <= 1:  # for input_ids, att_mask and so on
+            encoded_inputs[key] = encoded_inputs[key].reshape([-1, max_seq_len])
+        else:  # for bbox
+            encoded_inputs[key] = encoded_inputs[key].reshape(
+                [-1, max_seq_len, 4])
+    return encoded_inputs
+
+
+def preprocess(
+        tokenizer,
+        ori_img,
+        ocr_info,
+        img_size=(224, 224),
+        pad_token_label_id=-100,
+        max_seq_len=512,
+        add_special_ids=False,
+        return_attention_mask=True, ):
+    ocr_info = deepcopy(ocr_info)
+    height = ori_img.shape[0]
+    width = ori_img.shape[1]
+
+    img = cv2.resize(ori_img,
+                     (224, 224)).transpose([2, 0, 1]).astype(np.float32)
+
+    segment_offset_id = []
+    words_list = []
+    bbox_list = []
+    input_ids_list = []
+    token_type_ids_list = []
+
+    for info in ocr_info:
+        # x1, y1, x2, y2
+        bbox = info["bbox"]
+        bbox[0] = int(bbox[0] * 1000.0 / width)
+        bbox[2] = int(bbox[2] * 1000.0 / width)
+        bbox[1] = int(bbox[1] * 1000.0 / height)
+        bbox[3] = int(bbox[3] * 1000.0 / height)
+
+        text = info["text"]
+        encode_res = tokenizer.encode(
+            text, pad_to_max_seq_len=False, return_attention_mask=True)
+
+        if not add_special_ids:
+            # TODO: use tok.all_special_ids to remove
+            encode_res["input_ids"] = encode_res["input_ids"][1:-1]
+            encode_res["token_type_ids"] = encode_res["token_type_ids"][1:-1]
+            encode_res["attention_mask"] = encode_res["attention_mask"][1:-1]
+
+        input_ids_list.extend(encode_res["input_ids"])
+        token_type_ids_list.extend(encode_res["token_type_ids"])
+        bbox_list.extend([bbox] * len(encode_res["input_ids"]))
+        words_list.append(text)
+        segment_offset_id.append(len(input_ids_list))
+
+    encoded_inputs = {
+        "input_ids": input_ids_list,
+        "token_type_ids": token_type_ids_list,
+        "bbox": bbox_list,
+        "attention_mask": [1] * len(input_ids_list),
+    }
+
+    encoded_inputs = pad_sentences(
+        tokenizer,
+        encoded_inputs,
+        max_seq_len=max_seq_len,
+        return_attention_mask=return_attention_mask)
+
+    encoded_inputs = split_page(encoded_inputs)
+
+    fake_bs = encoded_inputs["input_ids"].shape[0]
+
+    encoded_inputs["image"] = paddle.to_tensor(img).unsqueeze(0).expand(
+        [fake_bs] + list(img.shape))
+
+    encoded_inputs["segment_offset_id"] = segment_offset_id
+
+    return encoded_inputs
+
+
+def postprocess(attention_mask, preds, label_map_path):
+    if isinstance(preds, paddle.Tensor):
+        preds = preds.numpy()
+    preds = np.argmax(preds, axis=2)
+
+    _, label_map = get_bio_label_maps(label_map_path)
+
+    preds_list = [[] for _ in range(preds.shape[0])]
+
+    # keep batch info
+    for i in range(preds.shape[0]):
+        for j in range(preds.shape[1]):
+            if attention_mask[i][j] == 1:
+                preds_list[i].append(label_map[preds[i][j]])
+
+    return preds_list
+
+
+def merge_preds_list_with_ocr_info(label_map_path, ocr_info, segment_offset_id,
+                                   preds_list):
+    # must ensure the preds_list is generated from the same image
+    preds = [p for pred in preds_list for p in pred]
+    label2id_map, _ = get_bio_label_maps(label_map_path)
+    for key in label2id_map:
+        if key.startswith("I-"):
+            label2id_map[key] = label2id_map["B" + key[1:]]
+
+    id2label_map = dict()
+    for key in label2id_map:
+        val = label2id_map[key]
+        if key == "O":
+            id2label_map[val] = key
+        if key.startswith("B-") or key.startswith("I-"):
+            id2label_map[val] = key[2:]
+        else:
+            id2label_map[val] = key
+
+    for idx in range(len(segment_offset_id)):
+        if idx == 0:
+            start_id = 0
+        else:
+            start_id = segment_offset_id[idx - 1]
+
+        end_id = segment_offset_id[idx]
+
+        curr_pred = preds[start_id:end_id]
+        curr_pred = [label2id_map[p] for p in curr_pred]
+
+        if len(curr_pred) <= 0:
+            pred_id = 0
+        else:
+            counts = np.bincount(curr_pred)
+            pred_id = np.argmax(counts)
+        ocr_info[idx]["pred_id"] = int(pred_id)
+        ocr_info[idx]["pred"] = id2label_map[pred_id]
+    return ocr_info
+
+
+@paddle.no_grad()
+def infer(args):
+    os.makedirs(args.output_dir, exist_ok=True)
+
+    # init token and model
+    tokenizer = LayoutXLMTokenizer.from_pretrained(args.model_name_or_path)
+    # model = LayoutXLMModel.from_pretrained(args.model_name_or_path)
+    model = LayoutXLMForTokenClassification.from_pretrained(
+        args.model_name_or_path)
+    model.eval()
+
+    # load ocr results json
+    ocr_results = dict()
+    with open(args.ocr_json_path, "r") as fin:
+        lines = fin.readlines()
+        for line in lines:
+            img_name, json_info = line.split("\t")
+            ocr_results[os.path.basename(img_name)] = json.loads(json_info)
+
+    # get infer img list
+    infer_imgs = get_image_file_list(args.infer_imgs)
+
+    # loop for infer
+    with open(os.path.join(args.output_dir, "infer_results.txt"), "w") as fout:
+        for idx, img_path in enumerate(infer_imgs):
+            print("process: [{}/{}]".format(idx, len(infer_imgs), img_path))
+
+            img = cv2.imread(img_path)
+
+            ocr_info = ocr_results[os.path.basename(img_path)]["ocr_info"]
+            inputs = preprocess(
+                tokenizer=tokenizer,
+                ori_img=img,
+                ocr_info=ocr_info,
+                max_seq_len=args.max_seq_length)
+
+            outputs = model(
+                input_ids=inputs["input_ids"],
+                bbox=inputs["bbox"],
+                image=inputs["image"],
+                token_type_ids=inputs["token_type_ids"],
+                attention_mask=inputs["attention_mask"])
+
+            preds = outputs[0]
+            preds = postprocess(inputs["attention_mask"], preds,
+                                args.label_map_path)
+            ocr_info = merge_preds_list_with_ocr_info(
+                args.label_map_path, ocr_info, inputs["segment_offset_id"],
+                preds)
+
+            fout.write(img_path + "\t" + json.dumps(
+                {
+                    "ocr_info": ocr_info,
+                }, ensure_ascii=False) + "\n")
+
+            img_res = draw_ser_results(img, ocr_info)
+            cv2.imwrite(
+                os.path.join(args.output_dir, os.path.basename(img_path)),
+                img_res)
+
+    return
+
+
+if __name__ == "__main__":
+    args = parse_args()
+    infer(args)