README.md 9.18 KB
Newer Older
MissPenguin's avatar
update  
MissPenguin committed
1
# 文档视觉问答(DocVQA)
littletomatodonkey's avatar
littletomatodonkey committed
2

MissPenguin's avatar
update  
MissPenguin committed
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
- [1. 简介](#1)
- [2. 性能](#2)
- [3. 效果演示](#3)
    - [3.1 SER](#31)
    - [3.2 RE](#32)
- [4. 安装](#4)
    - [4.1 安装依赖](#41)
    - [4.2 安装PaddleOCR](#42)
- [5. 使用](#5)
    - [5.1 数据和预训练模型准备](#51)
    - [5.2 SER](#52)
    - [5.3 RE](#53)
- [6. 参考链接](#6)



<a name="1"></a>
WenmuZhou's avatar
WenmuZhou committed
20
21
## 1. 简介

MissPenguin's avatar
update  
MissPenguin committed
22
VQA指视觉问答,主要针对图像内容进行提问和回答,DocVQA是VQA任务中的一种,DocVQA主要针对文本图像的文字内容提出问题。
WenmuZhou's avatar
add re  
WenmuZhou committed
23

MissPenguin's avatar
update  
MissPenguin committed
24
PP-Structure 里的DocVQA算法基于PaddleNLP自然语言处理算法库进行开发。
WenmuZhou's avatar
add re  
WenmuZhou committed
25
26

主要特性如下:
littletomatodonkey's avatar
littletomatodonkey committed
27
28

- 集成[LayoutXLM](https://arxiv.org/pdf/2104.08836.pdf)模型以及PP-OCR预测引擎。
WenmuZhou's avatar
WenmuZhou committed
29
30
- 支持基于多模态方法的语义实体识别 (Semantic Entity Recognition, SER) 以及关系抽取 (Relation Extraction, RE) 任务。基于 SER 任务,可以完成对图像中的文本识别与分类;基于 RE 任务,可以完成对图象中的文本内容的关系提取,如判断问题对(pair)。
- 支持SER任务和RE任务的自定义训练。
WenmuZhou's avatar
add re  
WenmuZhou committed
31
32
- 支持OCR+SER的端到端系统预测与评估。
- 支持OCR+SER+RE的端到端系统预测。
littletomatodonkey's avatar
littletomatodonkey committed
33
34
35
36
37


本项目是 [LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich Document Understanding](https://arxiv.org/pdf/2104.08836.pdf) 在 Paddle 2.2上的开源实现,
包含了在 [XFUND数据集](https://github.com/doc-analysis/XFUND) 上的微调代码。

MissPenguin's avatar
update  
MissPenguin committed
38
<a name="2"></a>
WenmuZhou's avatar
WenmuZhou committed
39
## 2. 性能
WenmuZhou's avatar
add re  
WenmuZhou committed
40

WenmuZhou's avatar
WenmuZhou committed
41
我们在 [XFUN](https://github.com/doc-analysis/XFUND) 的中文数据集上对算法进行了评估,性能如下
WenmuZhou's avatar
add re  
WenmuZhou committed
42

43
| 模型 | 任务 | hmean | 模型下载地址 |
WenmuZhou's avatar
WenmuZhou committed
44
|:---:|:---:|:---:| :---:|
45
| LayoutXLM | SER | 0.9038 | [链接](https://paddleocr.bj.bcebos.com/pplayout/ser_LayoutXLM_xfun_zh.tar) |
WenmuZhou's avatar
WenmuZhou committed
46
47
| LayoutXLM | RE | 0.7483 | [链接](https://paddleocr.bj.bcebos.com/pplayout/re_LayoutXLM_xfun_zh.tar) |
| LayoutLMv2 | SER | 0.8544 | [链接](https://paddleocr.bj.bcebos.com/pplayout/ser_LayoutLMv2_xfun_zh.tar)
WenmuZhou's avatar
WenmuZhou committed
48
| LayoutLMv2 | RE | 0.6777 | [链接](https://paddleocr.bj.bcebos.com/pplayout/re_LayoutLMv2_xfun_zh.tar) |
49
| LayoutLM | SER | 0.7731 | [链接](https://paddleocr.bj.bcebos.com/pplayout/ser_LayoutLM_xfun_zh.tar) |
WenmuZhou's avatar
add re  
WenmuZhou committed
50

MissPenguin's avatar
update  
MissPenguin committed
51
<a name="3"></a>
WenmuZhou's avatar
WenmuZhou committed
52
## 3. 效果演示
littletomatodonkey's avatar
littletomatodonkey committed
53
54
55

**注意:** 测试图片来源于XFUN数据集。

MissPenguin's avatar
update  
MissPenguin committed
56
<a name="31"></a>
WenmuZhou's avatar
WenmuZhou committed
57
### 3.1 SER
littletomatodonkey's avatar
littletomatodonkey committed
58

MissPenguin's avatar
update  
MissPenguin committed
59
![](../docs/vqa/result_ser/zh_val_0_ser.jpg) | ![](../docs/vqa/result_ser/zh_val_42_ser.jpg)
WenmuZhou's avatar
add re  
WenmuZhou committed
60
---|---
littletomatodonkey's avatar
littletomatodonkey committed
61

WenmuZhou's avatar
add re  
WenmuZhou committed
62
图中不同颜色的框表示不同的类别,对于XFUN数据集,有`QUESTION`, `ANSWER`, `HEADER` 3种类别
littletomatodonkey's avatar
littletomatodonkey committed
63

WenmuZhou's avatar
add re  
WenmuZhou committed
64
65
66
* 深紫色:HEADER
* 浅紫色:QUESTION
* 军绿色:ANSWER
littletomatodonkey's avatar
littletomatodonkey committed
67

WenmuZhou's avatar
add re  
WenmuZhou committed
68
在OCR检测框的左上方也标出了对应的类别和OCR识别结果。
littletomatodonkey's avatar
littletomatodonkey committed
69

MissPenguin's avatar
update  
MissPenguin committed
70
<a name="32"></a>
WenmuZhou's avatar
WenmuZhou committed
71
### 3.2 RE
littletomatodonkey's avatar
littletomatodonkey committed
72

MissPenguin's avatar
update  
MissPenguin committed
73
![](../docs/vqa/result_re/zh_val_21_re.jpg) | ![](../docs/vqa/result_re/zh_val_40_re.jpg)
WenmuZhou's avatar
add re  
WenmuZhou committed
74
---|---
littletomatodonkey's avatar
littletomatodonkey committed
75
76


WenmuZhou's avatar
add re  
WenmuZhou committed
77
图中红色框表示问题,蓝色框表示答案,问题和答案之间使用绿色线连接。在OCR检测框的左上方也标出了对应的类别和OCR识别结果。
littletomatodonkey's avatar
littletomatodonkey committed
78

MissPenguin's avatar
update  
MissPenguin committed
79
<a name="4"></a>
WenmuZhou's avatar
WenmuZhou committed
80
## 4. 安装
WenmuZhou's avatar
add re  
WenmuZhou committed
81

MissPenguin's avatar
update  
MissPenguin committed
82
<a name="41"></a>
WenmuZhou's avatar
WenmuZhou committed
83
### 4.1 安装依赖
littletomatodonkey's avatar
littletomatodonkey committed
84
85
86
87

- **(1) 安装PaddlePaddle**

```bash
WenmuZhou's avatar
WenmuZhou committed
88
python3 -m pip install --upgrade pip
littletomatodonkey's avatar
littletomatodonkey committed
89
90

# GPU安装
91
python3 -m pip install "paddlepaddle-gpu>=2.2" -i https://mirror.baidu.com/pypi/simple
littletomatodonkey's avatar
littletomatodonkey committed
92
93

# CPU安装
94
python3 -m pip install "paddlepaddle>=2.2" -i https://mirror.baidu.com/pypi/simple
littletomatodonkey's avatar
littletomatodonkey committed
95
96
97
98

```
更多需求,请参照[安装文档](https://www.paddlepaddle.org.cn/install/quick)中的说明进行操作。

MissPenguin's avatar
update  
MissPenguin committed
99
100
<a name="42"></a>
### 4.2 安装PaddleOCR
littletomatodonkey's avatar
littletomatodonkey committed
101
102
103
104

- **(1)pip快速安装PaddleOCR whl包(仅预测)**

```bash
105
python3 -m pip install paddleocr
littletomatodonkey's avatar
littletomatodonkey committed
106
107
```

littletomatodonkey's avatar
littletomatodonkey committed
108
- **(2)下载VQA源码(预测+训练)**
littletomatodonkey's avatar
littletomatodonkey committed
109
110
111
112
113
114
115
116
117
118

```bash
【推荐】git clone https://github.com/PaddlePaddle/PaddleOCR

# 如果因为网络问题无法pull成功,也可选择使用码云上的托管:
git clone https://gitee.com/paddlepaddle/PaddleOCR

# 注:码云托管代码可能无法实时同步本github项目更新,存在3~5天延时,请优先使用推荐方式。
```

119
- **(3)安装VQA的`requirements`**
littletomatodonkey's avatar
littletomatodonkey committed
120
121

```bash
122
python3 -m pip install -r ppstructure/vqa/requirements.txt
littletomatodonkey's avatar
littletomatodonkey committed
123
124
```

MissPenguin's avatar
update  
MissPenguin committed
125
<a name="5"></a>
WenmuZhou's avatar
WenmuZhou committed
126
## 5. 使用
littletomatodonkey's avatar
littletomatodonkey committed
127

MissPenguin's avatar
update  
MissPenguin committed
128
<a name="51"></a>
WenmuZhou's avatar
WenmuZhou committed
129
### 5.1 数据和预训练模型准备
littletomatodonkey's avatar
littletomatodonkey committed
130

131
132
133
134
如果希望直接体验预测过程,可以下载我们提供的预训练模型,跳过训练过程,直接预测即可。

* 下载处理好的数据集

littletomatodonkey's avatar
littletomatodonkey committed
135
136
137
138
139
140
141
142
143
处理好的XFUN中文数据集下载地址:[https://paddleocr.bj.bcebos.com/dataset/XFUND.tar](https://paddleocr.bj.bcebos.com/dataset/XFUND.tar)


下载并解压该数据集,解压后将数据集放置在当前目录下。

```shell
wget https://paddleocr.bj.bcebos.com/dataset/XFUND.tar
```

144
* 转换数据集
littletomatodonkey's avatar
littletomatodonkey committed
145

146
若需进行其他XFUN数据集的训练,可使用下面的命令进行数据集的转换
littletomatodonkey's avatar
littletomatodonkey committed
147

148
149
150
```bash
python3 ppstructure/vqa/helper/trans_xfun_data.py --ori_gt_path=path/to/json_path --output_path=path/to/save_path
```
littletomatodonkey's avatar
littletomatodonkey committed
151

MissPenguin's avatar
update  
MissPenguin committed
152
<a name="52"></a>
WenmuZhou's avatar
WenmuZhou committed
153
### 5.2 SER
littletomatodonkey's avatar
littletomatodonkey committed
154

155
156
157
158
159
160
启动训练之前,需要修改下面的四个字段

1. `Train.dataset.data_dir`:指向训练集图片存放目录
2. `Train.dataset.label_file_list`:指向训练集标注文件
3. `Eval.dataset.data_dir`:指指向验证集图片存放目录
4. `Eval.dataset.label_file_list`:指向验证集标注文件
littletomatodonkey's avatar
littletomatodonkey committed
161

162
* 启动训练
littletomatodonkey's avatar
littletomatodonkey committed
163
```shell
164
CUDA_VISIBLE_DEVICES=0 python3 tools/train.py -c configs/vqa/ser/layoutxlm.yml
littletomatodonkey's avatar
littletomatodonkey committed
165
166
```

167
168
最终会打印出`precision`, `recall`, `hmean`等指标。
`./output/ser_layoutxlm/`文件夹中会保存训练日志,最优的模型和最新epoch的模型。
littletomatodonkey's avatar
littletomatodonkey committed
169

zhoujun's avatar
zhoujun committed
170
171
* 恢复训练

172
173
恢复训练需要将之前训练好的模型所在文件夹路径赋值给 `Architecture.Backbone.checkpoints` 字段。

zhoujun's avatar
zhoujun committed
174
```shell
175
CUDA_VISIBLE_DEVICES=0 python3 tools/train.py -c configs/vqa/ser/layoutxlm.yml -o Architecture.Backbone.checkpoints=path/to/model_dir
zhoujun's avatar
zhoujun committed
176
177
178
179
```

* 评估

180
评估需要将待评估的模型所在文件夹路径赋值给 `Architecture.Backbone.checkpoints` 字段。
littletomatodonkey's avatar
littletomatodonkey committed
181
182

```shell
183
CUDA_VISIBLE_DEVICES=0 python3 tools/eval.py -c configs/vqa/ser/layoutxlm.yml -o Architecture.Backbone.checkpoints=path/to/model_dir
littletomatodonkey's avatar
littletomatodonkey committed
184
```
185
最终会打印出`precision`, `recall`, `hmean`等指标
littletomatodonkey's avatar
littletomatodonkey committed
186

187
* 使用`OCR引擎 + SER`串联预测
littletomatodonkey's avatar
littletomatodonkey committed
188

189
使用如下命令即可完成`OCR引擎 + SER`的串联预测
littletomatodonkey's avatar
littletomatodonkey committed
190
191

```shell
WenmuZhou's avatar
WenmuZhou committed
192
CUDA_VISIBLE_DEVICES=0 python3 tools/infer_vqa_token_ser.py -c configs/vqa/ser/layoutxlm.yml  -o Architecture.Backbone.checkpoints=ser_LayoutXLM_xfun_zh/ Global.infer_img=doc/vqa/input/zh_val_42.jpg
littletomatodonkey's avatar
littletomatodonkey committed
193
194
```

195
196
最终会在`config.Global.save_res_path`字段所配置的目录下保存预测结果可视化图像以及预测结果文本文件,预测结果文本文件名为`infer_results.txt`

littletomatodonkey's avatar
littletomatodonkey committed
197
198
*`OCR引擎 + SER`预测系统进行端到端评估

199
200
首先使用 `tools/infer_vqa_token_ser.py` 脚本完成数据集的预测,然后使用下面的命令进行评估。

littletomatodonkey's avatar
littletomatodonkey committed
201
202
```shell
export CUDA_VISIBLE_DEVICES=0
WenmuZhou's avatar
WenmuZhou committed
203
python3 helper/eval_with_label_end2end.py --gt_json_path XFUND/zh_val/xfun_normalize_val.json  --pred_json_path output_res/infer_results.txt
littletomatodonkey's avatar
littletomatodonkey committed
204
205
```

MissPenguin's avatar
update  
MissPenguin committed
206
<a name="53"></a>
WenmuZhou's avatar
WenmuZhou committed
207
### 5.3 RE
littletomatodonkey's avatar
littletomatodonkey committed
208

WenmuZhou's avatar
add re  
WenmuZhou committed
209
* 启动训练
littletomatodonkey's avatar
littletomatodonkey committed
210

211
启动训练之前,需要修改下面的四个字段
WenmuZhou's avatar
add re  
WenmuZhou committed
212

213
214
215
216
1. `Train.dataset.data_dir`:指向训练集图片存放目录
2. `Train.dataset.label_file_list`:指向训练集标注文件
3. `Eval.dataset.data_dir`:指指向验证集图片存放目录
4. `Eval.dataset.label_file_list`:指向验证集标注文件
zhoujun's avatar
zhoujun committed
217
218

```shell
219
CUDA_VISIBLE_DEVICES=0 python3 tools/train.py -c configs/vqa/re/layoutxlm.yml
zhoujun's avatar
zhoujun committed
220
221
```

222
223
224
225
226
227
最终会打印出`precision`, `recall`, `hmean`等指标。
`./output/re_layoutxlm/`文件夹中会保存训练日志,最优的模型和最新epoch的模型。

* 恢复训练

恢复训练需要将之前训练好的模型所在文件夹路径赋值给 `Architecture.Backbone.checkpoints` 字段。
WenmuZhou's avatar
add re  
WenmuZhou committed
228

zhoujun's avatar
zhoujun committed
229
```shell
230
CUDA_VISIBLE_DEVICES=0 python3 tools/train.py -c configs/vqa/re/layoutxlm.yml -o Architecture.Backbone.checkpoints=path/to/model_dir
zhoujun's avatar
zhoujun committed
231
232
```

233
* 评估
zhoujun's avatar
zhoujun committed
234

235
评估需要将待评估的模型所在文件夹路径赋值给 `Architecture.Backbone.checkpoints` 字段。
WenmuZhou's avatar
add re  
WenmuZhou committed
236
237

```shell
238
CUDA_VISIBLE_DEVICES=0 python3 tools/eval.py -c configs/vqa/re/layoutxlm.yml -o Architecture.Backbone.checkpoints=path/to/model_dir
WenmuZhou's avatar
add re  
WenmuZhou committed
239
```
240
最终会打印出`precision`, `recall`, `hmean`等指标
WenmuZhou's avatar
add re  
WenmuZhou committed
241

242
* 使用`OCR引擎 + SER + RE`串联预测
WenmuZhou's avatar
add re  
WenmuZhou committed
243

244
使用如下命令即可完成`OCR引擎 + SER + RE`的串联预测
WenmuZhou's avatar
add re  
WenmuZhou committed
245
246
```shell
export CUDA_VISIBLE_DEVICES=0
WenmuZhou's avatar
WenmuZhou committed
247
python3 tools/infer_vqa_token_ser_re.py -c configs/vqa/re/layoutxlm.yml -o Architecture.Backbone.checkpoints=re_LayoutXLM_xfun_zh/ Global.infer_img=doc/vqa/input/zh_val_21.jpg -c_ser configs/vqa/ser/layoutxlm.yml -o_ser Architecture.Backbone.checkpoints=ser_LayoutXLM_xfun_zh/
WenmuZhou's avatar
add re  
WenmuZhou committed
248
```
littletomatodonkey's avatar
littletomatodonkey committed
249

250
251
最终会在`config.Global.save_res_path`字段所配置的目录下保存预测结果可视化图像以及预测结果文本文件,预测结果文本文件名为`infer_results.txt`

MissPenguin's avatar
update  
MissPenguin committed
252
<a name="6"></a>
WenmuZhou's avatar
WenmuZhou committed
253
## 6. 参考链接
littletomatodonkey's avatar
littletomatodonkey committed
254
255
256
257

- LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich Document Understanding, https://arxiv.org/pdf/2104.08836.pdf
- microsoft/unilm/layoutxlm, https://github.com/microsoft/unilm/tree/master/layoutxlm
- XFUND dataset, https://github.com/doc-analysis/XFUND
MissPenguin's avatar
MissPenguin committed
258
259
260
261

## License

The content of this project itself is licensed under the [Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0)](https://creativecommons.org/licenses/by-nc-sa/4.0/)