README.md 4.37 KB
Newer Older
zhougaofeng's avatar
zhougaofeng committed
1
2
# magic_pdf

zhougaofeng's avatar
zhougaofeng committed
3
## 一、安装pdf文档解析
zhougaofeng's avatar
zhougaofeng committed
4

zhougaofeng's avatar
zhougaofeng committed
5
### 以下演示在223节点安装pdf解析模块(可以直接使用镜像:1177ea7959ce)
zhougaofeng's avatar
zhougaofeng committed
6

zhougaofeng's avatar
zhougaofeng committed
7
### 1、下载本项目
zhougaofeng's avatar
zhougaofeng committed
8

zhougaofeng's avatar
zhougaofeng committed
9
`git clone http://developer.sourcefind.cn/codes/zhiAn123/magic_pdf.git`
zhougaofeng's avatar
zhougaofeng committed
10

zhougaofeng's avatar
zhougaofeng committed
11
12
13
#### 安装依赖包

【提供的镜像暂未支持doclayout-YoLo模型,如需使用,请下载最新的库文件】
zhougaofeng's avatar
zhougaofeng committed
14
15

`pip install -U magic-pdf[full] --extra-index-url https://wheels.myhloli.com -i https://mirrors.aliyun.com/pypi/simple`
zhougaofeng's avatar
zhougaofeng committed
16
17
18

安装完毕后,请卸载需要使用光源库文件(torch、vllm等),以及nvdia-cuda类似不支持的库文件

zhougaofeng's avatar
zhougaofeng committed
19
### 2、下载需要的模型库
zhougaofeng's avatar
zhougaofeng committed
20

zhougaofeng's avatar
zhougaofeng committed
21

zhougaofeng's avatar
zhougaofeng committed
22
23
下载qwen模型:[快速下载通道](http://113.200.138.88:18080/aimodels/qwen/Qwen2-VL-7B-Instruct.git)

zhougaofeng's avatar
zhougaofeng committed
24
下载PDF解析需要的模型: [快速下载通道](http://113.200.138.88:18080/aimodels/opendatalab/PDF-Extract-Kit)
zhougaofeng's avatar
zhougaofeng committed
25

zhougaofeng's avatar
zhougaofeng committed
26
27
`pip install modelscope`

zhougaofeng's avatar
zhougaofeng committed
28
`wget https://gitee.com/myhloli/MinerU/raw/master/scripts/download_models.py -O download_models.py`
zhougaofeng's avatar
zhougaofeng committed
29

zhougaofeng's avatar
zhougaofeng committed
30
`python download_models.py`
zhougaofeng's avatar
zhougaofeng committed
31

zhougaofeng's avatar
zhougaofeng committed
32
【注意,download_models.py执行完毕,会把模型文件以及layout文件安装在/root/.cache下】
zhougaofeng's avatar
zhougaofeng committed
33

zhougaofeng's avatar
zhougaofeng committed
34
### 3、安装需要的依赖库
zhougaofeng's avatar
zhougaofeng committed
35

zhougaofeng's avatar
zhougaofeng committed
36
#### 进入主目录(以下内容都在主目录下进行)
zhougaofeng's avatar
zhougaofeng committed
37

zhougaofeng's avatar
zhougaofeng committed
38
`cd magic_pdf`
zhougaofeng's avatar
zhougaofeng committed
39

zhougaofeng's avatar
zhougaofeng committed
40
执行本地源码安装
zhougaofeng's avatar
zhougaofeng committed
41

zhougaofeng's avatar
zhougaofeng committed
42
#### pip install -e .
zhougaofeng's avatar
zhougaofeng committed
43

zhougaofeng's avatar
zhougaofeng committed
44
45
在国产计算卡运行时,下载已适配的[paddle](https://cancon.hpccube.com:65024/4/main/paddle)可以大幅提高文本解析速度(根据自身容器,自行选择已适配paddle的版本)

zhougaofeng's avatar
zhougaofeng committed
46
47
### 4、修改magic-pdf.json
在第二步执行download_models.py文件后,会在/root文件夹下,下载名为magic-pdf.json文件,修改文件内容
zhougaofeng's avatar
zhougaofeng committed
48

zhougaofeng's avatar
zhougaofeng committed
49
50
51
<div align=center>
    <img src="doc/image (9).png"/>
</div>
zhougaofeng's avatar
zhougaofeng committed
52

zhougaofeng's avatar
zhougaofeng committed
53
"models-dir":"[模型路径]" 路径指向**第二步下载的pdf解析模型路径下的models文件夹**
zhougaofeng's avatar
zhougaofeng committed
54

zhougaofeng's avatar
zhougaofeng committed
55
56
57
<div align=center>
    <img src="doc/image (10).png"/>
</div>
zhougaofeng's avatar
zhougaofeng committed
58

59
60
61
62
63
64
### 5、配置config.ini中的路由地址

vim magic_pdf/config.ini

默认如下:

65
66
67
<div align=center>
    <img src="doc/image13.png"/>
</div>
68
69
70
71

根据需要,自行配置路由地址

### 6、启动qwen-ocr模块:
zhougaofeng's avatar
zhougaofeng committed
72

zhougaofeng's avatar
zhougaofeng committed
73
74
#### 6.1、启动非vllm的qwen-ocr模块:

zhougaofeng's avatar
zhougaofeng committed
75
修改magic_pdf/magic_pdf/dict2md/ocr_server.py文件中模型路径地址
zhougaofeng's avatar
zhougaofeng committed
76
77
78
79

<div align=center>
    <img src="doc/image11.png"/>
</div>
zhougaofeng's avatar
zhougaofeng committed
80

zhougaofeng's avatar
zhougaofeng committed
81
##### qwen-ocr服务启动代码:
zhougaofeng's avatar
zhougaofeng committed
82

zhougaofeng's avatar
zhougaofeng committed
83
`python magic_pdf/dict2md/ocr_server.py`
zhougaofeng's avatar
zhougaofeng committed
84

85
默认使用0号DCU卡 ,可以通过--dcu_id 指定卡,-c 指定qwen模型地址,--config_path 指定config.ini路径
zhougaofeng's avatar
zhougaofeng committed
86

zhougaofeng's avatar
zhougaofeng committed
87
88
89
90
91
qwen-ocr模块启动成功:
<div align=center>
    <img src="doc/image (5).png"/>
</div>

zhougaofeng's avatar
zhougaofeng committed
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
#### 6.2、 启动vllm的qwen-ocr模块:

1、修改magic_pdf/magic_pdf/dict2md/ocr_vllm_server.py文件中模型路径地址
2、修改magic_pdf/magic_pdf/pre_proc/cut_image.py、magic_pdf/magic_pdf/tools/pdf_server.py、magic_pdf/magic_pdf/parse/ofd_parse.py文件
将注释更改如下
<div align=center>
    <img src="doc/image_vllm.png"/>
</div>

##### qwen-ocr-vllm服务启动代码:

`CUDA_VISIBLE_DEVICES=0 python magic_pdf/dict2md/ocr_vllm_server.py`

默认使用0号DCU卡,-c 指定qwen模型地址,--config_path 指定config.ini路径 


qwen-ocr模块启动成功:
<div align=center>
    <img src="doc/image (5).png"/>
</div>


114
### 7、启动pdf-server解析服务:
zhougaofeng's avatar
zhougaofeng committed
115
116
117

#### pdf-server解析服务启动代码:

zhougaofeng's avatar
zhougaofeng committed
118
`python magic_pdf/tools/pdf_server.py`
zhougaofeng's avatar
zhougaofeng committed
119

120
默认使用0号DCU卡 ,可以通过--dcu_id 指定卡,--config_path 指定config.ini路径
zhougaofeng's avatar
zhougaofeng committed
121

zhougaofeng's avatar
zhougaofeng committed
122
123
124
<div align=center>
    <img src="doc/image (6).png"/>
</div>
zhougaofeng's avatar
zhougaofeng committed
125

zhougaofeng's avatar
zhougaofeng committed
126
127
128
129
启动成功:
<div align=center>
    <img src="doc/image (7).png"/>
</div>
zhougaofeng's avatar
zhougaofeng committed
130

131
132
### 8、解析pdf

zhougaofeng's avatar
zhougaofeng committed
133
`python magic_pdf/parse/common_parse.py -p [文件/目录 路径] -o [输出地址]`
134
135
136
137
138
139
140

-p指定pdf路径,-o指定输出路径 --config_path 指定config.ini路径 

<div align=center>
    <img src="doc/image12.png"/>
</div>

zhougaofeng's avatar
zhougaofeng committed
141
142
143
<div align=center>
    <img src="doc/image (8).png"/>
</div>
144

zhougaofeng's avatar
zhougaofeng committed
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
## 二、安装ofd文档解析

`1、pip install easyofd`

2、安装需要的字体文件

查看容器内的字体:

`fc-list`

请配置magic_pdf/tools/font_tools.py文件,配置字体路径

<div align=center>
    <img src="doc/font.png"/>
</div>

zhougaofeng's avatar
zhougaofeng committed
161
162
163
164
若代码执行过程中报错,字体未找到
<div align=center>
    <img src="doc/fonts.png"/>
</div>
zhougaofeng's avatar
zhougaofeng committed
165

zhougaofeng's avatar
zhougaofeng committed
166
请执行:
zhougaofeng's avatar
zhougaofeng committed
167

zhougaofeng's avatar
zhougaofeng committed
168
169
170
171
`fc-list #查找字体文件`

`vim magic_pdf/tools/font_tools.py`

zhougaofeng's avatar
zhougaofeng committed
172
173
如果fc-list未找到需要的字体文件,请向容器内添加字体后,再修改font_tools.py文件

zhougaofeng's avatar
zhougaofeng committed
174
175
176
177




178