README.md 3.87 KB
Newer Older
yangql's avatar
yangql committed
1
# BERT-ORT
yangql's avatar
yangql committed
2
3
4
## 论文
`Bidirectional Encoder Representation from Transformers`
- https://browse.arxiv.org/pdf/1810.04805.pdf
yangql's avatar
yangql committed
5
6
7
8

## 模型结构
以往的预训练模型的结构会受到单向语言模型(从左到右或者从右到左)的限制,因而也限制了模型的表征能力,使其只能获取单方向的上下文信息。而BERT利用MLM进行预训练并且采用深层的双向Transformer组件(单向的Transformer一般被称为Transformer decoder,其每一个token(符号)只会attend到目前往左的token。而双向的Transformer则被称为Transformer encoder,其每一个token会attend到所有的token)来构建整个模型,因此最终生成能融合左右上下文信息的深层双向语言表征。

yangql's avatar
yangql committed
9
10
11
## 算法原理
BERT的全称为Bidirectional Encoder Representation from Transformers,是一个预训练的语言表征模型。它强调了不再像以往一样采用传统的单向语言模型或者把两个单向语言模型进行浅层拼接的方法进行预训练,而是采用新的masked language model(MLM),以致能生成深度的双向语言表征。

yangql's avatar
yangql committed
12
13
## 环境配置

yangql's avatar
yangql committed
14
### Docker
yangql's avatar
yangql committed
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

拉取镜像:

```
docker pull image.sourcefind.cn:5000/dcu/admin/base/custom:ort1.14.0_migraphx3.0.0-dtk22.10.1
```

创建并启动容器,安装相关依赖:

```
docker run --shm-size 16g --network=host --name=bert_ort --privileged --device=/dev/kfd --device=/dev/dri --group-add video --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -v $PWD/bert_ort:/home/bert_ort -it <Your Image ID> /bin/bash

# 激活dtk
source /opt/dtk/env.sh
```
yangql's avatar
yangql committed
30
## 数据集
yangql's avatar
yangql committed
31
通过对模型提出问题,模型完成问题回答任务。
yangql's avatar
yangql committed
32

yangql's avatar
yangql committed
33
34
## 推理
### Python版本推理
yangql's avatar
yangql committed
35
本次采用经典的Bert模型完成问题回答任务,模型和分词文件下载链接:https://pan.baidu.com/s/1yc30IzM4ocOpTpfFuUMR0w, 提取码:8f1a, 将bertsquad-10.onnx文件和uncased_L-12_H-768_A-12分词文件保存在Resource/文件夹下。下面介绍如何运行python代码示例,Python示例的详细说明见Doc目录下的Tutorial_Python.md。
yangql's avatar
yangql committed
36
#### 设置Python环境变量
yangql's avatar
yangql committed
37
38
39
```
export PYTHONPATH=/opt/dtk/lib:$PYTHONPATH
```
yangql's avatar
yangql committed
40
#### 运行示例
yangql's avatar
yangql committed
41
42
43
44
45
46
47
48
49
```python
# 进入bert ort工程根目录
cd <path_to_bert_ort> 

# 进入示例程序目录
cd Python/

# 安装依赖
pip install -r requirements.txt
yangql's avatar
yangql committed
50
51
52

# 运行示例

yangql's avatar
yangql committed
53
54
python bert.py
```
yangql's avatar
yangql committed
55

yangql's avatar
yangql committed
56
### C++版本推理
yangql's avatar
yangql committed
57
本次采用经典的Bert模型完成问题回答任务,模型和分词文件下载链接:https://pan.baidu.com/s/1yc30IzM4ocOpTpfFuUMR0w, 提取码:8f1a, 将bertsquad-10.onnx文件和uncased_L-12_H-768_A-12分词文件保存在Resource/文件夹下。下面介绍如何运行C++代码示例,C++示例的详细说明见Doc目录下的Tutorial_Cpp.md。
yangql's avatar
yangql committed
58
#### 构建工程
yangql's avatar
yangql committed
59
60
61
```
rbuild build -d depend
```
yangql's avatar
yangql committed
62
#### 设置环境变量
yangql's avatar
yangql committed
63
64
65
66
67
68
69
70
71
将依赖库依赖加入环境变量LD_LIBRARY_PATH,在~/.bashrc中添加如下语句:
```
export LD_LIBRARY_PATH=<path_to_bert_ort>/depend/lib64/:$LD_LIBRARY_PATH
```
然后执行:
```
source ~/.bashrc
source /opt/dtk/env.sh
```
yangql's avatar
yangql committed
72
#### 运行示例
yangql's avatar
yangql committed
73
```c++
yangql's avatar
yangql committed
74
75
76
77
78
79
80
81
82
# 进入bert ort工程根目录
cd <path_to_bert_ort> 

# 进入build目录
cd build/

# 执行示例程序
./Bert
```
yangql's avatar
yangql committed
83
84
85
86
87
88
89
90
91
## result
### python版本
```
“1”:"open-source exascale-class platform for accelerated computing",
"2":"(Tensorflow / PyTorch)",
"3":"scale"
```

### C++版本
yangql's avatar
yangql committed
92
93
94
95
96
97
98
99
```
question:What is ROCm?
answer:open-source exascale-class platform for accelerated computing
question:Which frameworks does ROCmsupport?
answer:tensorflow / pytorch
question:What is ROCm built for?
answer:scale
```
yangql's avatar
yangql committed
100

yangql's avatar
yangql committed
101
102
## 应用场景
### 算法类别
yangql's avatar
yangql committed
103
`对话问答`
yangql's avatar
yangql committed
104

yangql's avatar
yangql committed
105
### 热点应用行业
yangql's avatar
yangql committed
106
`零售`,`医疗`,`教育`
yangql's avatar
yangql committed
107
108
109
110

## 源码仓库及问题反馈
https://developer.hpccube.com/codes/modelzoo/bert_ort

yangql's avatar
yangql committed
111
## 参考资料
yangql's avatar
yangql committed
112
https://github.com/ROCmSoftwarePlatform/onnxruntime/blob/81120e9e8b377567daa00d55614c902f35b2ae8f/onnxruntime/python/tools/transformers/onnx_model_bert.py