README.md 4.86 KB
Newer Older
liucong's avatar
liucong committed
1
# BERT
liucong's avatar
liucong committed
2

liucong's avatar
liucong committed
3
4
5
6
## 论文
Bidirectional Encoder Representation from Transformers

- https://browse.arxiv.org/pdf/1810.04805.pdf
liucong's avatar
liucong committed
7
8

## 模型结构
liucong's avatar
liucong committed
9
以往的预训练模型的结构会受到单向语言模型(从左到右或者从右到左)的限制,因而也限制了模型的表征能力,使其只能获取单方向的上下文信息。而BERT利用MLM进行预训练并且采用深层的双向Transformer组件(单向的Transformer一般被称为Transformer decoder,其每一个token(符号)只会attend到目前往左的token。而双向的Transformer则被称为Transformer encoder,其每一个token会attend到所有的token)来构建整个模型,因此最终生成能融合上下文信息的深层双向语言表征。
liucong's avatar
liucong committed
10

liucong's avatar
liucong committed
11
<img src="./Doc/Images/Bert_01.png" style="zoom:100%;" align=middle>
liucong's avatar
liucong committed
12

liucong's avatar
liucong committed
13
## 算法原理
liucong's avatar
liucong committed
14

liucong's avatar
liucong committed
15
BERT的全称为Bidirectional Encoder Representation from Transformers,是一个预训练的语言表征模型。它强调了不再像以往一样采用传统的单向语言模型或者把两个单向语言模型进行浅层拼接的方法进行预训练,而是采用新的masked language model(MLM),以致能生成深度的双向语言表征。具体方法通过将Token Embedding、Segment Embedding、Position Embedding输入到BERT模型中进行推理,并经过下图所示的数据后处理得到最终的推理结果。
liucong's avatar
liucong committed
16

liucong's avatar
liucong committed
17
<img src="./Doc/Images/Bert_04.png" style="zoom:100%;" align=middle>
liucong's avatar
liucong committed
18

liucong's avatar
liucong committed
19
20
## 环境配置

liucong's avatar
liucong committed
21
### Docker(方法一)
liucong's avatar
liucong committed
22
23
24
25

拉取镜像:

```plaintext
dcuai's avatar
dcuai committed
26
docker pull image.sourcefind.cn:5000/dcu/admin/base/migraphx:4.3.0-ubuntu20.04-dtk24.04.1-py3.8
liucong's avatar
liucong committed
27
28
```

liucong's avatar
liucong committed
29
创建并启动容器:
liucong's avatar
liucong committed
30
31

```plaintext
dcuai's avatar
dcuai committed
32
docker run --shm-size 16g --network=host --name=bert_migraphx -v /opt/hyhal:/opt/hyhal:ro --privileged --device=/dev/kfd --device=/dev/dri --group-add video --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -v $PWD/bert_migraphx:/home/bert_migraphx -it <Your Image ID> /bin/bash
liucong's avatar
liucong committed
33
34
35
36
37

# 激活dtk
source /opt/dtk/env.sh
```

liucong's avatar
liucong committed
38
39
40
41
### Dockerfile(方法二)

```
cd ./docker
dcuai's avatar
dcuai committed
42
docker build --no-cache -t bert_migraphx:4.3.0 .
liucong's avatar
liucong committed
43

dcuai's avatar
dcuai committed
44
docker run --shm-size 16g --network=host --name=bert_migraphx -v /opt/hyhal:/opt/hyhal:ro --privileged --device=/dev/kfd --device=/dev/dri --group-add video --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -v $PWD/bert_migraphx:/home/bert_migraphx -it <Your Image ID> /bin/bash
liucong's avatar
liucong committed
45
46
47

# 激活dtk
source /opt/dtk/env.sh
liucong's avatar
liucong committed
48
49
```

liucong's avatar
liucong committed
50
51
52
53
54
55
56
57
58
59
60
## 数据集

在界面中根据提示输入问题,模型预测出答案。

## 推理

### Python版本推理

本次采用经典的Bert模型完成问题回答任务,模型和分词文件下载链接:https://pan.baidu.com/s/1yc30IzM4ocOpTpfFuUMR0w, 提取码:8f1a, 将bertsquad-10.onnx文件和uncased_L-12_H-768_A-12分词文件保存在Resource/文件夹下。下面介绍如何运行python代码示例,Python示例的详细说明见Doc目录下的Tutorial_Python.md。

#### 设置环境变量
liucong's avatar
liucong committed
61

liucong's avatar
liucong committed
62
63
64
```
export PYTHONPATH=/opt/dtk/lib:$PYTHONPATH
```
liucong's avatar
liucong committed
65

liucong's avatar
liucong committed
66
#### 运行示例
liucong's avatar
liucong committed
67
68

```python
liucong's avatar
liucong committed
69
70
# 进入bert migraphx工程根目录
cd <path_to_bert_migraphx> 
liucong's avatar
liucong committed
71
72

# 进入示例程序目录
liucong's avatar
liucong committed
73
cd Python/
liucong's avatar
liucong committed
74
75

# 安装依赖
liucong's avatar
liucong committed
76
pip install -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple
liucong's avatar
liucong committed
77

liucong's avatar
liucong committed
78
# 运行示例
liucong's avatar
liucong committed
79
python bert.py
80
81
```

liucong's avatar
liucong committed
82
### C++版本推理
liucong's avatar
liucong committed
83

liucong's avatar
liucong committed
84
本次采用经典的Bert模型完成问题回答任务,模型和分词文件下载链接:https://pan.baidu.com/s/1yc30IzM4ocOpTpfFuUMR0w, 提取码:8f1a, 将bertsquad-10.onnx文件和uncased_L-12_H-768_A-12分词文件保存在Resource/文件夹下。下面介绍如何运行C++代码示例,C++示例的详细说明见Doc目录下的Tutorial_Cpp.md。
liucong's avatar
liucong committed
85
86


liucong's avatar
liucong committed
87
#### 构建工程
88
89
90
91
92

```
rbuild build -d depend
```

liucong's avatar
liucong committed
93
#### 设置环境变量
94
95
96
97

将依赖库依赖加入环境变量LD_LIBRARY_PATH,在~/.bashrc中添加如下语句:

```
liucong's avatar
liucong committed
98
export LD_LIBRARY_PATH=<path_to_bert_migraphx>/depend/lib/:$LD_LIBRARY_PATH
liucong's avatar
liucong committed
99
100
```

101
然后执行:
liucong's avatar
liucong committed
102

103
104
105
106
```
source ~/.bashrc
```

liucong's avatar
liucong committed
107
#### 运行示例
liucong's avatar
liucong committed
108

109
```python
liucong's avatar
liucong committed
110
111
# 进入bert migraphx工程根目录
cd <path_to_bert_migraphx> 
112

liucong's avatar
liucong committed
113
# 进入build目录
liucong's avatar
liucong committed
114
cd build/
115

liucong's avatar
liucong committed
116
117
# 执行示例程序
./Bert
118
119
```

liucong's avatar
liucong committed
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
## result

### python版本

```
“1”:"open-source exascale-class platform for accelerated computing",
"2":"(Tensorflow / PyTorch)",
"3":"scale"
```

上述序号代表对应的问题:

"1"代表“What is ROCm?”

“2”代表“Which frameworks does ROCm support?”

“3”代表"What is ROCm built for?"

### C++版本
liucong's avatar
liucong committed
139

liucong's avatar
liucong committed
140
141
142
```
question:What is ROCm?
answer:open-source exascale-class platform for accelerated computing
liucong's avatar
liucong committed
143
question:Which frameworks does ROCm support?
liucong's avatar
liucong committed
144
145
146
147
answer:tensorflow / pytorch
question:What is ROCm built for?
answer:scale
```
liucong's avatar
liucong committed
148

liucong's avatar
liucong committed
149
150
151
152
### 精度



liucong's avatar
liucong committed
153
154
155
156
157
158
159
160
## 应用场景

### 算法类别

`对话问答`

### 热点应用行业

chenzk's avatar
chenzk committed
161
162
163
`政府,科研,金融,教育`

## 预训练权重
liucong's avatar
liucong committed
164

liucong's avatar
liucong committed
165
## 源码仓库及问题反馈
liucong's avatar
liucong committed
166

chenzk's avatar
chenzk committed
167
https://developer.sourcefind.cn/codes/modelzoo/bert_migraphx
liucong's avatar
liucong committed
168

liucong's avatar
liucong committed
169
## 参考资料
liucong's avatar
liucong committed
170

dcuai's avatar
dcuai committed
171
https://github.com/ROCmSoftwarePlatform/AMDMIGraphX/tree/develop/examples/nlp/python_bert_squad