README.md 4.46 KB
Newer Older
1
2
# Generative Pre-Training2(GPT2)

liucong's avatar
liucong committed
3
4
5
6
## 论文
Language Models are Unsupervised Multitask Learners

- https://d4mucfpksywv.cloudfront.net/better-language-models/language-models.pdf
7
8

## 模型结构
liucong's avatar
liucong committed
9
第二代生成式预训练模型(Generative Pre-Training2),GPT2主要使用Transformer的Decoder模块为特征提取器,并对Transformer Decoder进行了一些改动,原本的Decoder包含了两个Multi-Head Attention结构,而GPT2只保留了Mask Multi-Head Attention。
10

liucong's avatar
liucong committed
11
<img src="./Doc/Images/GPT_03.png" style="zoom:55%;" align=middle>
liucong's avatar
liucong committed
12

liucong's avatar
liucong committed
13
## 算法原理
liucong's avatar
liucong committed
14

liucong's avatar
liucong committed
15
GPT-2中使用了掩模自注意力(masked self-attention),通过屏蔽当前位置的右边token,使模型可以更好的预测下一个token。
16

liucong's avatar
liucong committed
17
<img src="./Doc/Images/GPT_04.png" style="zoom:70%;" align=middle>
18

liucong's avatar
liucong committed
19
20
21
22
23
24
25
## 环境配置

### Docker(方法一)

拉取镜像:

```
liucong's avatar
liucong committed
26
docker pull sugonhub/migraphx:3.2.1-centos7.6-dtk-23.04.1-py38
27
28
```

liucong's avatar
liucong committed
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
创建并启动容器:

```
docker run --shm-size 16g --network=host --name=gpt2_migraphx --privileged --device=/dev/kfd --device=/dev/dri --group-add video --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -v $PWD/gpt2_migraphx:/home/gpt2_migraphx -it <Your Image ID> /bin/bash
```

### Dockerfile(方法二)

```
cd ./docker
docker build --no-cache -t gpt2_migraphx:2.0 .

docker run --shm-size 16g --network=host --name=gpt2_migraphx --privileged --device=/dev/kfd --device=/dev/dri --group-add video --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -v $PWD/gpt2_migraphx:/home/gpt2_migraphx -it <Your Image ID> /bin/bash
```

## 推理

### Python版本推理

本次采用GPT-2模型进行诗词生成任务,模型文件下载链接:https://pan.baidu.com/s/1KWeoUuakCZ5dualK69qCcw , 提取码:4pmh ,并将GPT2_shici.onnx模型文件保存在Resource/文件夹下。下面介绍如何运行python代码示例,Python示例的详细说明见Doc目录下的Tutorial_Python.md。

#### 设置环境变量
liucong's avatar
liucong committed
51

liucong's avatar
liucong committed
52
53
54
```
export PYTHONPATH=/opt/dtk/lib:$PYTHONPATH
```
liucong's avatar
liucong committed
55

liucong's avatar
liucong committed
56
#### 运行示例
liucong's avatar
liucong committed
57

liucong's avatar
liucong committed
58
```
liucong's avatar
liucong committed
59
60
# 进入gpt2 migraphx工程根目录
cd <path_to_gpt2_migraphx> 
liucong's avatar
liucong committed
61
62

# 进入示例程序目录
liucong's avatar
liucong committed
63
cd Python/
liucong's avatar
liucong committed
64
65
66
67

# 安装依赖
pip install -r requirements.txt

liucong's avatar
liucong committed
68
# 运行示例
liucong's avatar
liucong committed
69
python gpt2.py
70
71
```

liucong's avatar
liucong committed
72
73
如下所示,采用交互式界面,通过输入开头诗词,GPT2模型可以生成后续的诗句。

liucong's avatar
liucong committed
74
75
76
77
78
79
80
81
82
83
```
user:江上归帆天际开
chatbot:江上归帆天际开,江头别棹日边回。风尘满地音书绝,鸿雁不来春又来。
user:我亦孤山冷淡郎
chatbot:我亦孤山冷淡郎,爱梅不作一般香。水边篱落坡仙笑,羔酒空浇入甲黄。
user:七言绝句
chatbot:七言绝句古无有,五字长城今在前。我欲从君乞妙语,笔端三昧要亲传。
user:春风吹絮满江南
chatbot:春风吹絮满江南,一片离情酒半酣。记得小桥和雪看,梅花无数簇晴岚。
```
liucong's avatar
liucong committed
84

liucong's avatar
liucong committed
85
### C++版本推理
liucong's avatar
liucong committed
86

liucong's avatar
liucong committed
87
本次采用GPT-2模型进行诗词生成任务,模型文件下载链接:https://pan.baidu.com/s/1KWeoUuakCZ5dualK69qCcw , 提取码:4pmh ,并将GPT2_shici.onnx模型文件保存在Resource/文件夹下。下面介绍如何运行C++代码示例,C++示例的详细说明见Doc目录下的Tutorial_Cpp.md。
liucong's avatar
liucong committed
88

89

liucong's avatar
liucong committed
90
#### 构建工程
91
92
93
94
95

```
rbuild build -d depend
```

liucong's avatar
liucong committed
96
#### 设置环境变量
97
98
99
100

将依赖库依赖加入环境变量LD_LIBRARY_PATH,在~/.bashrc中添加如下语句:

```
liucong's avatar
liucong committed
101
export LD_LIBRARY_PATH=<path_to_gpt2_migraphx>/depend/lib64/:$LD_LIBRARY_PATH
102
103
104
105
106
107
```

然后执行:

```
source ~/.bashrc
108
109
```

liucong's avatar
liucong committed
110
#### 运行示例
111
112

```python
liucong's avatar
liucong committed
113
114
# 进入gpt2 migraphx工程根目录
cd <path_to_gpt2_migraphx> 
115

liucong's avatar
liucong committed
116
# 进入build目录
liucong's avatar
liucong committed
117
cd build/
118

liucong's avatar
liucong committed
119
120
# 执行示例程序
./GPT2
121
122
123
```

如下所示,采用交互式界面,通过输入开头诗词,GPT2模型可以推理出后续的诗句。
124

liucong's avatar
liucong committed
125
126
127
128
129
130
131
132
133
134
```
question:江上归帆天际开
chatbot:江上归帆天际开,江头别棹日边回。风尘满地音书绝,鸿雁不来春又来。
question:我亦孤山冷淡郎
chatbot:我亦孤山冷淡郎,爱梅不作一般香。水边篱落坡仙笑,羔酒空浇入甲黄。
question:七言绝句
chatbot:七言绝句古无有,五字长城今在前。我欲从君乞妙语,笔端三昧要亲传。
question:春风吹絮满江南
chatbot:春风吹絮满江南,一片离情酒半酣。记得小桥和雪看,梅花无数簇晴岚。
```
135

liucong's avatar
liucong committed
136
137
138
139
140
141
142
143
144
145
## 应用场景

### 算法类别

自然语言处理

### 热点应用行业

nlp,智能聊天助手,科研

liucong's avatar
liucong committed
146
## 源码仓库及问题反馈
147
148
149

https://developer.hpccube.com/codes/modelzoo/gpt2_migraphx

liucong's avatar
liucong committed
150
## 参考
151
152

https://github.com/Morizeyao/GPT2-Chinese