Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
ModelZoo
GPT2_oneflow
Commits
dd23aa41
Commit
dd23aa41
authored
May 07, 2025
by
chenzk
Browse files
Update sf.md
parent
6b6be089
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
4 additions
and
4 deletions
+4
-4
README.md
README.md
+4
-4
No files found.
README.md
View file @
dd23aa41
...
...
@@ -8,7 +8,7 @@
## 模型结构
第二代生成式预训练模型(Generative Pre-Training2),GPT2使用 Transformer 的 Decoder 结构,并对 Transformer Decoder 进行了一些改动,原本的 Decoder 包含了两个 Multi-Head Attention 结构,GPT2 只保留了 Mask Multi-Head Attention。
<img
src=
"http://developer.
hpccube.com
/codes/modelzoo/GPT/-/raw/main/gpt2%E6%A8%A1%E5%9E%8B%E7%BB%93%E6%9E%84.png"
alt=
"gpt2模型结构.png"
style=
"zoom: 67%;"
/>
<img
src=
"http://developer.
sourcefind.cn
/codes/modelzoo/GPT/-/raw/main/gpt2%E6%A8%A1%E5%9E%8B%E7%BB%93%E6%9E%84.png"
alt=
"gpt2模型结构.png"
style=
"zoom: 67%;"
/>
我们为了用户可以使用OneFlow-Libai快速验证GPT2模型预训练,统计性能或验证精度,提供了一个GPT2网络示例,主要网络参数:
...
...
@@ -36,7 +36,7 @@ model.cfg.max_seq_length = 1024
GPT-2中使用掩模自注意力(masked self-attention),一般的自注意力模块允许某位置右侧的词计算时处于最大值。而掩模自注意力会阻止这种情况发生。
<img
src=
"http://developer.
hpccube.com
/codes/modelzoo/GPT/-/raw/main/gpt2%E7%AE%97%E6%B3%95%E5%8E%9F%E7%90%86.png"
alt=
"gpt2算法原理.png"
style=
"zoom:50%;"
/>
<img
src=
"http://developer.
sourcefind.cn
/codes/modelzoo/GPT/-/raw/main/gpt2%E7%AE%97%E6%B3%95%E5%8E%9F%E7%90%86.png"
alt=
"gpt2算法原理.png"
style=
"zoom:50%;"
/>
## 数据集
...
...
@@ -117,10 +117,10 @@ train.dist.pipeline_parallel_size = 1
## 源码仓库及问题反馈
-
https://developer.
hpccube.com
/codes/modelzoo/GPT
-
https://developer.
sourcefind.cn
/codes/modelzoo/GPT
## 参考
*
https://libai.readthedocs.io/en/latest/tutorials/get_started/quick_run.html
*
https://github.com/Oneflow-Inc/oneflow
*
https://github.com/Oneflow-Inc/libai/blob/main/docs/source/notes/FAQ.md
\ No newline at end of file
*
https://github.com/Oneflow-Inc/libai/blob/main/docs/source/notes/FAQ.md
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment