Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
ModelZoo
UMT5_pytorch
Commits
20f80172
"git@developer.sourcefind.cn:change/sglang.git" did not exist on "63ba630bbbb2d55787ac54ac0a01cbde993afc20"
Commit
20f80172
authored
May 22, 2024
by
wanglch
Browse files
Update README.md
parent
3f0a9b8d
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
6 additions
and
6 deletions
+6
-6
README.md
README.md
+6
-6
No files found.
README.md
View file @
20f80172
# UMT5
# UMT5
**注:执行下游任务是需要先进行预训练, 训练代码参考train_model.py。**
**注:执行下游任务是需要先进行预训练, 训练代码参考train_model.py。**
<div
align=
"center"
>
<div
align=
"center"
>
<img
align=
"center"
src=
docs/T5_task.png
>
<img
align=
"center"
src=
docs/T5_task.png
>
</div>
</div>
## 论文
## 论文
...
@@ -13,7 +13,7 @@ umT5:T5 的多语言版本,具备 T5 模型大部分的多功能性,在多
...
@@ -13,7 +13,7 @@ umT5:T5 的多语言版本,具备 T5 模型大部分的多功能性,在多
### MT5 模型结构
### MT5 模型结构
<div
align=
"center"
>
<div
align=
"center"
>
<img
align=
"center"
src=
docs/T5_structure.png
>
<img
align=
"center"
src=
docs/T5_structure.png
>
</div>
</div>
...
@@ -23,19 +23,19 @@ umT5:T5 的多语言版本,具备 T5 模型大部分的多功能性,在多
...
@@ -23,19 +23,19 @@ umT5:T5 的多语言版本,具备 T5 模型大部分的多功能性,在多
它主要的改动来自论文
[
GLU Variants Improve Transformer
](
https://arxiv.org/abs/2002.05202
)
,主要是借用了
[
Language Modeling with Gated Convolutional Networks
](
https://arxiv.org/abs/1612.08083
)
的
**GLU**
(Gated Linear Unit)来增强 FFN 部分的效果。具体来说,原来 T5 的 FFN 为(T5 没有 Bias):
它主要的改动来自论文
[
GLU Variants Improve Transformer
](
https://arxiv.org/abs/2002.05202
)
,主要是借用了
[
Language Modeling with Gated Convolutional Networks
](
https://arxiv.org/abs/1612.08083
)
的
**GLU**
(Gated Linear Unit)来增强 FFN 部分的效果。具体来说,原来 T5 的 FFN 为(T5 没有 Bias):
<div
align=
"center"
>
<div
align=
"center"
>
<img
align=
"center"
src=
docs/equation1.png
>
<img
align=
"center"
src=
docs/equation1.png
>
</div>
</div>
改为:
改为:
<div
align=
"center"
>
<div
align=
"center"
>
<img
align=
"center"
src=
docs/euqation2.png
>
<img
align=
"center"
src=
docs/euqation2.png
>
</div>
</div>
### T5 Transformer
### T5 Transformer
<div
align=
"center"
>
<div
align=
"center"
>
<img
align=
"center"
src=
docs/t5transformer.png
>
<img
align=
"center"
src=
docs/t5transformer.png
>
</div>
</div>
...
@@ -174,7 +174,7 @@ python umt5_summary.py
...
@@ -174,7 +174,7 @@ python umt5_summary.py
## result
## result
### 中文文本摘要任务
### 中文文本摘要任务
<div
align=
"center"
>
<div
align=
"center"
>
<img
align=
"center"
src=
docs/result.png
>
<img
align=
"center"
src=
docs/result.png
>
</div>
</div>
### 精度
### 精度
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment