Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in
Toggle navigation
Menu
Open sidebar
chenpangpang
transformers
Commits
a27195b1
Unverified
Commit
a27195b1
authored
Aug 16, 2022
by
flozi00
Committed by
GitHub
Aug 16, 2022
Browse files
Update longt5.mdx (#18634)
parent
fd9aa82b
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
1 addition
and
1 deletion
+1
-1
docs/source/en/model_doc/longt5.mdx
docs/source/en/model_doc/longt5.mdx
+1
-1
No files found.
docs/source/en/model_doc/longt5.mdx
View file @
a27195b1
...
@@ -37,7 +37,7 @@ Tips:
...
@@ -37,7 +37,7 @@ Tips:
- [`LongT5ForConditionalGeneration`] is an extension of [`T5ForConditionalGeneration`] exchanging the traditional
- [`LongT5ForConditionalGeneration`] is an extension of [`T5ForConditionalGeneration`] exchanging the traditional
encoder *self-attention* layer with efficient either *local* attention or *transient-global* (*tglobal*) attention.
encoder *self-attention* layer with efficient either *local* attention or *transient-global* (*tglobal*) attention.
- Unlike the T5 model, LongT5 does not use a task prefix. Furthermore, it uses a different pre-training objective
- Unlike the T5 model, LongT5 does not use a task prefix. Furthermore, it uses a different pre-training objective
inspired by the pre-training of
`
[PegasusForConditionalGeneration
]
`.
inspired by the pre-training of [
`
PegasusForConditionalGeneration`
]
.
- LongT5 model is designed to work efficiently and very well on long-range *sequence-to-sequence* tasks where the
- LongT5 model is designed to work efficiently and very well on long-range *sequence-to-sequence* tasks where the
input sequence exceeds commonly used 512 tokens. It is capable of handling input sequences of a length up to 16,384 tokens.
input sequence exceeds commonly used 512 tokens. It is capable of handling input sequences of a length up to 16,384 tokens.
- For *Local Attention*, the sparse sliding-window local attention operation allows a given token to attend only `r`
- For *Local Attention*, the sparse sliding-window local attention operation allows a given token to attend only `r`
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment