Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
OpenDAS
deepspeed
Commits
b73894de
"docs/vscode:/vscode.git/clone" did not exist on "008d9818a25bb667532cba1611093ccce1902b25"
Unverified
Commit
b73894de
authored
Sep 08, 2020
by
Arash Ashari
Committed by
GitHub
Sep 08, 2020
Browse files
adding sparse attention to feature index page (#377)
parent
9dadf38d
Changes
1
Show whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
19 additions
and
0 deletions
+19
-0
docs/_pages/features.md
docs/_pages/features.md
+19
-0
No files found.
docs/_pages/features.md
View file @
b73894de
...
@@ -221,3 +221,22 @@ Please see the [core API doc](https://deepspeed.readthedocs.io/) for more detail
...
@@ -221,3 +221,22 @@ Please see the [core API doc](https://deepspeed.readthedocs.io/) for more detail
}
}
}
}
```
```
## Sparse Attention
DeepSpeed offers sparse attention to support long sequences. Please refer to the
[
Sparse Attention
](
/tutorials/sparse_attention/
)
tutorial.
```
python
--
deepspeed_sparse_attention
```
```
python
"sparse_attention"
:
{
"mode"
:
"fixed"
,
"block"
:
16
,
"different_layout_per_head"
:
true
,
"num_local_blocks"
:
4
,
"num_global_blocks"
:
1
,
"attention"
:
"bidirectional"
,
"horizontal_global_attention"
:
false
,
"num_different_global_patterns"
:
4
}
```
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment