Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
OpenDAS
ColossalAI
Commits
1f698f44
Unverified
Commit
1f698f44
authored
Apr 14, 2022
by
Jiarui Fang
Committed by
GitHub
Apr 14, 2022
Browse files
[readme] polish readme (#764)
* [readme] polish readme * centering image
parent
920fe315
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
16 additions
and
8 deletions
+16
-8
README.md
README.md
+16
-8
No files found.
README.md
View file @
1f698f44
...
@@ -73,25 +73,33 @@ Colossal-AI provides a collection of parallel training components for you. We ai
...
@@ -73,25 +73,33 @@ Colossal-AI provides a collection of parallel training components for you. We ai
distributed deep learning models just like how you write your model on your laptop. We provide user-friendly tools to kickstart
distributed deep learning models just like how you write your model on your laptop. We provide user-friendly tools to kickstart
distributed training in a few lines.
distributed training in a few lines.
-
Data Parallelism
-
Parallelism strategies
-
Pipeline Parallelism
-
Data Parallelism
-
1D, 2D, 2.5D, 3D tensor parallelism
-
Pipeline Parallelism
-
Sequence parallelism
-
1D,
[
2D
](
https://arxiv.org/abs/2104.05343
)
,
[
2.5D
](
https://arxiv.org/abs/2105.14500
)
, 3D Tensor parallelism
-
Friendly trainer and engine
-
[
Sequence parallelism
](
https://arxiv.org/abs/2105.13120
)
-
Extensible for new parallelism
-
[
Zero Redundancy Optimizer (ZeRO)
](
https://arxiv.org/abs/2108.05818
)
-
Mixed Precision Training
-
Zero Redundancy Optimizer (ZeRO)
-
Heterogeneous Memory Menagement
-
[
PatrickStar
](
https://arxiv.org/abs/2108.05818
)
-
Friendly Usage
-
Configuration file based
<p
align=
"right"
>
(
<a
href=
"#top"
>
back to top
</a>
)
</p>
<p
align=
"right"
>
(
<a
href=
"#top"
>
back to top
</a>
)
</p>
## Demo
## Demo
### ViT
### ViT
<p
align=
"center"
>
<img
src=
"https://raw.githubusercontent.com/hpcaitech/public_assets/main/colossalai/img/ViT.png"
width=
"450"
/>
<img
src=
"https://raw.githubusercontent.com/hpcaitech/public_assets/main/colossalai/img/ViT.png"
width=
"450"
/>
</p>
-
14x larger batch size, and 5x faster training for Tensor Parallelism = 64
-
14x larger batch size, and 5x faster training for Tensor Parallelism = 64
### GPT-3
### GPT-3
<p
align=
"center"
>
<img
src=
"https://raw.githubusercontent.com/hpcaitech/public_assets/main/colossalai/img/GPT3.png"
width=
700/
>
<img
src=
"https://raw.githubusercontent.com/hpcaitech/public_assets/main/colossalai/img/GPT3.png"
width=
700/
>
</p>
-
Save 50% GPU resources, and 10.7% acceleration
-
Save 50% GPU resources, and 10.7% acceleration
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment