Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
OpenDAS
FastMoE
Commits
ae658b89
"git@developer.sourcefind.cn:OpenDAS/mmcv.git" did not exist on "f367d621c629eade67347bb4a0b9cb1ebe261641"
Commit
ae658b89
authored
Feb 01, 2021
by
Rick Ho
Browse files
add megatron example
parent
f9bec836
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
22 additions
and
0 deletions
+22
-0
examples/megatron/README.md
examples/megatron/README.md
+22
-0
No files found.
examples/megatron/README.md
0 → 100644
View file @
ae658b89
A modified version of Megatron-LM that can cope with FastMoE can be found in
[
this repository
](
https://github.com/laekov/fmoe-megatron
)
.
Using
`fmoe.megatron.create_moe_mlp`
to replace the
`ParallelMLP`
module in
Megatron's transformer model is all you need.
In our fork, the required modifications are located at line 425 of
`megatron/model/transformer.py`
as follow.
```
Python
# MLP
if args.num_experts == 1:
self.mlp = ParallelMLP(init_method,
output_layer_init_method)
else:
from fmoe.megatron import create_moe_mlp
self.mlp = create_moe_mlp(args)
```
When properly added
`--num-experts`
argument to
`megatron/arguments.py`
, FastMoE
is enabled without extra burden.
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment