- 13 Apr, 2022 1 commit
-
-
ver217 authored
* add tensor placement policies * polish comments * polish comments * update moe unit tests
-
- 12 Apr, 2022 2 commits
-
-
Frank Lee authored
-
Jiarui Fang authored
-
- 08 Apr, 2022 1 commit
-
-
HELSON authored
-
- 07 Apr, 2022 1 commit
-
-
HELSON authored
* adapt model weight initialization for methods in Pytorch nn.init
-
- 29 Mar, 2022 1 commit
-
-
ver217 authored
-
- 28 Mar, 2022 1 commit
-
-
HELSON authored
only process module's own parameters in Zero context add zero hooks for all modules that contrain parameters gather parameters only belonging to module itself
-
- 25 Mar, 2022 2 commits
-
-
Frank Lee authored
-
Jiarui Fang authored
-
- 18 Mar, 2022 6 commits
-
-
Frank Lee authored
-
ver217 authored
-
Jiarui Fang authored
* Revert "polish code" This reverts commit 8cf7ff08. * Revert "rename variables" This reverts commit e99af94a. * Revert "remove surplus imports" This reverts commit 46add4a5. * Revert "update sharded optim and fix zero init ctx" This reverts commit 57567ee7.
-
ver217 authored
-
ver217 authored
-
Frank Lee authored
-
- 16 Mar, 2022 1 commit
-
-
ver217 authored
-
- 14 Mar, 2022 2 commits
-
-
Jiarui Fang authored
-
ver217 authored
-
- 11 Mar, 2022 12 commits
-
-
Jiarui Fang authored
* place params on cpu after zero init context * polish code
-
Jiarui Fang authored
-
ver217 authored
-
ver217 authored
-
jiaruifang authored
-
jiaruifang authored
-
jiaruifang authored
-
jiaruifang authored
-
ver217 authored
-
jiaruifang authored
-
ver217 authored
-
Jiarui Fang authored
* add zero1 (#209) * add zero1 * add test zero1 * update zero stage 1 develop (#212) * Implement naive zero3 (#240) * naive zero3 works well * add zero3 param manager * add TODOs in comments * add gather full param ctx * fix sub module streams * add offload * fix bugs of hook and add unit tests * fix bugs of hook and add unit tests (#252) * add gather full param ctx * fix sub module streams * add offload * fix bugs of hook and add unit tests * polish code and add state dict hook * fix bug * update unit test * refactor reconstructed zero code * clip_grad support zero3 and add unit test * add unit test for Zero3ParameterManager * [WIP] initialize the shard param class * [WIP] Yet another sharded model implementation (#274) * [WIP] initialize the shard param class * [WIP] Yes another implementation of shardModel. Using a better hook method. * torch.concat -> torch.cat * fix test_zero_level_1.py::test_zero_level_1 unitest * remove deepspeed implementation and refactor for the reconstructed zero module * polish zero dp unittests Co-authored-by:
ver217 <lhx0217@gmail.com> Co-authored-by:
Frank Lee <somerlee.9@gmail.com>
-