Commits · e082d4752e3db2acf4348a4522daba077bb58aec · OpenDAS / deepspeed

30 Oct, 2020 1 commit

Add CPUAdam optimizer for zero-offload in deepspeed engine (#484) · f5aa2547

Reza Yazdani authored Oct 30, 2020



* add adamW to CPU-ADAM implementation

* supporting cpu-adam optimizer for zero-offload on deepspeed side

* bump DSE to match cpu-adam updates
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

f5aa2547

07 Oct, 2020 2 commits
- turning off different tests (temp) · 679fc135
  Jeff Rasley authored Oct 06, 2020
  
  679fc135
- temporarily disable lr unit tests · 11cf47ef
  Jeff Rasley authored Oct 06, 2020
  
  11cf47ef
29 Sep, 2020 1 commit
- Disable default installation of CPU Adam (#450) · 7b8be2a7
  Olatunji Ruwase authored Sep 29, 2020
```
* Disable default installation of CPU Adam

* Handle cpufeature import/use errors separately
```
  7b8be2a7
25 Sep, 2020 1 commit
- unit test rename (#442) · 5412a334
  Shaden Smith authored Sep 25, 2020
  
  5412a334
22 Sep, 2020 1 commit
- support dynamic sequence length in transformer kernels (#424) · f0f2a702
  RezaYazdaniAminabadi authored Sep 21, 2020
```
Co-authored-by: Conglong Li <conglong.li@gmail.com>
```
  f0f2a702
21 Sep, 2020 1 commit
- Add configurable intermediate size to transformer kernels (#423) · a148bd33
  RezaYazdaniAminabadi authored Sep 21, 2020
  
  a148bd33
18 Sep, 2020 3 commits
- Fix activation checkpoint unit tests for GPU systems (#421) · a825f996
  Shaden Smith authored Sep 18, 2020
  
  a825f996
- Revert "Activation checkpointing bugfix and unit tests (#420)" (#422) · a74a604a
  Jeff Rasley authored Sep 18, 2020
```
This reverts commit 01b6e27e

.
Co-authored-by: Shaden Smith <ShadenTSmith@gmail.com>
```
  a74a604a
- Activation checkpointing bugfix and unit tests (#420) · 01b6e27e
  Shaden Smith authored Sep 18, 2020
```
* Activation checkpointing bugfix and unit tests.

* Activation checkpointing bugfix and unit tests.
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
```
  01b6e27e
16 Sep, 2020 1 commit
- Add Linear warmup+decay lr schedule (#414) · 0e942df0
  Olatunji Ruwase authored Sep 16, 2020
```
Update lr schedule unit tests
```
  0e942df0
15 Sep, 2020 1 commit
- pytest skips for tests requiring certain ops (#411) · 91b4a93d
  Jeff Rasley authored Sep 15, 2020
```
* add pytest skips around tests that require certain ops to be installed
```
  91b4a93d
11 Sep, 2020 2 commits
- Revert "supporting different intermediate sizes other than 4 * hidden_dim (#389)" (#404) · 4ac9bf60
  Jeff Rasley authored Sep 11, 2020
```
This reverts commit e549be60.
```
  4ac9bf60
- supporting different intermediate sizes other than 4 * hidden_dim (#389) · e549be60
  RezaYazdaniAminabadi authored Sep 11, 2020
```
* supporting different intermediate sizes other than 4*hidden_dim

* run precommit

* uncommnet the unit tests
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
```
  e549be60
10 Sep, 2020 3 commits

fix for 16GB v100 nodes (#393) · b1d4bd73
Jeff Rasley authored Sep 10, 2020

b1d4bd73
Pipeline parallel training engine. (#392) · 65c2f974
Shaden Smith authored Sep 09, 2020
```
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
```
65c2f974

ZeRO-Offload release (#391) · 41db1c2f

Jeff Rasley authored Sep 09, 2020



* ZeRO-Offload (squash) (#381)
Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com>
Co-authored-by: Reza Yazdani <reyazda@microsoft.com>
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
Co-authored-by: Jie <37380896+jren73@users.noreply.github.com>
Co-authored-by: Arash Ashari <arashari@microsoft.com>
Co-authored-by: Reza Yazdani <reyazda@microsoft.com>
Co-authored-by: Samyam Rajbhandari <samyamr@microsoft.com>
Co-authored-by: Shaden Smith <Shaden.Smith@microsoft.com>
Co-authored-by: arashashari <arashashari@ArashMSLaptop.redmond.corp.microsoft.com>
Co-authored-by: RezaYazdaniAminabadi <44502768+RezaYazdaniAminabadi@users.noreply.github.com>
Co-authored-by: Reza Yazdani <reyazda@microsoft.com>
Co-authored-by: Samyam Rajbhandari <samyamr@microsoft.com>
Co-authored-by: Shaden Smith <Shaden.Smith@microsoft.com>

41db1c2f

09 Sep, 2020 1 commit

Add 1-bit Adam support to DeepSpeed (#380) · 01726ce2

Ammar Ahmad Awan authored Sep 09, 2020



* 1-bit adam (#353)
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
Co-authored-by: Your Name <you@example.com>
Co-authored-by: tanghl1994 <htang14@ur.rochester.edu>
Co-authored-by: Hank <tanghl1994@gmail.com>
Co-authored-by: root <root@node2x12b.cs.rochester.edu>
Co-authored-by: Ammar Ahmad Awan <awan.ammar@microsoft.com>

01726ce2

03 Sep, 2020 1 commit
- Update test_sparse_attention.py · 1ebcd6c5
  Jeff Rasley authored Sep 02, 2020
  
  1ebcd6c5
02 Sep, 2020 2 commits
- update DSE and rename SA tests · 1661e830
  Jeff Rasley authored Sep 02, 2020
  
  1661e830
- Sparse attn + ops/runtime refactor + v0.3.0 (#343) · e5bbc2e5
  Jeff Rasley authored Sep 01, 2020
```
* Sparse attn + ops/runtime refactor + v0.3.0
Co-authored-by: Arash Ashari <arashari@microsoft.com>
Co-authored-by: Arash Ashari <arashari@microsoft.com>
```
  e5bbc2e5
01 Sep, 2020 5 commits
- Switches BBS example to use mbsize=3 and gas=2 to fit in 16GB of memory. (#341) · 838f53b7
  Shaden Smith authored Sep 01, 2020
  
  838f53b7
- Update run_func_test.py · f4726b75
  Samyam Rajbhandari authored Aug 31, 2020
```
Renaming config files to gas3
```
  f4726b75
- Rename ds_config_func_bs8_zero0_gas10.json to ds_config_func_bs8_zero0_gas3.json · 6122a74e
  Samyam Rajbhandari authored Aug 31, 2020
  
  6122a74e
- Rename ds_config_func_bs8_zero2_gas10.json to ds_config_func_bs8_zero2_gas3.json · 7a356b28
  Samyam Rajbhandari authored Aug 31, 2020
  
  7a356b28
- Samyamr/grad acc stage2 (#338) · 7240abf3
  Samyam Rajbhandari authored Aug 31, 2020
```
* Adding gradient accumulation support for ZeRO Stage 2. Changing all Megatron-LM tests to also test gradient accumulation

* Gradient Accumulation support for Stage 2. Model tests added to test the feature

* formatting

* Update deepspeed_light.py

removing comment

* Update ds_config_func_bs8_zero1.json

reverting this file back. Its not needed for this PR

* defining baseline prefix
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
```
  7240abf3
10 Aug, 2020 1 commit
- Fix+tests for get_lr from lr_scheduler before training starts (#310) · cd68e6e5
  Jeff Rasley authored Aug 10, 2020
```
* add fix and tests for get_lr from lr_scheduler before training starts
```
  cd68e6e5
15 Jul, 2020 2 commits
- Empty grad fix (#291) · 376818ef
  Jeff Rasley authored Jul 15, 2020
```
* empty grad fix
* add unit tests for empty grad
```
  376818ef
- Fix bug in fp32 optimizer state loading (#289) · 607814fe
  Olatunji Ruwase authored Jul 15, 2020
  
  607814fe
14 Jul, 2020 1 commit

Support loading and saving ZeRO checkpoints with changing DP degree (#240) · 7ccc9daf

Olatunji Ruwase authored Jul 14, 2020



* Support saving and loading ZeRO checkpoints on different data
parallelism degree.

* Fix formatting

* Support checkpoint with varying GPU count in ZeRO stage 1

* Fix formatting

* Formatting fixes

* Update model tests

* Remove pprint

* Minor fix

* Fix formatting

* Update model tests
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

7ccc9daf

11 Jul, 2020 1 commit
- Support amp deepspeed backend (#286) · f5453124
  Jeff Rasley authored Jul 11, 2020
```
* add amp support for deepspeed (non-ZeRO)
* tests for amp mode
```
  f5453124
06 Jul, 2020 1 commit

ZeRO-2: Handle gradients of empty partitions (#275) · 4a3234e0

Olatunji Ruwase authored Jul 06, 2020



* Load non-DeepSpeed checkpoints into ZeRO optimizer

* Handle parameters smaller than DP

* Formatting fixes

* Handle empty partitions

* Fix perf bug
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>

4a3234e0

23 Jun, 2020 1 commit

Handle parameter groups smaller than DP (#273) · 88c319aa

Olatunji Ruwase authored Jun 23, 2020

* Load non-DeepSpeed checkpoints into ZeRO optimizer

* Handle parameters smaller than DP

* Formatting fixes

88c319aa

30 May, 2020 1 commit
- update tests · bbd8cd7d
  Jeff Rasley authored May 29, 2020
  
  bbd8cd7d
29 May, 2020 1 commit

Transformer kernel release (#242) · 734d8991

Jeff Rasley authored May 29, 2020



* Transformer kernels release
Co-authored-by: Shaden Smith <ShadenTSmith@gmail.com>
Co-authored-by: Elton Zheng <eltonz@microsoft.com>
Co-authored-by: Reza Yazdani <reyazda@microsoft.com>
Co-authored-by: RezaYazdaniAminabadi <44502768+RezaYazdaniAminabadi@users.noreply.github.com>
Co-authored-by: Tunji Ruwase <olruwase@microsoft.com>
Co-authored-by: Shaden Smith <ShadenTSmith@gmail.com>
Co-authored-by: Shaden Smith <Shaden.Smith@microsoft.com>
Co-authored-by: Samyam Rajbhandari <samyamr@microsoft.com>
Co-authored-by: Shaden Smith <ShadenTSmith@gmail.com>
Co-authored-by: Jeff Rasley <jerasley@microsoft.com>
Co-authored-by: Samyam Rajbhandari <samyamr@microsoft.com>
Co-authored-by: Shaden Smith <ShadenTSmith@gmail.com>
Co-authored-by: Elton Zheng <eltonz@microsoft.com>
Co-authored-by: Reza Yazdani <reyazda@microsoft.com>
Co-authored-by: RezaYazdaniAminabadi <44502768+RezaYazdaniAminabadi@users.noreply.github.com>
Co-authored-by: Tunji Ruwase <olruwase@microsoft.com>
Co-authored-by: Shaden Smith <Shaden.Smith@microsoft.com>
Co-authored-by: Samyam Rajbhandari <samyamr@microsoft.com>

734d8991

27 May, 2020 1 commit
- Support fp32 grad clipping and fix max_grad_norm confusion (#232) · abe2204d
  Jeff Rasley authored May 26, 2020
```
* updates to support fp32 grad clipping and disable max_grad_norm
```
  abe2204d
20 May, 2020 1 commit
- reduce size of megatron tests (#223) · 53ac7947
  Jeff Rasley authored May 20, 2020
  
  53ac7947
19 May, 2020 1 commit

ZeRO-2 (#217) · f2ac7eaf

Jeff Rasley authored May 19, 2020



Updates for ZeRO stage 2 + ZeRO stage 1 w. RS
Co-authored-by: Tunji Ruwase <olruwase@microsoft.com>
Co-authored-by: Samyam Rajbhandari <samyamr@microsoft.com>
Co-authored-by: Shaden Smith <ShadenTSmith@gmail.com>
Co-authored-by: Elton Zheng <eltonz@microsoft.com>
Co-authored-by: Shaden Smith <Shaden.Smith@microsoft.com>
Co-authored-by: yuxionghe <yuxhe@microsoft.com>
Co-authored-by: Arash Ashari <arashari@microsoft.com>

f2ac7eaf

18 May, 2020 1 commit

adding BingSqaud e2e test (#214) · c61e23b4

Arash Ashari authored May 18, 2020

* adding BingSqaud e2e test

* updating the draft test; bring final step under try section

* finalizinf test for base deepspeed and deepspeed with ZeRO

* applying the comment (thanks Jeff); fixed formatting

c61e23b4

11 May, 2020 1 commit
- Support dynamic loss scale args in fp16 optimizers (#212) · 0be026e3
  Olatunji Ruwase authored May 11, 2020
```
* Support dynamic loss scale args in fp16 optimizers

* Update names
```
  0be026e3