Commits · 899aa86368d9c0d9b3eda8b4186c78018cd56761 · OpenDAS / ColossalAI

05 May, 2023 2 commits

[CI] Update test_sharded_optim_with_sync_bn.py (#3688) · b49020c1
digger-yu authored May 05, 2023
```
fix spelling error in line23
change "cudnn_determinstic"=True to "cudnn_deterministic=True"
```
b49020c1

[booster] gemini plugin support shard checkpoint (#3610) · 307894f7

jiangmingyan authored May 05, 2023



* gemini plugin add shard checkpoint save/load

* gemini plugin add shard checkpoint save/load

* gemini plugin add shard checkpoint save/load

* gemini plugin add shard checkpoint save/load

* gemini plugin add shard checkpoint save/load

* gemini plugin add shard checkpoint save/load

* gemini plugin add shard checkpoint save/load

* gemini plugin add shard checkpoint save/load

* gemini plugin add shard checkpoint save/load

* gemini plugin add shard checkpoint save/load

* gemini plugin add shard checkpoint save/load

* gemini plugin add shard checkpoint save/load

* gemini plugin add shard checkpoint save/load

* gemini plugin add shard checkpoint save/load

* gemini plugin support shard checkpoint

* [API Refactoring]gemini plugin support shard checkpoint

* [API Refactoring]gemini plugin support shard checkpoint

* [API Refactoring]gemini plugin support shard checkpoint

* [API Refactoring]gemini plugin support shard checkpoint

* [API Refactoring]gemini plugin support shard checkpoint

* [API Refactoring]gemini plugin support shard checkpoint

* [API Refactoring]gemini plugin support shard checkpoint

* [API Refactoring]gemini plugin support shard checkpoint

* [API Refactoring]gemini plugin support shard checkpoint

* [API Refactoring]gemini plugin support shard checkpoint

* [API Refactoring]gemini plugin support shard checkpoint

* [API Refactoring]gemini plugin support shard checkpoint

* [API Refactoring]gemini plugin support shard checkpoint

---------
Co-authored-by: luchen <luchen@luchendeMBP.lan>
Co-authored-by: luchen <luchen@luchendeMacBook-Pro.local>

307894f7

26 Apr, 2023 1 commit

[gemini] accelerate inference (#3641) · 50793b35

Hongxin Liu authored Apr 26, 2023

* [gemini] support don't scatter after inference

* [chat] update colossalai strategy

* [chat] fix opt benchmark

* [chat] update opt benchmark

* [gemini] optimize inference

* [test] add gemini inference test

* [chat] fix unit test ci

* [chat] fix ci

* [chat] fix ci

* [chat] skip checkpoint test

50793b35

17 Apr, 2023 1 commit

[gemini] support save state dict in shards (#3581) · f313babd

Hongxin Liu authored Apr 17, 2023

* [gemini] support state dict shard

* [gemini] add test state dict shard

* [gemini] polish docstr

* [gemini] fix merge

* [gemini] polish code

f313babd

06 Apr, 2023 2 commits
- [test] refactor tests with spawn (#3452) · 80eba05b
  Frank Lee authored Apr 06, 2023
```
* [test] added spawn decorator

* polish code

* polish code

* polish code

* polish code

* polish code

* polish code
```
  80eba05b
- [test] reorganize zero/gemini tests (#3445) · 933048ad
  ver217 authored Apr 06, 2023
  
  933048ad
04 Apr, 2023 1 commit

[zero] reorganize zero/gemini folder structure (#3424) · 26b7aac0

ver217 authored Apr 04, 2023

* [zero] refactor low-level zero folder structure

* [zero] fix legacy zero import path

* [zero] fix legacy zero import path

* [zero] remove useless import

* [zero] refactor gemini folder structure

* [zero] refactor gemini folder structure

* [zero] refactor legacy zero import path

* [zero] refactor gemini folder structure

* [zero] refactor gemini folder structure

* [zero] refactor gemini folder structure

* [zero] refactor legacy zero import path

* [zero] fix test import path

* [zero] fix test

* [zero] fix circular import

* [zero] update import

26b7aac0

03 Apr, 2023 1 commit
- [test] fixed gemini plugin test (#3411) · 638a07a7
  Frank Lee authored Apr 03, 2023
```
* [test] fixed gemini plugin test

* polish code

* polish code
```
  638a07a7
29 Jan, 2023 2 commits

[zero] add zero wrappers (#2523) · b528eea0
HELSON authored Jan 29, 2023
```
* [zero] add zero wrappers

* change names

* add wrapper functions to init
```
b528eea0

[zero] fix gradient clipping in hybrid parallelism (#2521) · 077a5cdd

HELSON authored Jan 29, 2023

* [zero] fix gradient clipping in hybrid parallelism

* [testing] change model name to avoid pytest warning

* [hotfix] fix unit testing

077a5cdd

18 Jan, 2023 1 commit
- [zero] add unit testings for hybrid parallelism (#2486) · d565a248
  HELSON authored Jan 18, 2023
  
  d565a248
15 Jan, 2023 1 commit
- [zero] add unit test for low-level zero init (#2474) · 21c88220
  HELSON authored Jan 15, 2023
  
  21c88220
13 Jan, 2023 2 commits
- [zero] polish low level optimizer (#2473) · a5dc4253
  HELSON authored Jan 13, 2023
  
  a5dc4253
- [zero] low level optim supports ProcessGroup (#2464) · 867c8c2d
  Jiarui Fang authored Jan 13, 2023
  
  867c8c2d
26 Dec, 2022 1 commit
- [testing] add beit model for unit testings (#2196) · a3100bd5
  HELSON authored Dec 26, 2022
```
* [testing] add beit model

* [beit] fix bugs

* [beit] fix bugs

* [testing] fix bugs
```
  a3100bd5
20 Dec, 2022 1 commit
- [hotfix] fix auto policy of test_sharded_optim_v2 (#2157) · b87496a6
  Jiarui Fang authored Dec 20, 2022
  
  b87496a6
13 Dec, 2022 1 commit
- [Gemini] update API of the chunkmemstatscollector. (#2129) · c89c66a8
  Jiarui Fang authored Dec 14, 2022
  
  c89c66a8
06 Dec, 2022 2 commits
- [Gemini] remove GLOBAL_MODEL_DATA_TRACER (#2091) · 1fca5d79
  Jiarui Fang authored Dec 06, 2022
  
  1fca5d79
- [Gemini] use MemStats to store the tracing data. Seperate it from Collector. (#2084) · 33f44121
  Jiarui Fang authored Dec 06, 2022
  
  33f44121
30 Nov, 2022 1 commit
- [test] align model name with the file name. (#2045) · 1e885329
  Jiarui Fang authored Nov 30, 2022
  
  1e885329
29 Nov, 2022 1 commit

[zero] test gradient accumulation (#1964) · a1ce02d7

HELSON authored Nov 29, 2022

* [zero] fix memory leak for zero2

* [zero] test gradient accumulation

* [zero] remove grad clip test

a1ce02d7

18 Nov, 2022 1 commit
- [Gemini] add bert for MemtracerWrapper unintests (#1982) · 3712ac7f
  Jiarui Fang authored Nov 18, 2022
  
  3712ac7f
16 Nov, 2022 1 commit
- [zero] fix memory leak for zero2 (#1955) · 7066dfbf
  HELSON authored Nov 16, 2022
  
  7066dfbf
11 Nov, 2022 1 commit

[zero] migrate zero1&2 (#1878) · 6e51d296

HELSON authored Nov 11, 2022

* add zero1&2 optimizer

* rename test ditectory

* rename test files

* change tolerance in test

6e51d296

09 Oct, 2022 1 commit
- [feature] A new ZeRO implementation (#1644) · b28991dd
  HELSON authored Oct 09, 2022
  
  b28991dd
26 Sep, 2022 1 commit
- Revert "[feature] new zero implementation (#1623)" (#1643) · c5d39215
  Jiarui Fang authored Sep 26, 2022
```
This reverts commit 5be118f4.
```
  c5d39215
24 Sep, 2022 1 commit
- [feature] new zero implementation (#1623) · 5be118f4
  HELSON authored Sep 24, 2022
  
  5be118f4
22 Sep, 2022 1 commit
- [moe] fix MoE bugs (#1628) · f7f22487
  HELSON authored Sep 22, 2022
```
* remove forced FP32 modules

* correct no_shard-contexts' positions
```
  f7f22487
29 Jul, 2022 1 commit
- [zero] zero optim state_dict takes only_rank_0 (#1384) · 8dced41a
  ver217 authored Jul 29, 2022
```
* zero optim state_dict takes only_rank_0

* fix unit test
```
  8dced41a
28 Jul, 2022 1 commit
- [hotfix] fix zero optim save/load state dict (#1381) · 828b9e5e
  ver217 authored Jul 28, 2022
  
  828b9e5e
21 Jul, 2022 1 commit
- [colotensor] add Tensor.view op and its unit test (#1343) · 7a8702c0
  HELSON authored Jul 21, 2022
```
[colotensor] add megatron initialization for gpt2
```
  7a8702c0
18 Jul, 2022 1 commit

[hotfix] ZeroDDP use new process group (#1333) · 0c51ff2c

ver217 authored Jul 18, 2022

* process group supports getting ranks in group

* chunk mgr receives a process group

* update unit test

* fix unit tests

0c51ff2c

15 Jul, 2022 1 commit
- [hotfix] shared model returns cpu state_dict (#1328) · 7a053671
  ver217 authored Jul 15, 2022
  
  7a053671
04 Jul, 2022 1 commit
- [refactor] remove gpc dependency in colotensor's _ops (#1189) · 060b917d
  Jiarui Fang authored Jul 04, 2022
  
  060b917d
29 Jun, 2022 1 commit
- [refactor] move chunk and chunkmgr to directory gemini (#1182) · 372f7914
  Jiarui Fang authored Jun 29, 2022
  
  372f7914
24 Jun, 2022 2 commits
- [zero] sharded optim supports loading local state dict (#1170) · 9e1daa63
  ver217 authored Jun 24, 2022
```
* sharded optim supports loading local state dict

* polish code

* add unit test
```
  9e1daa63
- [zero] zero optim supports loading local state dict (#1171) · 561e9049
  ver217 authored Jun 24, 2022
```
* zero optim supports loading local state dict

* polish code

* add unit test
```
  561e9049
08 Jun, 2022 1 commit
- [test] ignore 8 gpu test (#1080) · 65ee6dcc
  Frank Lee authored Jun 08, 2022
```
* [test] ignore 8 gpu test

* polish code

* polish workflow

* polish workflow
```
  65ee6dcc
24 Apr, 2022 1 commit
- [gemini] add GeminiMemoryManger (#832) · e5ea3fde
  HELSON authored Apr 24, 2022
```
* refactor StatefulTensor, tensor utilities

* add unitest for GeminiMemoryManager
```
  e5ea3fde
19 Apr, 2022 1 commit
- Revert "[zero] add ZeroTensorShardStrategy (#793)" (#806) · e761ad2c
  Jiarui Fang authored Apr 19, 2022
  
  e761ad2c