Commits · 2ecc3d7a55a1d6b9701116370a195d741893db1c · OpenDAS / ColossalAI

21 Apr, 2022 4 commits

[tensor] lazy init (#823) · 2ecc3d7a
Jiarui Fang authored Apr 21, 2022

2ecc3d7a

[Tensor] apply ColoTensor on Torch functions (#821) · 660d2d1f

Jiarui Fang authored Apr 21, 2022

* Revert "[zero] add ZeroTensorShardStrategy (#793)"

This reverts commit 88759e28.

* [gemini] set cpu memory capacity

* [log] local throughput collecting

* polish

* polish

* polish

* polish code

* polish

* polish code

* add a new tensor structure and override linear for it

* polish

* polish

* polish

* polish

* polish

* polish

* polish

* polish

* polish

* polish

* polish

* [tensor] renaming and reorganize directory structure.

* rm useless dir

* polish

* polish

* [tensor] hander the function not wrapped

660d2d1f

[tensor] reorganize files (#820) · 0ce8924c
Jiarui Fang authored Apr 21, 2022

0ce8924c

[gemini] a new tensor structure (#818) · ab962b97

Jiarui Fang authored Apr 21, 2022

* Revert "[zero] add ZeroTensorShardStrategy (#793)"

This reverts commit 88759e28.

* [gemini] set cpu memory capacity

* [log] local throughput collecting

* polish

* polish

* polish

* polish code

* polish

* polish code

* add a new tensor structure and override linear for it

* polish

* polish

* polish

* polish

* polish

* polish

* polish

* polish

* polish

* polish

* polish

ab962b97

19 Apr, 2022 4 commits
- Revert "[zero] add ZeroTensorShardStrategy (#793)" (#806) · e761ad2c
  Jiarui Fang authored Apr 19, 2022
  
  e761ad2c
- [zero] add ZeroTensorShardStrategy (#793) · 88759e28
  HELSON authored Apr 19, 2022
  
  88759e28
- [refactor] moving grad acc logic to engine (#804) · 681addb5
  Jiarui Fang authored Apr 19, 2022
  
  681addb5
- [refactor] moving memtracer to gemini (#801) · 4d9332b4
  Jiarui Fang authored Apr 19, 2022
  
  4d9332b4
18 Apr, 2022 1 commit
- [hotfix] fix memory leak in zero (#781) · 4c4388c4
  HELSON authored Apr 18, 2022
  
  4c4388c4
14 Apr, 2022 5 commits
- [test] refactored with the new rerun decorator (#763) · 5a1a095b
  Frank Lee authored Apr 15, 2022
```
* [test] refactored with the new rerun decorator

* polish test case
```
  5a1a095b
- [gemini] init genimi individual directory (#754) · 10ef8afd
  Jiarui Fang authored Apr 14, 2022
  
  10ef8afd
- [hotfix] fix test_stateful_tensor_mgr (#762) · dcca614e
  ver217 authored Apr 14, 2022
  
  dcca614e
- [hotfix] fix reuse_fp16_shard of sharded model (#756) · a93a7d73
  ver217 authored Apr 14, 2022
```
* fix reuse_fp16_shard

* disable test stm

* polish code
```
  a93a7d73
- [zero] refactor memstats_collector (#746) · 84c6700b
  HELSON authored Apr 14, 2022
  
  84c6700b
13 Apr, 2022 2 commits
- [zero] add tensor placement policies (#743) · e396bb71
  ver217 authored Apr 13, 2022
```
* add tensor placement policies

* polish comments

* polish comments

* update moe unit tests
```
  e396bb71
- [zero] refactor ShardedParamV2 for convenience (#742) · 22c4b88d
  HELSON authored Apr 13, 2022
  
  22c4b88d
12 Apr, 2022 5 commits
- [bug] fixed DDP compatibility with torch 1.8 (#739) · f4f42d4c
  Frank Lee authored Apr 13, 2022
  
  f4f42d4c
- [utils] correct cpu memory used and capacity in the context of multi-process (#726) · 53cb5848
  Jiarui Fang authored Apr 12, 2022
  
  53cb5848
- [moe] add checkpoint for moe zero test (#729) · b9b469ea
  HELSON authored Apr 12, 2022
  
  b9b469ea
- [test] removed trivial outdated test · e88a498c
  FrankLeeeee authored Apr 12, 2022
  
  e88a498c
- [test] added missing decorators to model checkpointing tests · 62b4ce73
  FrankLeeeee authored Apr 12, 2022
  
  62b4ce73
11 Apr, 2022 6 commits
- [refactor] zero directory (#724) · 4d90a7b5
  Jiarui Fang authored Apr 11, 2022
  
  4d90a7b5
- [bug] fixed broken test_found_inf (#725) · 20ab1f55
  Frank Lee authored Apr 11, 2022
  
  20ab1f55
- [refactor] refactor the memory utils (#715) · 193dc8da
  Jiarui Fang authored Apr 11, 2022
  
  193dc8da
- [zero] check whether gradients have inf and nan in gpu (#712) · dbd96fe9
  HELSON authored Apr 11, 2022
  
  dbd96fe9
- [zero] improve adaptability for not-shard parameters (#708) · a9b8300d
  HELSON authored Apr 11, 2022
```
* adapt post grad hooks for not-shard parameters
* adapt optimizer for not-shard parameters
* offload gradients for not-replicated parameters
```
  a9b8300d
- [zero] refactor memstats collector (#706) · ab8c6b4a
  ver217 authored Apr 11, 2022
```
* refactor memstats collector

* fix disposable

* polish code
```
  ab8c6b4a
08 Apr, 2022 2 commits

[zero] adapt zero hooks for unsharded module (#699) · ee112fe1
HELSON authored Apr 08, 2022

ee112fe1

[zero] stateful tensor manager (#687) · 3c9cd5bb

ver217 authored Apr 08, 2022



* [WIP] stateful tensor manager

* add eviction strategy

* polish code

* polish code

* polish comment

* add unit test

* fix sampler bug

* polish code

* fix max sampling cnt resetting bug

* fix sampler bug

* polish code

* fix bug

* fix unit test
Co-authored-by: jiaruifang <fangjiarui123@gmail.com>

3c9cd5bb

07 Apr, 2022 1 commit
- [zero] fix init bugs in zero context (#686) · d7ecaf36
  HELSON authored Apr 07, 2022
```
* adapt model weight initialization for methods in Pytorch nn.init
```
  d7ecaf36
03 Apr, 2022 2 commits
- [hotfix] fix a bug in model data stats tracing (#655) · 0aab5230
  Jiarui Fang authored Apr 03, 2022
  
  0aab5230
- [refactor] pipeline, put runtime schedule into engine. (#627) · ade05a5d
  YuliangLiu0306 authored Apr 03, 2022
  
  ade05a5d
02 Apr, 2022 2 commits

[hotfix] fix bugs in testing (#659) · e5d615ae

HELSON authored Apr 02, 2022

* remove hybrid adam in test_moe_zero_optim

* fix activation checkpointing and its unitest

e5d615ae

fix bugs in CPU adam (#633) · b31daed4

HELSON authored Apr 02, 2022

* add cpu adam counter for all cpu adam

* fixed updating error in adam kernel

b31daed4

01 Apr, 2022 4 commits
- [zero] adapt zero for unsharded paramters (Optimizer part) (#601) · 055fbf5b
  HELSON authored Apr 01, 2022
  
  055fbf5b
- [model checkpoint] added unit tests for checkpoint save/load (#599) · 354b7954
  アマデウス authored Apr 01, 2022
  
  354b7954
- [zero] test zero tensor utils (#609) · 93f14d2a
  FredHuang99 authored Apr 01, 2022
  
  93f14d2a
- [refactor] memory utils (#577) · e956d93a
  Jiarui Fang authored Apr 01, 2022
  
  e956d93a
31 Mar, 2022 2 commits
- [zero] adapt zero for unsharded parameters (#561) · e6d50ec1
  HELSON authored Mar 31, 2022
```
* support existing sharded and unsharded parameters in zero

* add unitest for moe-zero model init

* polish moe gradient handler
```
  e6d50ec1
- [zero] trace states of fp16/32 grad and fp32 param (#571) · 7c6c427d
  ver217 authored Mar 31, 2022
  
  7c6c427d