Commits · 01e9f834f533be6d2bde61fa39e5b2c891dfd7ce · OpenDAS / ColossalAI

22 Apr, 2022 4 commits
- [dependency] removed torchvision (#833) · 01e9f834
  Frank Lee authored Apr 22, 2022
```
* [dependency] removed torchvision

* fixed transforms
```
  01e9f834
- Revert "[WIP] Applying ColoTensor on TP-1D-row Linear. (#831)" (#835) · cb5a4778
  Jiarui Fang authored Apr 22, 2022
```
This reverts commit ac88de6d.
```
  cb5a4778
- [WIP] Applying ColoTensor on TP-1D-row Linear. (#831) · ac88de6d
  Jiarui Fang authored Apr 22, 2022
```
* revert zero tensors back

* [tensor] init row 1d linear
```
  ac88de6d
- [tensor] ZeRO use ColoTensor as the base class. (#828) · 294a6060
  Jiarui Fang authored Apr 22, 2022
```
* [refactor] moving InsertPostInitMethodToModuleSubClasses to utils.

* [tensor] ZeRO use ColoTensor as the base class.

* polish
```
  294a6060
21 Apr, 2022 6 commits

[tensor]fix test_linear (#826) · 8e6fdb4f
Ziyue Jiang authored Apr 21, 2022

8e6fdb4f
[tensor] fix kwargs in colo_tensor torch_funtion (#825) · 1a9e2c2d
Ziyue Jiang authored Apr 21, 2022

1a9e2c2d
[tensor] lazy init (#823) · 2ecc3d7a
Jiarui Fang authored Apr 21, 2022

2ecc3d7a

[Tensor] apply ColoTensor on Torch functions (#821) · 660d2d1f

Jiarui Fang authored Apr 21, 2022

* Revert "[zero] add ZeroTensorShardStrategy (#793)"

This reverts commit 88759e28.

* [gemini] set cpu memory capacity

* [log] local throughput collecting

* polish

* polish

* polish

* polish code

* polish

* polish code

* add a new tensor structure and override linear for it

* polish

* polish

* polish

* polish

* polish

* polish

* polish

* polish

* polish

* polish

* polish

* [tensor] renaming and reorganize directory structure.

* rm useless dir

* polish

* polish

* [tensor] hander the function not wrapped

660d2d1f

[tensor] reorganize files (#820) · 0ce8924c
Jiarui Fang authored Apr 21, 2022

0ce8924c

[gemini] a new tensor structure (#818) · ab962b97

Jiarui Fang authored Apr 21, 2022

* Revert "[zero] add ZeroTensorShardStrategy (#793)"

This reverts commit 88759e28.

* [gemini] set cpu memory capacity

* [log] local throughput collecting

* polish

* polish

* polish

* polish code

* polish

* polish code

* add a new tensor structure and override linear for it

* polish

* polish

* polish

* polish

* polish

* polish

* polish

* polish

* polish

* polish

* polish

ab962b97

19 Apr, 2022 4 commits
- Revert "[zero] add ZeroTensorShardStrategy (#793)" (#806) · e761ad2c
  Jiarui Fang authored Apr 19, 2022
  
  e761ad2c
- [zero] add ZeroTensorShardStrategy (#793) · 88759e28
  HELSON authored Apr 19, 2022
  
  88759e28
- [refactor] moving grad acc logic to engine (#804) · 681addb5
  Jiarui Fang authored Apr 19, 2022
  
  681addb5
- [refactor] moving memtracer to gemini (#801) · 4d9332b4
  Jiarui Fang authored Apr 19, 2022
  
  4d9332b4
18 Apr, 2022 1 commit
- [hotfix] fix memory leak in zero (#781) · 4c4388c4
  HELSON authored Apr 18, 2022
  
  4c4388c4
14 Apr, 2022 5 commits
- [test] refactored with the new rerun decorator (#763) · 5a1a095b
  Frank Lee authored Apr 15, 2022
```
* [test] refactored with the new rerun decorator

* polish test case
```
  5a1a095b
- [gemini] init genimi individual directory (#754) · 10ef8afd
  Jiarui Fang authored Apr 14, 2022
  
  10ef8afd
- [hotfix] fix test_stateful_tensor_mgr (#762) · dcca614e
  ver217 authored Apr 14, 2022
  
  dcca614e
- [hotfix] fix reuse_fp16_shard of sharded model (#756) · a93a7d73
  ver217 authored Apr 14, 2022
```
* fix reuse_fp16_shard

* disable test stm

* polish code
```
  a93a7d73
- [zero] refactor memstats_collector (#746) · 84c6700b
  HELSON authored Apr 14, 2022
  
  84c6700b
13 Apr, 2022 2 commits
- [zero] add tensor placement policies (#743) · e396bb71
  ver217 authored Apr 13, 2022
```
* add tensor placement policies

* polish comments

* polish comments

* update moe unit tests
```
  e396bb71
- [zero] refactor ShardedParamV2 for convenience (#742) · 22c4b88d
  HELSON authored Apr 13, 2022
  
  22c4b88d
12 Apr, 2022 5 commits
- [bug] fixed DDP compatibility with torch 1.8 (#739) · f4f42d4c
  Frank Lee authored Apr 13, 2022
  
  f4f42d4c
- [utils] correct cpu memory used and capacity in the context of multi-process (#726) · 53cb5848
  Jiarui Fang authored Apr 12, 2022
  
  53cb5848
- [moe] add checkpoint for moe zero test (#729) · b9b469ea
  HELSON authored Apr 12, 2022
  
  b9b469ea
- [test] removed trivial outdated test · e88a498c
  FrankLeeeee authored Apr 12, 2022
  
  e88a498c
- [test] added missing decorators to model checkpointing tests · 62b4ce73
  FrankLeeeee authored Apr 12, 2022
  
  62b4ce73
11 Apr, 2022 6 commits
- [refactor] zero directory (#724) · 4d90a7b5
  Jiarui Fang authored Apr 11, 2022
  
  4d90a7b5
- [bug] fixed broken test_found_inf (#725) · 20ab1f55
  Frank Lee authored Apr 11, 2022
  
  20ab1f55
- [refactor] refactor the memory utils (#715) · 193dc8da
  Jiarui Fang authored Apr 11, 2022
  
  193dc8da
- [zero] check whether gradients have inf and nan in gpu (#712) · dbd96fe9
  HELSON authored Apr 11, 2022
  
  dbd96fe9
- [zero] improve adaptability for not-shard parameters (#708) · a9b8300d
  HELSON authored Apr 11, 2022
```
* adapt post grad hooks for not-shard parameters
* adapt optimizer for not-shard parameters
* offload gradients for not-replicated parameters
```
  a9b8300d
- [zero] refactor memstats collector (#706) · ab8c6b4a
  ver217 authored Apr 11, 2022
```
* refactor memstats collector

* fix disposable

* polish code
```
  ab8c6b4a
08 Apr, 2022 2 commits

[zero] adapt zero hooks for unsharded module (#699) · ee112fe1
HELSON authored Apr 08, 2022

ee112fe1

[zero] stateful tensor manager (#687) · 3c9cd5bb

ver217 authored Apr 08, 2022



* [WIP] stateful tensor manager

* add eviction strategy

* polish code

* polish code

* polish comment

* add unit test

* fix sampler bug

* polish code

* fix max sampling cnt resetting bug

* fix sampler bug

* polish code

* fix bug

* fix unit test
Co-authored-by: jiaruifang <fangjiarui123@gmail.com>

3c9cd5bb

07 Apr, 2022 1 commit
- [zero] fix init bugs in zero context (#686) · d7ecaf36
  HELSON authored Apr 07, 2022
```
* adapt model weight initialization for methods in Pytorch nn.init
```
  d7ecaf36
03 Apr, 2022 2 commits
- [hotfix] fix a bug in model data stats tracing (#655) · 0aab5230
  Jiarui Fang authored Apr 03, 2022
  
  0aab5230
- [refactor] pipeline, put runtime schedule into engine. (#627) · ade05a5d
  YuliangLiu0306 authored Apr 03, 2022
  
  ade05a5d
02 Apr, 2022 2 commits

[hotfix] fix bugs in testing (#659) · e5d615ae

HELSON authored Apr 02, 2022

* remove hybrid adam in test_moe_zero_optim

* fix activation checkpointing and its unitest

e5d615ae

fix bugs in CPU adam (#633) · b31daed4

HELSON authored Apr 02, 2022

* add cpu adam counter for all cpu adam

* fixed updating error in adam kernel

b31daed4