Commits · 80e37eec4215cd81a54a525af14a1030dd6177a1 · OpenDAS / ColossalAI

14 Apr, 2022 12 commits
- fix the ckpt bugs when using DDP (#769) · 80e37eec
  LuGY authored Apr 14, 2022
  
  80e37eec
- [readme] polish readme (#764) · 1f698f44
  Jiarui Fang authored Apr 14, 2022
```
* [readme] polish readme

* centering image
```
  1f698f44
- [compatibility] used backward-compatible API for global process group (#758) · 920fe315
  Frank Lee authored Apr 14, 2022
  
  920fe315
- [test] added a decorator for address already in use error with backward compatibility (#760) · 4ea49cb5
  Frank Lee authored Apr 14, 2022
```
* [test] added a decorator for address already in use error with backward compatibility

* [test] added a decorator for address already in use error with backward compatibility
```
  4ea49cb5
- [gemini] init genimi individual directory (#754) · 10ef8afd
  Jiarui Fang authored Apr 14, 2022
  
  10ef8afd
- [hotfix] fix test_stateful_tensor_mgr (#762) · dcca614e
  ver217 authored Apr 14, 2022
  
  dcca614e
- Automated submodule synchronization (#751) · 6978980f
  github-actions[bot] authored Apr 14, 2022
```
Co-authored-by: github-actions <github-actions@github.com>
```
  6978980f
- [hotfix] fix reuse_fp16_shard of sharded model (#756) · a93a7d73
  ver217 authored Apr 14, 2022
```
* fix reuse_fp16_shard

* disable test stm

* polish code
```
  a93a7d73
- [hotfix] fix auto tensor placement policy (#753) · 8f7ce94b
  ver217 authored Apr 14, 2022
  
  8f7ce94b
- [zero] refactor memstats_collector (#746) · 84c6700b
  HELSON authored Apr 14, 2022
  
  84c6700b
- [TP] allow layernorm without bias (#750) · b8899e09
  アマデウス authored Apr 14, 2022
  
  b8899e09
- [zero] use factory pattern for tensor_placement_policy (#752) · 3d7dc46d
  Jiarui Fang authored Apr 14, 2022
  
  3d7dc46d
13 Apr, 2022 6 commits
- fix prepare grads in sharded optim (#749) · 4b048a87
  ver217 authored Apr 13, 2022
  
  4b048a87
- fix initialize about zero · 09777254
  ver217 authored Apr 13, 2022
  
  09777254
- [zero] add tensor placement policies (#743) · e396bb71
  ver217 authored Apr 13, 2022
```
* add tensor placement policies

* polish comments

* polish comments

* update moe unit tests
```
  e396bb71
- [zero] refactor ShardedParamV2 for convenience (#742) · 22c4b88d
  HELSON authored Apr 13, 2022
  
  22c4b88d
- [utils] add synchronized cuda memory monitor (#740) · 340e59f9
  HELSON authored Apr 13, 2022
  
  340e59f9
- [hotfix] fix memory leak in backward of sharded model (#741) · e6212f56
  ver217 authored Apr 13, 2022
  
  e6212f56
12 Apr, 2022 13 commits
- [bug] fixed DDP compatibility with torch 1.8 (#739) · f4f42d4c
  Frank Lee authored Apr 13, 2022
  
  f4f42d4c
- [bug] fixed grad scaler compatibility with torch 1.8 (#735) · a4e91bc8
  Frank Lee authored Apr 12, 2022
  
  a4e91bc8
- [utils] correct cpu memory used and capacity in the context of multi-process (#726) · 53cb5848
  Jiarui Fang authored Apr 12, 2022
  
  53cb5848
- [hotfix] remove duplicated param register to stateful tensor manager (#728) · 7db3ccc7
  Jiarui Fang authored Apr 12, 2022
  
  7db3ccc7
- add video (#732) · 600e769a
  binmakeswell authored Apr 12, 2022
  
  600e769a
- [bug] removed zero installation requirements (#731) · a5c3f072
  Frank Lee authored Apr 12, 2022
  
  a5c3f072
- [moe] add checkpoint for moe zero test (#729) · b9b469ea
  HELSON authored Apr 12, 2022
  
  b9b469ea
- [doc] removed outdated installation command (#730) · 6f7d1362
  Frank Lee authored Apr 12, 2022
  
  6f7d1362
- [test] removed trivial outdated test · e88a498c
  FrankLeeeee authored Apr 12, 2022
  
  e88a498c
- [test] added missing decorators to model checkpointing tests · 62b4ce73
  FrankLeeeee authored Apr 12, 2022
  
  62b4ce73
- [util] fixed communication API depth with PyTorch 1.9 (#721) · 1cb7bdad
  Frank Lee authored Apr 12, 2022
  
  1cb7bdad
- [util] fixed activation checkpointing on torch 1.9 (#719) · 2412429d
  Frank Lee authored Apr 12, 2022
  
  2412429d
- [utils] support detection of number of processes on current node (#723) · 04ff5ea5
  Frank Lee authored Apr 12, 2022
  
  04ff5ea5
11 Apr, 2022 9 commits
- [refactor] zero directory (#724) · 4d90a7b5
  Jiarui Fang authored Apr 11, 2022
  
  4d90a7b5
- [bug] fixed broken test_found_inf (#725) · 20ab1f55
  Frank Lee authored Apr 11, 2022
  
  20ab1f55
- [refactor] refactor the memory utils (#715) · 193dc8da
  Jiarui Fang authored Apr 11, 2022
  
  193dc8da
- [zero] check whether gradients have inf and nan in gpu (#712) · dbd96fe9
  HELSON authored Apr 11, 2022
  
  dbd96fe9
- [hotfix] fix stm cuda model data size (#710) · 715b86ea
  ver217 authored Apr 11, 2022
  
  715b86ea
- [hotfix]fixed bugs of assigning grad states to non leaf nodes (#711) · 140263a3
  LuGY authored Apr 11, 2022
```
* fixed bugs of assigning grad states to non leaf nodes

* use detach()
```
  140263a3
- [compatibility] fixed tensor parallel compatibility with torch 1.9 (#700) · eda30a05
  Frank Lee authored Apr 11, 2022
  
  eda30a05
- [zero] improve adaptability for not-shard parameters (#708) · a9b8300d
  HELSON authored Apr 11, 2022
```
* adapt post grad hooks for not-shard parameters
* adapt optimizer for not-shard parameters
* offload gradients for not-replicated parameters
```
  a9b8300d
- [zero] refactor memstats collector (#706) · ab8c6b4a
  ver217 authored Apr 11, 2022
```
* refactor memstats collector

* fix disposable

* polish code
```
  ab8c6b4a