Commits · efba0f44b99f595731598d29792f5f229c7c0696 · OpenDAS / ColossalAI

05 Sep, 2023 2 commits
- [test] fix gemini checkpoint and gpt test (#4620) · bd186784
  Hongxin Liu authored Sep 05, 2023
  
  bd186784
- [zero] hotfix master param sync (#4618) · 807e01a4
  Hongxin Liu authored Sep 05, 2023
```
* [zero] add method to update master params

* [zero] update zero plugin

* [plugin] update low level zero plugin
```
  807e01a4
04 Sep, 2023 1 commit
- [checkpointio] support huggingface from_pretrained for all plugins (#4606) · e79b1e80
  Baizhou Zhang authored Sep 04, 2023
  
  e79b1e80
01 Sep, 2023 2 commits

[zero]fix zero ckptIO with offload (#4529) · cbac7822

LuGY authored Sep 01, 2023

* fix zero ckptio with offload

* fix load device

* saved tensors in ckpt should be on CPU

* fix unit test

* fix unit test

* add clear cache

* save memory for CI

cbac7822

[shardformer] support from_pretrained when loading model with HybridParallelPlugin (#4575) · 38ccb8b1
Baizhou Zhang authored Sep 01, 2023
```
* hybrid plugin support huggingface from_pretrained

* add huggingface compatibility tests

* add folder cleaning

* fix bugs
```
38ccb8b1

31 Aug, 2023 2 commits

[shardformer] support sharded optimizer checkpointIO of HybridParallelPlugin (#4540) · c9625dbb

Baizhou Zhang authored Aug 31, 2023

* implement sharded optimizer saving

* add more param info

* finish implementation of sharded optimizer saving

* fix bugs in optimizer sharded saving

* add pp+zero test

* param group loading

* greedy loading of optimizer

* fix bug when loading

* implement optimizer sharded saving

* add optimizer test & arrange checkpointIO utils

* fix gemini sharding state_dict

* add verbose option

* add loading of master params

* fix typehint

* fix master/working mapping in fp16 amp

c9625dbb

[shardformer] fix submodule replacement bug when enabling pp (#4544) · 2c787d7f
Baizhou Zhang authored Aug 31, 2023

2c787d7f

25 Aug, 2023 1 commit

[shardformer] support sharded checkpoint IO for models of HybridParallelPlugin (#4506) · 44eab2b2

Baizhou Zhang authored Aug 25, 2023

* add APIs

* implement save_sharded_model

* add test for hybrid checkpointio

* implement naive loading for sharded model

* implement efficient sharded model loading

* open a new file for hybrid checkpoint_io

* small fix

* fix circular importing

* fix docstring

* arrange arguments and apis

* small fix

44eab2b2

24 Aug, 2023 1 commit

[gemini] improve compatibility and add static placement policy (#4479) · 27061426

Hongxin Liu authored Aug 24, 2023

* [gemini] remove distributed-related part from colotensor (#4379)

* [gemini] remove process group dependency

* [gemini] remove tp part from colo tensor

* [gemini] patch inplace op

* [gemini] fix param op hook and update tests

* [test] remove useless tests

* [test] remove useless tests

* [misc] fix requirements

* [test] fix model zoo

* [test] fix model zoo

* [test] fix model zoo

* [test] fix model zoo

* [test] fix model zoo

* [misc] update requirements

* [gemini] refactor gemini optimizer and gemini ddp (#4398)

* [gemini] update optimizer interface

* [gemini] renaming gemini optimizer

* [gemini] refactor gemini ddp class

* [example] update gemini related example

* [example] update gemini related example

* [plugin] fix gemini plugin args

* [test] update gemini ckpt tests

* [gemini] fix checkpoint io

* [example] fix opt example requirements

* [example] fix opt example

* [example] fix opt example

* [example] fix opt example

* [gemini] add static placement policy (#4443)

* [gemini] add static placement policy

* [gemini] fix param offload

* [test] update gemini tests

* [plugin] update gemini plugin

* [plugin] update gemini plugin docstr

* [misc] fix flash attn requirement

* [test] fix gemini checkpoint io test

* [example] update resnet example result (#4457)

* [example] update bert example result (#4458)

* [doc] update gemini doc (#4468)

* [example] update gemini related examples (#4473)

* [example] update gpt example

* [example] update dreambooth example

* [example] update vit

* [example] update opt

* [example] update palm

* [example] update vit and opt benchmark

* [hotfix] fix bert in model zoo (#4480)

* [hotfix] fix bert in model zoo

* [test] remove chatglm gemini test

* [test] remove sam gemini test

* [test] remove vit gemini test

* [hotfix] fix opt tutorial example (#4497)

* [hotfix] fix opt tutorial example

* [hotfix] fix opt tutorial example

27061426

31 Jul, 2023 1 commit
- [zero] support shard optimizer state dict of zero (#4194) · 1a49a5ea
  LuGY authored Jul 11, 2023
```
* support shard optimizer of zero

* polish code

* support sync grad manually
```
  1a49a5ea
21 Jul, 2023 1 commit

[checkpointio] Sharded Optimizer Checkpoint for Gemini Plugin (#4302) · c6f60059

Baizhou Zhang authored Jul 21, 2023

* sharded optimizer checkpoint for gemini plugin

* modify test to reduce testing time

* update doc

* fix bug when keep_gatherd is true under GeminiPlugin

c6f60059

07 Jul, 2023 1 commit

Next commit [checkpointio] Unsharded Optimizer Checkpoint for Gemini Plugin (#4141) · 58913441

Baizhou Zhang authored Jul 07, 2023

* [checkpointio] unsharded optimizer checkpoint for Gemini plugin

* [checkpointio] unsharded optimizer checkpoint for Gemini using all_gather

58913441

04 Jul, 2023 1 commit
- [test] fixed tests failed due to dtensor change (#4082) · c4b1b659
  Frank Lee authored Jun 26, 2023
```
* [test] fixed tests failed due to dtensor change

* polish code
```
  c4b1b659
25 Jun, 2023 1 commit
- [gemini] fix argument naming during chunk configuration searching · 0bb0b481
  Baizhou Zhang authored Jun 25, 2023
  
  0bb0b481
16 Jun, 2023 2 commits
- [test] fixed codefactor format report (#4026) · a5883aa7
  Frank Lee authored Jun 16, 2023
  
  a5883aa7
- [checkpointio] sharded optimizer checkpoint for DDP plugin (#4002) · 822c3d4d
  Baizhou Zhang authored Jun 16, 2023
  
  822c3d4d
15 Jun, 2023 1 commit
- [checkpointio] General Checkpointing of Sharded Optimizers (#3984) · c9cff7e7
  Baizhou Zhang authored Jun 15, 2023
  
  c9cff7e7
23 May, 2023 1 commit
- [booster] torch fsdp fix ckpt (#3788) · 6b305a99
  wukong1992 authored May 23, 2023
  
  6b305a99
19 May, 2023 1 commit

[plugin] a workaround for zero plugins' optimizer checkpoint (#3780) · 3c07a284

Hongxin Liu authored May 19, 2023

* [test] refactor torch ddp checkpoint test

* [plugin] update low level zero optim checkpoint

* [plugin] update gemini optim checkpoint

3c07a284

18 May, 2023 1 commit

[plugin] torch ddp plugin supports sharded model checkpoint (#3775) · 5452df63

Hongxin Liu authored May 18, 2023

* [plugin] torch ddp plugin add save sharded model

* [test] fix torch ddp ckpt io test

* [test] fix torch ddp ckpt io test

* [test] fix low level zero plugin test

* [test] fix low level zero plugin test

* [test] add debug info

* [test] add debug info

* [test] add debug info

* [test] add debug info

* [test] add debug info

* [test] fix low level zero plugin test

* [test] fix low level zero plugin test

* [test] remove debug info

5452df63

15 May, 2023 1 commit

[devops] update torch version of CI (#3725) · afb239bb

Hongxin Liu authored May 15, 2023

* [test] fix flop tensor test

* [test] fix autochunk test

* [test] fix lazyinit test

* [devops] update torch version of CI

* [devops] enable testmon

* [devops] fix ci

* [devops] fix ci

* [test] fix checkpoint io test

* [test] fix cluster test

* [test] fix timm test

* [devops] fix ci

* [devops] fix ci

* [devops] fix ci

* [devops] fix ci

* [devops] force sync to test ci

* [test] skip fsdp test

afb239bb

10 May, 2023 1 commit

[booster] add tests for ddp and low level zero's checkpointio (#3715) · 20068ba1

jiangmingyan authored May 10, 2023

* [booster] update tests for booster

* [booster] update tests for booster

* [booster] update tests for booster

* [booster] update tests for booster

* [booster] update tests for booster

* [booster] update booster tutorials#3717, fix recursive check

20068ba1

05 May, 2023 1 commit

[booster] gemini plugin support shard checkpoint (#3610) · 307894f7

jiangmingyan authored May 05, 2023



* gemini plugin add shard checkpoint save/load

* gemini plugin add shard checkpoint save/load

* gemini plugin add shard checkpoint save/load

* gemini plugin add shard checkpoint save/load

* gemini plugin add shard checkpoint save/load

* gemini plugin add shard checkpoint save/load

* gemini plugin add shard checkpoint save/load

* gemini plugin add shard checkpoint save/load

* gemini plugin add shard checkpoint save/load

* gemini plugin add shard checkpoint save/load

* gemini plugin add shard checkpoint save/load

* gemini plugin add shard checkpoint save/load

* gemini plugin add shard checkpoint save/load

* gemini plugin add shard checkpoint save/load

* gemini plugin support shard checkpoint

* [API Refactoring]gemini plugin support shard checkpoint

* [API Refactoring]gemini plugin support shard checkpoint

* [API Refactoring]gemini plugin support shard checkpoint

* [API Refactoring]gemini plugin support shard checkpoint

* [API Refactoring]gemini plugin support shard checkpoint

* [API Refactoring]gemini plugin support shard checkpoint

* [API Refactoring]gemini plugin support shard checkpoint

* [API Refactoring]gemini plugin support shard checkpoint

* [API Refactoring]gemini plugin support shard checkpoint

* [API Refactoring]gemini plugin support shard checkpoint

* [API Refactoring]gemini plugin support shard checkpoint

* [API Refactoring]gemini plugin support shard checkpoint

* [API Refactoring]gemini plugin support shard checkpoint

---------
Co-authored-by: luchen <luchen@luchendeMBP.lan>
Co-authored-by: luchen <luchen@luchendeMacBook-Pro.local>

307894f7

06 Apr, 2023 2 commits

[checkpoint] support huggingface style sharded checkpoint (#3461) · 52a933e1

jiangmingyan authored Apr 06, 2023



* [checkpoint] support huggingface style sharded checkpoint

* [checkpoint] support huggingface style sharded checkpoint

* [checkpoint] support huggingface style sharded checkpoint

* [checkpoint] support huggingface style sharded checkpoint

* [checkpoint] support huggingface style sharded checkpoint

---------
Co-authored-by: luchen <luchen@luchendeMBP.lan>

52a933e1

[test] refactor tests with spawn (#3452) · 80eba05b

Frank Lee authored Apr 06, 2023

* [test] added spawn decorator

* polish code

* polish code

* polish code

* polish code

* polish code

* polish code

80eba05b

04 Apr, 2023 1 commit
- [checkpoint] refactored the API and added safetensors support (#3427) · 1beb85cc
  Frank Lee authored Apr 04, 2023
```
* [checkpoint] refactored the API and added safetensors support

* polish code
```
  1beb85cc
27 Mar, 2023 1 commit
- [booster] implemented the torch ddd + resnet example (#3232) · 73d3e4d3
  Frank Lee authored Mar 27, 2023
```
* [booster] implemented the torch ddd + resnet example

* polish code
```
  73d3e4d3
23 Mar, 2023 1 commit
- [api] implemented the checkpoint io module (#3205) · cd142fbe
  Frank Lee authored Mar 23, 2023
```
* [api] implemented the checkpoint io module

* polish code

* polish code
```
  cd142fbe