Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
OpenDAS
ColossalAI
Commits
14e5b11d
Unverified
Commit
14e5b11d
authored
Jun 10, 2022
by
Frank Lee
Committed by
GitHub
Jun 10, 2022
Browse files
[zero] fixed api consistency (#1098)
parent
cb18922c
Changes
2
Hide whitespace changes
Inline
Side-by-side
Showing
2 changed files
with
3 additions
and
3 deletions
+3
-3
colossalai/gemini/gemini_mgr.py
colossalai/gemini/gemini_mgr.py
+2
-2
colossalai/zero/zero_optimizer.py
colossalai/zero/zero_optimizer.py
+1
-1
No files found.
colossalai/gemini/gemini_mgr.py
View file @
14e5b11d
...
...
@@ -73,11 +73,11 @@ class GeminiManager:
def
_get_layout_info
(
self
,
compute_idx
:
int
,
warmup
:
bool
,
chunks
:
Tuple
[
Chunk
,
...],
group_name
:
str
):
cuda_demand
=
0
for
chunk
in
chunks
:
if
chunk
.
device_type
==
'cpu'
or
chunk
.
is_
free
:
if
chunk
.
device_type
==
'cpu'
or
chunk
.
is_
empty
:
cuda_demand
+=
chunk
.
mem
can_evict_chunks
=
[]
for
chunk
in
self
.
_chunk_manager
.
chunk_groups
[
group_name
]:
if
not
chunk
.
is_
free
and
chunk
.
device_type
==
'cuda'
and
chunk
.
can_move_device
:
if
not
chunk
.
is_
empty
and
chunk
.
device_type
==
'cuda'
and
chunk
.
can_move_device
:
can_evict_chunks
.
append
(
chunk
)
return
cuda_demand
,
can_evict_chunks
...
...
colossalai/zero/zero_optimizer.py
View file @
14e5b11d
...
...
@@ -136,7 +136,7 @@ class ZeroOptimizer(ColossalaiOptimizer):
fp32_params_used_cuda_margin_mem
=
0
for
fp16_param_chunk
,
fp32_param_chunk
in
zip
(
self
.
chunk_manager
.
chunk_groups
[
'fp16_param'
],
self
.
chunk_manager
.
chunk_groups
[
'fp32_param'
]):
if
fp32_param_chunk
.
is_
free
:
if
fp32_param_chunk
.
is_
empty
:
continue
if
fp32_params_used_cuda_margin_mem
+
fp32_param_chunk
.
mem
<
fp32_params_available_cuda_margin_mem
:
self
.
chunk_manager
.
move_chunk
(
fp32_param_chunk
,
get_current_device
())
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment