Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in
Toggle navigation
Menu
Open sidebar
OpenDAS
vllm_cscc
Commits
e5ac6a41
Unverified
Commit
e5ac6a41
authored
Oct 23, 2024
by
Tyler Michael Smith
Committed by
GitHub
Oct 23, 2024
Browse files
[Bugfix] Fix divide by zero when serving Mamba models (#9617)
Signed-off-by:
Tyler Michael Smith
<
tyler@neuralmagic.com
>
parent
dbdd3b5e
Changes
1
Show whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
2 additions
and
2 deletions
+2
-2
vllm/engine/llm_engine.py
vllm/engine/llm_engine.py
+2
-2
No files found.
vllm/engine/llm_engine.py
View file @
e5ac6a41
...
...
@@ -1612,7 +1612,7 @@ class LLMEngine:
# KV Cache Usage in %
num_total_gpu
=
self
.
cache_config
.
num_gpu_blocks
gpu_cache_usage_sys
=
0.
if
num_total_gpu
is
n
ot
None
:
if
num_total_gpu
:
# Guard against b
ot
h
None
and 0
num_free_gpu
=
sum
(
scheduler
.
block_manager
.
get_num_free_gpu_blocks
()
for
scheduler
in
self
.
scheduler
)
...
...
@@ -1620,7 +1620,7 @@ class LLMEngine:
num_total_cpu
=
self
.
cache_config
.
num_cpu_blocks
cpu_cache_usage_sys
=
0.
if
num_total_cpu
is
n
ot
None
and
num_total_cpu
>
0
:
if
num_total_cpu
:
# Guard against b
ot
h
None and
0
num_free_cpu
=
sum
(
scheduler
.
block_manager
.
get_num_free_cpu_blocks
()
for
scheduler
in
self
.
scheduler
)
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment