Skip to content

GitLab

Unverified Commit 2bb246b8 authored by Ning Xie's avatar Ning Xie Committed by GitHub
Browse files

[MISC] add cpu_kvcache_space_bytes to CacheConfig (#19812) 


Signed-off-by: default avatarAndy Xie <andy.xning@gmail.com>
parent 4c409cab
Show whitespace changes
Inline Side-by-side
Showing with 2 additions and 0 deletions
vllm/config.py
View file @ 2bb246b8
...@@ -1506,6 +1506,8 @@ class CacheConfig: ...@@ -1506,6 +1506,8 @@ class CacheConfig:
"""This enables dynamic calculation of `k_scale` and `v_scale` when """This enables dynamic calculation of `k_scale` and `v_scale` when
kv_cache_dtype is fp8. If `False`, the scales will be loaded from the model kv_cache_dtype is fp8. If `False`, the scales will be loaded from the model
checkpoint if available. Otherwise, the scales will default to 1.0.""" checkpoint if available. Otherwise, the scales will default to 1.0."""
cpu_kvcache_space_bytes: Optional[int] = None
"""(CPU backend only) CPU key-value cache space."""
# Will be set after profiling. # Will be set after profiling.
num_gpu_blocks: Optional[int] = field(default=None, init=False) num_gpu_blocks: Optional[int] = field(default=None, init=False)
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment