Unverified Commit 2bb246b8 authored by Ning Xie's avatar Ning Xie Committed by GitHub
Browse files

[MISC] add cpu_kvcache_space_bytes to CacheConfig (#19812)


Signed-off-by: default avatarAndy Xie <andy.xning@gmail.com>
parent 4c409cab
......@@ -1506,6 +1506,8 @@ class CacheConfig:
"""This enables dynamic calculation of `k_scale` and `v_scale` when
kv_cache_dtype is fp8. If `False`, the scales will be loaded from the model
checkpoint if available. Otherwise, the scales will default to 1.0."""
cpu_kvcache_space_bytes: Optional[int] = None
"""(CPU backend only) CPU key-value cache space."""
# Will be set after profiling.
num_gpu_blocks: Optional[int] = field(default=None, init=False)
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment