Skip to content

GitLab

Unverified Commit d3ea5011 authored by Woosuk Kwon's avatar Woosuk Kwon Committed by GitHub
Browse files

[V1][Minor] Print KV cache size in token counts (#13596) 


Signed-off-by: default avatarWoosuk Kwon <woosuk.kwon@berkeley.edu>
parent 34aad515
Show whitespace changes
Inline Side-by-side
Showing with 6 additions and 4 deletions
vllm/v1/core/kv_cache_utils.py
View file @ d3ea5011
...@@ -519,11 +519,13 @@ def _get_kv_cache_config_uniform_type(vllm_config: VllmConfig, ...@@ -519,11 +519,13 @@ def _get_kv_cache_config_uniform_type(vllm_config: VllmConfig,
"num_gpu_blocks_override=%d", num_blocks, num_gpu_blocks_override) "num_gpu_blocks_override=%d", num_blocks, num_gpu_blocks_override)
num_blocks = num_gpu_blocks_override num_blocks = num_gpu_blocks_override
logger.info("# GPU blocks: %d", num_blocks) num_tokens = num_blocks * vllm_config.cache_config.block_size
max_concurrency = (num_blocks * vllm_config.cache_config.block_size / num_tokens_str = f"{num_tokens:,}"
vllm_config.model_config.max_model_len) logger.info("GPU KV cache size: %s tokens", num_tokens_str)
max_model_len_str = f"{vllm_config.model_config.max_model_len:,}"
max_concurrency = num_tokens / vllm_config.model_config.max_model_len
logger.info("Maximum concurrency for %s tokens per request: %.2fx", logger.info("Maximum concurrency for %s tokens per request: %.2fx",
vllm_config.model_config.max_model_len, max_concurrency) max_model_len_str, max_concurrency)
per_layer_size = page_size * num_blocks per_layer_size = page_size * num_blocks
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment