Skip to content

GitLab

Unverified Commit 4078052f authored by Michael Goin's avatar Michael Goin Committed by GitHub
Browse files

[V1][Log] Add max request concurrency log to V1 (#12569) 


Signed-off-by: default avatarmgoin <michael@neuralmagic.com>
parent bd2107e3
Show whitespace changes
Inline Side-by-side
Showing with 4 additions and 0 deletions
vllm/v1/core/kv_cache_utils.py
View file @ 4078052f
...@@ -393,6 +393,10 @@ def _get_kv_cache_config_uniform_type(vllm_config: VllmConfig, ...@@ -393,6 +393,10 @@ def _get_kv_cache_config_uniform_type(vllm_config: VllmConfig,
num_blocks = num_gpu_blocks_override num_blocks = num_gpu_blocks_override
logger.info("# GPU blocks: %d", num_blocks) logger.info("# GPU blocks: %d", num_blocks)
max_concurrency = (num_blocks * vllm_config.cache_config.block_size /
vllm_config.model_config.max_model_len)
logger.info("Maximum concurrency for %s tokens per request: %.2fx",
vllm_config.model_config.max_model_len, max_concurrency)
per_layer_size = page_size * num_blocks per_layer_size = page_size * num_blocks
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment