Skip to content

GitLab

Commit c340a5df authored by zhuwenwen's avatar zhuwenwen
Browse files

revert DEFAULT_MAX_NUM_BATCHED_TOKENS = 2048

parent 6d25880c
Hide whitespace changes
Inline Side-by-side
Showing with 1 addition and 1 deletion
vllm/utils/__init__.py
View file @ c340a5df
...@@ -81,7 +81,7 @@ logger = init_logger(__name__) ...@@ -81,7 +81,7 @@ logger = init_logger(__name__)
# This value is chosen to have a balance between ITL and TTFT. Note it is # This value is chosen to have a balance between ITL and TTFT. Note it is
# not optimized for throughput. # not optimized for throughput.
DEFAULT_MAX_NUM_BATCHED_TOKENS = 10240 DEFAULT_MAX_NUM_BATCHED_TOKENS = 2048
POOLING_MODEL_MAX_NUM_BATCHED_TOKENS = 32768 POOLING_MODEL_MAX_NUM_BATCHED_TOKENS = 32768
MULTIMODAL_MODEL_MAX_NUM_BATCHED_TOKENS = 5120 MULTIMODAL_MODEL_MAX_NUM_BATCHED_TOKENS = 5120
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment