[V1]Enable APC by default only for text models (#10148)

Signed-off-by: Roger Wang <ywang@roblox.com>

[V1]Enable APC by default only for text models (#10148)
Signed-off-by: Roger Wang <ywang@roblox.com>
208ce622 · Roger Wang · GitHub · 1ff4aed5 · 208ce622
Unverified Commit 208ce622 authored Nov 08, 2024 by Roger Wang Committed by GitHub Nov 08, 2024
Show whitespace changes
Inline Side-by-side

Showing with 4 additions and 1 deletion

vllm/v1/engine/llm_engine.py vllm/v1/engine/llm_engine.py +4 -1

No files found.
--- a/vllm/v1/engine/llm_engine.py
+++ b/vllm/v1/engine/llm_engine.py
@@ -65,6 +65,9 @@ class LLMEngine:
        elif usage_context == UsageContext.OPENAI_API_SERVER:
            scheduler_config.max_num_seqs = 1024
            scheduler_config.max_num_batched_tokens = 2048
+
+        # TODO (ywang96): Enable APC by default when VLM supports it.
+        if not model_config.is_multimodal_model:
            cache_config.enable_prefix_caching = True

        logger.info(