Skip to content

GitLab

Unverified Commit 58e234a7 authored by Jee Jee Li's avatar Jee Jee Li Committed by GitHub
Browse files

[Misc] V1 LoRA support CPU offload (#15843) 


Signed-off-by: default avatarJee Jee Li <pandaleefree@gmail.com>
parent e86c414d
Hide whitespace changes
Inline Side-by-side
Showing with 3 additions and 3 deletions
vllm/config.py
View file @ 58e234a7
...@@ -2434,9 +2434,9 @@ class LoRAConfig: ...@@ -2434,9 +2434,9 @@ class LoRAConfig:
f"max_loras ({self.max_loras})") f"max_loras ({self.max_loras})")
def verify_with_cache_config(self, cache_config: CacheConfig): def verify_with_cache_config(self, cache_config: CacheConfig):
# TODO LoRA supports CPU offload. if cache_config.cpu_offload_gb > 0 and not envs.VLLM_USE_V1:
if cache_config.cpu_offload_gb > 0: raise ValueError(
raise ValueError("CPU offload is not supported with LoRA yet.") "V0 LoRA does not support CPU offload, please use V1.")
def verify_with_model_config(self, model_config: ModelConfig): def verify_with_model_config(self, model_config: ModelConfig):
if self.lora_dtype in (None, "auto"): if self.lora_dtype in (None, "auto"):
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment