Skip to content

GitLab

Unverified Commit 21906a6f authored by Ahmad Fahadh Ilyas's avatar Ahmad Fahadh Ilyas Committed by GitHub
Browse files

[Bugfix] Fix lora loading for Compressed Tensors in #9120 (#9179)

parent dc4aea67
Hide whitespace changes
Inline Side-by-side
Showing with 3 additions and 0 deletions
vllm/lora/layers.py
View file @ 21906a6f
...@@ -39,6 +39,9 @@ def _get_lora_device(base_layer: nn.Module) -> torch.device: ...@@ -39,6 +39,9 @@ def _get_lora_device(base_layer: nn.Module) -> torch.device:
# unquantizedLinear # unquantizedLinear
if hasattr(base_layer, "weight"): if hasattr(base_layer, "weight"):
return base_layer.weight.device return base_layer.weight.device
# Compressed Tensor
elif hasattr(base_layer, "weight_packed"):
return base_layer.weight_packed.device
# GPTQ/AWQ # GPTQ/AWQ
elif hasattr(base_layer, "qweight"): elif hasattr(base_layer, "qweight"):
return base_layer.qweight.device return base_layer.qweight.device
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment