Skip to content

GitLab

Unverified Commit a869baca authored by Cyrus Leung's avatar Cyrus Leung Committed by GitHub
Browse files

[Bugfix] Fix Llama GGUF initialization (#18717) 


Signed-off-by: default avatarDarkLight1337 <tlleungac@connect.ust.hk>
parent 82e2339b
Hide whitespace changes
Inline Side-by-side
Showing with 1 addition and 1 deletion
vllm/model_executor/models/llama.py
View file @ a869baca
...@@ -208,7 +208,7 @@ class LlamaAttention(nn.Module): ...@@ -208,7 +208,7 @@ class LlamaAttention(nn.Module):
quant_config: Optional[QuantizationConfig]) -> None: quant_config: Optional[QuantizationConfig]) -> None:
is_neox_style = True is_neox_style = True
is_gguf = quant_config and quant_config.get_name() == "gguf" is_gguf = quant_config and quant_config.get_name() == "gguf"
if is_gguf and self.config.model_type == "llama": if is_gguf and config.model_type == "llama":
is_neox_style = False is_neox_style = False
self.rotary_emb = get_rope( self.rotary_emb = get_rope(
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment