Commit 2a79e7d5 authored by zhangzbb's avatar zhangzbb
Browse files

Merge branch 'v0.15.1-dev_laibao' into 'v0.15.1-dev'

[BUGFIX] 修复 Qwen3-MoE Attention 中 fused RMS RoPE 的 epsilon 参数顺序错误

See merge request dcutoolkit/deeplearing/vllm!539
parents e55ba677 dec8b944
...@@ -391,9 +391,9 @@ class Qwen3MoeAttention(nn.Module): ...@@ -391,9 +391,9 @@ class Qwen3MoeAttention(nn.Module):
self.rotary_emb.is_neox_style, self.rotary_emb.is_neox_style,
self.q_norm.weight, self.q_norm.weight,
self.k_norm.weight, self.k_norm.weight,
self.q_norm.variance_epsilon,
None, None,
None, None,
self.q_norm.variance_epsilon,
) )
elif envs.VLLM_USE_FUSED_RMS_ROPE and positions.ndim == 2 and getattr( elif envs.VLLM_USE_FUSED_RMS_ROPE and positions.ndim == 2 and getattr(
self.rotary_emb, "mrope_section", None) is not None: self.rotary_emb, "mrope_section", None) is not None:
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment