Skip to content

GitLab

Unverified Commit 35b65cf0 authored by fzyzcjy's avatar fzyzcjy Committed by GitHub
Browse files

Use deepgemm instead of triton for fused_qkv_a_proj_with_mqa (#6890)

parent dd1012fc
Hide whitespace changes
Inline Side-by-side
Showing with 2 additions and 2 deletions
python/sglang/srt/layers/quantization/fp8_utils.py
View file @ 35b65cf0
...@@ -227,8 +227,8 @@ def deepgemm_w8a8_block_fp8_linear_with_fallback( ...@@ -227,8 +227,8 @@ def deepgemm_w8a8_block_fp8_linear_with_fallback(
output_dtype = input.dtype output_dtype = input.dtype
dtype_supported = output_dtype == torch.bfloat16 dtype_supported = output_dtype == torch.bfloat16
# TODO: add more robust shape check here # TODO: https://github.com/sgl-project/sglang/pull/6890#issuecomment-2943395737
shape_supported = weight.shape[0] % 128 == 0 and weight.shape[1] % 128 == 0 shape_supported = weight.shape[0] % 64 == 0 and weight.shape[1] % 128 == 0
if not (shape_supported and dtype_supported): if not (shape_supported and dtype_supported):
# fall back to triton # fall back to triton
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment