Fix nightly ci Gsm8k & Fix flashinfer backend kvcache quant (#4147)

ab7fba0e · yinfan98 · GitHub · bc1534ff · ab7fba0e
Unverified Commit ab7fba0e authored Mar 07, 2025 by yinfan98 Committed by GitHub Mar 06, 2025
Hide whitespace changes
Inline Side-by-side

Showing with 1 addition and 0 deletions

python/sglang/srt/layers/attention/flashinfer_backend.py python/sglang/srt/layers/attention/flashinfer_backend.py +1 -0

No files found.
--- a/python/sglang/srt/layers/attention/flashinfer_backend.py
+++ b/python/sglang/srt/layers/attention/flashinfer_backend.py
@@ -904,6 +904,7 @@ class FlashInferIndicesUpdaterPrefill:
            self.head_dim,
            1,
            q_data_type=self.q_data_type,
+            kv_data_type=self.data_type,
            custom_mask=custom_mask,
            non_blocking=True,
        )