[Fix] Fix seq_lens_sum for cuda graph runner in padded cases (#1789)

86a2c473 · Lianmin Zheng · GitHub · 1701b0db · 86a2c473
Unverified Commit 86a2c473 authored Oct 24, 2024 by Lianmin Zheng Committed by GitHub Oct 24, 2024
Hide whitespace changes
Inline Side-by-side

Showing with 1 addition and 1 deletion

python/sglang/srt/model_executor/cuda_graph_runner.py python/sglang/srt/model_executor/cuda_graph_runner.py +1 -1

No files found.
--- a/python/sglang/srt/model_executor/cuda_graph_runner.py
+++ b/python/sglang/srt/model_executor/cuda_graph_runner.py
@@ -307,7 +307,7 @@ class CudaGraphRunner:
            bs,
            self.req_pool_indices,
            self.seq_lens,
-            forward_batch.seq_lens_sum,
+            forward_batch.seq_lens_sum + (bs - raw_bs),
            self.encoder_lens,
        )