Fix prefill oom (#1743)

efb099cd · Liangsheng Yin · GitHub · 09603c6d · efb099cd
Unverified Commit efb099cd authored Oct 21, 2024 by Liangsheng Yin Committed by GitHub Oct 21, 2024
Show whitespace changes
Inline Side-by-side

Showing with 1 addition and 1 deletion

python/sglang/srt/managers/scheduler.py python/sglang/srt/managers/scheduler.py +1 -1

No files found.
--- a/python/sglang/srt/managers/scheduler.py
+++ b/python/sglang/srt/managers/scheduler.py
@@ -427,7 +427,7 @@ class Scheduler:
                if req.sampling_params.max_new_tokens is not None
                else 1 << 30
            ),
-            self.max_req_input_len - 1 - len(req.origin_input_ids),
+            self.max_req_input_len - len(req.origin_input_ids),
        )

        self.waiting_queue.append(req)