Skip to content

GitLab

"git@developer.sourcefind.cn:change/sglang.git" did not exist on "de89ef49da9314da0395287ff2e4ee2e9a94ccc2"
Unverified Commit 538fa0ae authored by Yixin Dong's avatar Yixin Dong Committed by GitHub
Browse files

[Fix] Avoid calling fill_vocab_mask for terminated requests (#2175)

parent 55842eb8
Hide whitespace changes
Inline Side-by-side
Showing with 4 additions and 1 deletion
python/sglang/srt/sampling/sampling_batch_info.py
View file @ 538fa0ae
...@@ -170,7 +170,10 @@ class SamplingBatchInfo: ...@@ -170,7 +170,10 @@ class SamplingBatchInfo:
for i, grammar in enumerate(self.grammars): for i, grammar in enumerate(self.grammars):
if grammar is not None: if grammar is not None:
grammar.fill_vocab_mask(self.vocab_mask, i) try:
grammar.fill_vocab_mask(self.vocab_mask, i)
except RuntimeError:
continue
def filter_batch(self, unfinished_indices: List[int], new_indices: torch.Tensor): def filter_batch(self, unfinished_indices: List[int], new_indices: torch.Tensor):
self.penalizer_orchestrator.filter(unfinished_indices, new_indices) self.penalizer_orchestrator.filter(unfinished_indices, new_indices)
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment