Unverified Commit 538fa0ae authored by Yixin Dong's avatar Yixin Dong Committed by GitHub
Browse files

[Fix] Avoid calling fill_vocab_mask for terminated requests (#2175)

parent 55842eb8
......@@ -170,7 +170,10 @@ class SamplingBatchInfo:
for i, grammar in enumerate(self.grammars):
if grammar is not None:
try:
grammar.fill_vocab_mask(self.vocab_mask, i)
except RuntimeError:
continue
def filter_batch(self, unfinished_indices: List[int], new_indices: torch.Tensor):
self.penalizer_orchestrator.filter(unfinished_indices, new_indices)
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment