Unverified Commit c87ebc3e authored by Nick Hill's avatar Nick Hill Committed by GitHub
Browse files

[BugFix] Ensure worker model loop is always stopped at the right time (#5987)

parent c4059ea5
......@@ -838,7 +838,7 @@ class LLMEngine:
# Tracing
self.do_tracing(scheduler_outputs)
if not request_outputs:
if not self.has_unfinished_requests():
# Stop the execute model loop in parallel workers until there are
# more requests to process. This avoids waiting indefinitely in
# torch.distributed ops which may otherwise timeout, and unblocks
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment