Unverified Commit b26cb1c5 authored by Zhiqiang Xie's avatar Zhiqiang Xie Committed by GitHub
Browse files

Fix problem of large page size with chunked prefill (#6046)

parent f8e46093
...@@ -499,12 +499,12 @@ class PrefillAdder: ...@@ -499,12 +499,12 @@ class PrefillAdder:
), ),
) )
else: else:
if self.rem_chunk_tokens == 0: # Make sure at least one page is available
trunc_len = self.rem_chunk_tokens - self.tree_cache.page_size + 1
if trunc_len <= 0:
return AddReqResult.OTHER return AddReqResult.OTHER
# Chunked prefill # Chunked prefill
trunc_len = self.rem_chunk_tokens
req.extend_input_len = trunc_len req.extend_input_len = trunc_len
req.fill_ids = req.fill_ids[: len(req.prefix_indices) + trunc_len] req.fill_ids = req.fill_ids[: len(req.prefix_indices) + trunc_len]
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment