Skip to content

GitLab

Unverified Commit 80d85c5d authored by 电脑星人's avatar 电脑星人 Committed by GitHub
Browse files

[Bugfix] Fix mrope_position_delta in non-last prefill chunk (#10403) 


Signed-off-by: default avatarimkero <kerorek@outlook.com>
parent 76aab90a
Hide whitespace changes
Inline Side-by-side
Showing with 1 addition and 1 deletion
vllm/model_executor/layers/rotary_embedding.py
View file @ 80d85c5d
...@@ -922,9 +922,9 @@ class MRotaryEmbedding(RotaryEmbedding): ...@@ -922,9 +922,9 @@ class MRotaryEmbedding(RotaryEmbedding):
torch.arange(text_len).view(1, -1).expand(3, -1) + st_idx) torch.arange(text_len).view(1, -1).expand(3, -1) + st_idx)
llm_positions = torch.cat(llm_pos_ids_list, dim=1).reshape(3, -1) llm_positions = torch.cat(llm_pos_ids_list, dim=1).reshape(3, -1)
llm_positions = llm_positions[:, context_len:seq_len]
mrope_position_delta = (llm_positions.max() + 1 - mrope_position_delta = (llm_positions.max() + 1 -
len(input_tokens)).item() len(input_tokens)).item()
llm_positions = llm_positions[:, context_len:seq_len]
return llm_positions.tolist(), mrope_position_delta return llm_positions.tolist(), mrope_position_delta
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment