Skip to content

GitLab

Unverified Commit 1dacedd2 authored by Marin's avatar Marin Committed by GitHub
Browse files

make sure logit bias is applied during eagle spec decoding verification (#11555)

parent b5e14b2b
Hide whitespace changes
Inline Side-by-side
Showing with 4 additions and 1 deletion
python/sglang/srt/speculative/eagle_info.py
View file @ 1dacedd2
...@@ -246,7 +246,10 @@ class EagleVerifyInput(SpecInput, EagleVerifyInputV2Mixin): ...@@ -246,7 +246,10 @@ class EagleVerifyInput(SpecInput, EagleVerifyInputV2Mixin):
) )
# Apply penalty # Apply penalty
if sampling_info.penalizer_orchestrator.is_required: if (
sampling_info.penalizer_orchestrator.is_required
or sampling_info.logit_bias is not None
):
# This is a relaxed version of penalties for speculative decoding. # This is a relaxed version of penalties for speculative decoding.
linear_penalty = torch.zeros( linear_penalty = torch.zeros(
(bs, logits_output.next_token_logits.shape[1]), (bs, logits_output.next_token_logits.shape[1]),
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment