Unverified Commit ec1ad004 authored by Thomas Parnell's avatar Thomas Parnell Committed by GitHub
Browse files

[Bugfix] Better error message for MLPSpeculator when `num_speculative_tokens`...


[Bugfix] Better error message for MLPSpeculator when `num_speculative_tokens` is set too high (#5894)
Signed-off-by: default avatarThomas Parnell <tpa@zurich.ibm.com>
parent 57f09a41
...@@ -956,9 +956,9 @@ class SpeculativeConfig: ...@@ -956,9 +956,9 @@ class SpeculativeConfig:
# Verify provided value doesn't exceed the maximum # Verify provided value doesn't exceed the maximum
# supported by the draft model. # supported by the draft model.
raise ValueError( raise ValueError(
"Expected both speculative_model and " "This speculative model supports a maximum of "
"num_speculative_tokens to be provided, but found " f"num_speculative_tokens={n_predict}, but "
f"{speculative_model=} and {num_speculative_tokens=}.") f"{num_speculative_tokens=} was provided.")
draft_model_config.max_model_len = ( draft_model_config.max_model_len = (
SpeculativeConfig._maybe_override_draft_max_model_len( SpeculativeConfig._maybe_override_draft_max_model_len(
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment