Skip to content

GitLab

Unverified Commit 0f9318f7 authored by Yineng Zhang's avatar Yineng Zhang Committed by GitHub
Browse files

feat: update auto_choose_speculative_params (#9470)

parent 849957bc
Hide whitespace changes
Inline Side-by-side
Showing with 6 additions and 2 deletions
python/sglang/srt/server_args.py
View file @ 0f9318f7
...@@ -2411,8 +2411,12 @@ def auto_choose_speculative_params(self: ServerArgs): ...@@ -2411,8 +2411,12 @@ def auto_choose_speculative_params(self: ServerArgs):
if arch in ["LlamaForCausalLM"]: if arch in ["LlamaForCausalLM"]:
# The default value for llama # The default value for llama
return (5, 4, 8) return (5, 4, 8)
elif arch in ["DeepseekV3ForCausalLM", "DeepseekV2ForCausalLM"]: elif arch in [
# The default value for deepseek "DeepseekV3ForCausalLM",
"DeepseekV2ForCausalLM",
"GptOssForCausalLM",
]:
# The default value for deepseek and gpt-oss
return (3, 1, 4) return (3, 1, 4)
elif arch in ["Grok1ForCausalLM", "Grok1VForCausalLM"]: elif arch in ["Grok1ForCausalLM", "Grok1VForCausalLM"]:
return (5, 4, 8) return (5, 4, 8)
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment