Unverified Commit aab35bcc authored by Xinyuan Tong's avatar Xinyuan Tong Committed by GitHub
Browse files

fix: draft model IMA by overide max_positional_embeddings (#10787)


Co-authored-by: default avatarQiaolin Yu <qy254@cornell.edu>
parent 42aedb02
......@@ -210,6 +210,11 @@ class ModelConfig:
):
logger.warning(msg)
self.context_len = context_length
if is_draft_model:
self.hf_text_config.max_position_embeddings = context_length
logger.warning(
f"Overriding the draft model's max_position_embeddings to {context_length}."
)
else:
raise ValueError(
f"{msg} To allow overriding this maximum, set the env var SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1"
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment