Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
change
sglang
Commits
19fd57bc
"tests/python/vscode:/vscode.git/clone" did not exist on "661f8177ceb885fd534e6c73b759da01f8937431"
Unverified
Commit
19fd57bc
authored
Mar 07, 2025
by
Adarsh Shirawalmath
Committed by
GitHub
Mar 06, 2025
Browse files
[docs] fix HF reference script command (#4148)
parent
9c58e68b
Changes
2
Hide whitespace changes
Inline
Side-by-side
Showing
2 changed files
with
7 additions
and
4 deletions
+7
-4
docs/references/supported_models.md
docs/references/supported_models.md
+1
-1
scripts/playground/reference_hf.py
scripts/playground/reference_hf.py
+6
-3
No files found.
docs/references/supported_models.md
View file @
19fd57bc
...
@@ -83,7 +83,7 @@ You should test the new vLM locally against hf models. See [`mmmu`](https://gith
...
@@ -83,7 +83,7 @@ You should test the new vLM locally against hf models. See [`mmmu`](https://gith
For interactive debugging, you can compare the outputs of huggingface/transformers and SGLang.
For interactive debugging, you can compare the outputs of huggingface/transformers and SGLang.
The following two commands should give the same text output and very similar prefill logits.
The following two commands should give the same text output and very similar prefill logits.
-
Get the reference output by
`python3 scripts/playground/reference_hf.py --model [new model]`
-
Get the reference output by
`python3 scripts/playground/reference_hf.py --model
-path
[new model]
--model-type {text,vlm}
`
-
Get the SGLang output by
`python3 -m sglang.bench_one_batch --correct --model [new model]`
-
Get the SGLang output by
`python3 -m sglang.bench_one_batch --correct --model [new model]`
#### Add the model to the test suite
#### Add the model to the test suite
...
...
scripts/playground/reference_hf.py
View file @
19fd57bc
"""
"""
Usage:
Usage: python3 scripts/playground/reference_hf.py --model-path MODEL_PATH --model-type {text,vlm} [--max-new-tokens NUM] [--dtype DTYPE]
python3 reference_hf.py --model TinyLlama/TinyLlama-1.1B-Chat-v0.4
--model-path MODEL_PATH: Path to model (default: TinyLlama/TinyLlama-1.1B-Chat-v0.4)
--model-type {text,vlm}: Model type, text or vlm (default: text)
--max-new-tokens NUM: Max new tokens to generate (default: 16)
--dtype DTYPE: Data type for computation (default: float16)
Note: '--model' is deprecated; use '--model-path'. Runs normal_text() for text, vlm_text_with_image() for vlm.
Reference output:
Reference output:
========== Prompt 0 ==========
========== Prompt 0 ==========
...
@@ -178,7 +182,6 @@ if __name__ == "__main__":
...
@@ -178,7 +182,6 @@ if __name__ == "__main__":
"--model-path"
,
"--model-path"
,
type
=
str
,
type
=
str
,
default
=
"TinyLlama/TinyLlama-1.1B-Chat-v0.4"
,
default
=
"TinyLlama/TinyLlama-1.1B-Chat-v0.4"
,
# default="meta-llama/Llama-2-7b-chat-hf",
)
)
parser
.
add_argument
(
"--max-new-tokens"
,
type
=
int
,
default
=
16
)
parser
.
add_argument
(
"--max-new-tokens"
,
type
=
int
,
default
=
16
)
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment