Skip to content

GitLab

Unverified Commit d544d141 authored by Chenyaaang's avatar Chenyaaang Committed by GitHub
Browse files

update benchmark_serving_structured_output to include auto backend (#16438) 


Signed-off-by: default avatarChenyaaang <chenyangli@google.com>
parent 3e397a94
Show whitespace changes
Inline Side-by-side
Showing with 9 additions and 7 deletions
benchmarks/benchmark_serving_structured_output.py
View file @ d544d141
...@@ -11,7 +11,7 @@ On the client side, run: ...@@ -11,7 +11,7 @@ On the client side, run:
--model <your_model> \ --model <your_model> \
--dataset json \ --dataset json \
--structured-output-ratio 1.0 \ --structured-output-ratio 1.0 \
--structured-output-backend xgrammar \ --structured-output-backend auto \
--request-rate 10 \ --request-rate 10 \
--num-prompts 1000 --num-prompts 1000
...@@ -997,11 +997,13 @@ if __name__ == "__main__": ...@@ -997,11 +997,13 @@ if __name__ == "__main__":
type=float, type=float,
default=1.0, default=1.0,
help="Ratio of Structured Outputs requests") help="Ratio of Structured Outputs requests")
parser.add_argument( parser.add_argument("--structured-output-backend",
"--structured-output-backend",
type=str, type=str,
choices=["outlines", "lm-format-enforcer", "xgrammar", "guidance"], choices=[
default="xgrammar", "outlines", "lm-format-enforcer", "xgrammar",
"guidance", "auto"
],
default="auto",
help="Backend to use for structured outputs") help="Backend to use for structured outputs")
args = parser.parse_args() args = parser.parse_args()
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment