Skip to content

GitLab

Unverified Commit fa7e2c30 authored by shaharmor98's avatar shaharmor98 Committed by GitHub
Browse files

fix bench_serving mishandling of internal states (#11376) 


Signed-off-by: default avatarShahar Mor <smor@nvidia.com>
parent 8f2cd177
Hide whitespace changes
Inline Side-by-side
Showing with 9 additions and 3 deletions
python/sglang/bench_serving.py
View file @ fa7e2c30
......@@ -1837,9 +1837,15 @@ async def benchmark(
server_info_json = server_info.json()
if "decode" in server_info_json:
server_info_json = server_info_json["decode"][0]
accept_length = server_info_json["internal_states"][0].get(
"avg_spec_accept_length", None
)
if (
"internal_states" in server_info_json
and server_info_json["internal_states"]
):
accept_length = server_info_json["internal_states"][0].get(
"avg_spec_accept_length", None
)
else:
accept_length = None
else:
accept_length = None
else:
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment