Skip to content

GitLab

"tests/vscode:/vscode.git/clone" did not exist on "f387f7b9da837dd91d5603afae2d4ab508180867"
Unverified Commit fa7e2c30 authored by shaharmor98's avatar shaharmor98 Committed by GitHub
Browse files

fix bench_serving mishandling of internal states (#11376) 


Signed-off-by: default avatarShahar Mor <smor@nvidia.com>
parent 8f2cd177
Hide whitespace changes
Inline Side-by-side
Showing with 9 additions and 3 deletions
python/sglang/bench_serving.py
View file @ fa7e2c30
...@@ -1837,9 +1837,15 @@ async def benchmark( ...@@ -1837,9 +1837,15 @@ async def benchmark(
server_info_json = server_info.json() server_info_json = server_info.json()
if "decode" in server_info_json: if "decode" in server_info_json:
server_info_json = server_info_json["decode"][0] server_info_json = server_info_json["decode"][0]
accept_length = server_info_json["internal_states"][0].get( if (
"avg_spec_accept_length", None "internal_states" in server_info_json
) and server_info_json["internal_states"]
):
accept_length = server_info_json["internal_states"][0].get(
"avg_spec_accept_length", None
)
else:
accept_length = None
else: else:
accept_length = None accept_length = None
else: else:
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment