Skip to content

GitLab

Unverified Commit fa7e2c30 authored by shaharmor98's avatar shaharmor98 Committed by GitHub
Browse files

fix bench_serving mishandling of internal states (#11376) 


Signed-off-by: default avatarShahar Mor <smor@nvidia.com>
parent 8f2cd177
Show whitespace changes
Inline Side-by-side
Showing with 9 additions and 3 deletions
python/sglang/bench_serving.py
View file @ fa7e2c30
...@@ -1837,6 +1837,10 @@ async def benchmark( ...@@ -1837,6 +1837,10 @@ async def benchmark(
server_info_json = server_info.json() server_info_json = server_info.json()
if "decode" in server_info_json: if "decode" in server_info_json:
server_info_json = server_info_json["decode"][0] server_info_json = server_info_json["decode"][0]
if (
"internal_states" in server_info_json
and server_info_json["internal_states"]
):
accept_length = server_info_json["internal_states"][0].get( accept_length = server_info_json["internal_states"][0].get(
"avg_spec_accept_length", None "avg_spec_accept_length", None
) )
...@@ -1844,6 +1848,8 @@ async def benchmark( ...@@ -1844,6 +1848,8 @@ async def benchmark(
accept_length = None accept_length = None
else: else:
accept_length = None accept_length = None
else:
accept_length = None
# Compute metrics and print results # Compute metrics and print results
benchmark_duration = time.perf_counter() - benchmark_start_time benchmark_duration = time.perf_counter() - benchmark_start_time
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment