Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in
Toggle navigation
Menu
Open sidebar
OpenDAS
vllm_cscc
Commits
c4bca740
Unverified
Commit
c4bca740
authored
Jun 28, 2024
by
mcalman
Committed by
GitHub
Jun 29, 2024
Browse files
[Bugfix] fix missing last itl in openai completions benchmark (#5926)
parent
7f83f40d
Changes
1
Show whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
5 additions
and
6 deletions
+5
-6
benchmarks/backend_request_func.py
benchmarks/backend_request_func.py
+5
-6
No files found.
benchmarks/backend_request_func.py
View file @
c4bca740
...
...
@@ -265,6 +265,9 @@ async def async_request_openai_completions(
else
:
data
=
json
.
loads
(
chunk
)
# NOTE: Some completion API might have a last
# usage summary response without a token so we
# want to check a token was generated
if
data
[
"choices"
][
0
][
"text"
]:
timestamp
=
time
.
perf_counter
()
# First token
...
...
@@ -273,10 +276,6 @@ async def async_request_openai_completions(
output
.
ttft
=
ttft
# Decoding phase
# NOTE: Some completion API might have a last
# usage summary response without a token so we
# do not want to include as inter-token-latency
elif
data
.
get
(
"usage"
,
None
)
is
None
:
output
.
itl
.
append
(
timestamp
-
most_recent_timestamp
)
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment