Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in
Toggle navigation
Menu
Open sidebar
OpenDAS
vllm_cscc
Commits
94bf9ae4
Unverified
Commit
94bf9ae4
authored
Oct 10, 2024
by
Andy Dai
Committed by
GitHub
Oct 11, 2024
Browse files
[Misc] Fix sampling from sonnet for long context case (#9235)
parent
f990bab2
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
4 additions
and
4 deletions
+4
-4
benchmarks/benchmark_serving.py
benchmarks/benchmark_serving.py
+4
-4
No files found.
benchmarks/benchmark_serving.py
View file @
94bf9ae4
...
...
@@ -176,9 +176,9 @@ def sample_sonnet_requests(
# Sample the rest of lines per request.
sampled_requests
:
List
[
Tuple
[
str
,
int
,
int
]]
=
[]
for
_
in
range
(
num_requests
):
sampled_lines
=
""
.
join
(
prefix_lines
+
random
.
sample
(
poem_lines
,
num_
input_lines
-
num_prefix_lines
))
num_lines_needed
=
num_input_lines
-
num_prefix_lines
sampled_lines
=
""
.
join
(
prefix_lines
+
random
.
choices
(
poem_lines
,
k
=
num_
lines_needed
))
prompt
=
f
"
{
base_prompt
}{
sampled_lines
}
"
message
=
[
...
...
@@ -536,7 +536,7 @@ async def benchmark(
# E.g., "Time to First Token"
metric_header
:
str
,
):
# This function print and add statistics of the specified
# This function print
s
and add
s
statistics of the specified
# metric.
if
metric_attribute_name
not
in
selected_percentile_metrics
:
return
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment