Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in
Toggle navigation
Menu
Open sidebar
OpenDAS
vllm_cscc
Commits
6e4cea1c
Unverified
Commit
6e4cea1c
authored
May 28, 2025
by
daniel-salib
Committed by
GitHub
May 28, 2025
Browse files
decrement server_load on listen for disconnect (#18784)
Signed-off-by:
Daniel Salib
<
danielsalib@meta.com
>
parent
435fa954
Changes
1
Show whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
5 additions
and
0 deletions
+5
-0
vllm/entrypoints/utils.py
vllm/entrypoints/utils.py
+5
-0
No files found.
vllm/entrypoints/utils.py
View file @
6e4cea1c
...
@@ -26,6 +26,11 @@ async def listen_for_disconnect(request: Request) -> None:
...
@@ -26,6 +26,11 @@ async def listen_for_disconnect(request: Request) -> None:
while
True
:
while
True
:
message
=
await
request
.
receive
()
message
=
await
request
.
receive
()
if
message
[
"type"
]
==
"http.disconnect"
:
if
message
[
"type"
]
==
"http.disconnect"
:
if
request
.
app
.
state
.
enable_server_load_tracking
:
# on timeout/cancellation the BackgroundTask in load_aware_call
# cannot decrement the server load metrics.
# Must be decremented by with_cancellation instead.
request
.
app
.
state
.
server_load_metrics
-=
1
break
break
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment