Skip to content

GitLab

Unverified Commit 68235126 authored by Qidong Su's avatar Qidong Su Committed by GitHub
Browse files

fix: pass back `usage` when using text mode with VLLM (#5336) 


Signed-off-by: default avatarQidong Su <soodoshll@gmail.com>
parent c8770464
Hide whitespace changes
Inline Side-by-side
Showing with 5 additions and 0 deletions
components/src/dynamo/vllm/handlers.py
View file @ 68235126
...@@ -1300,6 +1300,11 @@ class DecodeWorkerHandler(BaseWorkerHandler): ...@@ -1300,6 +1300,11 @@ class DecodeWorkerHandler(BaseWorkerHandler):
"choices": [choice_data], "choices": [choice_data],
} }
if output.finish_reason:
chunk["usage"] = BaseWorkerHandler._build_completion_usage(
request_output=res,
)
yield chunk yield chunk
except EngineDeadError as e: except EngineDeadError as e:
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment