Unverified Commit fe3b5bbc authored by Elinx's avatar Elinx Committed by GitHub
Browse files

[Bugfix] fix output parsing error for trtllm backend (#4137)


Co-authored-by: default avatarRoger Wang <ywang@roblox.com>
parent 8438e056
......@@ -135,6 +135,7 @@ async def async_request_trt_llm(
"data:")
data = json.loads(chunk)
output.generated_text += data["text_output"]
timestamp = time.perf_counter()
# First token
if ttft == 0.0:
......@@ -149,7 +150,6 @@ async def async_request_trt_llm(
most_recent_timestamp = timestamp
output.latency = most_recent_timestamp - st
output.generated_text = json.loads(data)["text_output"]
output.success = True
else:
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment