Skip to content

GitLab

Unverified Commit fe3b5bbc authored by Elinx's avatar Elinx Committed by GitHub
Browse files

[Bugfix] fix output parsing error for trtllm backend (#4137) 


Co-authored-by: default avatarRoger Wang <ywang@roblox.com>
parent 8438e056
Hide whitespace changes
Inline Side-by-side
Showing with 1 addition and 1 deletion
benchmarks/backend_request_func.py
View file @ fe3b5bbc
...@@ -135,6 +135,7 @@ async def async_request_trt_llm( ...@@ -135,6 +135,7 @@ async def async_request_trt_llm(
"data:") "data:")
data = json.loads(chunk) data = json.loads(chunk)
output.generated_text += data["text_output"]
timestamp = time.perf_counter() timestamp = time.perf_counter()
# First token # First token
if ttft == 0.0: if ttft == 0.0:
...@@ -149,7 +150,6 @@ async def async_request_trt_llm( ...@@ -149,7 +150,6 @@ async def async_request_trt_llm(
most_recent_timestamp = timestamp most_recent_timestamp = timestamp
output.latency = most_recent_timestamp - st output.latency = most_recent_timestamp - st
output.generated_text = json.loads(data)["text_output"]
output.success = True output.success = True
else: else:
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment