Unverified Commit f9b1757f authored by Ryan McCormick's avatar Ryan McCormick Committed by GitHub
Browse files

fix: Bring back ignore_eos/min_tokens support in trtllm component (#2023)


Signed-off-by: default avatarRyan McCormick <mccormick.codes@gmail.com>
Co-authored-by: default avatartanmayv25 <tanmay2592@gmail.com>
parent eebc7410
......@@ -132,6 +132,14 @@ class HandlerBase:
if max_tokens:
sampling_params.max_tokens = max_tokens
ignore_eos = request["stop_conditions"]["ignore_eos"]
if ignore_eos:
sampling_params.ignore_eos = ignore_eos
min_tokens = request["stop_conditions"]["min_tokens"]
if min_tokens:
sampling_params.min_tokens = min_tokens
# TODO: Instead of True, we should use streaming from the request.
# However, currently dynamo run does not send streaming in the request.
streaming = (
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment