Unverified Commit 008bb1e6 authored by Indrajit Bhosale's avatar Indrajit Bhosale Committed by GitHub
Browse files

fix: Add handling for ignore_eos sampling param in trtllm example base engine (#1726)

parent 45f0e424
......@@ -345,6 +345,10 @@ class BaseTensorrtLLMEngine:
if max_tokens:
sampling_params.max_tokens = max_tokens
ignore_eos = request.stop_conditions.ignore_eos
if ignore_eos:
sampling_params.ignore_eos = ignore_eos
# TODO: Disable streaming for context only requests when adding disagg support
async for res in self._llm_engine.llm.generate_async(
inputs=inputs,
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment