GitLab

Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
Co-authored-by: Keiven Chang <keivenchang@users.noreply.github.com>

fix(llm): support reading eos_token_ids from tokenizer_config.json for models like Qwen3.5 with <|im_end|> token (#8091)

Closes https://github.com/ai-dynamo/dynamo/issues/7737

Signed-off-by: Graham King <grahamk@nvidia.com>

Signed-off-by: Nikita Sukharev <kaonael@gmail.com>

Signed-off-by: Tushar Sharma <tusharma@nvidia.com>

Signed-off-by: Alexandre Milesi <milesial@users.noreply.github.com>

Signed-off-by: Vladislav Nosivskoy <vladnosiv@gmail.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>

Signed-off-by: zhangzhang <tangchenyu@xiaohongshu.com>
Co-authored-by: zhangzhang <tangchenyu@xiaohongshu.com>
Co-authored-by: Ayush Agarwal <ayushag@nvidia.com>

Signed-off-by: ayushag <ayushag@nvidia.com>

Signed-off-by: Alexandre Milesi <milesial@users.noreply.github.com>

Signed-off-by: ayushag <ayushag@nvidia.com>

Signed-off-by: Graham King <grahamk@nvidia.com>

Example:
```
dynamo-run out=<engine> <model> --kv-cache-block-size 64
```

In a distributed system this goes on the worker node and is propagated to ingress via the model deployment card.

Previously hard coded to 16, which is now the default.

- Load context_length from model. Closes #1172
- Store context length and KV cache block size in Model Deployment Card #1170

. New mistralrs and llamacpp version
. mistralrs: Handle Gemma 3 and Llama 4 as vision models
. Update the dynamo-run docs to use Qwen 3
. Our pre-processor now supports Llama 4's newer multi-modal `config.json`
. Upgrade minijinja to handle Qwen 3's prompt template

For Llama 4 we'll need to limit the max seq len. vllm says:
> To serve at least one request with the models's max seq len (10485760), (240.00 GiB KV cache is needed,...

I was able to run Llama 4 with llamacpp and a quantized GGUF, with Dynamo doing the pre-processing.

Signed-off-by: Neelay Shah <neelays@nvidia.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>