GitLab

Co-authored-by: kangclzjc <kangz@nvidia.com>

Signed-off-by: Matej Kosec <mkosec@nvidia.com>

Co-authored-by: Claude <noreply@anthropic.com>

Signed-off-by: Vladislav Nosivskoy <vladnosiv@gmail.com>

Signed-off-by: Alexandre Milesi <milesial@users.noreply.github.com>

Signed-off-by: Graham King <grahamk@nvidia.com>

Signed-off-by: ayushag <ayushag@nvidia.com>

Signed-off-by: Tushar Sharma <tusharma@nvidia.com>

Signed-off-by: Jie Hao <jihao@nvidia.com>

Signed-off-by: Alexandre Milesi <milesial@users.noreply.github.com>

Signed-off-by: Vladislav Nosivskoy <vladnosiv@gmail.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>

Signed-off-by: Krishnan Prashanth <kprashanth@nvidia.com>
Signed-off-by: KrishnanPrash <140860868+KrishnanPrash@users.noreply.github.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>

Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>

Signed-off-by: Ryan Lempka <rlempka@nvidia.com>

Signed-off-by: Ryan Lempka <rlempka@nvidia.com>
Signed-off-by: ryan-lempka <rlempka@nvidia.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>

Signed-off-by: Graham King <grahamk@nvidia.com>

Signed-off-by: ayushag <ayushag@nvidia.com>

Signed-off-by: Krishnan Prashanth <kprashanth@nvidia.com>

Signed-off-by: Chi McIsaac <chixie.mcisaac@gmail.com>

Signed-off-by: Graham King <grahamk@nvidia.com>

Signed-off-by: Graham King <grahamk@nvidia.com>

Signed-off-by: ayushag <ayushag@nvidia.com>

Signed-off-by: Graham King <grahamk@nvidia.com>

Signed-off-by: Graham King <grahamk@nvidia.com>

Signed-off-by: Krishnan Prashanth <kprashanth@nvidia.com>

Co-authored-by: Graham King <grahamk@nvidia.com>

Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>

- Add Granite to our tokenizer
- Fix pre-processor to load context length correctly
- Add strftime_now Jinja function for prompt templates
- Update llama.cpp
- Handle trtllm errors when not using trtllm

Support depends on the engine:

- `mistral.rs`, our default engine, doesn't support Granite yet.

- `llama.cpp` does and works very well:
```
dynamo-run out=llamacpp ~/llms/granite-3.3-2b-instruct-Q4_K_M.gguf --context-length 16384
```

- `vllm` also works very well:
```
dynamo-run in=http out=vllm ~/llms/granite-3.3-2b-instruct --context-length 16384
```

- `sglang` mostly works, but it doesn't catch the stop token, so we do in the HTTP ingress, and log an error. The Text ingress doesn't catch it because I disabled it to make the raw echo engine work. A bit of work to do here.

Closes: #1245 

Part of https://github.com/ai-dynamo/dynamo/issues/743

Signed-off-by: Olga Andreeva <124622579+oandreeva-nv@users.noreply.github.com>

This lets us do:
```
dynamo-run out=llamacpp <gguf_file>
```

Previously a `--model-config <hf-repo>` was also required, to configure our tokenizer.