GitLab

feat: Full Anthropic Messages API cache_control support (top-level, per-block, system block arrays) (#6629)
Signed-off-by: Matej Kosec <mkosec@nvidia.com>

Signed-off-by: Krishnan Prashanth <kprashanth@nvidia.com>

Signed-off-by: Tushar Sharma <tusharma@nvidia.com>

Co-authored-by: Claude <noreply@anthropic.com>

Signed-off-by: Satvik Matta <smatta@nvidia.com>

Signed-off-by: Alexandre Milesi <milesial@users.noreply.github.com>

Signed-off-by: Vladislav Nosivskoy <vladnosiv@gmail.com>

feat: Add support for skip_special_tokens parameter in v1/completions and v1/chat/completions endpoints (#4175)

Signed-off-by: Krishnan Prashanth <kprashanth@nvidia.com>

Signed-off-by: zhongdaor <zhongdaor@nvidia.com>

Signed-off-by: Nicholas Edelman <nedelman@nvidia.com>

Signed-off-by: ayushag <ayushag@nvidia.com>

Signed-off-by: ayushag <ayushag@nvidia.com>

Signed-off-by: Greg Clark <grclark@nvidia.com>

Signed-off-by: Chi McIsaac <chixie.mcisaac@gmail.com>

Signed-off-by: Graham King <grahamk@nvidia.com>

Signed-off-by: ayushag <ayushag@nvidia.com>

Signed-off-by: Ryan Lempka <rlempka@nvidia.com>

Signed-off-by: Ryan Lempka <rlempka@nvidia.com>

Changing the chat completions response objects from structs to types of dynamo_async_openai

Implement aggregator traits for them chat completion structs

add reasoning_content under message and delta message in lib/async-openai

Co-authored-by: Graham King <grahamk@nvidia.com>

Signed-off-by: Greg Clark <grclark@nvidia.com>

feat: Add frontend support for `min_tokens` and `ignore_eos` (outside of `nvext`) and Structured Output / Guided Decoding (#2380)
Signed-off-by: KrishnanPrash <140860868+KrishnanPrash@users.noreply.github.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
Co-authored-by: Ayush Agarwal <ayushag@nvidia.com>

Previously several parts of the stack ensured max tokens (for this single request) was set.

Now only text input sets it (to 8k). Everything else leaves as is, potentially blank. The engines themselves have very small defaults, 16 for vllm and 128 for sglang.

Also fix dynamo-run CUDA startup message to only print if we're using an engine that would benefit from it (mistralrs, llamacpp).

Co-authored-by: Biswa Panda <biswa.panda@gmail.com>

Co-authored-by: Graham King <grahamk@nvidia.com>

Co-authored-by: Graham King <grahamk@nvidia.com>

Signed-off-by: Neelay Shah <neelays@nvidia.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>

Signed-off-by: Ryan Olson <ryanolson@users.noreply.github.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
Co-authored-by: Neelay Shah <neelays@nvidia.com>