GitLab

Signed-off-by: Matej Kosec <mkosec@nvidia.com>

Signed-off-by: Yuewei Na <nv-yna@users.noreply.github.com>
Co-authored-by: Yuewei Na <nv-yna@users.noreply.github.com>

Signed-off-by: Graham King <grahamk@nvidia.com>

Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
Co-authored-by: Keiven Chang <keivenchang@users.noreply.github.com>

Signed-off-by: Tushar Sharma <tusharma@nvidia.com>

Signed-off-by: Alexandre Milesi <milesial@users.noreply.github.com>

fix: ModelDeploymentCard obtains full set of eos_token_ids by taking union from different files (#3192)
Signed-off-by: Guan Luo <gluo@nvidia.com>
Signed-off-by: Guan Luo <41310872+GuanLuo@users.noreply.github.com>

Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>
Co-authored-by: Keiven Chang <keivenchang@users.noreply.github.com>

Signed-off-by: Krishnan Prashanth <kprashanth@nvidia.com>

Signed-off-by: Alexandre Milesi <milesial@users.noreply.github.com>

Signed-off-by: Alexandre Milesi <30204471+milesial@users.noreply.github.com>

Signed-off-by: Chi McIsaac <chixie.mcisaac@gmail.com>

Signed-off-by: Graham King <grahamk@nvidia.com>

Signed-off-by: Graham King <grahamk@nvidia.com>

Signed-off-by: Krishnan Prashanth <kprashanth@nvidia.com>

Co-authored-by: Graham King <grahamk@nvidia.com>

feat: Add frontend support for `min_tokens` and `ignore_eos` (outside of `nvext`) and Structured Output / Guided Decoding (#2380)
Signed-off-by: KrishnanPrash <140860868+KrishnanPrash@users.noreply.github.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
Co-authored-by: Ayush Agarwal <ayushag@nvidia.com>

Co-authored-by: Keiven Chang <keivenchang@users.noreply.github.com>

Removed the hard coded sleeps, explained what we're testing.

Closes https://github.com/ai-dynamo/dynamo/issues/1132

The race condition is that `apply_event` sends a message on a channel, it does not directly apply the event. At some later point the tokio runtime schedules the task running the channel receiver, which applies the event. If that had not happened yet the test would fail.

Adding this to a Python script makes it register on the network so that `dynamo-run` can discover it and send it requests:
```
from dynamo.llm import register_llm

MODEL = "Qwen/Qwen2.5-0.5B-Instruct"
await register_llm(endpoint, MODEL, 3)
```

Full vllm example, with pre-processing in dynamo:
- `dynamo-run in=text out=dyn://dynamo.backend.generate`
- `cd lib/bindings/python/examples/hello_world`
- `python server_vllm.py`

This builds on top of the work to move pre-processor to ingress side. It means we can decouple Rust and Python using NATS as the bus.

The `register_llm` call does this:

- Download the model from HF if necessary
- Load the model deployment card from the HF folder or extract from GGUF
- Push the tokenizer config etc into NATS object store so ingress can access it from a different machine
- Publish the model deployment card to ETCD

Co-authored-by: Biswa Panda <biswa.panda@gmail.com>

Co-authored-by: Graham King <grahamk@nvidia.com>

Co-authored-by: Graham King <grahamk@nvidia.com>

Add backend type `EngineConfig::StaticCore` that wraps the engine in a preprocessor (prompt templating and tokenization).

Add example engine `echo_core` (`out=echo_core`) which takes and returns tokens. A nice side effect is that it echos the full prompt template with system prompt, whereas `echo_full` echos only user prompt.

![image](https://github.com/user-attachments/assets/27ec0a7b-a27d-4e69-96ea-1ffa0822ea90)

Signed-off-by: Ryan McCormick <rmccormick@nvidia.com>

Signed-off-by: Neelay Shah <neelays@nvidia.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>