GitLab

Signed-off-by: Tushar Sharma <tusharma@nvidia.com>

Signed-off-by: Nicholas Edelman <nedelman@nvidia.com>

Signed-off-by: Graham King <grahamk@nvidia.com>

feat: graceful exit leading to 500 and connection errors - liveness-return-false if-cancelled (#4914)
Signed-off-by: michaelfeil <me@michaelfeil.eu>

Signed-off-by: PeaBrane <yanrpei@gmail.com>

Signed-off-by: Graham King <grahamk@nvidia.com>

Signed-off-by: mohammedabdulwahhab <furkhan324@berkeley.edu>

Signed-off-by: Graham King <grahamk@nvidia.com>

Signed-off-by: Krishnan Prashanth <kprashanth@nvidia.com>

Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>

Signed-off-by: Graham King <grahamk@nvidia.com>

Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>

Signed-off-by: Keiven Chang <keivenchang@users.noreply.github.com>

Signed-off-by: Neelay Shah <neelays@nvidia.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Co-authored-by: Olga Andreeva <124622579+oandreeva-nv@users.noreply.github.com>

Signed-off-by: Greg Clark <grclark@nvidia.com>

- Stop advertising a model when it's last instance stops. Previously was when any instance stops.
- Faster locks on model manager.
- Move discovery code out of http, as it is used by all inputs.

Implements OpenAI embeddings (interface only).

- Adds ModelType::Embedding
- Adds OpenAI embedding request/response structs
- Adds support for embedding model discovery

For #1006

Prints this on startup:
```
2025-05-09T13:06:34.529Z DEBUG dynamo_run::input::http: Supported routes: ["GET /metrics", "GET /dynamo/alpha/list-models", "GET /v1/models", "POST /v1/chat/completions", "POST /v1/completions"]
```

Adds support for specifying default request parameters through a json template file that can be applied across all inference requests. This enables consistent parameter settings while still allowing per-request overrides.

Changes:
- Add --request-template CLI flag to specify template file path
- Integrate template support in HTTP, batch and text input modes
- Template values can be overridden by individual request parameters
- Example template.json:
```
{
    "model": "Qwen2.5-3B-Instruct",
    "temperature": 0.7,
    "max_completion_tokens": 4096
}
```

Co-authored-by: aflowers <aflowers@nvidia.com>

Signed-off-by: Neelay Shah <neelays@nvidia.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>

Co-authored-by: Biswa Ranjan Panda <biswaranjanp@nvidia.com>

Signed-off-by: Ryan Olson <ryanolson@users.noreply.github.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>

Signed-off-by: Ryan Olson <ryanolson@users.noreply.github.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
Co-authored-by: Neelay Shah <neelays@nvidia.com>