Commits · 86aff237ab4c098587c89ce9982f913ed32a87a8 · OpenDAS / dynamo

26 Feb, 2025 2 commits
- refactor: using async_openai · 86aff237
  Paul Hendricks authored Feb 26, 2025
```
Co-authored-by: Graham King <grahamk@nvidia.com>
```
  86aff237
- fix: Fix stream::until_deadline bug and improve metric examples (#280) · 494d5625
  Ryan McCormick authored Feb 26, 2025
```
Co-authored-by: Ryan Olson <rolson@nvidia.com>
```
  494d5625
25 Feb, 2025 6 commits

feat: sglang backend for tio (#271) · e97493eb

Graham King authored Feb 25, 2025

- Setup venv

```
uv venv
source .venv/bin/activate
uv pip install pip
uv pip install sgl-kernel --force-reinstall --no-deps
uv pip install "sglang[all]==0.4.2" --find-links https://flashinfer.ai/whl/cu124/torch2.4/flashinfer/
```

- Build: `cargo build --release --features sglang`

- Run single node (make sure you're in the venv): `./tio out=sglang ~/llm_models/my_model`

- Run Deepseek multi-gpu / multi-node:

Node 1:
```
tio in=http out=sglang --model-path ~/llm_models/DeepSeek-R1-Distill-Llama-70B/ --tensor-parallel-size 8 --num-nodes 2 --node-rank 0 --dist-init-addr 10.217.98.122:9876
```

Node 2:
```
tio in=none out=sglang --model-path ~/llm_models/DeepSeek-R1-Distill-Llama-70B/ --tensor-parallel-size 8 --num-nodes 2 --node-rank 1 --dist-init-addr 10.217.98.122:9876
```

e97493eb

feat: Add completion endpoint to http server and llmctl (#230) · b760c569
Alec authored Feb 25, 2025
```
Co-authored-by: aflowers <aflowers@nvidia.com>
```
b760c569

feat: enable metrics polling · 861c5098

GuanLuo authored Feb 25, 2025

Signed-off-by: Meenakshi Sharma <163925564+nvda-mesharma@users.noreply.github.com>
Signed-off-by: Neelay Shah <neelays@nvidia.com>
Co-authored-by: Neelay Shah <neelays@nvidia.com>
Co-authored-by: Ryan Olson <ryanolson@users.noreply.github.com>
Co-authored-by: Meenakshi Sharma <163925564+nvda-mesharma@users.noreply.github.com>
Co-authored-by: Biswa Panda <biswapanda@users.noreply.github.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>

861c5098

ci: Add rust checks to missing directories (#239) · c06b95ff
Ryan McCormick authored Feb 25, 2025
```
Signed-off-by: Ryan McCormick <rmccormick@nvidia.com>
```
c06b95ff

refactor: move libs to lib dir · 08fcd7e9

Neelay Shah authored Feb 24, 2025


Signed-off-by: Neelay Shah <neelays@nvidia.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>

08fcd7e9

refactor: remove python native runtime · 0bfd9a76
Neelay Shah authored Feb 24, 2025

0bfd9a76

24 Feb, 2025 1 commit
- feat: add rust based tokenizer · 4f6f63cd
  Biswa Panda authored Feb 24, 2025
  
  4f6f63cd
22 Feb, 2025 1 commit
- [fix] initialize vLLM engine before runtime (#238) · ccd153af
  Alec authored Feb 21, 2025
```
Co-authored-by: hongkuanz <hongkuanz@nvidia.com>
```
  ccd153af
21 Feb, 2025 3 commits
- feat: event plane + count · 3b7a462d
  Ryan Olson authored Feb 21, 2025
```
Signed-off-by: Ryan Olson <ryanolson@users.noreply.github.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
```
  3b7a462d
- build: Multi-stage VLLM Build (#225) · 6e0ccccb
  Ryan McCormick authored Feb 21, 2025
  
  6e0ccccb
- feat: Add http support to vllm kv router example (#217) · 15de1807
  Alec authored Feb 21, 2025
```
Co-authored-by: Sean Choi <choishsean@gmail.com>
Co-authored-by: aflowers <aflowers@nvidia.com>
```
  15de1807
20 Feb, 2025 3 commits
- feat: add cli args for example http service (#221) · 73c10ae9
  Biswa Panda authored Feb 20, 2025
```
Co-authored-by: Biswa Ranjan Panda <biswaranjanp@nvidia.com>
```
  73c10ae9
- feat: use vllm out of process engine · 60a73634
  ptarasiewiczNV authored Feb 20, 2025
```
Signed-off-by: Piotr Marcinkiewicz <piotrm@nvidia.com>
Co-authored-by: Piotr Marcinkiewicz <piotrm@nvidia.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
```
  60a73634
- feat: add local model card (#216) · 65a2dfab
  Biswa Panda authored Feb 20, 2025
  
  65a2dfab
18 Feb, 2025 5 commits
- feat: add openai endpoint to the vllm example (#183) · 9e4a548d
  ptarasiewiczNV authored Feb 18, 2025
```
Co-authored-by: Ryan Olson <rolson@nvidia.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
```
  9e4a548d
- fix: triton.code-workpace; update dependencies (#207) · 695754ae
  Ryan Olson authored Feb 18, 2025
  
  695754ae
- feat: http + llmctl (#181) · d0d35a9e
  Ryan Olson authored Feb 18, 2025
```
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
```
  d0d35a9e
- feat: Add KV publisher and receiver. Add KV aware routing example. · 8588e33a
  GuanLuo authored Feb 18, 2025
```
Signed-off-by: Neelay Shah <neelays@nvidia.com>
Co-authored-by: aflowers <aflowers@nvidia.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
Co-authored-by: hongkuanz <hongkuanz@nvidia.com>
Co-authored-by: Neelay Shah <neelays@nvidia.com>
```
  8588e33a
- feat: Add Dockerfile.vllm (#184) · d8aada0b
  ptarasiewiczNV authored Feb 18, 2025
  
  d8aada0b
13 Feb, 2025 2 commits
- fix: tcp updates + initial zmq (#176) · 2fd6592f
  Ryan Olson authored Feb 13, 2025
  
  2fd6592f
- feat: vLLM v0.7.2 patch supporting XpYd and heterogeneous P/D parallel configs (#167) · f38aa469
  ptarasiewiczNV authored Feb 13, 2025
```
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
Co-authored-by: Neelay Shah <neelays@nvidia.com>
```
  f38aa469
12 Feb, 2025 1 commit

fix: tcp retry and error handling updates (#169) · dddebc0d

Ryan Olson authored Feb 12, 2025


Signed-off-by: Ryan Olson <ryanolson@users.noreply.github.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>

dddebc0d

08 Feb, 2025 1 commit

refactor: Modify rust vllm example to use container · d2e7ae02

Ryan McCormick authored Feb 07, 2025


Co-authored-by: Piotr Tarasiewicz <ptarasiewicz@nvidia.com>
Co-authored-by: Neelay Shah <neelays@nvidia.com>

d2e7ae02

07 Feb, 2025 1 commit
- feat: vLLM DistributedRuntime Monolith and Disagg Workers Example · 45bc426c
  ptarasiewiczNV authored Feb 08, 2025
```
Co-authored-by: Neelay Shah <neelays@nvidia.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
```
  45bc426c
05 Feb, 2025 1 commit
- ci: Add Copyright Verification Scripts w/ Automation (#110) · c9130f8f
  J Wyman authored Feb 05, 2025
  
  c9130f8f
04 Feb, 2025 1 commit
- build: pip based installation of icp and runtime. Also make tritonserver optional · ccfe101e
  Neelay Shah authored Feb 04, 2025
```
Signed-off-by: Neelay Shah <neelays@nvidia.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
```
  ccfe101e
03 Feb, 2025 1 commit
- refactor: worker rename to runtime · 4017bd18
  Neelay Shah authored Feb 03, 2025
  
  4017bd18
01 Feb, 2025 4 commits
- Adjustment for multi node benchmark · 8492333b
  Piotr Marcinkiewicz authored Feb 02, 2025
```
Signed-off-by: Piotr Marcinkiewicz <piotrm@nvidia.com>
```
  8492333b
- docs: Add Llama 70B benchmark reproduction and results · 04e50aba
  ptarasiewiczNV authored Feb 01, 2025
```
Signed-off-by: Neelay Shah <neelays@nvidia.com>
Co-authored-by: Neelay Shah <neelays@nvidia.com>
Co-authored-by: Piotr Marcinkiewicz <piotrm@nvidia.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
```
  04e50aba
- refactor: Replace trtllm example prints with debug logs · 993954a6
  Ryan McCormick authored Feb 01, 2025
  
  993954a6
- feat(llm): adding initial TRTLLM disaggregation support · deb6c7e8
  ishandhanani authored Feb 01, 2025
```
Co-authored-by: nnshah1 <neelays@nvidia.com>
Co-authored-by: Ryan McCormick <rmccormick@nvidia.com>
```
  deb6c7e8
28 Jan, 2025 1 commit

Clean-up vLLM example to use Operator API (#68) · b9a0ce2c

ptarasiewiczNV authored Jan 28, 2025


Signed-off-by: ptarasiewiczNV <104908264+ptarasiewiczNV@users.noreply.github.com>
Co-authored-by: Neelay Shah <neelays@nvidia.com>

b9a0ce2c

27 Jan, 2025 1 commit
- refactor: Improve the usage of logging in worker (#29) · 022b6db5
  Tanmay Verma authored Jan 27, 2025
  
  022b6db5
25 Jan, 2025 1 commit
- chore: Update build version to v0.1.1 and add Teardown section to LLM example README (#66) · 1c1bd7da
  Ryan McCormick authored Jan 24, 2025
  
  1c1bd7da
24 Jan, 2025 2 commits
- docs: Add instructions for multi-node disaggregated deployment · 194abde3
  ptarasiewiczNV authored Jan 24, 2025
```
Signed-off-by: ptarasiewiczNV <104908264+ptarasiewiczNV@users.noreply.github.com>
Co-authored-by: Neelay Shah <neelays@nvidia.com>
```
  194abde3
- feat: Add vLLM workers to LLM example (#41) · f6d723f6
  Blazej authored Jan 24, 2025
```
Add example of LLM disaggregated serving
```
  f6d723f6
23 Jan, 2025 2 commits

vLLM example README (#52) · 8980ec37

ptarasiewiczNV authored Jan 23, 2025

Signed-off-by: ptarasiewiczNV <104908264+ptarasiewiczNV@users.noreply.github.com>
Co-authored-by: Piotr Marcinkiewicz <piotrm@nvidia.com>
Co-authored-by: Neelay Shah <neelays@nvidia.com>

8980ec37

fix: encoder repetition typo · 41527a81
Alec authored Jan 23, 2025
```
Co-authored-by: aflowers <aflowers@nvidia.com>
```
41527a81