"examples/multimodal/utils/protocol.py" did not exist on "5bcdb734aee094a3932ae798d230fc30ca1d4745"
- 25 Feb, 2025 6 commits
-
-
Graham King authored
- Setup venv ``` uv venv source .venv/bin/activate uv pip install pip uv pip install sgl-kernel --force-reinstall --no-deps uv pip install "sglang[all]==0.4.2" --find-links https://flashinfer.ai/whl/cu124/torch2.4/flashinfer/ ``` - Build: `cargo build --release --features sglang` - Run single node (make sure you're in the venv): `./tio out=sglang ~/llm_models/my_model` - Run Deepseek multi-gpu / multi-node: Node 1: ``` tio in=http out=sglang --model-path ~/llm_models/DeepSeek-R1-Distill-Llama-70B/ --tensor-parallel-size 8 --num-nodes 2 --node-rank 0 --dist-init-addr 10.217.98.122:9876 ``` Node 2: ``` tio in=none out=sglang --model-path ~/llm_models/DeepSeek-R1-Distill-Llama-70B/ --tensor-parallel-size 8 --num-nodes 2 --node-rank 1 --dist-init-addr 10.217.98.122:9876 ```
-
Alec authored
Co-authored-by:aflowers <aflowers@nvidia.com>
-
GuanLuo authored
Signed-off-by:
Meenakshi Sharma <163925564+nvda-mesharma@users.noreply.github.com> Signed-off-by:
Neelay Shah <neelays@nvidia.com> Co-authored-by:
Neelay Shah <neelays@nvidia.com> Co-authored-by:
Ryan Olson <ryanolson@users.noreply.github.com> Co-authored-by:
Meenakshi Sharma <163925564+nvda-mesharma@users.noreply.github.com> Co-authored-by:
Biswa Panda <biswapanda@users.noreply.github.com> Co-authored-by:
Ryan McCormick <rmccormick@nvidia.com>
-
Ryan McCormick authored
Signed-off-by:Ryan McCormick <rmccormick@nvidia.com>
-
Neelay Shah authored
Signed-off-by:
Neelay Shah <neelays@nvidia.com> Co-authored-by:
Ryan McCormick <rmccormick@nvidia.com>
-
Neelay Shah authored
-
- 24 Feb, 2025 1 commit
-
-
Biswa Panda authored
-
- 22 Feb, 2025 1 commit
-
-
Alec authored
Co-authored-by:hongkuanz <hongkuanz@nvidia.com>
-
- 21 Feb, 2025 3 commits
-
-
Ryan Olson authored
Signed-off-by:
Ryan Olson <ryanolson@users.noreply.github.com> Co-authored-by:
Ryan McCormick <rmccormick@nvidia.com>
-
Ryan McCormick authored
-
Alec authored
Co-authored-by:
Sean Choi <choishsean@gmail.com> Co-authored-by:
aflowers <aflowers@nvidia.com>
-
- 20 Feb, 2025 3 commits
-
-
Biswa Panda authored
Co-authored-by:Biswa Ranjan Panda <biswaranjanp@nvidia.com>
-
ptarasiewiczNV authored
Signed-off-by:
Piotr Marcinkiewicz <piotrm@nvidia.com> Co-authored-by:
Piotr Marcinkiewicz <piotrm@nvidia.com> Co-authored-by:
Ryan McCormick <rmccormick@nvidia.com>
-
Biswa Panda authored
-
- 18 Feb, 2025 5 commits
-
-
ptarasiewiczNV authored
Co-authored-by:
Ryan Olson <rolson@nvidia.com> Co-authored-by:
Ryan McCormick <rmccormick@nvidia.com>
-
Ryan Olson authored
-
Ryan Olson authored
Co-authored-by:Ryan McCormick <rmccormick@nvidia.com>
-
GuanLuo authored
Signed-off-by:
Neelay Shah <neelays@nvidia.com> Co-authored-by:
aflowers <aflowers@nvidia.com> Co-authored-by:
Ryan McCormick <rmccormick@nvidia.com> Co-authored-by:
hongkuanz <hongkuanz@nvidia.com> Co-authored-by:
Neelay Shah <neelays@nvidia.com>
-
ptarasiewiczNV authored
-
- 13 Feb, 2025 2 commits
-
-
Ryan Olson authored
-
ptarasiewiczNV authored
Co-authored-by:
Ryan McCormick <rmccormick@nvidia.com> Co-authored-by:
Neelay Shah <neelays@nvidia.com>
-
- 12 Feb, 2025 1 commit
-
-
Ryan Olson authored
Signed-off-by:
Ryan Olson <ryanolson@users.noreply.github.com> Co-authored-by:
Ryan McCormick <rmccormick@nvidia.com>
-
- 08 Feb, 2025 1 commit
-
-
Ryan McCormick authored
Co-authored-by:
Piotr Tarasiewicz <ptarasiewicz@nvidia.com> Co-authored-by:
Neelay Shah <neelays@nvidia.com>
-
- 07 Feb, 2025 1 commit
-
-
ptarasiewiczNV authored
Co-authored-by:
Neelay Shah <neelays@nvidia.com> Co-authored-by:
Ryan McCormick <rmccormick@nvidia.com>
-
- 05 Feb, 2025 1 commit
-
-
J Wyman authored
-
- 04 Feb, 2025 1 commit
-
-
Neelay Shah authored
Signed-off-by:
Neelay Shah <neelays@nvidia.com> Co-authored-by:
Ryan McCormick <rmccormick@nvidia.com>
-
- 03 Feb, 2025 1 commit
-
-
Neelay Shah authored
-
- 01 Feb, 2025 4 commits
-
-
Piotr Marcinkiewicz authored
Signed-off-by:Piotr Marcinkiewicz <piotrm@nvidia.com>
-
ptarasiewiczNV authored
Signed-off-by:
Neelay Shah <neelays@nvidia.com> Co-authored-by:
Neelay Shah <neelays@nvidia.com> Co-authored-by:
Piotr Marcinkiewicz <piotrm@nvidia.com> Co-authored-by:
Ryan McCormick <rmccormick@nvidia.com>
-
Ryan McCormick authored
-
ishandhanani authored
Co-authored-by:
nnshah1 <neelays@nvidia.com> Co-authored-by:
Ryan McCormick <rmccormick@nvidia.com>
-
- 28 Jan, 2025 1 commit
-
-
ptarasiewiczNV authored
Signed-off-by:
ptarasiewiczNV <104908264+ptarasiewiczNV@users.noreply.github.com> Co-authored-by:
Neelay Shah <neelays@nvidia.com>
-
- 27 Jan, 2025 1 commit
-
-
Tanmay Verma authored
-
- 25 Jan, 2025 1 commit
-
-
Ryan McCormick authored
-
- 24 Jan, 2025 2 commits
-
-
ptarasiewiczNV authored
Signed-off-by:
ptarasiewiczNV <104908264+ptarasiewiczNV@users.noreply.github.com> Co-authored-by:
Neelay Shah <neelays@nvidia.com>
-
Blazej authored
Add example of LLM disaggregated serving
-
- 23 Jan, 2025 3 commits
-
-
ptarasiewiczNV authored
Signed-off-by:
ptarasiewiczNV <104908264+ptarasiewiczNV@users.noreply.github.com> Co-authored-by:
Piotr Marcinkiewicz <piotrm@nvidia.com> Co-authored-by:
Neelay Shah <neelays@nvidia.com>
-
Alec authored
Co-authored-by:aflowers <aflowers@nvidia.com>
-
Neelay Shah authored
Co-authored-by:
rmccorm4 <21284872+rmccorm4@users.noreply.github.com> Co-authored-by:
piotrm-nvidia <62554872+piotrm-nvidia@users.noreply.github.com>
-
- 22 Jan, 2025 1 commit
-
-
Anant Sharma authored
-