GitLab

refactor: change trtllm example kv routing use python bindings | deal with trtllm partial blocks | trtllm event change (#866)

We were observing a 40% performance drop compared with trtllm serve when benchmarking with isl=1000 and osl=200 at a concurrency level > 128.

The number of the tokenization worker is the bottleneck. After bumping the tokenization processors number to 5, dynamo's benchmarking perf could match the trtllm serve's perf.

Signed-off-by: Hongkuan Zhou <tedzhouhk@gmail.com>
Co-authored-by: ishandhanani <82981111+ishandhanani@users.noreply.github.com>
Co-authored-by: ishandhanani <ishandhanani@gmail.com>
Co-authored-by: Ubuntu <ubuntu@dev-inst-2w1vokvyuts83rzn4n1k7mnzew9.us-central1-a.c.brevdevprod.internal>
Co-authored-by: Biswa Panda <biswa.panda@gmail.com>
Co-authored-by: Anant Sharma <anants@nvidia.com>

Co-authored-by: ishandhanani <ishandhanani@gmail.com>

Co-authored-by: mohammedabdulwahhab <furkhan324@berkeley.edu>

Co-authored-by: Julien Mancuso <jmancuso@nvidia.com>

Signed-off-by: cormick <cormick1080@gmail.com>

Co-authored-by: ishandhanani <82981111+ishandhanani@users.noreply.github.com>

Co-authored-by: ishandhanani <82981111+ishandhanani@users.noreply.github.com>

Co-authored-by: Ziqi Fan <ziqif@nvidia.com>

Signed-off-by: hhzhang16 <54051230+hhzhang16@users.noreply.github.com>
Co-authored-by: mohammedabdulwahhab <furkhan324@berkeley.edu>

Signed-off-by: mohammedabdulwahhab <furkhan324@berkeley.edu>
Co-authored-by: hhzhang16 <54051230+hhzhang16@users.noreply.github.com>
Co-authored-by: Julien Mancuso <jmancuso@nvidia.com>

feat: ETCD prefix watcher + python binding + runtime reconfiguration for router and disagg router  (#581)
Signed-off-by: Hongkuan Zhou <tedzhouhk@gmail.com>
Co-authored-by: Neelay Shah <neelays@nvidia.com>

Signed-off-by: Tanmay Verma <tanmay2592@gmail.com>
Signed-off-by: Tanmay Verma <tanmayv@nvidia.com>
Co-authored-by: Neelay Shah <neelays@nvidia.com>

Signed-off-by: Jacky <18255193+kthui@users.noreply.github.com>
Co-authored-by: Neelay Shah <neelays@nvidia.com>
Co-authored-by: Meenakshi Sharma <163925564+nvda-mesharma@users.noreply.github.com>

Signed-off-by: Jacky <18255193+kthui@users.noreply.github.com>
Signed-off-by: Pavithra Vijayakrishnan <160681768+pvijayakrish@users.noreply.github.com>
Signed-off-by: Chantal D Gama Rose <cdgamarose@nvidia.com>
Signed-off-by: hhzhang16 <54051230+hhzhang16@users.noreply.github.com>
Co-authored-by: Julien Mancuso <jmancuso@nvidia.com>
Co-authored-by: mohammedabdulwahhab <furkhan324@berkeley.edu>
Co-authored-by: mabdulwahhab <mabdulwahhab@nvidia.com>
Co-authored-by: Tushar Sharma <tusharma@nvidia.com>
Co-authored-by: Jacky <18255193+kthui@users.noreply.github.com>
Co-authored-by: Neelay Shah <neelays@nvidia.com>
Co-authored-by: Pavithra Vijayakrishnan <160681768+pvijayakrish@users.noreply.github.com>
Co-authored-by: cdgamarose-nv <cdgamarose@nvidia.com>
Co-authored-by: Anant Sharma <anants@nvidia.com>
Co-authored-by: julienmancuso <161955438+julienmancuso@users.noreply.github.com>
Co-authored-by: Suman Tatiraju <167138127+statiraju@users.noreply.github.com>
Co-authored-by: Piotr Marcinkiewicz <piotrm@nvidia.com>
Co-authored-by: ishandhanani <82981111+ishandhanani@users.noreply.github.com>
Co-authored-by: Tanmay Verma <tanmayv@nvidia.com>