Commits · e60f550b3825cbce2d3c7e882b029e2c1d914d8d · OpenDAS / vllm_cscc

15 May, 2025 1 commit
- [v1] Support multiple KV cache groups in GPU model runner (#17945) · e60f550b
  Chen Zhang authored May 15, 2025
```
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
```
  e60f550b
14 May, 2025 3 commits
- Modularize fused experts and integrate PPLX kernels (#15956) · f9c069c8
  bnellnm authored May 14, 2025
  
  f9c069c8
- [core][distributed] add ep group and all2all interface (#18077) · 6266c57b
  youkaichao authored May 14, 2025
```
Signed-off-by: youkaichao <youkaichao@gmail.com>
```
  6266c57b
- [Misc] Remove unused numpy tensor (#18084) · 6e27c6d8
  Roger Wang authored May 13, 2025
```
Signed-off-by: Roger Wang <hey@rogerw.me>
```
  6e27c6d8
13 May, 2025 1 commit
- [Bugfix][V1] Only get input embeddings w/ multi-modal models if first PP (#17916) · 8dd0671b
  Jin Huang authored May 13, 2025
```
Signed-off-by: Jin Huang <jinhun@amazon.com>
Co-authored-by: Jin Huang <jinhun@amazon.com>
```
  8dd0671b
12 May, 2025 2 commits

[P/D] NIXL Integration (#17751) · d1911020

Robert Shaw authored May 12, 2025


Signed-off-by: ApostaC <yihua98@uchicago.edu>
Signed-off-by: Tyler Michael Smith <tyler@neuralmagic.com>
Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com>
Signed-off-by: Robert Shaw <rshaw@neuralmagic.com>
Signed-off-by: mgoin <mgoin64@gmail.com>
Signed-off-by: Nick Hill <nhill@redhat.com>
Signed-off-by: Brent Salisbury <bsalisbu@redhat.com>
Co-authored-by: Tyler Michael Smith <tyler@neuralmagic.com>
Co-authored-by: ApostaC <yihua98@uchicago.edu>
Co-authored-by: Robert Shaw <rshaw@neuralmagic.com>
Co-authored-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
Co-authored-by: Tyler Michael Smith <tysmith@redhat.com>
Co-authored-by: Brent Salisbury <bsalisbu@redhat.com>

d1911020

[Bugfix][TPU] Use np array when updating cache slot_mapping (#17971) · 43078301
Siyuan Liu authored May 11, 2025
```
Signed-off-by: Siyuan Liu <lsiyuan@google.com>
```
43078301

10 May, 2025 1 commit
- [v1] Pass BlockTable and KVCacheSpec to AttentionMetadataBuilders (#17483) · 950751a9
  Chen Zhang authored May 11, 2025
```
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
```
  950751a9
08 May, 2025 1 commit

[Core] Support full cuda graph in v1 (#16072) · 7ea2adb8

Chanh Nguyen authored May 07, 2025


Signed-off-by: Chanh Nguyen <cnguyen@linkedin.com>
Co-authored-by: Chanh Nguyen <cnguyen@linkedin.com>

7ea2adb8

07 May, 2025 2 commits

[Hardware][TPU][V1] Multi-LoRA implementation for the V1 TPU backend (#14238) · c20ef40f

Akshat Tripathi authored May 07, 2025


Signed-off-by: Akshat Tripathi <akshat@krai.ai>
Signed-off-by: Chengji Yao <chengjiyao@google.com>
Co-authored-by: Chengji Yao <chengjiyao@google.com>

c20ef40f

[Misc] Split model loader (#17712) · 822de7fb
Jee Jee Li authored May 07, 2025
```
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
```
822de7fb

06 May, 2025 2 commits
- [v1] AttentionMetadata for each layer (#17394) · cba31c47
  Chen Zhang authored May 06, 2025
```
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
```
  cba31c47
- [V1][PP] Support PP for MultiprocExecutor (#14219) · a6fed020
  Li, Jiang authored May 06, 2025
```
Signed-off-by: jiang1.li <jiang1.li@intel.com>
Signed-off-by: jiang.li <jiang1.li@intel.com>
```
  a6fed020
05 May, 2025 1 commit
- [TPU][V1] Add support for top-logprobs (#17072) · 5941e0b7
  Nicolò Lucchesi authored May 05, 2025
```
Signed-off-by: NickLucche <nlucches@redhat.com>
```
  5941e0b7
04 May, 2025 1 commit
- Add full API docs and improve the UX of navigating them (#17485) · d6484ef3
  Harry Mellor authored May 04, 2025
```
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
```
  d6484ef3
30 Apr, 2025 1 commit
- [V1][Feature] Enable Speculative Decoding with Structured Outputs (#14702) · 34120f5a
  Benjamin Chislett authored Apr 29, 2025
```
Signed-off-by: Benjamin Chislett <benjamin.chislett@centml.ai>
Signed-off-by: Benjamin Chislett <chislett.ben@gmail.com>
```
  34120f5a
29 Apr, 2025 2 commits
- [V1][Spec Decode] Apply torch.compile & cudagraph to EAGLE (#17211) · 70788bdb
  Bryan Lu authored Apr 29, 2025
```
Signed-off-by: Bryan Lu <yuzhelu@amazon.com>
```
  70788bdb
- [V1] Remove num_input_tokens from attn_metadata (#17193) · 24e6ad3f
  Chen Zhang authored Apr 30, 2025
```
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
```
  24e6ad3f
27 Apr, 2025 2 commits

[Feature] support sequence parallelism using compilation pass (#16155) · 690fe019

cascade authored Apr 27, 2025


Signed-off-by: cascade812 <cascade812@outlook.com>
Signed-off-by: Tyler Michael Smith <tyler@neuralmagic.com>
Co-authored-by: Tyler Michael Smith <tyler@neuralmagic.com>

690fe019

[Bugfix] Get a specific type of layer from forward context (#17222) · 838cedad
Chen Zhang authored Apr 27, 2025
```
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
```
838cedad

26 Apr, 2025 2 commits
- [Core] Remove prompt string from engine core data structures (#17214) · df6f3ce8
  Nick Hill authored Apr 25, 2025
```
Signed-off-by: Nick Hill <nhill@redhat.com>
```
  df6f3ce8
- [Minor][Spec Decode] Add use_eagle to SpeculativeConfig (#17213) · 1cf0719e
  Woosuk Kwon authored Apr 25, 2025
```
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
```
  1cf0719e
25 Apr, 2025 2 commits
- [V1][Spec Decode] EAGLE-3 Support (#16937) · a0e619e6
  Benjamin Chislett authored Apr 25, 2025
```
Signed-off-by: Bryan Lu <yuzhelu@amazon.com>
Signed-off-by: Benjamin Chislett <benjamin.chislett@centml.ai>
Co-authored-by: Bryan Lu <yuzhelu@amazon.com>
```
  a0e619e6
- [V1] Move usage stats to worker and start logging TPU hardware (#16211) · 48cb2109
  Daniel Li authored Apr 25, 2025
  
  48cb2109
24 Apr, 2025 2 commits
- Improve static type checking in `LoRAModelRunnerMixin` (#17104) · 21f4f1c9
  Harry Mellor authored Apr 24, 2025
```
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
```
  21f4f1c9
- [Chore] Remove Sampler from Model Code (#17084) · b411418f
  Woosuk Kwon authored Apr 24, 2025
```
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
```
  b411418f
23 Apr, 2025 3 commits
- [V1][Spec Decode] Always use argmax for sampling draft tokens (#16899) · 41fb013d
  Woosuk Kwon authored Apr 23, 2025
```
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
```
  41fb013d
- [BugFix][V1] Fix int32 token index overflow when preparing input ids (#16806) · 32d4b669
  Yong Hoon Shin authored Apr 23, 2025
  
  32d4b669
- [Core][V1][TPU] Enable structured decoding on TPU V1 (#16499) · 83d93371
  Chenyaaang authored Apr 22, 2025
```
Signed-off-by: Chenyaaang <chenyangli@google.com>
```
  83d93371
22 Apr, 2025 3 commits

[Perf] Optimize `_update_states` for GPU model runner (#16910) · a114bf20
SnowCharm authored Apr 22, 2025
```
Signed-off-by: snowcharm <snowcharmqq@gmail.com>
```
a114bf20

[V1] V1 FlashInfer Attention (#16684) · 986537f1

Michael Goin authored Apr 21, 2025


Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Aurick Qiao <qiao@aurick.net>

986537f1

[TPU][V1] Capture multimodal encoder during model compilation (#15051) · 21020752

Nicolò Lucchesi authored Apr 22, 2025


Signed-off-by: Michael Goin <mgoin64@gmail.com>
Signed-off-by: NickLucche <nlucches@redhat.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: Siyuan Liu <lsiyuan@google.com>

21020752

21 Apr, 2025 2 commits
- [V1][Spec Decode] Handle draft tokens beyond max_model_len (#16087) · 3a0fba5c
  Woosuk Kwon authored Apr 21, 2025
```
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
```
  3a0fba5c
- Restore buffers when wake up from level 2 sleep (#16564) (#16889) · d41faaf9
  Han Zhang authored Apr 21, 2025
```
Signed-off-by: Han <zh950713@gmail.com>
```
  d41faaf9
19 Apr, 2025 1 commit

[Model][VLM] Add Qwen2.5-Omni model support (thinker only) (#15130) · 2c1bd848

Yang Fan authored Apr 19, 2025


Signed-off-by: fyabc <suyang.fy@alibaba-inc.com>
Signed-off-by: Roger Wang <ywang@roblox.com>
Co-authored-by: Roger Wang <136131678+ywang96@users.noreply.github.com>
Co-authored-by: Roger Wang <ywang@roblox.com>
Co-authored-by: Xiong Wang <wangxiongts@163.com>

2c1bd848

17 Apr, 2025 5 commits

[P/D][V1] KV Connector API V1 (#15960) · 3408e471

Yihua Cheng authored Apr 17, 2025


Signed-off-by: ApostaC <yihua98@uchicago.edu>
Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com>
Signed-off-by: remi <remi@mistral.ai>
Co-authored-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
Co-authored-by: Rémi Delacourt <54138269+Flechman@users.noreply.github.com>
Co-authored-by: Tyler Michael Smith <tysmith@redhat.com>

3408e471

[MLA] Simplification to batch P/D reordering (#16673) · 0377b831
Nick Hill authored Apr 17, 2025
```
Signed-off-by: Nick Hill <nhill@redhat.com>
```
0377b831

[V1][TPU] Enable Top K (#15489) · eb5819b2

Nicolò Lucchesi authored Apr 17, 2025


Signed-off-by: NickLucche <nlucches@redhat.com>
Signed-off-by: Hyesoo Yang <hyeygit@gmail.com>
Co-authored-by: Hyesoo Yang <hyeygit@gmail.com>

eb5819b2

[TPU][V1] Fix padding recompilation when `max-num-batched-tokens` is not even (#16726) · 5989f468
Nicolò Lucchesi authored Apr 17, 2025
```
Signed-off-by: NickLucche <nlucches@redhat.com>
```
5989f468
[Misc] Remove redundant comment (#16703) · 8a7368e0
Jade Zheng authored Apr 17, 2025
```
Signed-off-by: Jade Zheng <zheng.shoujian@outlook.com>
```
8a7368e0