Commits · 57504a4bcf82cf544a4261778fc0505e492e4484 · OpenDAS / vllm_cscc

11 Apr, 2025 2 commits
- [Kernel] support merge_attn_states CUDA kernel, 3x speedup (#16173) · e9528f6d
  DefTruth authored Apr 11, 2025
```
Signed-off-by: DefTruth <qiustudent_r@163.com>
```
  e9528f6d
- Enforce valid max_num_batched_tokens when disable_chunked_mm_input=True (#16447) · aa3b3d76
  Michael Goin authored Apr 11, 2025
```
Signed-off-by: mgoin <mgoin64@gmail.com>
```
  aa3b3d76
10 Apr, 2025 6 commits
- [TPU][V1] Disable per-request seed/Generator (#16172) · 3cc9af88
  Nicolò Lucchesi authored Apr 10, 2025
```
Signed-off-by: NickLucche <nlucches@redhat.com>
```
  3cc9af88
- [V1] Zero-copy tensor/ndarray serialization/transmission (#13790) · dd143ef5
  Nick Hill authored Apr 10, 2025
```
Signed-off-by: Nick Hill <nhill@redhat.com>
```
  dd143ef5
- [V1][Spec Decode] Eagle Model loading (#16035) · e8224f3d
  Lily Liu authored Apr 10, 2025
```
Signed-off-by: LiuXiaoxuanPKU <lilyliupku@gmail.com>
```
  e8224f3d
- [TPU][V1] Use `language_model` interface for getting text backbone in MM (#16410) · c1b57855
  Nicolò Lucchesi authored Apr 10, 2025
```
Signed-off-by: NickLucche <nlucches@redhat.com>
```
  c1b57855
- [Bugfix] Fix validation error for text-only Mllama 3.2 (#16377) · a5d11a54
  Cyrus Leung authored Apr 10, 2025
```
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
```
  a5d11a54
- [TPU][V1] Refine tpu_model_runner to mitigate future recompilation issues (#16275) · a4547485
  Chengji Yao authored Apr 09, 2025
```
Signed-off-by: Chengji Yao <chengjiyao@google.com>
```
  a4547485
09 Apr, 2025 5 commits
- [Hardware] add platform-specific request validation api (#16291) · cb391d85
  Joe Runde authored Apr 09, 2025
```
Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
```
  cb391d85
- [BugFix] fix some typos found by typos. (#16314) · 04149cce
  yihong authored Apr 09, 2025
```
Signed-off-by: yihong0618 <zouzou0208@gmail.com>
```
  04149cce
- [Bugfix] Avoid transferring cached multi-modal items from P0 to P1 (#16273) · e484e028
  Cyrus Leung authored Apr 09, 2025
```
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
```
  e484e028
- [Core] Upgrade to xgrammar 0.1.18, add cache size limit (#16283) · cb84e45a
  Russell Bryant authored Apr 08, 2025
```
Signed-off-by: Russell Bryant <rbryant@redhat.com>
```
  cb84e45a
- [Feature] Estimate max-model-len use available KV cache memory (#16168) · 4716377f
  rongfu.leng authored Apr 09, 2025
```
Signed-off-by: rongfu.leng <rongfu.leng@daocloud.io>
```
  4716377f
08 Apr, 2025 7 commits
- [BugFix] Fix Llama4 - Index Error When Single Request Near Max Context (#16209) · e1a2c699
  Lucas Wilkinson authored Apr 08, 2025
```
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
```
  e1a2c699
- [Bugfix] Proper input validation for multi-modal encoder-decoder models (#16156) · 4ebc0b96
  Cyrus Leung authored Apr 09, 2025
```
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
```
  4ebc0b96
- [V1] Add `disable_chunked_mm_input` arg to disable partial mm input prefill (#15837) · 8e5314a4
  Michael Goin authored Apr 08, 2025
```
Signed-off-by: mgoin <mgoin64@gmail.com>
```
  8e5314a4
- [torch.compile][TPU] Make @support_torch_compile work for XLA backend (#15782) · 87918e40
  Siyuan Liu authored Apr 07, 2025
```
Signed-off-by: Siyuan Liu <lsiyuan@google.com>
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: mgoin <mgoin64@gmail.com>
```
  87918e40
- Add warning for Attention backends that do not support irope yet (#16212) · 05a015d6
  Yong Hoon Shin authored Apr 07, 2025
  
  05a015d6
- [V1] Scatter and gather placeholders in the model runner (#16076) · f2ebb6f5
  Roger Wang authored Apr 07, 2025
```
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: mgoin <mgoin64@gmail.com>
Signed-off-by: Roger Wang <ywang@roblox.com>
Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
Co-authored-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Jennifer Zhao <ai.jenniferzhao@gmail.com>
```
  f2ebb6f5
- [core] do not send error across process (#16174) · a865bc1c
  youkaichao authored Apr 08, 2025
```
Signed-off-by: youkaichao <youkaichao@gmail.com>
```
  a865bc1c
07 Apr, 2025 6 commits

[V1][BugFix] Exit properly if engine core fails during startup (#16137) · 7f6d47c1
Nick Hill authored Apr 07, 2025
```
Signed-off-by: Nick Hill <nhill@redhat.com>
```
7f6d47c1
[Bugfix] Fix guidance backend for Qwen models (#16210) · 3147586e
Benjamin Chislett authored Apr 07, 2025
```
Signed-off-by: Benjamin Chislett <benjamin.chislett@centml.ai>
```
3147586e

Upstream Llama4 Support to Main (#16113) · 55dcce91

Lu Fang authored Apr 07, 2025


Signed-off-by: Aston Zhang <22279212+astonzhang@users.noreply.github.com>
Signed-off-by: Chris Thi <chris.c.thi@gmail.com>
Signed-off-by: drisspg <drisspguessous@gmail.com>
Signed-off-by: Jon Swenson <jmswen@gmail.com>
Signed-off-by: Keyun Tong <tongkeyun@gmail.com>
Signed-off-by: Lu Fang <fanglu@meta.com>
Signed-off-by: Xiaodong Wang <xdwang@meta.com>
Signed-off-by: Yang Chen <yangche@fb.com>
Signed-off-by: Ye (Charlotte) Qi <yeq@meta.com>
Signed-off-by: Yong Hoon Shin <yhshin@meta.com>
Signed-off-by: Zijing Liu <liuzijing2014@gmail.com>
Signed-off-by: Lu Fang <lufang@fb.com>
Signed-off-by: Lu Fang <fanglu@fb.com>
Signed-off-by: Lucia Fang <fanglu@fb.com>
Signed-off-by: Roger Wang <ywang@roblox.com>
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Co-authored-by: Lu Fang <fanglu@fb.com>
Co-authored-by: Roger Wang <ywang@roblox.com>
Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>

55dcce91

[V1][Structured Output] Add `supports_structured_output()` method to Platform (#16148) · e9ba99f2
Shanshan Shen authored Apr 07, 2025
```
Signed-off-by: shen-shanshan <467638484@qq.com>
```
e9ba99f2
[V1][Minor] Minor simplification for get_computed_blocks (#16139) · 3749e287
Woosuk Kwon authored Apr 06, 2025
```
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
```
3749e287

[Metrics] Add bucket for `request_latency`, `time_to_first_token` and... · 86fc2321

Kay Yan authored Apr 07, 2025


[Metrics] Add bucket for `request_latency`, `time_to_first_token` and `time_per_output_token` (#15202)
Signed-off-by: Kay Yan <kay.yan@daocloud.io>

86fc2321

06 Apr, 2025 2 commits
- [V1][Minor] Optimize get_cached_block (#16135) · b10e5198
  Woosuk Kwon authored Apr 06, 2025
  
  b10e5198
- [CI][V1] Fix passing `tokenizer` as kwarg to `validate_guidance_grammar` (#16117) · 9ca710e5
  Roger Wang authored Apr 06, 2025
```
Signed-off-by: Roger Wang <ywang@roblox.com>
```
  9ca710e5
04 Apr, 2025 7 commits
- [V1][Spec Decode] Update N-gram Proposer Interface (#15750) · 63375f0c
  Woosuk Kwon authored Apr 04, 2025
```
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
```
  63375f0c
- [Bugfix][TPU] Fix V1 TPU worker for sliding window (#16059) · 70ad3f9e
  Michael Goin authored Apr 04, 2025
```
Signed-off-by: Michael Goin <mgoin64@gmail.com>
```
  70ad3f9e
- Revert "[V1] Scatter and gather placeholders in the model runner" (#16075) · af51d80f
  Roger Wang authored Apr 04, 2025
  
  af51d80f
- [V1] Scatter and gather placeholders in the model runner (#15712) · f5722a50
  Cyrus Leung authored Apr 05, 2025
```
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: mgoin <mgoin64@gmail.com>
Signed-off-by: Roger Wang <ywang@roblox.com>
Co-authored-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Roger Wang <ywang@roblox.com>
```
  f5722a50
- [V1] DP scale-out (1/N): Use zmq ROUTER/DEALER sockets for input queue (#15906) · 651cf0fe
  Nick Hill authored Apr 04, 2025
```
Signed-off-by: Nick Hill <nhill@redhat.com>
```
  651cf0fe
- [V1][Spec Decode] Avoid logging useless nan metrics (#16023) · a35a8a83
  Mark McLoughlin authored Apr 04, 2025
```
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
```
  a35a8a83
- [TPU][V1] Remove ragged attention kernel parameter hard coding (#16041) · fadc59c0
  Chengji Yao authored Apr 04, 2025
```
Signed-off-by: Chengji Yao <chengjiyao@google.com>
```
  fadc59c0
03 Apr, 2025 5 commits
- [TPU] Support sliding window and logit soft capping in the paged attention kernel for TPU. (#15732) · b6be6f8d
  iefgnoix authored Apr 03, 2025
```
Signed-off-by: Xiongfei Wei <isaacwxf23@gmail.com>
```
  b6be6f8d
- [Bugfix][V1] Fix bug from putting llm_engine.model_executor in a background process (#15367) · 463bbb18
  wwl2755 authored Apr 03, 2025
```
Signed-off-by: wwl2755 <wangwenlong2755@gmail.com>
```
  463bbb18
- [ROCM][KERNEL] Paged attention for V1 (#15720) · e73ff24e
  Aleksandr Malyshev authored Apr 02, 2025
```
Signed-off-by: Aleksandr Malyshev <maleksan@amd.com>
Signed-off-by: root <root@banff-cyxtera-s65-4.amd.com>
Co-authored-by: Aleksandr Malyshev <maleksan@amd.com>
Co-authored-by: root <root@banff-cyxtera-s65-4.amd.com>
```
  e73ff24e
- [V1][TPU] Do not compile sampling more than needed (#15883) · bd7599d3
  Nicolò Lucchesi authored Apr 03, 2025
```
Signed-off-by: NickLucche <nlucches@redhat.com>
```
  bd7599d3
- [TPU] optimize the all-reduce performance (#15903) · 01b61136
  Chengji Yao authored Apr 02, 2025
```
Signed-off-by: Chengji Yao <chengjiyao@google.com>
```
  01b61136