Commits · aa2cd2c43d1d19ece0f3b36ad716c3a9b8a2def0 · OpenDAS / vllm_cscc

26 Jan, 2025 6 commits

[Bugfix] Disable w16a16 2of4 sparse CompressedTensors24 (#12417) · aa2cd2c4
Tyler Michael Smith authored Jan 26, 2025
```
Signed-off-by: Tyler Michael Smith <tyler@neuralmagic.com>
Co-authored-by: mgoin <michael@neuralmagic.com>
```
aa2cd2c4

[Frontend] generation_config.json for maximum tokens(#12242) · 9ddc3522

Matthew Hendrey authored Jan 26, 2025


Signed-off-by: Matthew Hendrey <matthew.hendrey@gmail.com>
Signed-off-by: Shangming Cai <caishangming@linux.alibaba.com>
Signed-off-by: youkaichao <youkaichao@gmail.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Yuan Tang <terrytangyuan@gmail.com>
Signed-off-by: Isotr0py <2037008807@qq.com>
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com>
Co-authored-by: shangmingc <caishangming@linux.alibaba.com>
Co-authored-by: youkaichao <youkaichao@gmail.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Yuan Tang <terrytangyuan@gmail.com>
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
Co-authored-by: Chen Zhang <zhangch99@outlook.com>
Co-authored-by: wangxiyuan <wangxiyuan1007@gmail.com>

9ddc3522

[Misc] Revert FA on ViT #12355 and #12435 (#12445) · a5255270
Roger Wang authored Jan 26, 2025

a5255270
[V1][Bugfix] Fix assertion when mm hashing is turned off (#12439) · 0ee349b5
Roger Wang authored Jan 26, 2025
```
Signed-off-by: Roger Wang <ywang@roblox.com>
```
0ee349b5
[V1][Perf] Reduce scheduling overhead in model runner after cuda sync (#12094) · fa63e710
Keyun Tong authored Jan 26, 2025
```
Signed-off-by: Keyun Tong <tongkeyun@gmail.com>
```
fa63e710

[Misc][Bugfix] FA3 support to ViT MHA layer (#12435) · 2a0309a6

Roger Wang authored Jan 25, 2025


Signed-off-by: Roger Wang <ywang@roblox.com>
Signed-off-by: Isotr0py <2037008807@qq.com>
Co-authored-by: Isotr0py <2037008807@qq.com>

2a0309a6

25 Jan, 2025 5 commits
- [TPU][CI] Update torchxla version in requirement-tpu.txt (#12422) · 324960a9
  Siyuan Liu authored Jan 24, 2025
```
Signed-off-by: Siyuan Liu <lsiyuan@google.com>
```
  324960a9
- [Misc] Add FA2 support to ViT MHA layer (#12355) · f1fc0510
  Isotr0py authored Jan 25, 2025
```
Signed-off-by: Isotr0py <2037008807@qq.com>
```
  f1fc0510
- [ROCm][MoE] MI300 tuned configs Mixtral-8x(7B,22B) | fp16, fp8 (#12408) · bf21481d
  Divakar Verma authored Jan 24, 2025
```
Signed-off-by: Divakar Verma <divakar.verma@amd.com>
```
  bf21481d
- [Bugfix] Fix BLIP-2 processing (#12412) · fb30ee92
  Cyrus Leung authored Jan 25, 2025
```
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
```
  fb30ee92
- [Bugfix][Kernel] Fix moe align block issue for mixtral (#12413) · 221d388c
  ElizaWszola authored Jan 24, 2025
  
  221d388c
24 Jan, 2025 15 commits
- [Bugfix][Kernel] FA3 Fix - RuntimeError: This flash attention build only... · 3132a933
  Lucas Wilkinson authored Jan 24, 2025
```
[Bugfix][Kernel] FA3 Fix - RuntimeError: This flash attention build only supports pack_gqa (for build size reasons). (#12405)
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
```
  3132a933
- [Misc] Remove deprecated code (#12383) · df5dafaa
  Cyrus Leung authored Jan 25, 2025
```
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
```
  df5dafaa
- [Bugfix][Kernel] Fix CUDA 11.8 being broken by FA3 build (#12375) · ab5bbf5a
  Lucas Wilkinson authored Jan 24, 2025
```
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
```
  ab5bbf5a
- [Misc] Enable proxy support in benchmark script (#12356) · 3bb8e2c9
  Junichi Sato authored Jan 24, 2025
```
Signed-off-by: Junichi Sato <junichi.sato@sbintuitions.co.jp>
```
  3bb8e2c9
- [ci/build] sync default value for wheel size (#12398) · e784c6b9
  youkaichao authored Jan 24, 2025
```
Signed-off-by: youkaichao <youkaichao@gmail.com>
```
  e784c6b9
- [Hardware][Gaudi][Doc] Add missing step in setup instructions (#12382) · 9a0f3bdb
  Mohit Deopujari authored Jan 24, 2025
  
  9a0f3bdb
- [ci/build] fix wheel size check (#12396) · c7c98510
  youkaichao authored Jan 24, 2025
```
Signed-off-by: youkaichao <youkaichao@gmail.com>
```
  c7c98510
- [Misc] Use VisionArena Dataset for VLM Benchmarking (#12389) · 3c818bdb
  Roger Wang authored Jan 24, 2025
```
Signed-off-by: Roger Wang <ywang@roblox.com>
```
  3c818bdb
- [perf] fix perf regression from #12253 (#12380) · 6dd94dbe
  youkaichao authored Jan 24, 2025
```
Signed-off-by: youkaichao <youkaichao@gmail.com>
```
  6dd94dbe
- [V1] Increase default batch size for H100/H200 (#12369) · 0e74d797
  Woosuk Kwon authored Jan 23, 2025
```
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
```
  0e74d797
- Update compressed-tensors version (#12367) · 55ef66ed
  Dipika Sikka authored Jan 23, 2025
  
  55ef66ed
- [Bugfix] Path join when building local path for S3 clone (#12353) · 5e5630a4
  omer-dayan authored Jan 24, 2025
```
Signed-off-by: Omer Dayan (SW-GPU) <omer@run.ai>
```
  5e5630a4
- Set weights_only=True when using torch.load() (#12366) · d3d6bb13
  Russell Bryant authored Jan 23, 2025
```
Signed-off-by: Russell Bryant <rbryant@redhat.com>
```
  d3d6bb13
- [V1][Frontend] Coalesce bunched `RequestOutput`s (#12298) · 24b0205f
  Nick Hill authored Jan 23, 2025
```
Signed-off-by: Nick Hill <nhill@redhat.com>
Co-authored-by: Robert Shaw <rshaw@neuralmagic.com>
```
  24b0205f
- [Docs] Update spec decode + structured output in compat matrix (#12373) · c5cffcd0
  Russell Bryant authored Jan 23, 2025
```
Signed-off-by: Russell Bryant <rbryant@redhat.com>
```
  c5cffcd0
23 Jan, 2025 14 commits
- [Docs] Add meetup slides (#12345) · 682b55bc
  Woosuk Kwon authored Jan 23, 2025
```
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
```
  682b55bc
- [Misc] Fix OpenAI API Compatibility Issues in Benchmark Script (#12357) · 9726ad67
  Junichi Sato authored Jan 24, 2025
```
Signed-off-by: Junichi Sato <junichi.sato@sbintuitions.co.jp>
```
  9726ad67
- [BugFix] Fix parameter names and `process_after_weight_loading` for W4A16 MoE... · eb5cb5e5
  Dipika Sikka authored Jan 23, 2025
```
[BugFix] Fix parameter names and `process_after_weight_loading` for W4A16 MoE Group Act Order  (#11528)
Signed-off-by: ElizaWszola <eliza@neuralmagic.com>
Co-authored-by: ElizaWszola <eliza@neuralmagic.com>
Co-authored-by: Michael Goin <michael@neuralmagic.com>
```
  eb5cb5e5
- [Docs] Document Phi-4 support (#12362) · 2cbeedad
  Isotr0py authored Jan 24, 2025
```
Signed-off-by: Isotr0py <2037008807@qq.com>
```
  2cbeedad
- [TPU] Update TPU CI to use torchxla nightly on 20250122 (#12334) · 2c85529b
  Siyuan Liu authored Jan 23, 2025
```
Signed-off-by: Siyuan Liu <lsiyuan@google.com>
```
  2c85529b
- [FP8][Kernel] Dynamic kv cache scaling factors computation (#11906) · e97f802b
  Gregory Shtrasberg authored Jan 23, 2025
```
Signed-off-by: Gregory Shtrasberg <Gregory.Shtrasberg@amd.com>
Co-authored-by: Micah Williamson <micah.williamson@amd.com>
```
  e97f802b
- [torch.compile] decouple compile sizes and cudagraph sizes (#12243) · 6e650f56
  youkaichao authored Jan 24, 2025
```
Signed-off-by: youkaichao <youkaichao@gmail.com>
```
  6e650f56
- [core] add wake_up doc and some sanity check (#12361) · 3f50c148
  youkaichao authored Jan 24, 2025
```
Signed-off-by: youkaichao <youkaichao@gmail.com>
```
  3f50c148
- [Bugfix] Fix broken internvl2 inference with v1 (#12360) · 8c01b802
  Isotr0py authored Jan 24, 2025
```
Signed-off-by: Isotr0py <2037008807@qq.com>
```
  8c01b802
- [V1] Simplify M-RoPE (#12352) · 99d01a5e
  Roger Wang authored Jan 23, 2025
```
Signed-off-by: Roger Wang <ywang@roblox.com>
Co-authored-by: imkero <kerorek@outlook.com>
```
  99d01a5e
- [Doc] Troubleshooting errors during model inspection (#12351) · d07efb31
  Cyrus Leung authored Jan 23, 2025
```
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
```
  d07efb31
- [Kernel] Flash Attention 3 Support (#12093) · 978b45f3
  Lucas Wilkinson authored Jan 23, 2025
```
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
```
  978b45f3
- [Bugfix] Fix k_proj's bias for whisper self attention (#12342) · c5b4b11d
  Isotr0py authored Jan 23, 2025
```
Signed-off-by: Isotr0py <2037008807@qq.com>
```
  c5b4b11d
- [Hardware][Gaudi][BugFix] Fix dataclass error due to triton package update (#12338) · 8ae5ff20
  liuzhenwei authored Jan 23, 2025
```
Signed-off-by: zhenwei <zhenweiliu@habana.ai>
```
  8ae5ff20