Commits · dd66fd2b01e1195b7ccc8ffcd4b5d49ff1946a56 · OpenDAS / vllm_cscc

28 Jan, 2025 6 commits
- [CI] fix pre-commit error (#12494) · dd66fd2b
  Mengqing Cao authored Jan 28, 2025
```
Signed-off-by: Mengqing Cao <cmq0113@163.com>
```
  dd66fd2b
- [FEATURE] Enables offline /score for embedding models (#12021) · 0f465ab5
  Gabriel Marinho authored Jan 28, 2025
```
Signed-off-by: Gabriel Marinho <gmarinho@ibm.com>
```
  0f465ab5
- [CI/Build] Fixed the xla nightly issue report in #12451 (#12453) · 23a7cbc8
  Hossein Sarshar authored Jan 27, 2025
  
  23a7cbc8
- Fix bad path in prometheus example (#12481) · 426a5c36
  Michael Goin authored Jan 27, 2025
```
Signed-off-by: mgoin <michael@neuralmagic.com>
```
  426a5c36
- [Neuron][Kernel] NKI-based flash-attention kernel with paged KV cache (#11277) · ddee88d0
  Liangfu Chen authored Jan 27, 2025
```
Signed-off-by: Liangfu Chen <liangfc@amazon.com>
Co-authored-by: Jiangfei Duan <jfduan@outlook.com>
```
  ddee88d0
- Update `pre-commit` hooks (#12475) · 823ab796
  Harry Mellor authored Jan 28, 2025
```
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
```
  823ab796
27 Jan, 2025 13 commits
- [Feature] [Spec decode]: Enable MLPSpeculator/Medusa and `prompt_logprobs`... · 6116ca8c
  Nicolò Lucchesi authored Jan 27, 2025
```
[Feature] [Spec decode]: Enable MLPSpeculator/Medusa and `prompt_logprobs` with ChunkedPrefill (#10132)
Signed-off-by: NickLucche <nlucches@redhat.com>
Signed-off-by: wallashss <wallashss@ibm.com>
Co-authored-by: wallashss <wallashss@ibm.com>
```
  6116ca8c
- [FlashInfer] Upgrade to 0.2.0 (#11194) · 2bc3fbba
  Bowen Wang authored Jan 28, 2025
```
Signed-off-by: Bowen Wang <abmfy@icloud.com>
Signed-off-by: youkaichao <youkaichao@gmail.com>
Co-authored-by: youkaichao <youkaichao@gmail.com>
```
  2bc3fbba
- [V1][CI/Test] Do basic test for top-p & top-k sampling (#12469) · 3f1fc742
  Woosuk Kwon authored Jan 27, 2025
```
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
```
  3f1fc742
- [V1][Metrics] Add initial Prometheus logger (#12416) · 01ba9270
  Mark McLoughlin authored Jan 27, 2025
```
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
```
  01ba9270
- [Build] Only build 9.0a for scaled_mm and sparse kernels (#12339) · 103bd17a
  Lucas Wilkinson authored Jan 27, 2025
```
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
```
  103bd17a
- [Bugfix] Fix gpt2 GGUF inference (#12467) · ce69f7f7
  Isotr0py authored Jan 27, 2025
```
Signed-off-by: Isotr0py <2037008807@qq.com>
```
  ce69f7f7
- [V1][Minor] Minor optimizations for update_from_output (#12454) · 624a1e47
  Woosuk Kwon authored Jan 27, 2025
```
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
```
  624a1e47
- [Bugfix] Fix missing seq_start_loc in xformers prefill metadata (#12464) · 372bf089
  Isotr0py authored Jan 27, 2025
```
Signed-off-by: Isotr0py <2037008807@qq.com>
```
  372bf089
- [Bugfix] Fix Granite 3.0 MoE model loading (#12446) · 5204ff5c
  Cyrus Leung authored Jan 27, 2025
```
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
```
  5204ff5c
- [Frontend] Support scores endpoint in run_batch (#12430) · 0cc6b383
  Pooya Davoodi authored Jan 26, 2025
```
Signed-off-by: Pooya Davoodi <pooya.davoodi@parasail.io>
```
  0cc6b383
- [V1] Avoid list creation in input preparation (#12457) · 28e07508
  Woosuk Kwon authored Jan 26, 2025
```
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
```
  28e07508
- [DOC] Add link to vLLM blog (#12460) · 582cf787
  Yuan Tang authored Jan 26, 2025
```
Signed-off-by: Yuan Tang <terrytangyuan@gmail.com>
```
  582cf787
- [Frontend] Rerank API (Jina- and Cohere-compatible API) (#12376) · 0034b09c
  Kyle Mistele authored Jan 26, 2025
```
Signed-off-by: Kyle Mistele <kyle@mistele.com>
```
  0034b09c
26 Jan, 2025 9 commits

[Build/CI] Fix libcuda.so linkage (#12424) · 72bac730
Tyler Michael Smith authored Jan 26, 2025
```
Signed-off-by: Tyler Michael Smith <tyler@neuralmagic.com>
```
72bac730
[Bugfix][Kernel] Fix perf regression caused by PR #12405 (#12434) · 68f11149
Lucas Wilkinson authored Jan 26, 2025
```
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
```
68f11149
[Bugfix/CI] Fix broken kernels/test_mha.py (#12450) · 72f48804
Tyler Michael Smith authored Jan 26, 2025

72f48804
[Bugfix] Disable w16a16 2of4 sparse CompressedTensors24 (#12417) · aa2cd2c4
Tyler Michael Smith authored Jan 26, 2025
```
Signed-off-by: Tyler Michael Smith <tyler@neuralmagic.com>
Co-authored-by: mgoin <michael@neuralmagic.com>
```
aa2cd2c4

[Frontend] generation_config.json for maximum tokens(#12242) · 9ddc3522

Matthew Hendrey authored Jan 26, 2025


Signed-off-by: Matthew Hendrey <matthew.hendrey@gmail.com>
Signed-off-by: Shangming Cai <caishangming@linux.alibaba.com>
Signed-off-by: youkaichao <youkaichao@gmail.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Yuan Tang <terrytangyuan@gmail.com>
Signed-off-by: Isotr0py <2037008807@qq.com>
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com>
Co-authored-by: shangmingc <caishangming@linux.alibaba.com>
Co-authored-by: youkaichao <youkaichao@gmail.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Yuan Tang <terrytangyuan@gmail.com>
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
Co-authored-by: Chen Zhang <zhangch99@outlook.com>
Co-authored-by: wangxiyuan <wangxiyuan1007@gmail.com>

9ddc3522

[Misc] Revert FA on ViT #12355 and #12435 (#12445) · a5255270
Roger Wang authored Jan 26, 2025

a5255270
[V1][Bugfix] Fix assertion when mm hashing is turned off (#12439) · 0ee349b5
Roger Wang authored Jan 26, 2025
```
Signed-off-by: Roger Wang <ywang@roblox.com>
```
0ee349b5
[V1][Perf] Reduce scheduling overhead in model runner after cuda sync (#12094) · fa63e710
Keyun Tong authored Jan 26, 2025
```
Signed-off-by: Keyun Tong <tongkeyun@gmail.com>
```
fa63e710

[Misc][Bugfix] FA3 support to ViT MHA layer (#12435) · 2a0309a6

Roger Wang authored Jan 25, 2025


Signed-off-by: Roger Wang <ywang@roblox.com>
Signed-off-by: Isotr0py <2037008807@qq.com>
Co-authored-by: Isotr0py <2037008807@qq.com>

2a0309a6

25 Jan, 2025 5 commits
- [TPU][CI] Update torchxla version in requirement-tpu.txt (#12422) · 324960a9
  Siyuan Liu authored Jan 24, 2025
```
Signed-off-by: Siyuan Liu <lsiyuan@google.com>
```
  324960a9
- [Misc] Add FA2 support to ViT MHA layer (#12355) · f1fc0510
  Isotr0py authored Jan 25, 2025
```
Signed-off-by: Isotr0py <2037008807@qq.com>
```
  f1fc0510
- [ROCm][MoE] MI300 tuned configs Mixtral-8x(7B,22B) | fp16, fp8 (#12408) · bf21481d
  Divakar Verma authored Jan 24, 2025
```
Signed-off-by: Divakar Verma <divakar.verma@amd.com>
```
  bf21481d
- [Bugfix] Fix BLIP-2 processing (#12412) · fb30ee92
  Cyrus Leung authored Jan 25, 2025
```
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
```
  fb30ee92
- [Bugfix][Kernel] Fix moe align block issue for mixtral (#12413) · 221d388c
  ElizaWszola authored Jan 24, 2025
  
  221d388c
24 Jan, 2025 7 commits
- [Bugfix][Kernel] FA3 Fix - RuntimeError: This flash attention build only... · 3132a933
  Lucas Wilkinson authored Jan 24, 2025
```
[Bugfix][Kernel] FA3 Fix - RuntimeError: This flash attention build only supports pack_gqa (for build size reasons). (#12405)
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
```
  3132a933
- [Misc] Remove deprecated code (#12383) · df5dafaa
  Cyrus Leung authored Jan 25, 2025
```
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
```
  df5dafaa
- [Bugfix][Kernel] Fix CUDA 11.8 being broken by FA3 build (#12375) · ab5bbf5a
  Lucas Wilkinson authored Jan 24, 2025
```
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
```
  ab5bbf5a
- [Misc] Enable proxy support in benchmark script (#12356) · 3bb8e2c9
  Junichi Sato authored Jan 24, 2025
```
Signed-off-by: Junichi Sato <junichi.sato@sbintuitions.co.jp>
```
  3bb8e2c9
- [ci/build] sync default value for wheel size (#12398) · e784c6b9
  youkaichao authored Jan 24, 2025
```
Signed-off-by: youkaichao <youkaichao@gmail.com>
```
  e784c6b9
- [Hardware][Gaudi][Doc] Add missing step in setup instructions (#12382) · 9a0f3bdb
  Mohit Deopujari authored Jan 24, 2025
  
  9a0f3bdb
- [ci/build] fix wheel size check (#12396) · c7c98510
  youkaichao authored Jan 24, 2025
```
Signed-off-by: youkaichao <youkaichao@gmail.com>
```
  c7c98510