Commits · e0cc5f259a8bec0d66ed0bc3e25ca245377679a1 · OpenDAS / vllm_cscc

29 Jan, 2025 14 commits

[V1][BugFix] Free encoder cache for aborted requests (#12545) · e0cc5f25
Woosuk Kwon authored Jan 29, 2025
```
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
```
e0cc5f25
Revert "[Build/CI] Fix libcuda.so linkage" (#12552) · 73aa6cfd
Tyler Michael Smith authored Jan 29, 2025

73aa6cfd
[Kernel] add triton fused moe kernel for gptq/awq (#12185) · 27b78c73
Jinzhen Lin authored Jan 29, 2025

27b78c73
[Hardware][NV] Fix Modelopt model loading for k-v-scales for Llama models. (#11787) · b02fd288
Pavani Majety authored Jan 29, 2025
```
Signed-off-by: Pavani Majety <pmajety@nvidia.com>
Co-authored-by: mgoin <michael@neuralmagic.com>
```
b02fd288
[Frontend] Support override generation config in args (#12409) · ff7424f4
Yanyi Liu authored Jan 29, 2025
```
Signed-off-by: liuyanyi <wolfsonliu@163.com>
```
ff7424f4

[Model] Refactoring of MiniCPM-V and add MiniCPM-o-2.6 support for vLLM (#12069) · d93bf4da

Alphi authored Jan 29, 2025


Signed-off-by: hzh <hezhihui_thu@163.com>
Signed-off-by: Sungjae Lee <33976427+llsj14@users.noreply.github.com>
Signed-off-by: shaochangxu.scx <shaochangxu.scx@antgroup.com>
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: NickLucche <nlucches@redhat.com>
Signed-off-by: Isotr0py <2037008807@qq.com>
Signed-off-by: Roger Wang <ywang@roblox.com>
Signed-off-by: Rafael Vasquez <rafvasq21@gmail.com>
Signed-off-by: Akshat Tripathi <akshat@krai.ai>
Signed-off-by: Oleg Mosalov <oleg@krai.ai>
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
Signed-off-by: rshaw@neuralmagic.com <rshaw@neuralmagic.com>
Signed-off-by: Yida Wu <yidawu@alumni.cmu.edu>
Signed-off-by: Chenguang Li <757486878@qq.com>
Signed-off-by: youkaichao <youkaichao@gmail.com>
Signed-off-by: Alex-Brooks <Alex.brooks@ibm.com>
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Shanshan Shen <467638484@qq.com>
Signed-off-by: elijah <f1renze.142857@gmail.com>
Signed-off-by: Yikun <yikunkero@gmail.com>
Signed-off-by: mgoin <michael@neuralmagic.com>
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
Signed-off-by: Konrad Zawora <kzawora@habana.ai>
Signed-off-by: tjtanaa <tunjian.tan@embeddedllm.com>
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com>
Signed-off-by: Rui Qiao <ruisearch42@gmail.com>
Co-authored-by: Sungjae Lee <33976427+llsj14@users.noreply.github.com>
Co-authored-by: shaochangxu <85155497+shaochangxu@users.noreply.github.com>
Co-authored-by: shaochangxu.scx <shaochangxu.scx@antgroup.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com>
Co-authored-by: sixgod <evethwillbeok@outlook.com>
Co-authored-by: Isotr0py <2037008807@qq.com>
Co-authored-by: Roger Wang <136131678+ywang96@users.noreply.github.com>
Co-authored-by: Rafael Vasquez <rafvasq21@gmail.com>
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
Co-authored-by: Akshat Tripathi <Akshat.tripathi6568@gmail.com>
Co-authored-by: Oleg Mosalov <oleg@krai.ai>
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com>
Co-authored-by: Avshalom Manevich <12231371+avshalomman@users.noreply.github.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-neuralmagic@users.noreply.github.com>
Co-authored-by: Yangcheng Li <liyangcheng.lyc@alibaba-inc.com>
Co-authored-by: Siyuan Li <94890248+liaoyanqing666@users.noreply.github.com>
Co-authored-by: Concurrensee <yida.wu@amd.com>
Co-authored-by: Chenguang Li <757486878@qq.com>
Co-authored-by: youkaichao <youkaichao@gmail.com>
Co-authored-by: Alex Brooks <alex.brooks@ibm.com>
Co-authored-by: Chen Zhang <zhangch99@outlook.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Shanshan Shen <467638484@qq.com>
Co-authored-by: elijah <30852919+e1ijah1@users.noreply.github.com>
Co-authored-by: Yikun Jiang <yikunkero@gmail.com>
Co-authored-by: Steve Luo <36296769+SunflowerAries@users.noreply.github.com>
Co-authored-by: mgoin <michael@neuralmagic.com>
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
Co-authored-by: Konrad Zawora <kzawora@habana.ai>
Co-authored-by: TJian <tunjian1996@gmail.com>
Co-authored-by: tjtanaa <tunjian.tan@embeddedllm.com>
Co-authored-by: wangxiyuan <wangxiyuan1007@gmail.com>
Co-authored-by: maang-h <55082429+maang-h@users.noreply.github.com>
Co-authored-by: Elfie Guo <164945471+elfiegg@users.noreply.github.com>
Co-authored-by: Rui Qiao <161574667+ruisearch42@users.noreply.github.com>
Co-authored-by: Roger Wang <ywang@roblox.com>

d93bf4da

[Bugfix] handle alignment of arguments in convert_sparse_cross_attention_mask_to_dense (#12347) · 036ca94c

Travis Johnson authored Jan 29, 2025


Signed-off-by: Travis Johnson <tsjohnso@us.ibm.com>
Signed-off-by: Wallas Santos <wallashss@ibm.com>
Co-authored-by: Wallas Santos <wallashss@ibm.com>

036ca94c

Fix the pydantic logging validator (#12420) · ef001d98
Maximilien de Bayser authored Jan 29, 2025
```
Signed-off-by: Max de Bayser <mbayser@br.ibm.com>
```
ef001d98
[V1] Improve Error Message for Unsupported Config (#12535) · 5f671cb4
Robert Shaw authored Jan 28, 2025
```
Co-authored-by: Michael Goin <michael@neuralmagic.com>
```
5f671cb4
Bugfix for whisper quantization due to fake k_proj bias (#12524) · bd02164c
Michael Goin authored Jan 28, 2025
```
Signed-off-by: mgoin <michael@neuralmagic.com>
```
bd02164c
[V1][Metrics] Add TTFT and TPOT histograms (#12530) · 46fb0567
Mark McLoughlin authored Jan 29, 2025
```
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
```
46fb0567
[Doc] Convert docs to use colon fences (#12471) · dd6a3a02
Harry Mellor authored Jan 29, 2025
```
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
```
dd6a3a02

[Frontend] Support reasoning content for deepseek r1 (#12473) · a7e3eba6

Ce Gao authored Jan 29, 2025


Signed-off-by: Ce Gao <cegao@tensorchord.ai>
Co-authored-by: Rafael Vasquez <rafvasq21@gmail.com>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
Co-authored-by: Michael Goin <mgoin@redhat.com>

a7e3eba6

[TPU] Add example for profiling TPU inference (#12531) · fbb5bd4c
Michael Goin authored Jan 28, 2025
```
Signed-off-by: mgoin <mgoin@redhat.com>
```
fbb5bd4c

28 Jan, 2025 16 commits
- [Kernel] Pipe attn_logits_soft_cap through paged attention TPU kernels (#12482) · 80fcc3ed
  fenghuizhang authored Jan 28, 2025
```
Signed-off-by: Fenghui Zhang <fhzhang@google.com>
```
  80fcc3ed
- [V1][Metrics] Add per-request prompt/generation_tokens histograms (#12516) · c386c43c
  Mark McLoughlin authored Jan 28, 2025
```
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
```
  c386c43c
- Do not run `suggestion` `pre-commit` hook multiple times (#12521) · f26d7907
  Harry Mellor authored Jan 28, 2025
```
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
```
  f26d7907
- Replace missed warning_once for rerank API (#12472) · 0f657bdc
  Michael Goin authored Jan 28, 2025
```
Signed-off-by: mgoin <michael@neuralmagic.com>
```
  0f657bdc
- [V1][Metrics] Hook up IterationStats for Prometheus metrics (#12478) · 3fd1fb63
  Mark McLoughlin authored Jan 28, 2025
```
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
```
  3fd1fb63
- [Doc] Fix typo for x86 CPU installation (#12514) · 925d2f19
  Jun Duan authored Jan 28, 2025
```
Signed-off-by: Jun Duan <jun.duan.phd@outlook.com>
```
  925d2f19
- [VLM] Merged multi-modal processor and V1 support for Qwen-VL (#12504) · 8f58a513
  Cyrus Leung authored Jan 29, 2025
```
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
```
  8f58a513
- [Core] Make raw_request optional in ServingCompletion (#12503) · 2079e43b
  Sebastian Schoennenbeck authored Jan 28, 2025
```
Signed-off-by: Sebastian Schönnenbeck <sebastian.schoennenbeck@comma-soft.com>
```
  2079e43b
- [V1] Include Engine Version in Logs (#12496) · e29d4358
  Robert Shaw authored Jan 28, 2025
```
Signed-off-by: rshaw@neuralmagic.com <rshaw@neuralmagic.com>
```
  e29d4358
- Update README.md with V1 alpha release (#12495) · 8cbc4249
  Roger Wang authored Jan 28, 2025
  
  8cbc4249
- [CI] fix pre-commit error (#12494) · dd66fd2b
  Mengqing Cao authored Jan 28, 2025
```
Signed-off-by: Mengqing Cao <cmq0113@163.com>
```
  dd66fd2b
- [FEATURE] Enables offline /score for embedding models (#12021) · 0f465ab5
  Gabriel Marinho authored Jan 28, 2025
```
Signed-off-by: Gabriel Marinho <gmarinho@ibm.com>
```
  0f465ab5
- [CI/Build] Fixed the xla nightly issue report in #12451 (#12453) · 23a7cbc8
  Hossein Sarshar authored Jan 27, 2025
  
  23a7cbc8
- Fix bad path in prometheus example (#12481) · 426a5c36
  Michael Goin authored Jan 27, 2025
```
Signed-off-by: mgoin <michael@neuralmagic.com>
```
  426a5c36
- [Neuron][Kernel] NKI-based flash-attention kernel with paged KV cache (#11277) · ddee88d0
  Liangfu Chen authored Jan 27, 2025
```
Signed-off-by: Liangfu Chen <liangfc@amazon.com>
Co-authored-by: Jiangfei Duan <jfduan@outlook.com>
```
  ddee88d0
- Update `pre-commit` hooks (#12475) · 823ab796
  Harry Mellor authored Jan 28, 2025
```
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
```
  823ab796
27 Jan, 2025 10 commits
- [Feature] [Spec decode]: Enable MLPSpeculator/Medusa and `prompt_logprobs`... · 6116ca8c
  Nicolò Lucchesi authored Jan 27, 2025
```
[Feature] [Spec decode]: Enable MLPSpeculator/Medusa and `prompt_logprobs` with ChunkedPrefill (#10132)
Signed-off-by: NickLucche <nlucches@redhat.com>
Signed-off-by: wallashss <wallashss@ibm.com>
Co-authored-by: wallashss <wallashss@ibm.com>
```
  6116ca8c
- [FlashInfer] Upgrade to 0.2.0 (#11194) · 2bc3fbba
  Bowen Wang authored Jan 28, 2025
```
Signed-off-by: Bowen Wang <abmfy@icloud.com>
Signed-off-by: youkaichao <youkaichao@gmail.com>
Co-authored-by: youkaichao <youkaichao@gmail.com>
```
  2bc3fbba
- [V1][CI/Test] Do basic test for top-p & top-k sampling (#12469) · 3f1fc742
  Woosuk Kwon authored Jan 27, 2025
```
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
```
  3f1fc742
- [V1][Metrics] Add initial Prometheus logger (#12416) · 01ba9270
  Mark McLoughlin authored Jan 27, 2025
```
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
```
  01ba9270
- [Build] Only build 9.0a for scaled_mm and sparse kernels (#12339) · 103bd17a
  Lucas Wilkinson authored Jan 27, 2025
```
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
```
  103bd17a
- [Bugfix] Fix gpt2 GGUF inference (#12467) · ce69f7f7
  Isotr0py authored Jan 27, 2025
```
Signed-off-by: Isotr0py <2037008807@qq.com>
```
  ce69f7f7
- [V1][Minor] Minor optimizations for update_from_output (#12454) · 624a1e47
  Woosuk Kwon authored Jan 27, 2025
```
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
```
  624a1e47
- [Bugfix] Fix missing seq_start_loc in xformers prefill metadata (#12464) · 372bf089
  Isotr0py authored Jan 27, 2025
```
Signed-off-by: Isotr0py <2037008807@qq.com>
```
  372bf089
- [Bugfix] Fix Granite 3.0 MoE model loading (#12446) · 5204ff5c
  Cyrus Leung authored Jan 27, 2025
```
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
```
  5204ff5c
- [Frontend] Support scores endpoint in run_batch (#12430) · 0cc6b383
  Pooya Davoodi authored Jan 26, 2025
```
Signed-off-by: Pooya Davoodi <pooya.davoodi@parasail.io>
```
  0cc6b383