Commits · afd0da2186c1d58fb48e138df0a2f548612b5d7d · OpenDAS / vllm_cscc

01 Feb, 2025 1 commit

[Attention] Deepseek v3 MLA support with FP8 compute (#12601) · baeded25

Lucas Wilkinson authored Feb 01, 2025



This PR implements the Deepseek V3 support by performing matrix absorption the fp8 weights 

---------
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
Co-authored-by: simon-mo <simon.mo@hey.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: Zhuohan Li <zhuohan123@gmail.com>
Co-authored-by: Tyler Michael Smith <tysmith@redhat.com>
Co-authored-by: Alexander Matveev <59768536+alexm-neuralmagic@users.noreply.github.com>

baeded25

31 Jan, 2025 2 commits

[Attention] MLA decode optimizations (#12528) · cabaf4ef

Lucas Wilkinson authored Jan 31, 2025


Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
Signed-off-by: simon-mo <xmo@berkeley.edu>
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
Co-authored-by: simon-mo <simon.mo@hey.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: Zhuohan Li <zhuohan123@gmail.com>
Co-authored-by: Tyler Michael Smith <tysmith@redhat.com>
Co-authored-by: Alexander Matveev <59768536+alexm-neuralmagic@users.noreply.github.com>
Co-authored-by: simon-mo <xmo@berkeley.edu>

cabaf4ef

[ROCm][AMD][Model] llama 3.2 support upstreaming (#12421) · a1fc18c0
Aleksandr Malyshev authored Jan 30, 2025
```
Signed-off-by: Aleksandr Malyshev <maleksan@amd.com>
Co-authored-by: Aleksandr Malyshev <maleksan@amd.com>
```
a1fc18c0

29 Jan, 2025 4 commits

[Hardware][NV] Fix Modelopt model loading for k-v-scales for Llama models. (#11787) · b02fd288
Pavani Majety authored Jan 29, 2025
```
Signed-off-by: Pavani Majety <pmajety@nvidia.com>
Co-authored-by: mgoin <michael@neuralmagic.com>
```
b02fd288

[Model] Refactoring of MiniCPM-V and add MiniCPM-o-2.6 support for vLLM (#12069) · d93bf4da

Alphi authored Jan 29, 2025


Signed-off-by: hzh <hezhihui_thu@163.com>
Signed-off-by: Sungjae Lee <33976427+llsj14@users.noreply.github.com>
Signed-off-by: shaochangxu.scx <shaochangxu.scx@antgroup.com>
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: NickLucche <nlucches@redhat.com>
Signed-off-by: Isotr0py <2037008807@qq.com>
Signed-off-by: Roger Wang <ywang@roblox.com>
Signed-off-by: Rafael Vasquez <rafvasq21@gmail.com>
Signed-off-by: Akshat Tripathi <akshat@krai.ai>
Signed-off-by: Oleg Mosalov <oleg@krai.ai>
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
Signed-off-by: rshaw@neuralmagic.com <rshaw@neuralmagic.com>
Signed-off-by: Yida Wu <yidawu@alumni.cmu.edu>
Signed-off-by: Chenguang Li <757486878@qq.com>
Signed-off-by: youkaichao <youkaichao@gmail.com>
Signed-off-by: Alex-Brooks <Alex.brooks@ibm.com>
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Shanshan Shen <467638484@qq.com>
Signed-off-by: elijah <f1renze.142857@gmail.com>
Signed-off-by: Yikun <yikunkero@gmail.com>
Signed-off-by: mgoin <michael@neuralmagic.com>
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
Signed-off-by: Konrad Zawora <kzawora@habana.ai>
Signed-off-by: tjtanaa <tunjian.tan@embeddedllm.com>
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com>
Signed-off-by: Rui Qiao <ruisearch42@gmail.com>
Co-authored-by: Sungjae Lee <33976427+llsj14@users.noreply.github.com>
Co-authored-by: shaochangxu <85155497+shaochangxu@users.noreply.github.com>
Co-authored-by: shaochangxu.scx <shaochangxu.scx@antgroup.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com>
Co-authored-by: sixgod <evethwillbeok@outlook.com>
Co-authored-by: Isotr0py <2037008807@qq.com>
Co-authored-by: Roger Wang <136131678+ywang96@users.noreply.github.com>
Co-authored-by: Rafael Vasquez <rafvasq21@gmail.com>
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
Co-authored-by: Akshat Tripathi <Akshat.tripathi6568@gmail.com>
Co-authored-by: Oleg Mosalov <oleg@krai.ai>
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com>
Co-authored-by: Avshalom Manevich <12231371+avshalomman@users.noreply.github.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-neuralmagic@users.noreply.github.com>
Co-authored-by: Yangcheng Li <liyangcheng.lyc@alibaba-inc.com>
Co-authored-by: Siyuan Li <94890248+liaoyanqing666@users.noreply.github.com>
Co-authored-by: Concurrensee <yida.wu@amd.com>
Co-authored-by: Chenguang Li <757486878@qq.com>
Co-authored-by: youkaichao <youkaichao@gmail.com>
Co-authored-by: Alex Brooks <alex.brooks@ibm.com>
Co-authored-by: Chen Zhang <zhangch99@outlook.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Shanshan Shen <467638484@qq.com>
Co-authored-by: elijah <30852919+e1ijah1@users.noreply.github.com>
Co-authored-by: Yikun Jiang <yikunkero@gmail.com>
Co-authored-by: Steve Luo <36296769+SunflowerAries@users.noreply.github.com>
Co-authored-by: mgoin <michael@neuralmagic.com>
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
Co-authored-by: Konrad Zawora <kzawora@habana.ai>
Co-authored-by: TJian <tunjian1996@gmail.com>
Co-authored-by: tjtanaa <tunjian.tan@embeddedllm.com>
Co-authored-by: wangxiyuan <wangxiyuan1007@gmail.com>
Co-authored-by: maang-h <55082429+maang-h@users.noreply.github.com>
Co-authored-by: Elfie Guo <164945471+elfiegg@users.noreply.github.com>
Co-authored-by: Rui Qiao <161574667+ruisearch42@users.noreply.github.com>
Co-authored-by: Roger Wang <ywang@roblox.com>

d93bf4da

[Bugfix] handle alignment of arguments in convert_sparse_cross_attention_mask_to_dense (#12347) · 036ca94c

Travis Johnson authored Jan 29, 2025


Signed-off-by: Travis Johnson <tsjohnso@us.ibm.com>
Signed-off-by: Wallas Santos <wallashss@ibm.com>
Co-authored-by: Wallas Santos <wallashss@ibm.com>

036ca94c

Bugfix for whisper quantization due to fake k_proj bias (#12524) · bd02164c
Michael Goin authored Jan 28, 2025
```
Signed-off-by: mgoin <michael@neuralmagic.com>
```
bd02164c

28 Jan, 2025 2 commits
- [VLM] Merged multi-modal processor and V1 support for Qwen-VL (#12504) · 8f58a513
  Cyrus Leung authored Jan 29, 2025
```
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
```
  8f58a513
- Update `pre-commit` hooks (#12475) · 823ab796
  Harry Mellor authored Jan 28, 2025
```
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
```
  823ab796
27 Jan, 2025 2 commits
- [Bugfix] Fix gpt2 GGUF inference (#12467) · ce69f7f7
  Isotr0py authored Jan 27, 2025
```
Signed-off-by: Isotr0py <2037008807@qq.com>
```
  ce69f7f7
- [Bugfix] Fix Granite 3.0 MoE model loading (#12446) · 5204ff5c
  Cyrus Leung authored Jan 27, 2025
```
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
```
  5204ff5c
25 Jan, 2025 1 commit
- [Bugfix] Fix BLIP-2 processing (#12412) · fb30ee92
  Cyrus Leung authored Jan 25, 2025
```
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
```
  fb30ee92
23 Jan, 2025 3 commits
- [FP8][Kernel] Dynamic kv cache scaling factors computation (#11906) · e97f802b
  Gregory Shtrasberg authored Jan 23, 2025
```
Signed-off-by: Gregory Shtrasberg <Gregory.Shtrasberg@amd.com>
Co-authored-by: Micah Williamson <micah.williamson@amd.com>
```
  e97f802b
- [Bugfix] Fix k_proj's bias for whisper self attention (#12342) · c5b4b11d
  Isotr0py authored Jan 23, 2025
```
Signed-off-by: Isotr0py <2037008807@qq.com>
```
  c5b4b11d
- support qwen2-vl video · 1a11f127
  zhuwenwen authored Jan 23, 2025
  
  1a11f127
22 Jan, 2025 6 commits
- [Doc] Add docs for prompt replacement (#12318) · 6609cdf0
  Cyrus Leung authored Jan 22, 2025
```
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
```
  6609cdf0
- [Bugfix][VLM] Fix mixed-modality inference backward compatibility for V0 (#12313) · 16366ee8
  Roger Wang authored Jan 22, 2025
```
Signed-off-by: Roger Wang <ywang@roblox.com>
```
  16366ee8
- [Model][Bugfix]: correct Aria model output (#12309) · 528dbcac
  zhou fan authored Jan 22, 2025
```
Signed-off-by: xffxff <1247714429@qq.com>
```
  528dbcac
- [VLM] Avoid unnecessary tokenization (#12310) · cd7b6f08
  Cyrus Leung authored Jan 22, 2025
```
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
```
  cd7b6f08
- [ci/lint] Add back default arg for pre-commit (#12279) · 64ea24d0
  Kevin H. Luu authored Jan 21, 2025
```
Signed-off-by: kevin <kevin@anyscale.com>
```
  64ea24d0
- [VLM] Simplify post-processing of replacement info (#12269) · df76e5af
  Cyrus Leung authored Jan 22, 2025
```
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
```
  df76e5af
21 Jan, 2025 4 commits
- [Misc] Set default backend to SDPA for get_vit_attn_backend (#12235) · fa9ee081
  wangxiyuan authored Jan 22, 2025
```
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com>
```
  fa9ee081
- [Misc] Remove redundant TypeVar from base model (#12248) · f2e9f2a3
  Cyrus Leung authored Jan 21, 2025
```
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
```
  f2e9f2a3
- [Misc]Add BNB quantization for PaliGemmaForConditionalGeneration (#12237) · 1f1542af
  Jee Jee Li authored Jan 21, 2025
```
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
```
  1f1542af
- [Misc] Rename `MultiModalInputsV2 -> MultiModalInputs` (#12244) · 96912550
  Cyrus Leung authored Jan 21, 2025
```
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
```
  96912550
20 Jan, 2025 2 commits
- [Model] Upgrade Aria to transformers 4.48 (#12203) · b37d8279
  Cyrus Leung authored Jan 20, 2025
```
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
```
  b37d8279
- [Model] Add Qwen2 PRM model support (#12202) · 83609791
  Isotr0py authored Jan 20, 2025
```
Signed-off-by: Isotr0py <2037008807@qq.com>
```
  83609791
19 Jan, 2025 4 commits

[Model] Support for fairseq2 Llama (#11442) · bbe5f9de

Martin Gleize authored Jan 19, 2025


Signed-off-by: Martin Gleize <mgleize@meta.com>
Co-authored-by: mgleize user <mgleize@a100-st-p4de24xlarge-4.fair-a100.hpcaas>

bbe5f9de

[V1] Add V1 support of Qwen2-VL (#12128) · 81763c58

Roger Wang authored Jan 19, 2025


Signed-off-by: Roger Wang <ywang@roblox.com>
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Co-authored-by: imkero <kerorek@outlook.com>
Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>

81763c58

[Misc] Add BNB support to GLM4-V model (#12184) · edaae198
Isotr0py authored Jan 19, 2025
```
Signed-off-by: Isotr0py <2037008807@qq.com>
```
edaae198
[Bugfix] Fix multi-modal processors for transformers 4.48 (#12187) · 630eb5b5
Cyrus Leung authored Jan 19, 2025

630eb5b5

18 Jan, 2025 1 commit
- [Model] Port deepseek-vl2 processor, remove dependency (#12169) · 02798eca
  Isotr0py authored Jan 18, 2025
```
Signed-off-by: Isotr0py <2037008807@qq.com>
```
  02798eca
16 Jan, 2025 5 commits
- [Model] Add support for deepseek-vl2-tiny model (#12068) · 62b06ba2
  Isotr0py authored Jan 17, 2025
```
Signed-off-by: Isotr0py <2037008807@qq.com>
```
  62b06ba2
- [Bugfix] Fix max image feature size for Llava-one-vision (#12104) · 874f7c29
  Roger Wang authored Jan 16, 2025
```
Signed-off-by: Roger Wang <ywang@roblox.com>
```
  874f7c29
- [Bugfix] Remove hardcoded `head_size=256` for Deepseek v2 and v3 (#12067) · dd7c9ad8
  Isotr0py authored Jan 16, 2025
```
Signed-off-by: Isotr0py <2037008807@qq.com>
```
  dd7c9ad8
- Various cosmetic/comment fixes (#12089) · 9aa1519f
  Michael Goin authored Jan 16, 2025
```
Signed-off-by: mgoin <michael@neuralmagic.com>
```
  9aa1519f
- 增加w8a8相关修改 · 083b80ea
  zhuwenwen authored Jan 16, 2025
  
  083b80ea
15 Jan, 2025 3 commits
- [Misc][Quark] Upstream Quark format to VLLM (#10765) · de0526f6
  kewang-xlnx authored Jan 16, 2025
```
Signed-off-by: kewang-xlnx <kewang@xilinx.com>
Signed-off-by: kewang2 <kewang2@amd.com>
Co-authored-by: kewang2 <kewang2@amd.com>
Co-authored-by: Michael Goin <michael@neuralmagic.com>
```
  de0526f6
- [Model]: Support internlm3 (#12037) · 97eb97b5
  RunningLeon authored Jan 15, 2025
  
  97eb97b5
- [Kernel] Support MulAndSilu (#11624) · 42f5e7c5
  Jee Jee Li authored Jan 15, 2025
```
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
```
  42f5e7c5