Commits · f256ebe4df6757d76f1f1642d7e110268a2f8190 · OpenDAS / vllm_cscc

31 Jan, 2025 2 commits

[Attention] MLA decode optimizations (#12528) · cabaf4ef

Lucas Wilkinson authored Jan 31, 2025


Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
Signed-off-by: simon-mo <xmo@berkeley.edu>
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
Co-authored-by: simon-mo <simon.mo@hey.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: Zhuohan Li <zhuohan123@gmail.com>
Co-authored-by: Tyler Michael Smith <tysmith@redhat.com>
Co-authored-by: Alexander Matveev <59768536+alexm-neuralmagic@users.noreply.github.com>
Co-authored-by: simon-mo <xmo@berkeley.edu>

cabaf4ef

[Kernel] Update `cutlass_scaled_mm` to support 2d group (blockwise) scaling (#11868) · 9798b2fb
Lucas Wilkinson authored Jan 30, 2025

9798b2fb

29 Jan, 2025 1 commit
- [Kernel] add triton fused moe kernel for gptq/awq (#12185) · 27b78c73
  Jinzhen Lin authored Jan 29, 2025
  
  27b78c73
28 Jan, 2025 1 commit
- Update `pre-commit` hooks (#12475) · 823ab796
  Harry Mellor authored Jan 28, 2025
```
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
```
  823ab796
27 Jan, 2025 1 commit

[FlashInfer] Upgrade to 0.2.0 (#11194) · 2bc3fbba

Bowen Wang authored Jan 28, 2025


Signed-off-by: Bowen Wang <abmfy@icloud.com>
Signed-off-by: youkaichao <youkaichao@gmail.com>
Co-authored-by: youkaichao <youkaichao@gmail.com>

2bc3fbba

26 Jan, 2025 2 commits
- [Bugfix/CI] Fix broken kernels/test_mha.py (#12450) · 72f48804
  Tyler Michael Smith authored Jan 26, 2025
  
  72f48804
- [Bugfix] Disable w16a16 2of4 sparse CompressedTensors24 (#12417) · aa2cd2c4
  Tyler Michael Smith authored Jan 26, 2025
```
Signed-off-by: Tyler Michael Smith <tyler@neuralmagic.com>
Co-authored-by: mgoin <michael@neuralmagic.com>
```
  aa2cd2c4
25 Jan, 2025 1 commit
- [Misc] Add FA2 support to ViT MHA layer (#12355) · f1fc0510
  Isotr0py authored Jan 25, 2025
```
Signed-off-by: Isotr0py <2037008807@qq.com>
```
  f1fc0510
24 Jan, 2025 1 commit
- [Bugfix][Kernel] Fix CUDA 11.8 being broken by FA3 build (#12375) · ab5bbf5a
  Lucas Wilkinson authored Jan 24, 2025
```
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
```
  ab5bbf5a
23 Jan, 2025 3 commits
- [FP8][Kernel] Dynamic kv cache scaling factors computation (#11906) · e97f802b
  Gregory Shtrasberg authored Jan 23, 2025
```
Signed-off-by: Gregory Shtrasberg <Gregory.Shtrasberg@amd.com>
Co-authored-by: Micah Williamson <micah.williamson@amd.com>
```
  e97f802b
- [Kernel] Flash Attention 3 Support (#12093) · 978b45f3
  Lucas Wilkinson authored Jan 23, 2025
```
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
```
  978b45f3
- [AMD][Quantization] Add TritonScaledMMLinearKernel since int8 is broken for AMD (#12282) · 68c4421b
  rasmith authored Jan 22, 2025
```
Signed-off-by: Randall Smith <Randall.Smith@amd.com>
```
  68c4421b
16 Jan, 2025 1 commit
- [Bugfix] Remove hardcoded `head_size=256` for Deepseek v2 and v3 (#12067) · dd7c9ad8
  Isotr0py authored Jan 16, 2025
```
Signed-off-by: Isotr0py <2037008807@qq.com>
```
  dd7c9ad8
15 Jan, 2025 2 commits

[Platform] Do not raise error if _Backend is not found (#12023) · 3adf0ffd

wangxiyuan authored Jan 15, 2025


Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com>
Signed-off-by: Mengqing Cao <cmq0113@163.com>
Co-authored-by: Mengqing Cao <cmq0113@163.com>

3adf0ffd

[Kernel] Support MulAndSilu (#11624) · 42f5e7c5
Jee Jee Li authored Jan 15, 2025
```
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
```
42f5e7c5

12 Jan, 2025 1 commit
- [Hardware][TPU] workaround fix for MoE on TPU (#11764) · 263a870e
  Avshalom Manevich authored Jan 12, 2025
  
  263a870e
10 Jan, 2025 1 commit
- [torch.compile] Hide KV cache behind torch.compile boundary (#11677) · cf5f000d
  Chen Zhang authored Jan 10, 2025
```
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
```
  cf5f000d
09 Jan, 2025 1 commit

[platform] Allow platform specify attention backend (#11609) · 405eb8e3

wangxiyuan authored Jan 09, 2025


Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com>
Signed-off-by: Mengqing Cao <cmq0113@163.com>
Co-authored-by: Mengqing Cao <cmq0113@163.com>

405eb8e3

06 Jan, 2025 1 commit
- [Kernel] Move attn_type to Attention.__init__() (#11690) · e20c92bb
  Chen Zhang authored Jan 07, 2025
```
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
```
  e20c92bb
01 Jan, 2025 1 commit
- [V1] Implement Cascade Attention (#11635) · 73001445
  Woosuk Kwon authored Jan 01, 2025
```
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
```
  73001445
30 Dec, 2024 1 commit
- [platforms] enable platform plugins (#11602) · b12e87f9
  youkaichao authored Dec 30, 2024
```
Signed-off-by: youkaichao <youkaichao@gmail.com>
```
  b12e87f9
26 Dec, 2024 1 commit

[Model] [Quantization] Support deepseek_v3 w8a8 fp8 block-wise quantization (#11523) · 2072924d

Michael Goin authored Dec 26, 2024


Signed-off-by: mgoin <michael@neuralmagic.com>
Signed-off-by: simon-mo <simon.mo@hey.com>
Signed-off-by: simon-mo <xmo@berkeley.edu>
Co-authored-by: simon-mo <simon.mo@hey.com>
Co-authored-by: simon-mo <xmo@berkeley.edu>
Co-authored-by: HandH1998 <1335248067@qq.com>

2072924d

19 Dec, 2024 1 commit
- [Bugfix][Build/CI] Fix sparse CUTLASS compilation on CUDA [12.0, 12.2) (#11311) · 5a9da2e6
  Tyler Michael Smith authored Dec 18, 2024
```
Signed-off-by: Tyler Michael Smith <tyler@neuralmagic.com>
```
  5a9da2e6
18 Dec, 2024 1 commit

[Kernel]: Cutlass 2:4 Sparsity + FP8/Int8 Quant Support (#10995) · 60508ffd

Dipika Sikka authored Dec 18, 2024


Co-authored-by: Faraz Shahsavan <faraz.shahsavan@gmail.com>
Co-authored-by: ilmarkov <markovilya197@gmail.com>
Co-authored-by: Rahul Tuli <rahul@neuralmagic.com>
Co-authored-by: rshaw@neuralmagic.com <rshaw@neuralmagic.com>

60508ffd

13 Dec, 2024 1 commit

[torch.compile] Dynamic fp8 + rms_norm fusion (#10906) · 30870b4f

Luka Govedič authored Dec 12, 2024


Signed-off-by: luka <luka@neuralmagic.com>
Co-authored-by: Varun Sundar Rabindranath <varun@neuralmagic.com>

30870b4f

07 Dec, 2024 1 commit
- [BugFix][Kernel]: fix illegal memory access in causal_conv1d when conv_states is None (#10928) · 78029b34
  zhou fan authored Dec 08, 2024
```
Signed-off-by: xffxff <1247714429@qq.com>
```
  78029b34
02 Dec, 2024 1 commit
- [Kernel] Use `out` arg in flash_attn_varlen_func (#10811) · 073a4bd1
  Woosuk Kwon authored Dec 01, 2024
```
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
```
  073a4bd1
25 Nov, 2024 2 commits
- [Bugfix] Fix chunked prefill with model dtype float32 on Turing Devices (#9850) · c27df94e
  Wallas Henrique authored Nov 25, 2024
```
Signed-off-by: Wallas Santos <wallashss@ibm.com>
Co-authored-by: Michael Goin <michael@neuralmagic.com>
```
  c27df94e
- [misc] move functions to config.py (#10624) · 05d1f8c9
  youkaichao authored Nov 25, 2024
```
Signed-off-by: youkaichao <youkaichao@gmail.com>
```
  05d1f8c9
22 Nov, 2024 1 commit
- [torch.compile] support all attention backends (#10558) · eebad39f
  youkaichao authored Nov 22, 2024
```
Signed-off-by: youkaichao <youkaichao@gmail.com>
```
  eebad39f
20 Nov, 2024 1 commit
- [Bugfix] Marlin 2:4 temp fix for large M dim (>256) (#10464) · d200972e
  Lucas Wilkinson authored Nov 19, 2024
```
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
```
  d200972e
19 Nov, 2024 2 commits
- [Model][Quantization] HQQ support through Marlin kernel expansion (#9766) · b00b33d7
  ElizaWszola authored Nov 19, 2024
```
Signed-off-by: ElizaWszola <eliza@neuralmagic.com>
```
  b00b33d7
- [Platform][Refactor] Extract func `get_default_attn_backend` to `Platform` (#10358) · 8c1fb507
  Mengqing Cao authored Nov 19, 2024
```
Signed-off-by: Mengqing Cao <cmq0113@163.com>
```
  8c1fb507
18 Nov, 2024 1 commit
- [Kernel] Initial Machete W4A8 support + Refactors (#9855) · 96d999fb
  Lucas Wilkinson authored Nov 18, 2024
```
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
```
  96d999fb
15 Nov, 2024 2 commits
- [Misc] Bump up test_fused_moe tolerance (#10364) · 79ee45b4
  ElizaWszola authored Nov 15, 2024
```
Signed-off-by: ElizaWszola <eliza@neuralmagic.com>
```
  79ee45b4
- [bugfix] Fix static asymmetric quantization case (#10334) · bf2ddc66
  Luka Govedič authored Nov 14, 2024
```
Signed-off-by: Daniël de Kok <me@danieldk.eu>
Signed-off-by: luka <luka@neuralmagic.com>
Co-authored-by: Daniël de Kok <me@danieldk.eu>
```
  bf2ddc66
09 Nov, 2024 1 commit

[Kernel][Triton] Add Triton implementation for scaled_mm_triton to support fp8... · 127c0748

rasmith authored Nov 08, 2024


[Kernel][Triton] Add Triton implementation for scaled_mm_triton to support fp8 and int8 SmoothQuant, symmetric case (#9857)
Signed-off-by: Randall Smith <Randall.Smith@amd.com>

127c0748

08 Nov, 2024 1 commit
- [torch.compile] Fuse RMSNorm with quant (#9138) · 4f93dfe9
  Luka Govedič authored Nov 08, 2024
```
Signed-off-by: luka <luka@neuralmagic.com>
Co-authored-by: youkaichao <youkaichao@126.com>
```
  4f93dfe9
06 Nov, 2024 2 commits
- [V1] Make v1 more testable (#9888) · d58268c5
  Joe Runde authored Nov 06, 2024
```
Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
```
  d58268c5
- [CI/Build] drop support for Python 3.8 EOL (#8464) · 21063c11
  Aaron Pham authored Nov 06, 2024
```
Signed-off-by: Aaron Pham <contact@aarnphm.xyz>
```
  21063c11