Commits · 2f808e69ab16a8775cd58849b15b465d4f11b92e · OpenDAS / vllm_cscc · GitLab

23 Jul, 2024 3 commits
- [Misc] Add ignored layers for `fp8` quantization (#6657) · 0eb0757b
  Michael Goin authored Jul 23, 2024
  
  0eb0757b
- [Misc] Support FP8 kv cache scales from compressed-tensors (#6528) · 9e0b558a
  Michael Goin authored Jul 23, 2024
  
  9e0b558a
- [Bugfix] Fix null `modules_to_not_convert` in FBGEMM Fp8 quantization (#6665) · c5e83309
  Cheng Li authored Jul 22, 2024
  
  c5e83309
21 Jul, 2024 1 commit
- [Kernel][Core] Add AWQ support to the Marlin kernel (#6612) · 396d92d5
  Alexander Matveev authored Jul 21, 2024
  
  396d92d5
20 Jul, 2024 4 commits
- [ Bugfix ] Fix AutoFP8 fp8 marlin (#6609) · 082ecd80
  Robert Shaw authored Jul 20, 2024
  
  082ecd80
- [Misc] Fix input_scale typing in w8a8_utils.py (#6579) · f952bbc8
  Michael Goin authored Jul 20, 2024
  
  f952bbc8
- [ Kernel ] Enable `fp8-marlin` for `fbgemm-fp8` models (#6606) · 9364f74e
  Robert Shaw authored Jul 20, 2024
  
  9364f74e
- [ Misc ] `fbgemm` checkpoints (#6559) · 683e3cb9
  Robert Shaw authored Jul 20, 2024
  
  683e3cb9
19 Jul, 2024 2 commits
- [ Kernel ] Enable Dynamic Per Token `fp8` (#6547) · 4cc24f01
  Robert Shaw authored Jul 19, 2024
  
  4cc24f01
- [ Misc ] non-uniform quantization via `compressed-tensors` for `Llama` (#6515) · dbe55885
  Robert Shaw authored Jul 18, 2024
  
  dbe55885
18 Jul, 2024 3 commits
- [Kernel] Implement fallback for FP8 channelwise using torch._scaled_mm (#6552) · 4ffffccb
  Tyler Michael Smith authored Jul 18, 2024
  
  4ffffccb
- [ Misc ] Improve Min Capability Checking in `compressed-tensors` (#6522) · 58ca6632
  Robert Shaw authored Jul 18, 2024
  
  58ca6632
- [ Kernel ] Fp8 Channelwise Weight Support (#6487) · 18fecc35
  Robert Shaw authored Jul 17, 2024
  
  18fecc35
16 Jul, 2024 1 commit
- [Kernel][Attention] Separate `Attention.kv_scale` into `k_scale` and `v_scale` (#6081) · 978aed53
  Michael Goin authored Jul 16, 2024
  
  978aed53
15 Jul, 2024 1 commit
- [Kernel] Use CUTLASS kernels for the FP8 layers with Bias (#6270) · c8fd97f2
  Tyler Michael Smith authored Jul 15, 2024
  
  c8fd97f2
14 Jul, 2024 1 commit
- [ Misc ] Apply MoE Refactor to Deepseekv2 To Support Fp8 (#6417) · fb6af8bc
  Robert Shaw authored Jul 13, 2024
  
  fb6af8bc
13 Jul, 2024 1 commit
- [ Misc ] More Cleanup of Marlin (#6359) · babf52da
  Robert Shaw authored Jul 13, 2024
```
Co-authored-by: Robert Shaw <rshaw@neuralmagic.com>
```
  babf52da
12 Jul, 2024 1 commit
- [ Misc ] Support Models With Bias in `compressed-tensors` integration (#6356) · aea19f09
  Robert Shaw authored Jul 12, 2024
  
  aea19f09
11 Jul, 2024 1 commit
- [ Misc ] Refactor Marlin Python Utilities (#6082) · b675069d
  Robert Shaw authored Jul 11, 2024
```
Co-authored-by: Robert Shaw <rshaw@neuralmagic.com>
```
  b675069d
07 Jul, 2024 1 commit
- [ Misc ] Support Fp8 via `llm-compressor` (#6110) · abfe705a
  Robert Shaw authored Jul 07, 2024
```
Co-authored-by: Robert Shaw <rshaw@neuralmagic>
```
  abfe705a
03 Jul, 2024 3 commits
- [ Misc ] Clean Up `CompressedTensorsW8A8` (#6113) · 62963d12
  Robert Shaw authored Jul 03, 2024
  
  62963d12
- [Kernel] Expand FP8 support to Ampere GPUs using FP8 Marlin (#5975) · 47f0954a
  Michael Goin authored Jul 03, 2024
  
  47f0954a
- [hardware][misc] introduce platform abstraction (#6080) · 482045ee
  youkaichao authored Jul 02, 2024
  
  482045ee
02 Jul, 2024 2 commits
- [CORE] Quantized lm-head Framework (#4442) · ee93f4f9
  Qubitium-ModelCloud authored Jul 03, 2024
```
Co-authored-by: Robert Shaw <rshaw@neuralmagic.com>
Co-authored-by: ZX <zx@lbx.dev>
```
  ee93f4f9
- [ Misc ] Refactor MoE to isolate Fp8 From Mixtral (#5970) · 7c008c51
  Robert Shaw authored Jul 02, 2024
```
Co-authored-by: Robert Shaw <rshaw@neuralmagic>
Co-authored-by: Michael Goin <michael@neuralmagic.com>
```
  7c008c51
01 Jul, 2024 1 commit
- [misc][cuda] use nvml to avoid accidentally cuda initialization (#6007) · 614aa512
  youkaichao authored Jun 30, 2024
  
  614aa512
30 Jun, 2024 2 commits
- [ Misc ] Refactor w8a8 to use `process_weights_after_load` (Simplify Weight Loading) (#5940) · af9ad46f
  Robert Shaw authored Jun 30, 2024
```
Co-authored-by: Robert Shaw <rshaw@neuralmagic>
```
  af9ad46f
- [Misc] Fix `get_min_capability` (#5971) · 7836fdcc
  Dipika Sikka authored Jun 30, 2024
  
  7836fdcc
29 Jun, 2024 1 commit

[ CI/Build ] Added E2E Test For Compressed Tensors (#5839) · 8dbfcd35

Robert Shaw authored Jun 29, 2024


Co-authored-by: Michael Goin <michael@neuralmagic.com>
Co-authored-by: Robert Shaw <rshaw@neuralmagic>

8dbfcd35

28 Jun, 2024 1 commit
- [ Bugfix ] Enabling Loading Models With Fused QKV/MLP on Disk with FP8 (#5921) · 2cd402e1
  Robert Shaw authored Jun 28, 2024
```
Co-authored-by: Robert Shaw <rshaw@neuralmagic>
```
  2cd402e1
25 Jun, 2024 1 commit
- [Misc] Update `w4a16` `compressed-tensors` support to include `w8a16` (#5794) · dd248f76
  Dipika Sikka authored Jun 25, 2024
  
  dd248f76
20 Jun, 2024 1 commit
- [Bugfix] Fix the CUDA version check for FP8 support in the CUTLASS kernels (#5715) · 3f3b6b21
  Tyler Michael Smith authored Jun 20, 2024
  
  3f3b6b21
19 Jun, 2024 1 commit
- [Misc] Add per channel support for static activation quantization; update w8a8... · 4a30d7e3
  Dipika Sikka authored Jun 19, 2024
```
[Misc] Add per channel support for static activation quantization; update w8a8 schemes to share base classes (#5650)
```
  4a30d7e3
18 Jun, 2024 1 commit
- [Misc] Add channel-wise quantization support for w8a8 dynamic per token... · 95db455e
  Dipika Sikka authored Jun 18, 2024
```
[Misc] Add channel-wise quantization support for w8a8 dynamic per token activation quantization (#5542)
```
  95db455e
17 Jun, 2024 1 commit
- [Kernel] `compressed-tensors` marlin 24 support (#5435) · 890d8d96
  Dipika Sikka authored Jun 17, 2024
  
  890d8d96
15 Jun, 2024 1 commit
- [mypy] Enable type checking for test directory (#5017) · 0e9164b4
  Cyrus Leung authored Jun 15, 2024
  
  0e9164b4
14 Jun, 2024 2 commits
- [ Misc ] Rs/compressed tensors cleanup (#5432) · 15985680
  Robert Shaw authored Jun 14, 2024
```
Co-authored-by: mgoin <michael@neuralmagic.com>
Co-authored-by: Dipika Sikka <dipikasikka1@gmail.com>
```
  15985680
- [Kernel] Fix CUTLASS 3.x custom broadcast load epilogue (#5516) · 703475f6
  Tyler Michael Smith authored Jun 14, 2024
  
  703475f6
13 Jun, 2024 2 commits

[Kernel] Disable CUTLASS kernels for fp8 (#5505) · e38042d4
Tyler Michael Smith authored Jun 13, 2024

e38042d4

[Kernel] Factor out epilogues from cutlass kernels (#5391) · 85657b56

Tyler Michael Smith authored Jun 13, 2024


Co-authored-by: Michael Goin <michael@neuralmagic.com>
Co-authored-by: youkaichao <youkaichao@gmail.com>
Co-authored-by: zifeitong <zifei.tong@parasail.io>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-neuralmagic@users.noreply.github.com>

85657b56