Skip to content

GitLab

"docs/source/en/training/lora.mdx" did not exist on "263b9680413900379512ca1c2daa15aa1451f57e"
Unverified Commit 56383649 authored by Rex's avatar Rex Committed by GitHub
Browse files

Refactor 2 awq gemm kernels into m16nXk32 (#2723) 


Co-authored-by: default avatarChunan Zeng <chunanzeng@Chunans-Air.attlocal.net>
parent 4ca2c358
Expand all Hide whitespace changes
Inline Side-by-side
Showing with 73 additions and 295 deletions
csrc/quantization/awq/gemm_kernels.cu
View file @ 56383649
This diff is collapsed.
vllm/model_executor/layers/quantization/awq.py
View file @ 56383649
...@@ -145,8 +145,8 @@ class AWQLinearMethod(LinearMethodBase): ...@@ -145,8 +145,8 @@ class AWQLinearMethod(LinearMethodBase):
x: torch.Tensor, x: torch.Tensor,
bias: Optional[torch.Tensor] = None) -> torch.Tensor: bias: Optional[torch.Tensor] = None) -> torch.Tensor:
qweight = weights["qweight"] qweight = weights["qweight"]
qzeros = weights["qzeros"]
scales = weights["scales"] scales = weights["scales"]
qzeros = weights["qzeros"]
pack_factor = self.quant_config.pack_factor pack_factor = self.quant_config.pack_factor
out_shape = (x.shape[:-1] + (qweight.shape[-1] * pack_factor, )) out_shape = (x.shape[:-1] + (qweight.shape[-1] * pack_factor, ))
reshaped_x = x.reshape(-1, x.shape[-1]) reshaped_x = x.reshape(-1, x.shape[-1])
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment