[Bugfix] fix for deepseek w4a16 (#8906)

Co-authored-by: mgoin <michael@neuralmagic.com>

[Bugfix] fix for deepseek w4a16 (#8906)
Co-authored-by: mgoin <michael@neuralmagic.com>
c5d55356 · Lucas Wilkinson · GitHub · 172d1cd2 · c5d55356
Unverified Commit c5d55356 authored Sep 27, 2024 by Lucas Wilkinson Committed by GitHub Sep 27, 2024
Show whitespace changes
Inline Side-by-side

Showing with 5 additions and 4 deletions

vllm/model_executor/layers/quantization/kernels/marlin.py vllm/model_executor/layers/quantization/kernels/marlin.py +5 -4

No files found.
--- a/vllm/model_executor/layers/quantization/kernels/marlin.py
+++ b/vllm/model_executor/layers/quantization/kernels/marlin.py
@@ -38,9 +38,10 @@ class MarlinLinearKernel(MPLinearKernel):
                            "Marlin, supported group sizes are: "\
                            f"{MARLIN_SUPPORTED_GROUP_SIZES}"
-        return check_marlin_supports_shape(c.partition_weight_shape[0],
+        return check_marlin_supports_shape(
-                                           c.partition_weight_shape[1],
+            c.partition_weight_shape[1],  # out_features
-                                           c.full_weight_shape[1],
+            c.partition_weight_shape[0],  # in_features
+            c.full_weight_shape[0],  # in_features
            c.group_size)
    # note assumes that