Skip to content

GitLab

Unverified Commit 082ecd80 authored by Robert Shaw's avatar Robert Shaw Committed by GitHub
Browse files

[ Bugfix ] Fix AutoFP8 fp8 marlin (#6609)

parent f952bbc8
Show whitespace changes
Inline Side-by-side
Showing with 2 additions and 1 deletion
vllm/model_executor/layers/quantization/utils/marlin_utils_fp8.py
View file @ 082ecd80
...@@ -76,7 +76,8 @@ def prepare_fp8_layer_for_marlin(layer: torch.nn.Module) -> None: ...@@ -76,7 +76,8 @@ def prepare_fp8_layer_for_marlin(layer: torch.nn.Module) -> None:
# WEIGHT SCALES # WEIGHT SCALES
# Currently Marlin doesn't support per-tensor scales, so we # Currently Marlin doesn't support per-tensor scales, so we
# expand it to channelwise # expand it to channelwise
is_channelwise = layer.weight_scale.shape[0] == part_size_n is_channelwise = (len(layer.weight_scale.shape) > 0
and layer.weight_scale.shape[0] == part_size_n)
if is_channelwise: if is_channelwise:
scales = layer.weight_scale scales = layer.weight_scale
else: else:
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment