Skip to content

GitLab

Unverified Commit 6f170f11 authored by Isotr0py's avatar Isotr0py Committed by GitHub
Browse files

[Bugfix] Fix bnb 8bit model weights loading (#19917) 


Signed-off-by: default avatarIsotr0py <2037008807@qq.com>
parent 8ca81bb0
Hide whitespace changes
Inline Side-by-side
Showing with 2 additions and 2 deletions
vllm/model_executor/model_loader/bitsandbytes_loader.py
View file @ 6f170f11
...@@ -577,10 +577,10 @@ def dequantize_dq(quant_states: dict) -> None: ...@@ -577,10 +577,10 @@ def dequantize_dq(quant_states: dict) -> None:
thereby avoiding this computational overhead during inference. This comes thereby avoiding this computational overhead during inference. This comes
at the cost of increased memory usage. at the cost of increased memory usage.
""" """
from bitsandbytes.functional import dequantize_blockwise from bitsandbytes.functional import QuantState, dequantize_blockwise
for _, quant_state in quant_states.items(): for _, quant_state in quant_states.items():
# Copied from: https://github.com/bitsandbytes-foundation/bitsandbytes/blob/0.45.3/bitsandbytes/functional.py#L1352-#L1356 # Copied from: https://github.com/bitsandbytes-foundation/bitsandbytes/blob/0.45.3/bitsandbytes/functional.py#L1352-#L1356
if quant_state.nested: if isinstance(quant_state, QuantState) and quant_state.nested:
absmax = dequantize_blockwise(quant_state.absmax, absmax = dequantize_blockwise(quant_state.absmax,
quant_state.state2) quant_state.state2)
absmax += quant_state.offset absmax += quant_state.offset
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment