Unverified Commit 6f170f11 authored by Isotr0py's avatar Isotr0py Committed by GitHub
Browse files

[Bugfix] Fix bnb 8bit model weights loading (#19917)


Signed-off-by: default avatarIsotr0py <2037008807@qq.com>
parent 8ca81bb0
......@@ -577,10 +577,10 @@ def dequantize_dq(quant_states: dict) -> None:
thereby avoiding this computational overhead during inference. This comes
at the cost of increased memory usage.
"""
from bitsandbytes.functional import dequantize_blockwise
from bitsandbytes.functional import QuantState, dequantize_blockwise
for _, quant_state in quant_states.items():
# Copied from: https://github.com/bitsandbytes-foundation/bitsandbytes/blob/0.45.3/bitsandbytes/functional.py#L1352-#L1356
if quant_state.nested:
if isinstance(quant_state, QuantState) and quant_state.nested:
absmax = dequantize_blockwise(quant_state.absmax,
quant_state.state2)
absmax += quant_state.offset
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment