Skip to content

GitLab

Unverified Commit 70dc14fb authored by zixuanzhang226's avatar zixuanzhang226 Committed by GitHub
Browse files

[Model] support bitsandbytes quantization with minicpm3 model (#10682) 


Signed-off-by: default avatarUbuntu <zixuanzhang@bytedance.com>
parent cb4e1c3f
Hide whitespace changes
Inline Side-by-side
Showing with 6 additions and 0 deletions
vllm/model_executor/models/minicpm3.py
View file @ 70dc14fb
...@@ -241,6 +241,12 @@ class MiniCPM3ForCausalLM(MiniCPMForCausalLM): ...@@ -241,6 +241,12 @@ class MiniCPM3ForCausalLM(MiniCPMForCausalLM):
# `embedding_modules` and `embedding_padding_modules` # `embedding_modules` and `embedding_padding_modules`
# are inherited from MiniCPMForCausalLM # are inherited from MiniCPMForCausalLM
bitsandbytes_stacked_params_mapping = {
# shard_name, weight_name, index
"gate_proj": ("gate_up_proj", 0),
"up_proj": ("gate_up_proj", 1),
}
def _init_model(self, *, vllm_config: VllmConfig, prefix: str = ""): def _init_model(self, *, vllm_config: VllmConfig, prefix: str = ""):
self.model = MiniCPM3Model(vllm_config=vllm_config, self.model = MiniCPM3Model(vllm_config=vllm_config,
prefix=maybe_prefix(prefix, "model")) prefix=maybe_prefix(prefix, "model"))
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment