Skip to content

GitLab

Unverified Commit 7bab4bb0 authored by Jee Jee Li's avatar Jee Jee Li Committed by GitHub
Browse files

[Misc] Add Qwen2MoeForCausalLM moe tuning support (#14276) 


Signed-off-by: default avatarJee Jee Li <pandaleefree@gmail.com>
parent e17e4488
Show whitespace changes
Inline Side-by-side
Showing with 5 additions and 0 deletions
benchmarks/kernels/benchmark_moe.py
View file @ 7bab4bb0
...@@ -509,6 +509,11 @@ def main(args: argparse.Namespace): ...@@ -509,6 +509,11 @@ def main(args: argparse.Namespace):
intermediate_size = config.moe_intermediate_size intermediate_size = config.moe_intermediate_size
shard_intermediate_size = 2 * intermediate_size // args.tp_size shard_intermediate_size = 2 * intermediate_size // args.tp_size
block_quant_shape = config.quantization_config['weight_block_size'] block_quant_shape = config.quantization_config['weight_block_size']
elif config.architectures[0] == "Qwen2MoeForCausalLM":
E = config.num_experts
topk = config.num_experts_per_tok
intermediate_size = config.moe_intermediate_size
shard_intermediate_size = 2 * intermediate_size // args.tp_size
else: else:
# Default: Mixtral. # Default: Mixtral.
E = config.num_local_experts E = config.num_local_experts
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment