Skip to content

GitLab

Commit 6d8c8719 authored by zhuwenwen's avatar zhuwenwen
Browse files

set VLLM_USE_FUSED_RMS_ROPE=1

parent ad60a973
Hide whitespace changes
Inline Side-by-side
Showing with 1 addition and 1 deletion
vllm/envs.py
View file @ 6d8c8719
...@@ -1687,7 +1687,7 @@ environment_variables: dict[str, Callable[[], Any]] = { ...@@ -1687,7 +1687,7 @@ environment_variables: dict[str, Callable[[], Any]] = {
("true", "1")), ("true", "1")),
# vLLM will use fused RMS + RoPE kernel # vLLM will use fused RMS + RoPE kernel
"VLLM_USE_FUSED_RMS_ROPE": "VLLM_USE_FUSED_RMS_ROPE":
lambda: (os.environ.get("VLLM_USE_FUSED_RMS_ROPE", "False").lower() in lambda: (os.environ.get("VLLM_USE_FUSED_RMS_ROPE", "True").lower() in
("true", "1")), ("true", "1")),
# vLLM will use Marlin W16A16 kernel for MoE experts # vLLM will use Marlin W16A16 kernel for MoE experts
"VLLM_USE_MARLIN_W16A16_MOE": "VLLM_USE_MARLIN_W16A16_MOE":
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment