Skip to content

GitLab

Unverified Commit a65a934e authored by Zhewen Li's avatar Zhewen Li Committed by GitHub
Browse files

[CI/Build] Temporary fix to LM Eval Small Models (#28324) 


Signed-off-by: default avatarzhewenli <zhewenli@meta.com>
parent 4a8d6bd1
Show whitespace changes
Inline Side-by-side
Showing with 8 additions and 3 deletions
tests/evals/gsm8k/configs/Qwen1.5-MoE-W4A16-CT.yaml
View file @ a65a934e
...@@ -3,3 +3,6 @@ accuracy_threshold: 0.45 ...@@ -3,3 +3,6 @@ accuracy_threshold: 0.45
num_questions: 1319 num_questions: 1319
num_fewshot: 5 num_fewshot: 5
max_model_len: 4096 max_model_len: 4096
# Duo stream incompatabilbe with this model: https://github.com/vllm-project/vllm/issues/28220
env:
VLLM_DISABLE_SHARED_EXPERTS_STREAM: "1"
tests/evals/gsm8k/test_gsm8k_correctness.py
View file @ a65a934e
...@@ -62,9 +62,11 @@ def test_gsm8k_correctness_param(config_filename, tp_size): ...@@ -62,9 +62,11 @@ def test_gsm8k_correctness_param(config_filename, tp_size):
str(tp_size), str(tp_size),
] ]
env_dict = eval_config.get("env", None)
# Launch server and run evaluation # Launch server and run evaluation
with RemoteOpenAIServer( with RemoteOpenAIServer(
eval_config["model_name"], server_args, max_wait_seconds=480 eval_config["model_name"], server_args, env_dict=env_dict, max_wait_seconds=480
) as remote_server: ) as remote_server:
server_url = remote_server.url_for("v1") server_url = remote_server.url_for("v1")
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment