Unverified Commit 83dd28aa authored by Nick Hill's avatar Nick Hill Committed by GitHub
Browse files

[CI] Adjust threshold for flaky ngram spec decoding test (#24528)


Signed-off-by: default avatarNick Hill <nhill@redhat.com>
parent f88e8401
......@@ -117,9 +117,9 @@ def test_ngram_correctness(
print(f"ref_output: {ref_output.outputs[0].text}")
print(f"spec_output: {spec_output.outputs[0].text}")
# Heuristic: expect at least 70% of the prompts to match exactly
# Heuristic: expect at least 68% of the prompts to match exactly
# Upon failure, inspect the outputs to check for inaccuracy.
assert matches > int(0.7 * len(ref_outputs))
assert matches >= int(0.68 * len(ref_outputs))
del spec_llm
torch.cuda.empty_cache()
cleanup_dist_env_and_memory()
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment