Skip to content

GitLab

Unverified Commit f33251ff authored by Silvia Colabrese's avatar Silvia Colabrese Committed by GitHub
Browse files

[Bugfix] Fix Mistral-small `--format` (#36782) 


Signed-off-by: default avatar12010486 <silvia.colabrese@intel.com>
parent e584dce5
Hide whitespace changes
Inline Side-by-side
Showing with 6 additions and 6 deletions
examples/offline_inference/mistral-small.py
View file @ f33251ff
...@@ -62,9 +62,9 @@ def run_simple_demo(args: argparse.Namespace): ...@@ -62,9 +62,9 @@ def run_simple_demo(args: argparse.Namespace):
llm = LLM( llm = LLM(
model=model_name, model=model_name,
tokenizer_mode="mistral" if args.format == "mistral" else "auto", tokenizer_mode="mistral" if args.format == "mistral" else "hf",
config_format="mistral" if args.format == "mistral" else "auto", config_format="mistral" if args.format == "mistral" else "hf",
load_format="mistral" if args.format == "mistral" else "auto", load_format="mistral" if args.format == "mistral" else "hf",
limit_mm_per_prompt={"image": 1}, limit_mm_per_prompt={"image": 1},
max_model_len=4096, max_model_len=4096,
max_num_seqs=2, max_num_seqs=2,
...@@ -102,9 +102,9 @@ def run_advanced_demo(args: argparse.Namespace): ...@@ -102,9 +102,9 @@ def run_advanced_demo(args: argparse.Namespace):
sampling_params = SamplingParams(max_tokens=8192, temperature=0.7) sampling_params = SamplingParams(max_tokens=8192, temperature=0.7)
llm = LLM( llm = LLM(
model=model_name, model=model_name,
tokenizer_mode="mistral" if args.format == "mistral" else "auto", tokenizer_mode="mistral" if args.format == "mistral" else "hf",
config_format="mistral" if args.format == "mistral" else "auto", config_format="mistral" if args.format == "mistral" else "hf",
load_format="mistral" if args.format == "mistral" else "auto", load_format="mistral" if args.format == "mistral" else "hf",
limit_mm_per_prompt={"image": max_img_per_msg}, limit_mm_per_prompt={"image": max_img_per_msg},
max_model_len=max_img_per_msg * max_tokens_per_img, max_model_len=max_img_per_msg * max_tokens_per_img,
tensor_parallel_size=2, tensor_parallel_size=2,
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment