Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in
Toggle navigation
Menu
Open sidebar
OpenDAS
vllm_cscc
Commits
34868b10
Unverified
Commit
34868b10
authored
Mar 19, 2025
by
Roger Wang
Committed by
GitHub
Mar 20, 2025
Browse files
[Doc] Update Mistral Small 3.1/Pixtral example (#15184)
Signed-off-by:
Roger Wang
<
ywang@roblox.com
>
parent
1f16b7fe
Changes
1
Show whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
8 additions
and
2 deletions
+8
-2
examples/offline_inference/mistral-small.py
examples/offline_inference/mistral-small.py
+8
-2
No files found.
examples/offline_inference/
pixtra
l.py
→
examples/offline_inference/
mistral-smal
l.py
View file @
34868b10
...
...
@@ -6,14 +6,16 @@ import argparse
from
vllm
import
LLM
from
vllm.sampling_params
import
SamplingParams
# This script is an offline demo for running Mistral-Small-3
# This script is an offline demo for running Mistral-Small-3
.1
#
# If you want to run a server/client setup, please follow this code:
#
# - Server:
#
# ```bash
# vllm serve mistralai/Mistral-Small-3.1-24B-Instruct-2503 --tokenizer-mode mistral --limit-mm-per-prompt 'image=4' --max-model-len 16384
# vllm serve mistralai/Mistral-Small-3.1-24B-Instruct-2503 \
# --tokenizer-mode mistral --config-format mistral --load-format mistral \
# --limit-mm-per-prompt 'image=4' --max-model-len 16384
# ```
#
# - Client:
...
...
@@ -51,6 +53,8 @@ def run_simple_demo(args: argparse.Namespace):
llm
=
LLM
(
model
=
model_name
,
tokenizer_mode
=
"mistral"
,
config_format
=
"mistral"
,
load_format
=
"mistral"
,
max_model_len
=
4096
,
max_num_seqs
=
2
,
disable_mm_preprocessor_cache
=
args
.
disable_mm_preprocessor_cache
,
...
...
@@ -91,6 +95,8 @@ def run_advanced_demo(args: argparse.Namespace):
llm
=
LLM
(
model
=
model_name
,
tokenizer_mode
=
"mistral"
,
config_format
=
"mistral"
,
load_format
=
"mistral"
,
limit_mm_per_prompt
=
{
"image"
:
max_img_per_msg
},
max_model_len
=
max_img_per_msg
*
max_tokens_per_img
,
disable_mm_preprocessor_cache
=
args
.
disable_mm_preprocessor_cache
,
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment