supported_models.md 43.2 KB
Newer Older
1
(supported-models)=
Woosuk Kwon's avatar
Woosuk Kwon committed
2

3
# Supported Models
Woosuk Kwon's avatar
Woosuk Kwon committed
4

5
vLLM supports [generative](generative-models) and [pooling](pooling-models) models across various tasks.
6
If a model supports more than one task, you can set the task via the `--task` argument.
7
8

For each task, we list the model architectures that have been implemented in vLLM.
Woosuk Kwon's avatar
Woosuk Kwon committed
9
10
Alongside each architecture, we include some popular models that use it.

11
## Model Implementation
12

13
### vLLM
14

15
If vLLM natively supports a model, its implementation can be found in <gh-file:vllm/model_executor/models>.
16

17
These models are what we list in <project:#supported-text-models> and <project:#supported-mm-models>.
18

19
(transformers-backend)=
20

21
### Transformers
22

23
vLLM also supports model implementations that are available in Transformers. This does not currently work for all models, but most decoder language models are supported, and vision language model support is planned!
24

25
To check if the modeling backend is Transformers, you can simply do this:
26

27
```python
28
29
from vllm import LLM
llm = LLM(model=..., task="generate")  # Name or path of your model
30
llm.apply_model(lambda model: print(type(model)))
31
32
```

33
If it is `TransformersForCausalLM` then it means it's based on Transformers!
34

35
:::{tip}
36
You can force the use of `TransformersForCausalLM` by setting `model_impl="transformers"` for <project:#offline-inference> or `--model-impl transformers` for the <project:#openai-compatible-server>.
37
38
:::

39
40
41
:::{note}
vLLM may not fully optimise the Transformers implementation so you may see degraded performance if comparing a native model to a Transformers model in vLLM.
:::
42

43
#### Custom models
44

45
If a model is neither supported natively by vLLM or Transformers, it can still be used in vLLM!
46

47
For a model to be compatible with the Transformers backend for vLLM it must:
48

49
50
51
52
53
- be a Transformers compatible custom model (see [Transformers - Customizing models](https://huggingface.co/docs/transformers/en/custom_models)):
  * The model directory must have the correct structure (e.g. `config.json` is present).
  * `config.json` must contain `auto_map.AutoModel`.
- be a Transformers backend for vLLM compatible model (see <project:#writing-custom-models>):
  * Customisation should be done in the base model (e.g. in `MyModel`, not `MyModelForCausalLM`).
54

55
If the compatible model is:
56

Elad Segal's avatar
Elad Segal committed
57
- on the Hugging Face Model Hub, simply set `trust_remote_code=True` for <project:#offline-inference> or `--trust-remote-code` for the <project:#openai-compatible-server>.
58
- in a local directory, simply pass directory path to `model=<MODEL_DIR>` for <project:#offline-inference> or `vllm serve <MODEL_DIR>` for the <project:#openai-compatible-server>.
59

60
This means that, with the Transformers backend for vLLM, new models can be used before they are officially supported in Transformers or vLLM!
61

62
63
64
65
66
(writing-custom-models)=

#### Writing custom models

This section details the necessary modifications to make to a Transformers compatible custom model that make it compatible with the Transformers backend for vLLM. (We assume that a Transformers compatible custom model has already been created, see [Transformers - Customizing models](https://huggingface.co/docs/transformers/en/custom_models)).
67

68
To make your model compatible with the Transformers backend, it needs:
69

70
71
72
73
1. `kwargs` passed down through all modules from `MyModel` to `MyAttention`.
2. `MyAttention` must use `ALL_ATTENTION_FUNCTIONS` to call attention.
3. `MyModel` must contain `_supports_attention_backend = True`.

74
75
```{code-block} python
:caption: modeling_my_model.py
76
77
78
79
80
81

from transformers import PreTrainedModel
from torch import nn

class MyAttention(nn.Module):

82
  def forward(self, hidden_states, **kwargs):
83
    ...
84
    attention_interface = ALL_ATTENTION_FUNCTIONS[self.config._attn_implementation]
85
86
87
88
89
90
91
92
93
94
95
96
97
    attn_output, attn_weights = attention_interface(
      self,
      query_states,
      key_states,
      value_states,
      **kwargs,
    )
    ...

class MyModel(PreTrainedModel):
  _supports_attention_backend = True
```

98
Here is what happens in the background when this model is loaded:
99

100
101
102
1. The config is loaded.
2. `MyModel` Python class is loaded from the `auto_map` in config, and we check that the model `is_backend_compatible()`.
3. `MyModel` is loaded into `TransformersForCausalLM` (see <gh-file:vllm/model_executor/models/transformers.py>) which sets `self.config._attn_implementation = "vllm"` so that vLLM's attention layer is used.
103

104
105
106
That's it!

For your model to be compatible with vLLM's tensor parallel and/or pipeline parallel features, you must add `base_model_tp_plan` and/or `base_model_pp_plan` to your model's config class:
107
108
109
110
111
112
113
114

```{code-block} python
:caption: configuration_my_model.py

from transformers import PretrainedConfig

class MyConfig(PretrainedConfig):
  base_model_tp_plan = {
115
116
117
118
119
120
121
122
123
124
125
    "layers.*.self_attn.k_proj": "colwise",
    "layers.*.self_attn.v_proj": "colwise",
    "layers.*.self_attn.o_proj": "rowwise",
    "layers.*.mlp.gate_proj": "colwise",
    "layers.*.mlp.up_proj": "colwise",
    "layers.*.mlp.down_proj": "rowwise",
  }
  base_model_pp_plan = {
    "embed_tokens": (["input_ids"], ["inputs_embeds"]),
    "layers": (["hidden_states", "attention_mask"], ["hidden_states"]),
    "norm": (["hidden_states"], ["hidden_states"]),
126
127
128
  }
```

129
130
131
132
133
134
135
136
137
138
139
- `base_model_tp_plan` is a `dict` that maps fully qualified layer name patterns to tensor parallel styles (currently only `"colwise"` and `"rowwise"` are supported).
- `base_model_pp_plan` is a `dict` that maps direct child layer names to `tuple`s of `list`s of `str`s:
  * You only need to do this for layers which are not present on all pipeline stages
  * vLLM assumes that there will be only one `nn.ModuleList`, which is distributed across the pipeline stages
  * The `list` in the first element of the `tuple` contains the names of the input arguments
  * The `list` in the last element of the `tuple` contains the names of the variables the layer outputs to in your modeling code

## Loading a Model

### Hugging Face Hub

Reid's avatar
Reid committed
140
By default, vLLM loads models from [Hugging Face (HF) Hub](https://huggingface.co/models). To change the download path for models, you can set the `HF_HOME` environment variable; for more details, refer to [their official documentation](https://huggingface.co/docs/huggingface_hub/package_reference/environment_variables#hfhome).
141
142
143
144
145

To determine whether a given model is natively supported, you can check the `config.json` file inside the HF repository.
If the `"architectures"` field contains a model architecture listed below, then it should be natively supported.

Models do not _need_ to be natively supported to be used in vLLM.
146
The [Transformers backend](#transformers-backend) enables you to run models directly using their Transformers implementation (or even remote code on the Hugging Face Model Hub!).
147

148
:::{tip}
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
The easiest way to check if your model is really supported at runtime is to run the program below:

```python
from vllm import LLM

# For generative models (task=generate) only
llm = LLM(model=..., task="generate")  # Name or path of your model
output = llm.generate("Hello, my name is")
print(output)

# For pooling models (task={embed,classify,reward,score}) only
llm = LLM(model=..., task="embed")  # Name or path of your model
output = llm.encode("Hello, my name is")
print(output)
```

If vLLM successfully returns text (for generative models) or hidden states (for pooling models), it indicates that your model is supported.
166
:::
167

168
169
Otherwise, please refer to [Adding a New Model](#new-model) for instructions on how to implement your model in vLLM.
Alternatively, you can [open an issue on GitHub](https://github.com/vllm-project/vllm/issues/new/choose) to request vLLM support.
170

171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
#### Download a model

If you prefer, you can use the Hugging Face CLI to [download a model](https://huggingface.co/docs/huggingface_hub/guides/cli#huggingface-cli-download) or specific files from a model repository:

```console
# Download a model
huggingface-cli download HuggingFaceH4/zephyr-7b-beta

# Specify a custom cache directory
huggingface-cli download HuggingFaceH4/zephyr-7b-beta --cache-dir ./path/to/cache

# Download a specific file from a model repo
huggingface-cli download HuggingFaceH4/zephyr-7b-beta eval_results.json
```

#### List the downloaded models

Use the Hugging Face CLI to [manage models](https://huggingface.co/docs/huggingface_hub/guides/manage-cache#scan-your-cache) stored in local cache:

```console
# List cached models
huggingface-cli scan-cache

# Show detailed (verbose) output
huggingface-cli scan-cache -v

# Specify a custom cache directory
huggingface-cli scan-cache --dir ~/.cache/huggingface/hub
```

#### Delete a cached model

Use the Hugging Face CLI to interactively [delete downloaded model](https://huggingface.co/docs/huggingface_hub/guides/manage-cache#clean-your-cache) from the cache:

```console
# The `delete-cache` command requires extra dependencies to work with the TUI.
# Please run `pip install huggingface_hub[cli]` to install them.

# Launch the interactive TUI to select models to delete
$ huggingface-cli delete-cache
? Select revisions to delete: 1 revisions selected counting for 438.9M.
  ○ None of the following (if selected, nothing will be deleted).
Model BAAI/bge-base-en-v1.5 (438.9M, used 1 week ago)
❯ ◉ a5beb1e3: main # modified 1 week ago

Model BAAI/bge-large-en-v1.5 (1.3G, used 1 week ago)
  ○ d4aa6901: main # modified 1 week ago

Model BAAI/bge-reranker-base (1.1G, used 4 weeks ago)
  ○ 2cfc18c9: main # modified 4 weeks ago

Press <space> to select, <enter> to validate and <ctrl+c> to quit without modification.

# Need to confirm after selected
? Select revisions to delete: 1 revision(s) selected.
? 1 revisions selected counting for 438.9M. Confirm deletion ? Yes
Start deletion.
Done. Deleted 1 repo(s) and 0 revision(s) for a total of 438.9M.
```

Reid's avatar
Reid committed
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
#### Using a proxy

Here are some tips for loading/downloading models from Hugging Face using a proxy:

- Set the proxy globally for your session (or set it in the profile file):

```shell
export http_proxy=http://your.proxy.server:port
export https_proxy=http://your.proxy.server:port
```

- Set the proxy for just the current command:

```shell
https_proxy=http://your.proxy.server:port huggingface-cli download <model_name>

# or use vllm cmd directly
https_proxy=http://your.proxy.server:port  vllm serve <model_name> --disable-log-requests
```

- Set the proxy in Python interpreter:

```python
import os

os.environ['http_proxy'] = 'http://your.proxy.server:port'
os.environ['https_proxy'] = 'http://your.proxy.server:port'
```

260
### ModelScope
261

262
To use models from [ModelScope](https://www.modelscope.cn) instead of Hugging Face Hub, set an environment variable:
263

264
```shell
265
export VLLM_USE_MODELSCOPE=True
266
```
267

268
And use with `trust_remote_code=True`.
269

270
271
```python
from vllm import LLM
272

273
llm = LLM(model=..., revision=..., task=..., trust_remote_code=True)
274

275
276
277
# For generative models (task=generate) only
output = llm.generate("Hello, my name is")
print(output)
278

279
# For pooling models (task={embed,classify,reward,score}) only
280
281
282
output = llm.encode("Hello, my name is")
print(output)
```
283

284
285
286
287
288
289
290
291
292
293
(feature-status-legend)=

## Feature Status Legend

- ✅︎ indicates that the feature is supported for the model.

- 🚧 indicates that the feature is planned but not yet supported for the model.

- ⚠️ indicates that the feature is available but may have known issues or limitations.

294
295
(supported-text-models)=

296
## List of Text-only Language Models
297

298
### Generative Models
299

300
See [this page](#generative-models) for more information on how to use generative models.
301

302
303
304
#### Text Generation

Specified using `--task generate`.
305

306
:::{list-table}
307
308
309
:widths: 25 25 50 5 5
:header-rows: 1

310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
- * Architecture
  * Models
  * Example HF Models
  * [LoRA](#lora-adapter)
  * [PP](#distributed-serving)
- * `AquilaForCausalLM`
  * Aquila, Aquila2
  * `BAAI/Aquila-7B`, `BAAI/AquilaChat-7B`, etc.
  * ✅︎
  * ✅︎
- * `ArcticForCausalLM`
  * Arctic
  * `Snowflake/snowflake-arctic-base`, `Snowflake/snowflake-arctic-instruct`, etc.
  *
  * ✅︎
- * `BaiChuanForCausalLM`
  * Baichuan2, Baichuan
  * `baichuan-inc/Baichuan2-13B-Chat`, `baichuan-inc/Baichuan-7B`, etc.
  * ✅︎
  * ✅︎
330
331
332
333
334
- * `BambaForCausalLM`
  * Bamba
  * `ibm-ai-platform/Bamba-9B-fp8`, `ibm-ai-platform/Bamba-9B`
  *
  *
335
336
337
338
339
340
341
342
343
344
- * `BloomForCausalLM`
  * BLOOM, BLOOMZ, BLOOMChat
  * `bigscience/bloom`, `bigscience/bloomz`, etc.
  *
  * ✅︎
- * `BartForConditionalGeneration`
  * BART
  * `facebook/bart-base`, `facebook/bart-large-cnn`, etc.
  *
  *
345
- * `ChatGLMModel`, `ChatGLMForConditionalGeneration`
346
  * ChatGLM
347
  * `THUDM/chatglm2-6b`, `THUDM/chatglm3-6b`, `ShieldLM-6B-chatglm3`, etc.
348
349
350
351
352
353
354
355
356
357
358
359
360
361
  * ✅︎
  * ✅︎
- * `CohereForCausalLM`, `Cohere2ForCausalLM`
  * Command-R
  * `CohereForAI/c4ai-command-r-v01`, `CohereForAI/c4ai-command-r7b-12-2024`, etc.
  * ✅︎
  * ✅︎
- * `DbrxForCausalLM`
  * DBRX
  * `databricks/dbrx-base`, `databricks/dbrx-instruct`, etc.
  *
  * ✅︎
- * `DeciLMForCausalLM`
  * DeciLM
362
  * `nvidia/Llama-3_3-Nemotron-Super-49B-v1`, etc.
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
  *
  * ✅︎
- * `DeepseekForCausalLM`
  * DeepSeek
  * `deepseek-ai/deepseek-llm-67b-base`, `deepseek-ai/deepseek-llm-7b-chat` etc.
  *
  * ✅︎
- * `DeepseekV2ForCausalLM`
  * DeepSeek-V2
  * `deepseek-ai/DeepSeek-V2`, `deepseek-ai/DeepSeek-V2-Chat` etc.
  *
  * ✅︎
- * `DeepseekV3ForCausalLM`
  * DeepSeek-V3
  * `deepseek-ai/DeepSeek-V3-Base`, `deepseek-ai/DeepSeek-V3` etc.
  *
  * ✅︎
- * `ExaoneForCausalLM`
  * EXAONE-3
  * `LGAI-EXAONE/EXAONE-3.0-7.8B-Instruct`, etc.
  * ✅︎
  * ✅︎
- * `FalconForCausalLM`
  * Falcon
  * `tiiuae/falcon-7b`, `tiiuae/falcon-40b`, `tiiuae/falcon-rw-7b`, etc.
  *
  * ✅︎
- * `FalconMambaForCausalLM`
  * FalconMamba
  * `tiiuae/falcon-mamba-7b`, `tiiuae/falcon-mamba-7b-instruct`, etc.
  * ✅︎
  * ✅︎
Dhia Eddine Rhaiem's avatar
Dhia Eddine Rhaiem committed
395
396
397
398
399
- * `FalconH1ForCausalLM`
  * Falcon-H1
  * `tiiuae/Falcon-H1-34B-Base`, `tiiuae/Falcon-H1-34B-Instruct`, etc.
  * ✅︎
  * ✅︎
400
401
- * `GemmaForCausalLM`
  * Gemma
402
  * `google/gemma-2b`, `google/gemma-1.1-2b-it`, etc.
403
404
405
  * ✅︎
  * ✅︎
- * `Gemma2ForCausalLM`
406
  * Gemma 2
407
408
409
  * `google/gemma-2-9b`, `google/gemma-2-27b`, etc.
  * ✅︎
  * ✅︎
410
411
412
413
414
- * `Gemma3ForCausalLM`
  * Gemma 3
  * `google/gemma-3-1b-it`, etc.
  * ✅︎
  * ✅︎
415
416
417
418
419
- * `GlmForCausalLM`
  * GLM-4
  * `THUDM/glm-4-9b-chat-hf`, etc.
  * ✅︎
  * ✅︎
Yuxuan Zhang's avatar
Yuxuan Zhang committed
420
421
- * `Glm4ForCausalLM`
  * GLM-4-0414
intervitens's avatar
intervitens committed
422
  * `THUDM/GLM-4-32B-0414`, etc.
Yuxuan Zhang's avatar
Yuxuan Zhang committed
423
424
  * ✅︎
  * ✅︎
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
- * `GPT2LMHeadModel`
  * GPT-2
  * `gpt2`, `gpt2-xl`, etc.
  *
  * ✅︎
- * `GPTBigCodeForCausalLM`
  * StarCoder, SantaCoder, WizardCoder
  * `bigcode/starcoder`, `bigcode/gpt_bigcode-santacoder`, `WizardLM/WizardCoder-15B-V1.0`, etc.
  * ✅︎
  * ✅︎
- * `GPTJForCausalLM`
  * GPT-J
  * `EleutherAI/gpt-j-6b`, `nomic-ai/gpt4all-j`, etc.
  *
  * ✅︎
- * `GPTNeoXForCausalLM`
  * GPT-NeoX, Pythia, OpenAssistant, Dolly V2, StableLM
  * `EleutherAI/gpt-neox-20b`, `EleutherAI/pythia-12b`, `OpenAssistant/oasst-sft-4-pythia-12b-epoch-3.5`, `databricks/dolly-v2-12b`, `stabilityai/stablelm-tuned-alpha-7b`, etc.
  *
  * ✅︎
- * `GraniteForCausalLM`
  * Granite 3.0, Granite 3.1, PowerLM
  * `ibm-granite/granite-3.0-2b-base`, `ibm-granite/granite-3.1-8b-instruct`, `ibm/PowerLM-3b`, etc.
  * ✅︎
  * ✅︎
- * `GraniteMoeForCausalLM`
  * Granite 3.0 MoE, PowerMoE
  * `ibm-granite/granite-3.0-1b-a400m-base`, `ibm-granite/granite-3.0-3b-a800m-instruct`, `ibm/PowerMoE-3b`, etc.
  * ✅︎
  * ✅︎
455
456
457
458
459
- * `GraniteMoeHybridForCausalLM`
  * Granite 4.0 MoE Hybrid
  * `ibm-granite/granite-4.0-tiny-preview`, etc.
  * ✅︎
  * ✅︎
460
461
462
463
464
- * `GraniteMoeSharedForCausalLM`
  * Granite MoE Shared
  * `ibm-research/moe-7b-1b-active-shared-experts` (test model)
  * ✅︎
  * ✅︎
465
466
467
468
469
- * `GritLM`
  * GritLM
  * `parasail-ai/GritLM-7B-vllm`.
  * ✅︎
  * ✅︎
Michael Goin's avatar
Michael Goin committed
470
471
472
473
474
- * `Grok1ModelForCausalLM`
  * Grok1
  * `hpcai-tech/grok-1`.
  * ✅︎
  * ✅︎
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
- * `InternLMForCausalLM`
  * InternLM
  * `internlm/internlm-7b`, `internlm/internlm-chat-7b`, etc.
  * ✅︎
  * ✅︎
- * `InternLM2ForCausalLM`
  * InternLM2
  * `internlm/internlm2-7b`, `internlm/internlm2-chat-7b`, etc.
  * ✅︎
  * ✅︎
- * `InternLM3ForCausalLM`
  * InternLM3
  * `internlm/internlm3-8b-instruct`, etc.
  * ✅︎
  * ✅︎
- * `JAISLMHeadModel`
  * Jais
  * `inceptionai/jais-13b`, `inceptionai/jais-13b-chat`, `inceptionai/jais-30b-v3`, `inceptionai/jais-30b-chat-v3`, etc.
  *
  * ✅︎
- * `JambaForCausalLM`
  * Jamba
  * `ai21labs/AI21-Jamba-1.5-Large`, `ai21labs/AI21-Jamba-1.5-Mini`, `ai21labs/Jamba-v0.1`, etc.
  * ✅︎
  * ✅︎
- * `LlamaForCausalLM`
  * Llama 3.1, Llama 3, Llama 2, LLaMA, Yi
  * `meta-llama/Meta-Llama-3.1-405B-Instruct`, `meta-llama/Meta-Llama-3.1-70B`, `meta-llama/Meta-Llama-3-70B-Instruct`, `meta-llama/Llama-2-70b-hf`, `01-ai/Yi-34B`, etc.
  * ✅︎
  * ✅︎
- * `MambaForCausalLM`
  * Mamba
  * `state-spaces/mamba-130m-hf`, `state-spaces/mamba-790m-hf`, `state-spaces/mamba-2.8b-hf`, etc.
  *
  * ✅︎
- * `MiniCPMForCausalLM`
  * MiniCPM
  * `openbmb/MiniCPM-2B-sft-bf16`, `openbmb/MiniCPM-2B-dpo-bf16`, `openbmb/MiniCPM-S-1B-sft`, etc.
  * ✅︎
  * ✅︎
- * `MiniCPM3ForCausalLM`
  * MiniCPM3
  * `openbmb/MiniCPM3-4B`, etc.
  * ✅︎
  * ✅︎
- * `MistralForCausalLM`
  * Mistral, Mistral-Instruct
  * `mistralai/Mistral-7B-v0.1`, `mistralai/Mistral-7B-Instruct-v0.1`, etc.
  * ✅︎
  * ✅︎
- * `MixtralForCausalLM`
  * Mixtral-8x7B, Mixtral-8x7B-Instruct
  * `mistralai/Mixtral-8x7B-v0.1`, `mistralai/Mixtral-8x7B-Instruct-v0.1`, `mistral-community/Mixtral-8x22B-v0.1`, etc.
  * ✅︎
  * ✅︎
- * `MPTForCausalLM`
  * MPT, MPT-Instruct, MPT-Chat, MPT-StoryWriter
  * `mosaicml/mpt-7b`, `mosaicml/mpt-7b-storywriter`, `mosaicml/mpt-30b`, etc.
  *
  * ✅︎
- * `NemotronForCausalLM`
  * Nemotron-3, Nemotron-4, Minitron
  * `nvidia/Minitron-8B-Base`, `mgoin/Nemotron-4-340B-Base-hf-FP8`, etc.
  * ✅︎
  * ✅︎
- * `OLMoForCausalLM`
  * OLMo
  * `allenai/OLMo-1B-hf`, `allenai/OLMo-7B-hf`, etc.
  *
  * ✅︎
- * `OLMo2ForCausalLM`
  * OLMo2
547
  * `allenai/OLMo-2-0425-1B`, etc.
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
  *
  * ✅︎
- * `OLMoEForCausalLM`
  * OLMoE
  * `allenai/OLMoE-1B-7B-0924`, `allenai/OLMoE-1B-7B-0924-Instruct`, etc.
  * ✅︎
  * ✅︎
- * `OPTForCausalLM`
  * OPT, OPT-IML
  * `facebook/opt-66b`, `facebook/opt-iml-max-30b`, etc.
  *
  * ✅︎
- * `OrionForCausalLM`
  * Orion
  * `OrionStarAI/Orion-14B-Base`, `OrionStarAI/Orion-14B-Chat`, etc.
  *
  * ✅︎
- * `PhiForCausalLM`
  * Phi
  * `microsoft/phi-1_5`, `microsoft/phi-2`, etc.
  * ✅︎
  * ✅︎
- * `Phi3ForCausalLM`
  * Phi-4, Phi-3
572
  * `microsoft/Phi-4-mini-instruct`, `microsoft/Phi-4`, `microsoft/Phi-3-mini-4k-instruct`, `microsoft/Phi-3-mini-128k-instruct`, `microsoft/Phi-3-medium-128k-instruct`, etc.
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
  * ✅︎
  * ✅︎
- * `Phi3SmallForCausalLM`
  * Phi-3-Small
  * `microsoft/Phi-3-small-8k-instruct`, `microsoft/Phi-3-small-128k-instruct`, etc.
  *
  * ✅︎
- * `PhiMoEForCausalLM`
  * Phi-3.5-MoE
  * `microsoft/Phi-3.5-MoE-instruct`, etc.
  * ✅︎
  * ✅︎
- * `PersimmonForCausalLM`
  * Persimmon
  * `adept/persimmon-8b-base`, `adept/persimmon-8b-chat`, etc.
  *
  * ✅︎
Shinichi Hemmi's avatar
Shinichi Hemmi committed
590
591
592
593
594
- * `Plamo2ForCausalLM`
  * PLaMo2
  * `pfnet/plamo-2-1b`, `pfnet/plamo-2-8b`, etc.
  *
  *
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
- * `QWenLMHeadModel`
  * Qwen
  * `Qwen/Qwen-7B`, `Qwen/Qwen-7B-Chat`, etc.
  * ✅︎
  * ✅︎
- * `Qwen2ForCausalLM`
  * QwQ, Qwen2
  * `Qwen/QwQ-32B-Preview`, `Qwen/Qwen2-7B-Instruct`, `Qwen/Qwen2-7B`, etc.
  * ✅︎
  * ✅︎
- * `Qwen2MoeForCausalLM`
  * Qwen2MoE
  * `Qwen/Qwen1.5-MoE-A2.7B`, `Qwen/Qwen1.5-MoE-A2.7B-Chat`, etc.
  *
  * ✅︎
610
611
612
613
614
615
616
- * `Qwen3ForCausalLM`
  * Qwen3
  * `Qwen/Qwen3-8B`, etc.
  * ✅︎
  * ✅︎
- * `Qwen3MoeForCausalLM`
  * Qwen3MoE
Jee Jee Li's avatar
Jee Jee Li committed
617
618
  * `Qwen/Qwen3-30B-A3B`, etc.
  *
619
  * ✅︎
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
- * `StableLmForCausalLM`
  * StableLM
  * `stabilityai/stablelm-3b-4e1t`, `stabilityai/stablelm-base-alpha-7b-v2`, etc.
  *
  * ✅︎
- * `Starcoder2ForCausalLM`
  * Starcoder2
  * `bigcode/starcoder2-3b`, `bigcode/starcoder2-7b`, `bigcode/starcoder2-15b`, etc.
  *
  * ✅︎
- * `SolarForCausalLM`
  * Solar Pro
  * `upstage/solar-pro-preview-instruct`, etc.
  * ✅︎
  * ✅︎
- * `TeleChat2ForCausalLM`
  * TeleChat2
637
  * `Tele-AI/TeleChat2-3B`, `Tele-AI/TeleChat2-7B`, `Tele-AI/TeleChat2-35B`, etc.
638
639
  * ✅︎
  * ✅︎
640
641
642
643
644
- * `TeleFLMForCausalLM`
  * TeleFLM
  * `CofeAI/FLM-2-52B-Instruct-2407`, `CofeAI/Tele-FLM`, etc.
  * ✅︎
  * ✅︎
645
646
647
648
649
- * `XverseForCausalLM`
  * XVERSE
  * `xverse/XVERSE-7B-Chat`, `xverse/XVERSE-13B-Chat`, `xverse/XVERSE-65B-Chat`, etc.
  * ✅︎
  * ✅︎
650
651
652
653
654
- * `MiniMaxText01ForCausalLM`
  * MiniMax-Text
  * `MiniMaxAI/MiniMax-Text-01`, etc.
  *
  * ✅︎
655
656
657
658
659
- * `Zamba2ForCausalLM`
  * Zamba2
  * `Zyphra/Zamba2-7B-instruct`, `Zyphra/Zamba2-2.7B-instruct`, `Zyphra/Zamba2-1.2B-instruct`, etc.
  *
  *
660
661
662
663
664
- * `MiMoForCausalLM`
  * MiMo
  * `XiaomiMiMo/MiMo-7B-RL`, etc.
  *
  *
665
666
667
:::

:::{note}
668
Currently, the ROCm version of vLLM supports Mistral and Mixtral only for context lengths up to 4096.
669
:::
670

671
### Pooling Models
672

673
See [this page](pooling-models) for more information on how to use pooling models.
674

675
:::{important}
676
677
Since some model architectures support both generative and pooling tasks,
you should explicitly specify the task type to ensure that the model is used in pooling mode instead of generative mode.
678
:::
679

680
681
682
#### Text Embedding

Specified using `--task embed`.
683

684
:::{list-table}
685
686
687
:widths: 25 25 50 5 5
:header-rows: 1

688
689
690
691
692
693
694
- * Architecture
  * Models
  * Example HF Models
  * [LoRA](#lora-adapter)
  * [PP](#distributed-serving)
- * `BertModel`
  * BERT-based
695
  * `BAAI/bge-base-en-v1.5`, `Snowflake/snowflake-arctic-embed-xs`, etc.
696
697
698
  *
  *
- * `Gemma2Model`
699
  * Gemma 2-based
700
701
702
703
704
705
706
707
  * `BAAI/bge-multilingual-gemma2`, etc.
  *
  * ✅︎
- * `GritLM`
  * GritLM
  * `parasail-ai/GritLM-7B-vllm`.
  * ✅︎
  * ✅︎
708
- * `GteModel`
709
  * Arctic-Embed-2.0-M
710
711
712
  * `Snowflake/snowflake-arctic-embed-m-v2.0`.
  *
  *
713
714
715
716
717
718
719
720
721
722
- * `GteNewModel`
  * mGTE-TRM (see note)
  * `Alibaba-NLP/gte-multilingual-base`, etc.
  *
  *
- * `ModernBertModel`
  * ModernBERT-based
  * `Alibaba-NLP/gte-modernbert-base`, etc.
  *
  *
723
- * `NomicBertModel`
724
  * Nomic BERT
725
726
727
  * `nomic-ai/nomic-embed-text-v1`, `nomic-ai/nomic-embed-text-v2-moe`, `Snowflake/snowflake-arctic-embed-m-long`, etc.
  *
  *
728
729
730
731
732
733
734
735
736
737
738
739
- * `LlamaModel`, `LlamaForCausalLM`, `MistralModel`, etc.
  * Llama-based
  * `intfloat/e5-mistral-7b-instruct`, etc.
  * ✅︎
  * ✅︎
- * `Qwen2Model`, `Qwen2ForCausalLM`
  * Qwen2-based
  * `ssmits/Qwen2-7B-Instruct-embed-base` (see note), `Alibaba-NLP/gte-Qwen2-7B-instruct` (see note), etc.
  * ✅︎
  * ✅︎
- * `RobertaModel`, `RobertaForMaskedLM`
  * RoBERTa-based
740
  * `sentence-transformers/all-roberta-large-v1`, etc.
741
742
743
744
  *
  *
- * `XLMRobertaModel`
  * XLM-RoBERTa-based
745
  * `intfloat/multilingual-e5-large`, `jinaai/jina-reranker-v2-base-multilingual`, `Snowflake/snowflake-arctic-embed-l-v2.0`, `jinaai/jina-embeddings-v3`(see note), etc.
746
747
748
749
750
  *
  *
:::

:::{note}
751
752
`ssmits/Qwen2-7B-Instruct-embed-base` has an improperly defined Sentence Transformers config.
You should manually set mean pooling by passing `--override-pooler-config '{"pooling_type": "MEAN"}'`.
753
:::
754

755
:::{note}
756
757
The HF implementation of `Alibaba-NLP/gte-Qwen2-1.5B-instruct` is hardcoded to use causal attention despite what is shown in `config.json`. To compare vLLM vs HF results,
you should set `--hf-overrides '{"is_causal": true}'` in vLLM so that the two implementations are consistent with each other.
758

759
760
For both the 1.5B and 7B variants, you also need to enable `--trust-remote-code` for the correct tokenizer to be loaded.
See [relevant issue on HF Transformers](https://github.com/huggingface/transformers/issues/34882).
761
:::
762

763
764
765
766
:::{note}
`jinaai/jina-embeddings-v3` supports multiple tasks through lora, while vllm temporarily only supports text-matching tasks by merging lora weights.
:::

767
768
769
770
:::{note}
The second-generation GTE model (mGTE-TRM) is named `NewModel`. The name `NewModel` is too generic, you should set `--hf-overrides '{"architectures": ["GteNewModel"]}'` to specify the use of the `GteNewModel` architecture.
:::

771
If your model is not in the above list, we will try to automatically convert the model using
772
{func}`~vllm.model_executor.models.adapters.as_embedding_model`. By default, the embeddings
773
774
of the whole prompt are extracted from the normalized hidden state corresponding to the last token.

775
776
777
#### Reward Modeling

Specified using `--task reward`.
778

779
:::{list-table}
780
781
782
:widths: 25 25 50 5 5
:header-rows: 1

783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
- * Architecture
  * Models
  * Example HF Models
  * [LoRA](#lora-adapter)
  * [PP](#distributed-serving)
- * `InternLM2ForRewardModel`
  * InternLM2-based
  * `internlm/internlm2-1_8b-reward`, `internlm/internlm2-7b-reward`, etc.
  * ✅︎
  * ✅︎
- * `LlamaForCausalLM`
  * Llama-based
  * `peiyi9979/math-shepherd-mistral-7b-prm`, etc.
  * ✅︎
  * ✅︎
- * `Qwen2ForRewardModel`
  * Qwen2-based
  * `Qwen/Qwen2.5-Math-RM-72B`, etc.
  * ✅︎
  * ✅︎
- * `Qwen2ForProcessRewardModel`
  * Qwen2-based
  * `Qwen/Qwen2.5-Math-PRM-7B`, `Qwen/Qwen2.5-Math-PRM-72B`, etc.
  * ✅︎
  * ✅︎
:::
809

810
If your model is not in the above list, we will try to automatically convert the model using
811
{func}`~vllm.model_executor.models.adapters.as_reward_model`. By default, we return the hidden states of each token directly.
812

813
:::{important}
814
815
For process-supervised reward models such as `peiyi9979/math-shepherd-mistral-7b-prm`, the pooling config should be set explicitly,
e.g.: `--override-pooler-config '{"pooling_type": "STEP", "step_tag_id": 123, "returned_token_ids": [456, 789]}'`.
816
:::
817

818
819
820
#### Classification

Specified using `--task classify`.
821

822
:::{list-table}
823
824
825
:widths: 25 25 50 5 5
:header-rows: 1

826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
- * Architecture
  * Models
  * Example HF Models
  * [LoRA](#lora-adapter)
  * [PP](#distributed-serving)
- * `JambaForSequenceClassification`
  * Jamba
  * `ai21labs/Jamba-tiny-reward-dev`, etc.
  * ✅︎
  * ✅︎
- * `Qwen2ForSequenceClassification`
  * Qwen2-based
  * `jason9693/Qwen2.5-1.5B-apeach`, etc.
  * ✅︎
  * ✅︎
:::
842

843
If your model is not in the above list, we will try to automatically convert the model using
844
{func}`~vllm.model_executor.models.adapters.as_classification_model`. By default, the class probabilities are extracted from the softmaxed hidden state corresponding to the last token.
845

846
847
848
#### Sentence Pair Scoring

Specified using `--task score`.
849

850
:::{list-table}
851
852
853
:widths: 25 25 50 5 5
:header-rows: 1

854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
- * Architecture
  * Models
  * Example HF Models
  * [LoRA](#lora-adapter)
  * [PP](#distributed-serving)
- * `BertForSequenceClassification`
  * BERT-based
  * `cross-encoder/ms-marco-MiniLM-L-6-v2`, etc.
  *
  *
- * `RobertaForSequenceClassification`
  * RoBERTa-based
  * `cross-encoder/quora-roberta-base`, etc.
  *
  *
- * `XLMRobertaForSequenceClassification`
  * XLM-RoBERTa-based
  * `BAAI/bge-reranker-v2-m3`, etc.
  *
  *
xsank's avatar
xsank committed
874
875
876
877
878
- * `ModernBertForSequenceClassification`
  * ModernBert-based
  * `Alibaba-NLP/gte-reranker-modernbert-base`, etc.
  *
  *
879
:::
880

881
(supported-mm-models)=
882

883
## List of Multimodal Language Models
884
885
886

The following modalities are supported depending on the model:

887
888
889
890
- **T**ext
- **I**mage
- **V**ideo
- **A**udio
891

892
Any combination of modalities joined by `+` are supported.
Cyrus Leung's avatar
Cyrus Leung committed
893

894
- e.g.: `T + I` means that the model supports text-only, image-only, and text-with-image inputs.
Cyrus Leung's avatar
Cyrus Leung committed
895

896
On the other hand, modalities separated by `/` are mutually exclusive.
Cyrus Leung's avatar
Cyrus Leung committed
897

898
- e.g.: `T / I` means that the model supports text-only and image-only inputs, but not text-with-image inputs.
Cyrus Leung's avatar
Cyrus Leung committed
899

900
See [this page](#multimodal-inputs) on how to pass multi-modal inputs to the model.
901

902
:::{important}
903
**To enable multiple multi-modal items per text prompt in vLLM V0**, you have to set `limit_mm_per_prompt` (offline inference)
904
or `--limit-mm-per-prompt` (online serving). For example, to enable passing up to 4 images per text prompt:
905
906

Offline inference:
907

908
```python
Reid's avatar
Reid committed
909
910
from vllm import LLM

911
912
913
914
915
916
llm = LLM(
    model="Qwen/Qwen2-VL-7B-Instruct",
    limit_mm_per_prompt={"image": 4},
)
```

917
Online serving:
918

919
```bash
920
vllm serve Qwen/Qwen2-VL-7B-Instruct --limit-mm-per-prompt '{"image":4}'
921
922
```

923
924
**This is no longer required if you are using vLLM V1.**

925
926
927
:::

:::{note}
928
vLLM currently only supports adding LoRA to the language backbone of multimodal models.
929
:::
930

931
### Generative Models
932

933
See [this page](#generative-models) for more information on how to use generative models.
934

935
936
937
#### Text Generation

Specified using `--task generate`.
938

939
:::{list-table}
940
941
942
:widths: 25 25 15 20 5 5 5
:header-rows: 1

943
944
945
946
947
948
949
950
951
952
953
954
955
956
- * Architecture
  * Models
  * Inputs
  * Example HF Models
  * [LoRA](#lora-adapter)
  * [PP](#distributed-serving)
  * [V1](gh-issue:8779)
- * `AriaForConditionalGeneration`
  * Aria
  * T + I<sup>+</sup>
  * `rhymes-ai/Aria`
  *
  * ✅︎
  * ✅︎
Jennifer Zhao's avatar
Jennifer Zhao committed
957
958
959
960
961
962
963
- * `AyaVisionForConditionalGeneration`
  * Aya Vision
  * T + I<sup>+</sup>
  * `CohereForAI/aya-vision-8b`, `CohereForAI/aya-vision-32b`, etc.
  *
  * ✅︎
  * ✅︎
964
965
966
967
968
969
970
971
972
973
974
975
976
977
- * `Blip2ForConditionalGeneration`
  * BLIP-2
  * T + I<sup>E</sup>
  * `Salesforce/blip2-opt-2.7b`, `Salesforce/blip2-opt-6.7b`, etc.
  *
  * ✅︎
  * ✅︎
- * `ChameleonForConditionalGeneration`
  * Chameleon
  * T + I
  * `facebook/chameleon-7b` etc.
  *
  * ✅︎
  * ✅︎
978
- * `DeepseekVLV2ForCausalLM`<sup>^</sup>
979
980
  * DeepSeek-VL2
  * T + I<sup>+</sup>
981
  * `deepseek-ai/deepseek-vl2-tiny`, `deepseek-ai/deepseek-vl2-small`, `deepseek-ai/deepseek-vl2` etc.
982
983
984
  *
  * ✅︎
  * ✅︎
985
986
987
988
989
990
991
- * `Florence2ForConditionalGeneration`
  * Florence-2
  * T + I
  * `microsoft/Florence-2-base`, `microsoft/Florence-2-large` etc.
  *
  *
  *
992
993
994
995
996
997
998
- * `FuyuForCausalLM`
  * Fuyu
  * T + I
  * `adept/fuyu-8b` etc.
  *
  * ✅︎
  * ✅︎
999
1000
1001
1002
1003
1004
- * `Gemma3ForConditionalGeneration`
  * Gemma 3
  * T + I<sup>+</sup>
  * `google/gemma-3-4b-it`, `google/gemma-3-27b-it`, etc.
  * ✅︎
  * ✅︎
1005
  * ⚠️
1006
- * `GLM4VForCausalLM`<sup>^</sup>
1007
1008
  * GLM-4V
  * T + I
1009
  * `THUDM/glm-4v-9b`, `THUDM/cogagent-9b-20241220` etc.
1010
1011
  * ✅︎
  * ✅︎
1012
  * ✅︎
1013
1014
1015
1016
1017
1018
1019
- * `GraniteSpeechForConditionalGeneration`
  * Granite Speech
  * T + A
  * `ibm-granite/granite-speech-3.3-8b`
  * ✅︎
  * ✅︎
  * ✅︎
1020
1021
1022
1023
1024
1025
- * `H2OVLChatModel`
  * H2OVL
  * T + I<sup>E+</sup>
  * `h2oai/h2ovl-mississippi-800m`, `h2oai/h2ovl-mississippi-2b`, etc.
  *
  * ✅︎
1026
  * ✅︎\*
1027
1028
1029
1030
1031
1032
- * `Idefics3ForConditionalGeneration`
  * Idefics3
  * T + I
  * `HuggingFaceM4/Idefics3-8B-Llama3` etc.
  * ✅︎
  *
1033
  * ✅︎
1034
- * `InternVLChatModel`
1035
  * InternVL 3.0, InternVideo 2.5, InternVL 2.5, Mono-InternVL, InternVL 2.0
1036
  * T + I<sup>E+</sup>
1037
  * `OpenGVLab/InternVL3-9B`, `OpenGVLab/InternVideo2_5_Chat_8B`, `OpenGVLab/InternVL2_5-4B`, `OpenGVLab/Mono-InternVL-2B`, `OpenGVLab/InternVL2-4B`, etc.
1038
1039
1040
  *
  * ✅︎
  * ✅︎
1041
1042
1043
1044
1045
1046
1047
- * `KimiVLForConditionalGeneration`
  * Kimi-VL-A3B-Instruct, Kimi-VL-A3B-Thinking
  * T + I<sup>+</sup>
  * `moonshotai/Kimi-VL-A3B-Instruct`, `moonshotai/Kimi-VL-A3B-Thinking`
  *
  *
  * ✅︎
1048
- * `Llama4ForConditionalGeneration`
1049
  * Llama 4
1050
1051
1052
1053
1054
  * T + I<sup>+</sup>
  * `meta-llama/Llama-4-Scout-17B-16E-Instruct`, `meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8`, `meta-llama/Llama-4-Maverick-17B-128E-Instruct`, etc.
  *
  * ✅︎
  * ✅︎
1055
1056
1057
1058
1059
1060
1061
1062
1063
1064
1065
1066
1067
1068
1069
1070
1071
1072
1073
1074
1075
1076
1077
1078
1079
1080
1081
1082
- * `LlavaForConditionalGeneration`
  * LLaVA-1.5
  * T + I<sup>E+</sup>
  * `llava-hf/llava-1.5-7b-hf`, `TIGER-Lab/Mantis-8B-siglip-llama3` (see note), etc.
  *
  * ✅︎
  * ✅︎
- * `LlavaNextForConditionalGeneration`
  * LLaVA-NeXT
  * T + I<sup>E+</sup>
  * `llava-hf/llava-v1.6-mistral-7b-hf`, `llava-hf/llava-v1.6-vicuna-7b-hf`, etc.
  *
  * ✅︎
  * ✅︎
- * `LlavaNextVideoForConditionalGeneration`
  * LLaVA-NeXT-Video
  * T + V
  * `llava-hf/LLaVA-NeXT-Video-7B-hf`, etc.
  *
  * ✅︎
  * ✅︎
- * `LlavaOnevisionForConditionalGeneration`
  * LLaVA-Onevision
  * T + I<sup>+</sup> + V<sup>+</sup>
  * `llava-hf/llava-onevision-qwen2-7b-ov-hf`, `llava-hf/llava-onevision-qwen2-0.5b-ov-hf`, etc.
  *
  * ✅︎
  * ✅︎
1083
1084
1085
1086
1087
1088
- * `MiniCPMO`
  * MiniCPM-O
  * T + I<sup>E+</sup> + V<sup>E+</sup> + A<sup>E+</sup>
  * `openbmb/MiniCPM-o-2_6`, etc.
  * ✅︎
  * ✅︎
1089
  * ✅︎
1090
1091
- * `MiniCPMV`
  * MiniCPM-V
1092
  * T + I<sup>E+</sup> + V<sup>E+</sup>
1093
1094
1095
  * `openbmb/MiniCPM-V-2` (see note), `openbmb/MiniCPM-Llama3-V-2_5`, `openbmb/MiniCPM-V-2_6`, etc.
  * ✅︎
  * ✅︎
1096
  * ✅︎
1097
1098
1099
1100
1101
1102
1103
- * `MiniMaxVL01ForConditionalGeneration`
  * MiniMax-VL
  * T + I<sup>E+</sup>
  * `MiniMaxAI/MiniMax-VL-01`, etc.
  *
  * ✅︎
  * ✅︎
1104
1105
1106
1107
- * `Mistral3ForConditionalGeneration`
  * Mistral3
  * T + I<sup>+</sup>
  * `mistralai/Mistral-Small-3.1-24B-Instruct-2503`, etc.
1108
  * ✅︎
1109
  * ✅︎
1110
  * ✅︎
1111
1112
1113
1114
1115
1116
1117
1118
1119
- * `MllamaForConditionalGeneration`
  * Llama 3.2
  * T + I<sup>+</sup>
  * `meta-llama/Llama-3.2-90B-Vision-Instruct`, `meta-llama/Llama-3.2-11B-Vision`, etc.
  *
  *
  *
- * `MolmoForCausalLM`
  * Molmo
1120
  * T + I<sup>+</sup>
1121
  * `allenai/Molmo-7B-D-0924`, `allenai/Molmo-7B-O-0924`, etc.
1122
1123
1124
1125
1126
  * ✅︎
  * ✅︎
  * ✅︎
- * `NVLM_D_Model`
  * NVLM-D 1.0
1127
  * T + I<sup>+</sup>
1128
1129
1130
1131
  * `nvidia/NVLM-D-72B`, etc.
  *
  * ✅︎
  * ✅︎
1132
1133
- * `Ovis`
  * Ovis2, Ovis1.6
1134
  * T + I<sup>+</sup>
1135
  * `AIDC-AI/Ovis2-1B`, `AIDC-AI/Ovis1.6-Llama3.2-3B`, etc.
1136
1137
1138
  *
  *
  * ✅︎
1139
- * `PaliGemmaForConditionalGeneration`
1140
  * PaliGemma, PaliGemma 2
1141
1142
1143
1144
  * T + I<sup>E</sup>
  * `google/paligemma-3b-pt-224`, `google/paligemma-3b-mix-224`, `google/paligemma2-3b-ft-docci-448`, etc.
  *
  * ✅︎
1145
  * ⚠️
1146
1147
1148
1149
1150
1151
1152
- * `Phi3VForCausalLM`
  * Phi-3-Vision, Phi-3.5-Vision
  * T + I<sup>E+</sup>
  * `microsoft/Phi-3-vision-128k-instruct`, `microsoft/Phi-3.5-vision-instruct`, etc.
  *
  * ✅︎
  * ✅︎
1153
1154
1155
1156
1157
1158
- * `Phi4MMForCausalLM`
  * Phi-4-multimodal
  * T + I<sup>+</sup> / T + A<sup>+</sup> / I<sup>+</sup> + A<sup>+</sup>
  * `microsoft/Phi-4-multimodal-instruct`, etc.
  * ✅︎
  *
1159
  * ✅︎
1160
1161
1162
- * `PixtralForConditionalGeneration`
  * Pixtral
  * T + I<sup>+</sup>
1163
  * `mistralai/Mistral-Small-3.1-24B-Instruct-2503`, `mistral-community/pixtral-12b`, etc.
1164
1165
1166
  *
  * ✅︎
  * ✅︎
1167
- * `QwenVLForConditionalGeneration`<sup>^</sup>
1168
1169
1170
1171
1172
1173
1174
1175
1176
1177
1178
1179
1180
1181
1182
1183
1184
1185
1186
1187
  * Qwen-VL
  * T + I<sup>E+</sup>
  * `Qwen/Qwen-VL`, `Qwen/Qwen-VL-Chat`, etc.
  * ✅︎
  * ✅︎
  * ✅︎
- * `Qwen2AudioForConditionalGeneration`
  * Qwen2-Audio
  * T + A<sup>+</sup>
  * `Qwen/Qwen2-Audio-7B-Instruct`
  *
  * ✅︎
  * ✅︎
- * `Qwen2VLForConditionalGeneration`
  * QVQ, Qwen2-VL
  * T + I<sup>E+</sup> + V<sup>E+</sup>
  * `Qwen/QVQ-72B-Preview`, `Qwen/Qwen2-VL-7B-Instruct`, `Qwen/Qwen2-VL-72B-Instruct`, etc.
  * ✅︎
  * ✅︎
  * ✅︎
Roger Wang's avatar
Roger Wang committed
1188
1189
1190
1191
- * `Qwen2_5_VLForConditionalGeneration`
  * Qwen2.5-VL
  * T + I<sup>E+</sup> + V<sup>E+</sup>
  * `Qwen/Qwen2.5-VL-3B-Instruct`, `Qwen/Qwen2.5-VL-72B-Instruct`, etc.
1192
  * ✅︎
Roger Wang's avatar
Roger Wang committed
1193
1194
  * ✅︎
  * ✅︎
1195
1196
1197
1198
1199
1200
1201
- * `Qwen2_5OmniThinkerForConditionalGeneration`
  * Qwen2.5-Omni
  * T + I<sup>E+</sup> + V<sup>E+</sup> + A<sup>+</sup>
  * `Qwen/Qwen2.5-Omni-7B`
  *
  * ✅︎
  * ✅︎\*
1202
1203
1204
1205
1206
1207
1208
- * `SkyworkR1VChatModel`
  * Skywork-R1V-38B
  * T + I
  * `Skywork/Skywork-R1V-38B`
  *
  * ✅︎
  * ✅︎
1209
1210
1211
1212
1213
1214
1215
- * `SmolVLMForConditionalGeneration`
  * SmolVLM2
  * T + I
  * `SmolVLM2-2.2B-Instruct`
  *
  * ✅︎
  * ✅︎
1216
1217
1218
- * `UltravoxModel`
  * Ultravox
  * T + A<sup>E+</sup>
1219
  * `fixie-ai/ultravox-v0_5-llama-3_2-1b`
1220
  * ✅︎
1221
1222
1223
  * ✅︎
  * ✅︎
:::
1224

1225
1226
1227
<sup>^</sup> You need to set the architecture name via `--hf-overrides` to match the one in vLLM.  
&nbsp;&nbsp;&nbsp;&nbsp;• For example, to use DeepSeek-VL2 series models:  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;`--hf-overrides '{"architectures": ["DeepseekVLV2ForCausalLM"]}'`  
1228
1229
<sup>E</sup> Pre-computed embeddings can be inputted for this modality.  
<sup>+</sup> Multiple items can be inputted per text prompt for this modality.
1230

1231
:::{warning}
1232
1233
1234
1235
1236
1237
1238
1239
1240
1241
1242
Both V0 and V1 support `Gemma3ForConditionalGeneration` for text-only inputs.
However, there are differences in how they handle text + image inputs:

V0 correctly implements the model's attention pattern:
- Uses bidirectional attention between the image tokens corresponding to the same image
- Uses causal attention for other tokens
- Implemented via (naive) PyTorch SDPA with masking tensors
- Note: May use significant memory for long prompts with image

V1 currently uses a simplified attention pattern:
- Uses causal attention for all tokens, including image tokens
1243
- Generates reasonable outputs but does not match the original model's attention for text + image inputs, especially when `{"do_pan_and_scan": true}`
1244
1245
1246
- Will be updated in the future to support the correct behavior

This limitation exists because the model's mixed attention pattern (bidirectional for images, causal otherwise) is not yet supported by vLLM's attention backends.
1247
1248
1249
:::

:::{note}
1250
`h2oai/h2ovl-mississippi-2b` will be available in V1 once we support head size 80.
1251
:::
1252

1253
1254
1255
1256
:::{note}
To use `TIGER-Lab/Mantis-8B-siglip-llama3`, you have to pass `--hf_overrides '{"architectures": ["MantisForConditionalGeneration"]}'` when running vLLM.
:::

Eyshika Agarwal's avatar
Eyshika Agarwal committed
1257
:::{warning}
1258
1259
1260
1261
1262
1263
1264
1265
1266
1267
1268
1269
1270
1271
1272
1273
1274
1275
1276
1277
1278
1279
1280
1281
1282
1283
1284
The output quality of `AllenAI/Molmo-7B-D-0924` (especially in object localization tasks) has deteriorated in recent updates.

For the best results, we recommend using the following dependency versions (tested on A10 and L40):

```text
# Core vLLM-compatible dependencies with Molmo accuracy setup (tested on L40)
torch==2.5.1
torchvision==0.20.1
transformers==4.48.1
tokenizers==0.21.0
tiktoken==0.7.0
vllm==0.7.0

# Optional but recommended for improved performance and stability
triton==3.1.0
xformers==0.0.28.post3
uvloop==0.21.0
protobuf==5.29.3
openai==1.60.2
opencv-python-headless==4.11.0.86
pillow==10.4.0

# Installed FlashAttention (for float16 only)
flash-attn>=2.5.6  # Not used in float32, but should be documented
```

**Note:** Make sure you understand the security implications of using outdated packages.
Eyshika Agarwal's avatar
Eyshika Agarwal committed
1285
1286
:::

1287
1288
1289
1290
1291
1292
1293
:::{note}
The official `openbmb/MiniCPM-V-2` doesn't work yet, so we need to use a fork (`HwwwH/MiniCPM-V-2`) for now.
For more details, please see: <gh-pr:4087#issuecomment-2250397630>
:::

:::{warning}
Our PaliGemma implementations have the same problem as Gemma 3 (see above) for both V0 and V1.
1294
1295
:::

1296
:::{note}
1297
1298
To use Qwen2.5-Omni, you have to install Hugging Face Transformers library from source via
`pip install git+https://github.com/huggingface/transformers.git`.
1299
1300

Read audio from video pre-processing is currently supported on V0 (but not V1), because overlapping modalities is not yet supported in V1.
1301
`--mm-processor-kwargs '{"use_audio_in_video": true}'`.
1302
1303
:::

1304
### Pooling Models
1305

1306
See [this page](pooling-models) for more information on how to use pooling models.
1307

1308
:::{important}
1309
1310
Since some model architectures support both generative and pooling tasks,
you should explicitly specify the task type to ensure that the model is used in pooling mode instead of generative mode.
1311
:::
1312

1313
1314
1315
#### Text Embedding

Specified using `--task embed`.
1316

1317
Any text generation model can be converted into an embedding model by passing `--task embed`.
1318

1319
:::{note}
1320
To get the best results, you should use pooling models that are specifically trained as such.
1321
:::
1322
1323

The following table lists those that are tested in vLLM.
1324

1325
:::{list-table}
1326
1327
1328
:widths: 25 25 15 25 5 5
:header-rows: 1

1329
1330
1331
1332
1333
1334
1335
1336
1337
1338
1339
1340
1341
1342
1343
1344
1345
1346
1347
1348
1349
1350
1351
1352
1353
- * Architecture
  * Models
  * Inputs
  * Example HF Models
  * [LoRA](#lora-adapter)
  * [PP](#distributed-serving)
- * `LlavaNextForConditionalGeneration`
  * LLaVA-NeXT-based
  * T / I
  * `royokong/e5-v`
  *
  * ✅︎
- * `Phi3VForCausalLM`
  * Phi-3-Vision-based
  * T + I
  * `TIGER-Lab/VLM2Vec-Full`
  * 🚧
  * ✅︎
- * `Qwen2VLForConditionalGeneration`
  * Qwen2-VL-based
  * T + I
  * `MrLight/dse-qwen2-2b-mrl-v1`
  *
  * ✅︎
:::
1354

1355
1356
1357
#### Transcription

Specified using `--task transcription`.
1358
1359
1360
1361
1362
1363
1364
1365
1366
1367
1368
1369
1370
1371
1372
1373
1374
1375
1376

Speech2Text models trained specifically for Automatic Speech Recognition.

:::{list-table}
:widths: 25 25 25 5 5
:header-rows: 1

- * Architecture
  * Models
  * Example HF Models
  * [LoRA](#lora-adapter)
  * [PP](#distributed-serving)
- * `Whisper`
  * Whisper-based
  * `openai/whisper-large-v3-turbo`
  * 🚧
  * 🚧
:::

1377
_________________
1378

1379
## Model Support Policy
1380
1381
1382
1383

At vLLM, we are committed to facilitating the integration and support of third-party models within our ecosystem. Our approach is designed to balance the need for robustness and the practical limitations of supporting a wide range of models. Here’s how we manage third-party model support:

1. **Community-Driven Support**: We encourage community contributions for adding new models. When a user requests support for a new model, we welcome pull requests (PRs) from the community. These contributions are evaluated primarily on the sensibility of the output they generate, rather than strict consistency with existing implementations such as those in transformers. **Call for contribution:** PRs coming directly from model vendors are greatly appreciated!
1384

1385
1386
2. **Best-Effort Consistency**: While we aim to maintain a level of consistency between the models implemented in vLLM and other frameworks like transformers, complete alignment is not always feasible. Factors like acceleration techniques and the use of low-precision computations can introduce discrepancies. Our commitment is to ensure that the implemented models are functional and produce sensible results.

1387
    :::{tip}
1388
    When comparing the output of `model.generate` from Hugging Face Transformers with the output of `llm.generate` from vLLM, note that the former reads the model's generation config file (i.e., [generation_config.json](https://github.com/huggingface/transformers/blob/19dabe96362803fb0a9ae7073d03533966598b17/src/transformers/generation/utils.py#L1945)) and applies the default parameters for generation, while the latter only uses the parameters passed to the function. Ensure all sampling parameters are identical when comparing outputs.
1389
    :::
1390

1391
3. **Issue Resolution and Model Updates**: Users are encouraged to report any bugs or issues they encounter with third-party models. Proposed fixes should be submitted via PRs, with a clear explanation of the problem and the rationale behind the proposed solution. If a fix for one model impacts another, we rely on the community to highlight and address these cross-model dependencies. Note: for bugfix PRs, it is good etiquette to inform the original author to seek their feedback.
1392

1393
4. **Monitoring and Updates**: Users interested in specific models should monitor the commit history for those models (e.g., by tracking changes in the main/vllm/model_executor/models directory). This proactive approach helps users stay informed about updates and changes that may affect the models they use.
1394

1395
1396
1397
1398
1399
1400
1401
1402
5. **Selective Focus**: Our resources are primarily directed towards models with significant user interest and impact. Models that are less frequently used may receive less attention, and we rely on the community to play a more active role in their upkeep and improvement.

Through this approach, vLLM fosters a collaborative environment where both the core development team and the broader community contribute to the robustness and diversity of the third-party models supported in our ecosystem.

Note that, as an inference engine, vLLM does not introduce new models. Therefore, all models supported by vLLM are third-party models in this regard.

We have the following levels of testing for models:

1403
1. **Strict Consistency**: We compare the output of the model with the output of the model in the HuggingFace Transformers library under greedy decoding. This is the most stringent test. Please refer to [models tests](https://github.com/vllm-project/vllm/blob/main/tests/models) for the models that have passed this test.
1404
2. **Output Sensibility**: We check if the output of the model is sensible and coherent, by measuring the perplexity of the output and checking for any obvious errors. This is a less stringent test.
Reid's avatar
Reid committed
1405
3. **Runtime Functionality**: We check if the model can be loaded and run without errors. This is the least stringent test. Please refer to [functionality tests](gh-dir:tests) and [examples](gh-dir:examples) for the models that have passed this test.
1406
4. **Community Feedback**: We rely on the community to provide feedback on the models. If a model is broken or not working as expected, we encourage users to raise issues to report it or open pull requests to fix it. The rest of the models fall under this category.