supported_models.md 28 KB
Newer Older
1
(supported-models)=
Woosuk Kwon's avatar
Woosuk Kwon committed
2

3
# Supported Models
Woosuk Kwon's avatar
Woosuk Kwon committed
4

5
vLLM supports generative and pooling models across various tasks.
6
If a model supports more than one task, you can set the task via the {code}`--task` argument.
7
8

For each task, we list the model architectures that have been implemented in vLLM.
Woosuk Kwon's avatar
Woosuk Kwon committed
9
10
Alongside each architecture, we include some popular models that use it.

11
## Loading a Model
12

13
### HuggingFace Hub
14

15
By default, vLLM loads models from [HuggingFace (HF) Hub](https://huggingface.co/models).
16

17
18
To determine whether a given model is supported, you can check the {code}`config.json` file inside the HF repository.
If the {code}`"architectures"` field contains a model architecture listed below, then it should be supported in theory.
19

20
21
````{tip}
The easiest way to check if your model is really supported at runtime is to run the program below:
22

23
24
```python
from vllm import LLM
25

26
27
28
29
# For generative models (task=generate) only
llm = LLM(model=..., task="generate")  # Name or path of your model
output = llm.generate("Hello, my name is")
print(output)
30

31
# For pooling models (task={embed,classify,reward,score}) only
32
33
34
35
llm = LLM(model=..., task="embed")  # Name or path of your model
output = llm.encode("Hello, my name is")
print(output)
```
36

37
38
If vLLM successfully returns text (for generative models) or hidden states (for pooling models), it indicates that your model is supported.
````
39

40
41
Otherwise, please refer to [Adding a New Model](#adding-a-new-model) and [Enabling Multimodal Inputs](#enabling-multimodal-inputs) for instructions on how to implement your model in vLLM.
Alternatively, you can [open an issue on GitHub](https://github.com/vllm-project/vllm/issues/new/choose) to request vLLM support.
42

43
### ModelScope
44

45
To use models from [ModelScope](https://www.modelscope.cn) instead of HuggingFace Hub, set an environment variable:
46

47
48
49
```shell
$ export VLLM_USE_MODELSCOPE=True
```
50

51
And use with {code}`trust_remote_code=True`.
52

53
54
```python
from vllm import LLM
55

56
llm = LLM(model=..., revision=..., task=..., trust_remote_code=True)
57

58
59
60
# For generative models (task=generate) only
output = llm.generate("Hello, my name is")
print(output)
61

62
# For pooling models (task={embed,classify,reward,score}) only
63
64
65
output = llm.encode("Hello, my name is")
print(output)
```
66

67
## List of Text-only Language Models
68

69
### Generative Models
70

71
See [this page](#generative-models) for more information on how to use generative models.
72

73
#### Text Generation (`--task generate`)
74

75
```{eval-rst}
Woosuk Kwon's avatar
Woosuk Kwon committed
76
.. list-table::
77
  :widths: 25 25 50 5 5
Woosuk Kwon's avatar
Woosuk Kwon committed
78
79
80
81
  :header-rows: 1

  * - Architecture
    - Models
82
    - Example HF Models
83
84
    - :ref:`LoRA <lora-adapter>`
    - :ref:`PP <distributed-serving>`
85
  * - :code:`AquilaForCausalLM`
86
    - Aquila, Aquila2
87
    - :code:`BAAI/Aquila-7B`, :code:`BAAI/AquilaChat-7B`, etc.
88
    - ✅︎
89
    - ✅︎
90
91
92
93
  * - :code:`ArcticForCausalLM`
    - Arctic
    - :code:`Snowflake/snowflake-arctic-base`, :code:`Snowflake/snowflake-arctic-instruct`, etc.
    -
94
    - ✅︎
Zhuohan Li's avatar
Zhuohan Li committed
95
  * - :code:`BaiChuanForCausalLM`
96
    - Baichuan2, Baichuan
97
    - :code:`baichuan-inc/Baichuan2-13B-Chat`, :code:`baichuan-inc/Baichuan-7B`, etc.
Jee Li's avatar
Jee Li committed
98
    - ✅︎
99
    - ✅︎
100
101
102
103
  * - :code:`BloomForCausalLM`
    - BLOOM, BLOOMZ, BLOOMChat
    - :code:`bigscience/bloom`, :code:`bigscience/bloomz`, etc.
    -
104
    - ✅︎
105
106
107
  * - :code:`BartForConditionalGeneration`
    - BART
    - :code:`facebook/bart-base`, :code:`facebook/bart-large-cnn`, etc.
108
109
    -
    -
110
111
112
  * - :code:`ChatGLMModel`
    - ChatGLM
    - :code:`THUDM/chatglm2-6b`, :code:`THUDM/chatglm3-6b`, etc.
Jee Li's avatar
Jee Li committed
113
    - ✅︎
114
    - ✅︎
115
  * - :code:`CohereForCausalLM`,:code:`Cohere2ForCausalLM`
116
    - Command-R
117
    - :code:`CohereForAI/c4ai-command-r-v01`, :code:`CohereForAI/c4ai-command-r7b-12-2024`, etc.
118
119
    - ✅︎
    - ✅︎
120
121
122
  * - :code:`DbrxForCausalLM`
    - DBRX
    - :code:`databricks/dbrx-base`, :code:`databricks/dbrx-instruct`, etc.
123
    -
124
    - ✅︎
125
126
127
  * - :code:`DeciLMForCausalLM`
    - DeciLM
    - :code:`Deci/DeciLM-7B`, :code:`Deci/DeciLM-7B-instruct`, etc.
128
    -
129
    - ✅︎
130
131
132
  * - :code:`DeepseekForCausalLM`
    - DeepSeek
    - :code:`deepseek-ai/deepseek-llm-67b-base`, :code:`deepseek-ai/deepseek-llm-7b-chat` etc.
133
    -
134
    - ✅︎
135
136
137
  * - :code:`DeepseekV2ForCausalLM`
    - DeepSeek-V2
    - :code:`deepseek-ai/DeepSeek-V2`, :code:`deepseek-ai/DeepSeek-V2-Chat` etc.
138
    -
139
    - ✅︎
140
141
142
143
144
  * - :code:`DeepseekV3ForCausalLM`
    - DeepSeek-V3
    - :code:`deepseek-ai/DeepSeek-V3-Base`, :code:`deepseek-ai/DeepSeek-V3` etc.
    -
    - ✅︎
145
146
147
148
  * - :code:`ExaoneForCausalLM`
    - EXAONE-3
    - :code:`LGAI-EXAONE/EXAONE-3.0-7.8B-Instruct`, etc.
    - ✅︎
149
    - ✅︎
Zhuohan Li's avatar
Zhuohan Li committed
150
151
  * - :code:`FalconForCausalLM`
    - Falcon
152
    - :code:`tiiuae/falcon-7b`, :code:`tiiuae/falcon-40b`, :code:`tiiuae/falcon-rw-7b`, etc.
153
    -
154
    - ✅︎
155
156
157
158
  * - :code:`FalconMambaForCausalLM`
    - FalconMamba
    - :code:`tiiuae/falcon-mamba-7b`, :code:`tiiuae/falcon-mamba-7b-instruct`, etc.
    - ✅︎
159
    - ✅︎
160
161
162
  * - :code:`GemmaForCausalLM`
    - Gemma
    - :code:`google/gemma-2b`, :code:`google/gemma-7b`, etc.
163
    - ✅︎
164
    - ✅︎
Woosuk Kwon's avatar
Woosuk Kwon committed
165
166
167
168
  * - :code:`Gemma2ForCausalLM`
    - Gemma2
    - :code:`google/gemma-2-9b`, :code:`google/gemma-2-27b`, etc.
    - ✅︎
169
    - ✅︎
170
171
172
173
174
  * - :code:`GlmForCausalLM`
    - GLM-4
    - :code:`THUDM/glm-4-9b-chat-hf`, etc.
    - ✅︎
    - ✅︎
Woosuk Kwon's avatar
Woosuk Kwon committed
175
176
  * - :code:`GPT2LMHeadModel`
    - GPT-2
177
    - :code:`gpt2`, :code:`gpt2-xl`, etc.
178
    -
179
    - ✅︎
180
181
182
  * - :code:`GPTBigCodeForCausalLM`
    - StarCoder, SantaCoder, WizardCoder
    - :code:`bigcode/starcoder`, :code:`bigcode/gpt_bigcode-santacoder`, :code:`WizardLM/WizardCoder-15B-V1.0`, etc.
183
    - ✅︎
184
    - ✅︎
185
186
187
  * - :code:`GPTJForCausalLM`
    - GPT-J
    - :code:`EleutherAI/gpt-j-6b`, :code:`nomic-ai/gpt4all-j`, etc.
188
    -
189
    - ✅︎
Woosuk Kwon's avatar
Woosuk Kwon committed
190
191
  * - :code:`GPTNeoXForCausalLM`
    - GPT-NeoX, Pythia, OpenAssistant, Dolly V2, StableLM
192
    - :code:`EleutherAI/gpt-neox-20b`, :code:`EleutherAI/pythia-12b`, :code:`OpenAssistant/oasst-sft-4-pythia-12b-epoch-3.5`, :code:`databricks/dolly-v2-12b`, :code:`stabilityai/stablelm-tuned-alpha-7b`, etc.
193
    -
194
    - ✅︎
195
  * - :code:`GraniteForCausalLM`
196
197
    - Granite 3.0, Granite 3.1, PowerLM
    - :code:`ibm-granite/granite-3.0-2b-base`, :code:`ibm-granite/granite-3.1-8b-instruct`, :code:`ibm/PowerLM-3b`, etc.
198
    - ✅︎
199
    - ✅︎
200
  * - :code:`GraniteMoeForCausalLM`
201
202
    - Granite 3.0 MoE, PowerMoE
    - :code:`ibm-granite/granite-3.0-1b-a400m-base`, :code:`ibm-granite/granite-3.0-3b-a800m-instruct`, :code:`ibm/PowerMoE-3b`, etc.
203
    - ✅︎
204
    - ✅︎
205
206
207
208
209
  * - :code:`GritLM`
    - GritLM
    - :code:`parasail-ai/GritLM-7B-vllm`.
    - ✅︎
    - ✅︎
210
211
212
  * - :code:`InternLMForCausalLM`
    - InternLM
    - :code:`internlm/internlm-7b`, :code:`internlm/internlm-chat-7b`, etc.
213
    - ✅︎
214
    - ✅︎
Fengzhe Zhou's avatar
Fengzhe Zhou committed
215
216
217
  * - :code:`InternLM2ForCausalLM`
    - InternLM2
    - :code:`internlm/internlm2-7b`, :code:`internlm/internlm2-chat-7b`, etc.
218
    - ✅︎
219
    - ✅︎
220
221
  * - :code:`JAISLMHeadModel`
    - Jais
222
    - :code:`inceptionai/jais-13b`, :code:`inceptionai/jais-13b-chat`, :code:`inceptionai/jais-30b-v3`, :code:`inceptionai/jais-30b-chat-v3`, etc.
223
    -
224
    - ✅︎
Mor Zusman's avatar
Mor Zusman committed
225
226
  * - :code:`JambaForCausalLM`
    - Jamba
227
    - :code:`ai21labs/AI21-Jamba-1.5-Large`, :code:`ai21labs/AI21-Jamba-1.5-Mini`, :code:`ai21labs/Jamba-v0.1`, etc.
Mor Zusman's avatar
Mor Zusman committed
228
    - ✅︎
229
    - ✅︎
Woosuk Kwon's avatar
Woosuk Kwon committed
230
  * - :code:`LlamaForCausalLM`
231
232
    - Llama 3.1, Llama 3, Llama 2, LLaMA, Yi
    - :code:`meta-llama/Meta-Llama-3.1-405B-Instruct`, :code:`meta-llama/Meta-Llama-3.1-70B`, :code:`meta-llama/Meta-Llama-3-70B-Instruct`, :code:`meta-llama/Llama-2-70b-hf`, :code:`01-ai/Yi-34B`, etc.
233
    - ✅︎
234
    - ✅︎
235
236
237
  * - :code:`MambaForCausalLM`
    - Mamba
    - :code:`state-spaces/mamba-130m-hf`, :code:`state-spaces/mamba-790m-hf`, :code:`state-spaces/mamba-2.8b-hf`, etc.
238
    -
239
    - ✅︎
ywfang's avatar
ywfang committed
240
241
  * - :code:`MiniCPMForCausalLM`
    - MiniCPM
242
    - :code:`openbmb/MiniCPM-2B-sft-bf16`, :code:`openbmb/MiniCPM-2B-dpo-bf16`, :code:`openbmb/MiniCPM-S-1B-sft`, etc.
243
244
    - ✅︎
    - ✅︎
ywfang's avatar
ywfang committed
245
246
247
  * - :code:`MiniCPM3ForCausalLM`
    - MiniCPM3
    - :code:`openbmb/MiniCPM3-4B`, etc.
248
249
    - ✅︎
    - ✅︎
250
251
252
  * - :code:`MistralForCausalLM`
    - Mistral, Mistral-Instruct
    - :code:`mistralai/Mistral-7B-v0.1`, :code:`mistralai/Mistral-7B-Instruct-v0.1`, etc.
253
    - ✅︎
254
    - ✅︎
Woosuk Kwon's avatar
Woosuk Kwon committed
255
256
  * - :code:`MixtralForCausalLM`
    - Mixtral-8x7B, Mixtral-8x7B-Instruct
257
    - :code:`mistralai/Mixtral-8x7B-v0.1`, :code:`mistralai/Mixtral-8x7B-Instruct-v0.1`, :code:`mistral-community/Mixtral-8x22B-v0.1`, etc.
258
    - ✅︎
259
    - ✅︎
Woosuk Kwon's avatar
Woosuk Kwon committed
260
  * - :code:`MPTForCausalLM`
261
262
    - MPT, MPT-Instruct, MPT-Chat, MPT-StoryWriter
    - :code:`mosaicml/mpt-7b`, :code:`mosaicml/mpt-7b-storywriter`, :code:`mosaicml/mpt-30b`, etc.
263
    -
264
    - ✅︎
265
266
267
268
  * - :code:`NemotronForCausalLM`
    - Nemotron-3, Nemotron-4, Minitron
    - :code:`nvidia/Minitron-8B-Base`, :code:`mgoin/Nemotron-4-340B-Base-hf-FP8`, etc.
    - ✅︎
269
    - ✅︎
Isotr0py's avatar
Isotr0py committed
270
271
  * - :code:`OLMoForCausalLM`
    - OLMo
272
    - :code:`allenai/OLMo-1B-hf`, :code:`allenai/OLMo-7B-hf`, etc.
273
    -
274
    - ✅︎
275
276
277
278
279
  * - :code:`OLMo2ForCausalLM`
    - OLMo2
    - :code:`allenai/OLMo2-7B-1124`, etc.
    -
    - ✅︎
280
281
282
283
284
  * - :code:`OLMoEForCausalLM`
    - OLMoE
    - :code:`allenai/OLMoE-1B-7B-0924`, :code:`allenai/OLMoE-1B-7B-0924-Instruct`, etc.
    - ✅︎
    - ✅︎
Woosuk Kwon's avatar
Woosuk Kwon committed
285
286
  * - :code:`OPTForCausalLM`
    - OPT, OPT-IML
287
    - :code:`facebook/opt-66b`, :code:`facebook/opt-iml-max-30b`, etc.
288
    -
289
    - ✅︎
张大成's avatar
张大成 committed
290
291
292
  * - :code:`OrionForCausalLM`
    - Orion
    - :code:`OrionStarAI/Orion-14B-Base`, :code:`OrionStarAI/Orion-14B-Chat`, etc.
293
    -
294
    - ✅︎
295
  * - :code:`PhiForCausalLM`
296
297
    - Phi
    - :code:`microsoft/phi-1_5`, :code:`microsoft/phi-2`, etc.
298
    - ✅︎
299
    - ✅︎
300
301
  * - :code:`Phi3ForCausalLM`
    - Phi-3
302
    - :code:`microsoft/Phi-3-mini-4k-instruct`, :code:`microsoft/Phi-3-mini-128k-instruct`, :code:`microsoft/Phi-3-medium-128k-instruct`, etc.
303
304
    - ✅︎
    - ✅︎
305
306
307
  * - :code:`Phi3SmallForCausalLM`
    - Phi-3-Small
    - :code:`microsoft/Phi-3-small-8k-instruct`, :code:`microsoft/Phi-3-small-128k-instruct`, etc.
308
    -
309
    - ✅︎
310
311
312
  * - :code:`PhiMoEForCausalLM`
    - Phi-3.5-MoE
    - :code:`microsoft/Phi-3.5-MoE-instruct`, etc.
313
314
    - ✅︎
    - ✅︎
315
316
317
  * - :code:`PersimmonForCausalLM`
    - Persimmon
    - :code:`adept/persimmon-8b-base`, :code:`adept/persimmon-8b-chat`, etc.
318
    -
319
    - ✅︎
320
  * - :code:`QWenLMHeadModel`
321
322
    - Qwen
    - :code:`Qwen/Qwen-7B`, :code:`Qwen/Qwen-7B-Chat`, etc.
323
    - ✅︎
324
    - ✅︎
Junyang Lin's avatar
Junyang Lin committed
325
326
  * - :code:`Qwen2ForCausalLM`
    - Qwen2
327
    - :code:`Qwen/QwQ-32B-Preview`, :code:`Qwen/Qwen2-7B-Instruct`, :code:`Qwen/Qwen2-7B`, etc.
328
    - ✅︎
329
    - ✅︎
330
331
332
  * - :code:`Qwen2MoeForCausalLM`
    - Qwen2MoE
    - :code:`Qwen/Qwen1.5-MoE-A2.7B`, :code:`Qwen/Qwen1.5-MoE-A2.7B-Chat`, etc.
333
    -
334
    - ✅︎
335
  * - :code:`StableLmForCausalLM`
Hyunsung Lee's avatar
Hyunsung Lee committed
336
    - StableLM
337
    - :code:`stabilityai/stablelm-3b-4e1t`, :code:`stabilityai/stablelm-base-alpha-7b-v2`, etc.
338
    -
339
    - ✅︎
340
341
342
343
  * - :code:`Starcoder2ForCausalLM`
    - Starcoder2
    - :code:`bigcode/starcoder2-3b`, :code:`bigcode/starcoder2-7b`, :code:`bigcode/starcoder2-15b`, etc.
    -
344
    - ✅︎
345
  * - :code:`SolarForCausalLM`
346
    - Solar Pro
347
    - :code:`upstage/solar-pro-preview-instruct`, etc.
348
349
    - ✅︎
    - ✅︎
350
351
352
353
354
  * - :code:`TeleChat2ForCausalLM`
    - TeleChat2
    - :code:`TeleAI/TeleChat2-3B`, :code:`TeleAI/TeleChat2-7B`, :code:`TeleAI/TeleChat2-35B`, etc.
    - ✅︎
    - ✅︎
355
  * - :code:`XverseForCausalLM`
356
    - XVERSE
357
    - :code:`xverse/XVERSE-7B-Chat`, :code:`xverse/XVERSE-13B-Chat`, :code:`xverse/XVERSE-65B-Chat`, etc.
358
359
    - ✅︎
    - ✅︎
360
```
361

362
363
364
```{note}
Currently, the ROCm version of vLLM supports Mistral and Mixtral only for context lengths up to 4096.
```
365

366
### Pooling Models
367

368
See [this page](pooling-models) for more information on how to use pooling models.
369

370
371
372
373
```{important}
Since some model architectures support both generative and pooling tasks,
you should explicitly specify the task type to ensure that the model is used in pooling mode instead of generative mode.
```
374

375
#### Text Embedding (`--task embed`)
376

377
```{eval-rst}
378
379
380
381
382
383
.. list-table::
  :widths: 25 25 50 5 5
  :header-rows: 1

  * - Architecture
    - Models
384
    - Example HF Models
385
386
    - :ref:`LoRA <lora-adapter>`
    - :ref:`PP <distributed-serving>`
387
388
389
  * - :code:`BertModel`
    - BERT-based
    - :code:`BAAI/bge-base-en-v1.5`, etc.
390
391
    -
    -
392
393
394
  * - :code:`Gemma2Model`
    - Gemma2-based
    - :code:`BAAI/bge-multilingual-gemma2`, etc.
395
    -
396
    - ✅︎
397
398
399
400
401
  * - :code:`GritLM`
    - GritLM
    - :code:`parasail-ai/GritLM-7B-vllm`.
    - ✅︎
    - ✅︎
402
403
  * - :code:`LlamaModel`, :code:`LlamaForCausalLM`, :code:`MistralModel`, etc.
    - Llama-based
404
    - :code:`intfloat/e5-mistral-7b-instruct`, etc.
405
    - ✅︎
406
    - ✅︎
407
408
  * - :code:`Qwen2Model`, :code:`Qwen2ForCausalLM`
    - Qwen2-based
409
    - :code:`ssmits/Qwen2-7B-Instruct-embed-base` (see note), :code:`Alibaba-NLP/gte-Qwen2-7B-instruct` (see note), etc.
410
411
    - ✅︎
    - ✅︎
412
413
414
  * - :code:`RobertaModel`, :code:`RobertaForMaskedLM`
    - RoBERTa-based
    - :code:`sentence-transformers/all-roberta-large-v1`, :code:`sentence-transformers/all-roberta-large-v1`, etc.
415
416
    -
    -
417
418
419
  * - :code:`XLMRobertaModel`
    - XLM-RoBERTa-based
    - :code:`intfloat/multilingual-e5-large`, etc.
420
421
422
    -
    -
```
423

424
425
426
427
```{note}
{code}`ssmits/Qwen2-7B-Instruct-embed-base` has an improperly defined Sentence Transformers config.
You should manually set mean pooling by passing {code}`--override-pooler-config '{"pooling_type": "MEAN"}'`.
```
428

429
430
431
```{note}
Unlike base Qwen2, {code}`Alibaba-NLP/gte-Qwen2-7B-instruct` uses bi-directional attention.
You can set {code}`--hf-overrides '{"is_causal": false}'` to change the attention mask accordingly.
432

433
434
435
On the other hand, its 1.5B variant ({code}`Alibaba-NLP/gte-Qwen2-1.5B-instruct`) uses causal attention
despite being described otherwise on its model card.
```
436

437
438
439
440
If your model is not in the above list, we will try to automatically convert the model using
:func:`vllm.model_executor.models.adapters.as_embedding_model`. By default, the embeddings
of the whole prompt are extracted from the normalized hidden state corresponding to the last token.

441
#### Reward Modeling (`--task reward`)
442

443
```{eval-rst}
444
445
446
447
448
449
.. list-table::
  :widths: 25 25 50 5 5
  :header-rows: 1

  * - Architecture
    - Models
450
    - Example HF Models
451
452
    - :ref:`LoRA <lora-adapter>`
    - :ref:`PP <distributed-serving>`
453
454
455
456
457
  * - :code:`LlamaForCausalLM`
    - Llama-based
    - :code:`peiyi9979/math-shepherd-mistral-7b-prm`, etc.
    - ✅︎
    - ✅︎
458
459
460
  * - :code:`Qwen2ForRewardModel`
    - Qwen2-based
    - :code:`Qwen/Qwen2.5-Math-RM-72B`, etc.
461
    - ✅︎
462
    - ✅︎
463
```
464

465
466
467
If your model is not in the above list, we will try to automatically convert the model using
:func:`vllm.model_executor.models.adapters.as_reward_model`. By default, we return the hidden states of each token directly.

468
469
470
471
```{important}
For process-supervised reward models such as {code}`peiyi9979/math-shepherd-mistral-7b-prm`, the pooling config should be set explicitly,
e.g.: {code}`--override-pooler-config '{"pooling_type": "STEP", "step_tag_id": 123, "returned_token_ids": [456, 789]}'`.
```
472

473
#### Classification (`--task classify`)
474

475
```{eval-rst}
476
477
478
479
480
481
482
.. list-table::
  :widths: 25 25 50 5 5
  :header-rows: 1

  * - Architecture
    - Models
    - Example HF Models
483
484
    - :ref:`LoRA <lora-adapter>`
    - :ref:`PP <distributed-serving>`
485
486
487
488
489
  * - :code:`JambaForSequenceClassification`
    - Jamba
    - :code:`ai21labs/Jamba-tiny-reward-dev`, etc.
    - ✅︎
    - ✅︎
490
491
492
  * - :code:`Qwen2ForSequenceClassification`
    - Qwen2-based
    - :code:`jason9693/Qwen2.5-1.5B-apeach`, etc.
493
    - ✅︎
494
    - ✅︎
495
```
496

497
498
499
If your model is not in the above list, we will try to automatically convert the model using
:func:`vllm.model_executor.models.adapters.as_classification_model`. By default, the class probabilities are extracted from the softmaxed hidden state corresponding to the last token.

500
#### Sentence Pair Scoring (`--task score`)
501

502
```{eval-rst}
503
504
505
506
507
508
509
.. list-table::
  :widths: 25 25 50 5 5
  :header-rows: 1

  * - Architecture
    - Models
    - Example HF Models
510
511
    - :ref:`LoRA <lora-adapter>`
    - :ref:`PP <distributed-serving>`
512
513
514
  * - :code:`BertForSequenceClassification`
    - BERT-based
    - :code:`cross-encoder/ms-marco-MiniLM-L-6-v2`, etc.
515
516
    -
    -
517
518
519
  * - :code:`RobertaForSequenceClassification`
    - RoBERTa-based
    - :code:`cross-encoder/quora-roberta-base`, etc.
520
521
    -
    -
522
523
524
  * - :code:`XLMRobertaForSequenceClassification`
    - XLM-RoBERTa-based
    - :code:`BAAI/bge-reranker-v2-m3`, etc.
525
526
527
    -
    -
```
528

529
(supported-mm-models)=
530

531
## List of Multimodal Language Models
532
533
534

The following modalities are supported depending on the model:

535
536
537
538
- **T**ext
- **I**mage
- **V**ideo
- **A**udio
539

540
Any combination of modalities joined by {code}`+` are supported.
Cyrus Leung's avatar
Cyrus Leung committed
541

542
- e.g.: {code}`T + I` means that the model supports text-only, image-only, and text-with-image inputs.
Cyrus Leung's avatar
Cyrus Leung committed
543

544
On the other hand, modalities separated by {code}`/` are mutually exclusive.
Cyrus Leung's avatar
Cyrus Leung committed
545

546
- e.g.: {code}`T / I` means that the model supports text-only and image-only inputs, but not text-with-image inputs.
Cyrus Leung's avatar
Cyrus Leung committed
547

548
See [this page](#multimodal-inputs) on how to pass multi-modal inputs to the model.
549

550
### Generative Models
551

552
See [this page](#generative-models) for more information on how to use generative models.
553

554
#### Text Generation (`--task generate`)
555

556
```{eval-rst}
557
.. list-table::
558
  :widths: 25 25 15 20 5 5 5
559
560
561
562
  :header-rows: 1

  * - Architecture
    - Models
563
564
    - Inputs
    - Example HF Models
565
566
    - :ref:`LoRA <lora-adapter>`
    - :ref:`PP <distributed-serving>`
567
    - V1
568
569
570
571
  * - :code:`AriaForConditionalGeneration`
    - Aria
    - T + I
    - :code:`rhymes-ai/Aria`
572
    -
573
    - ✅︎
574
    -
575
576
  * - :code:`Blip2ForConditionalGeneration`
    - BLIP-2
577
    - T + I\ :sup:`E`
578
579
    - :code:`Salesforce/blip2-opt-2.7b`, :code:`Salesforce/blip2-opt-6.7b`, etc.
    -
580
    - ✅︎
581
    -
582
583
  * - :code:`ChameleonForConditionalGeneration`
    - Chameleon
584
    - T + I
585
    - :code:`facebook/chameleon-7b` etc.
586
    -
587
    - ✅︎
588
    -
589
590
  * - :code:`FuyuForCausalLM`
    - Fuyu
591
    - T + I
592
    - :code:`adept/fuyu-8b` etc.
593
    -
594
    - ✅︎
595
    -
596
597
  * - :code:`ChatGLMModel`
    - GLM-4V
598
    - T + I
599
    - :code:`THUDM/glm-4v-9b` etc.
600
    - ✅︎
601
    - ✅︎
602
    -
603
604
605
606
  * - :code:`H2OVLChatModel`
    - H2OVL
    - T + I\ :sup:`E+`
    - :code:`h2oai/h2ovl-mississippi-800m`, :code:`h2oai/h2ovl-mississippi-2b`, etc.
607
    -
608
    - ✅︎
609
    -
610
611
612
613
  * - :code:`Idefics3ForConditionalGeneration`
    - Idefics3
    - T + I
    - :code:`HuggingFaceM4/Idefics3-8B-Llama3` etc.
614
    - ✅︎
615
    -
616
    -
617
  * - :code:`InternVLChatModel`
618
    - InternVL 2.5, Mono-InternVL, InternVL 2.0
619
    - T + I\ :sup:`E+`
620
    - :code:`OpenGVLab/InternVL2_5-4B`, :code:`OpenGVLab/Mono-InternVL-2B`, :code:`OpenGVLab/InternVL2-4B`, etc.
621
    -
622
    - ✅︎
623
    - ✅︎
624
625
  * - :code:`LlavaForConditionalGeneration`
    - LLaVA-1.5
626
    - T + I\ :sup:`E+`
627
    - :code:`llava-hf/llava-1.5-7b-hf`, :code:`TIGER-Lab/Mantis-8B-siglip-llama3` (see note), etc.
628
    -
629
    - ✅︎
630
    - ✅︎
631
632
  * - :code:`LlavaNextForConditionalGeneration`
    - LLaVA-NeXT
633
    - T + I\ :sup:`E+`
634
635
    - :code:`llava-hf/llava-v1.6-mistral-7b-hf`, :code:`llava-hf/llava-v1.6-vicuna-7b-hf`, etc.
    -
636
    - ✅︎
637
    -
638
639
  * - :code:`LlavaNextVideoForConditionalGeneration`
    - LLaVA-NeXT-Video
640
    - T + V
641
    - :code:`llava-hf/LLaVA-NeXT-Video-7B-hf`, etc.
642
    -
643
    - ✅︎
644
    -
645
646
  * - :code:`LlavaOnevisionForConditionalGeneration`
    - LLaVA-Onevision
647
    - T + I\ :sup:`+` + V\ :sup:`+`
648
    - :code:`llava-hf/llava-onevision-qwen2-7b-ov-hf`, :code:`llava-hf/llava-onevision-qwen2-0.5b-ov-hf`, etc.
649
    -
650
    - ✅︎
651
    -
652
653
  * - :code:`MiniCPMV`
    - MiniCPM-V
654
    - T + I\ :sup:`E+`
655
    - :code:`openbmb/MiniCPM-V-2` (see note), :code:`openbmb/MiniCPM-Llama3-V-2_5`, :code:`openbmb/MiniCPM-V-2_6`, etc.
656
657
    - ✅︎
    - ✅︎
658
    -
659
660
  * - :code:`MllamaForConditionalGeneration`
    - Llama 3.2
661
    - T + I\ :sup:`+`
662
663
    - :code:`meta-llama/Llama-3.2-90B-Vision-Instruct`, :code:`meta-llama/Llama-3.2-11B-Vision`, etc.
    -
664
    -
665
    -
666
667
  * - :code:`MolmoForCausalLM`
    - Molmo
668
    - T + I
669
670
671
    - :code:`allenai/Molmo-7B-D-0924`, :code:`allenai/Molmo-72B-0924`, etc.
    -
    - ✅︎
672
    - ✅︎
673
674
  * - :code:`NVLM_D_Model`
    - NVLM-D 1.0
675
    - T + I\ :sup:`E+`
676
    - :code:`nvidia/NVLM-D-72B`, etc.
677
    -
678
    - ✅︎
679
    - ✅︎
Roger Wang's avatar
Roger Wang committed
680
  * - :code:`PaliGemmaForConditionalGeneration`
Jani Monoses's avatar
Jani Monoses committed
681
    - PaliGemma, PaliGemma 2
682
    - T + I\ :sup:`E`
Jani Monoses's avatar
Jani Monoses committed
683
    - :code:`google/paligemma-3b-pt-224`, :code:`google/paligemma-3b-mix-224`, :code:`google/paligemma2-3b-ft-docci-448`, etc.
684
    -
685
    - ✅︎
686
    -
687
  * - :code:`Phi3VForCausalLM`
688
    - Phi-3-Vision, Phi-3.5-Vision
689
    - T + I\ :sup:`E+`
690
    - :code:`microsoft/Phi-3-vision-128k-instruct`, :code:`microsoft/Phi-3.5-vision-instruct` etc.
691
    -
692
    - ✅︎
693
    - ✅︎
Patrick von Platen's avatar
Patrick von Platen committed
694
695
  * - :code:`PixtralForConditionalGeneration`
    - Pixtral
696
    - T + I\ :sup:`+`
697
    - :code:`mistralai/Pixtral-12B-2409`, :code:`mistral-community/pixtral-12b` etc.
Patrick von Platen's avatar
Patrick von Platen committed
698
    -
699
    - ✅︎
700
    - ✅︎
701
  * - :code:`QWenLMHeadModel`
702
    - Qwen-VL
703
    - T + I\ :sup:`E+`
704
    - :code:`Qwen/Qwen-VL`, :code:`Qwen/Qwen-VL-Chat`, etc.
705
    - ✅︎
706
    - ✅︎
707
    -
708
709
710
711
712
713
  * - :code:`Qwen2AudioForConditionalGeneration`
    - Qwen2-Audio
    - T + A\ :sup:`+`
    - :code:`Qwen/Qwen2-Audio-7B-Instruct`
    -
    - ✅︎
714
    -
715
  * - :code:`Qwen2VLForConditionalGeneration`
716
    - Qwen2-VL
717
    - T + I\ :sup:`E+` + V\ :sup:`E+`
718
    - :code:`Qwen/QVQ-72B-Preview`, :code:`Qwen/Qwen2-VL-7B-Instruct`, :code:`Qwen/Qwen2-VL-72B-Instruct`, etc.
719
    - ✅︎
720
    - ✅︎
721
    -
722
  * - :code:`UltravoxModel`
723
    - Ultravox
724
    - T + A\ :sup:`E+`
725
    - :code:`fixie-ai/ultravox-v0_3`
726
    -
727
    - ✅︎
728
729
    -
```
730

731
732
```{eval-rst}
:sup:`E` Pre-computed embeddings can be inputted for this modality.
733

734
735
:sup:`+` Multiple items can be inputted per text prompt for this modality.
```
736

737
738
739
````{important}
To enable multiple multi-modal items per text prompt, you have to set {code}`limit_mm_per_prompt` (offline inference)
or {code}`--limit-mm-per-prompt` (online inference). For example, to enable passing up to 4 images per text prompt:
740

741
742
743
744
745
746
```python
llm = LLM(
    model="Qwen/Qwen2-VL-7B-Instruct",
    limit_mm_per_prompt={"image": 4},
)
```
747

748
749
750
751
```bash
vllm serve Qwen/Qwen2-VL-7B-Instruct --limit-mm-per-prompt image=4
```
````
752

753
754
755
```{note}
vLLM currently only supports adding LoRA to the language backbone of multimodal models.
```
756

757
758
759
760
```{note}
To use {code}`TIGER-Lab/Mantis-8B-siglip-llama3`, you have to install their GitHub repo ({code}`pip install git+https://github.com/TIGER-AI-Lab/Mantis.git`)
and pass {code}`--hf_overrides '{"architectures": ["MantisForConditionalGeneration"]}'` when running vLLM.
```
761

762
763
```{note}
The official {code}`openbmb/MiniCPM-V-2` doesn't work yet, so we need to use a fork ({code}`HwwwH/MiniCPM-V-2`) for now.
764
For more details, please see: <gh-pr:4087#issuecomment-2250397630>
765
```
Alphi's avatar
Alphi committed
766

767
### Pooling Models
768

769
See [this page](pooling-models) for more information on how to use pooling models.
770

771
772
773
774
```{important}
Since some model architectures support both generative and pooling tasks,
you should explicitly specify the task type to ensure that the model is used in pooling mode instead of generative mode.
```
775

776
#### Text Embedding (`--task embed`)
777

778
Any text generation model can be converted into an embedding model by passing {code}`--task embed`.
779

780
781
782
```{note}
To get the best results, you should use pooling models that are specifically trained as such.
```
783
784

The following table lists those that are tested in vLLM.
785

786
```{eval-rst}
787
788
789
790
791
792
793
794
.. list-table::
  :widths: 25 25 15 25 5 5
  :header-rows: 1

  * - Architecture
    - Models
    - Inputs
    - Example HF Models
795
796
    - :ref:`LoRA <lora-adapter>`
    - :ref:`PP <distributed-serving>`
Cyrus Leung's avatar
Cyrus Leung committed
797
798
799
800
  * - :code:`LlavaNextForConditionalGeneration`
    - LLaVA-NeXT-based
    - T / I
    - :code:`royokong/e5-v`
801
    -
Cyrus Leung's avatar
Cyrus Leung committed
802
    - ✅︎
803
804
805
806
807
808
  * - :code:`Phi3VForCausalLM`
    - Phi-3-Vision-based
    - T + I
    - :code:`TIGER-Lab/VLM2Vec-Full`
    - 🚧
    - ✅︎
809
810
811
812
  * - :code:`Qwen2VLForConditionalGeneration`
    - Qwen2-VL-based
    - T + I
    - :code:`MrLight/dse-qwen2-2b-mrl-v1`
813
    -
814
    - ✅︎
815
```
816

817
______________________________________________________________________
818

819
# Model Support Policy
820
821
822
823
824
825

At vLLM, we are committed to facilitating the integration and support of third-party models within our ecosystem. Our approach is designed to balance the need for robustness and the practical limitations of supporting a wide range of models. Here’s how we manage third-party model support:

1. **Community-Driven Support**: We encourage community contributions for adding new models. When a user requests support for a new model, we welcome pull requests (PRs) from the community. These contributions are evaluated primarily on the sensibility of the output they generate, rather than strict consistency with existing implementations such as those in transformers. **Call for contribution:** PRs coming directly from model vendors are greatly appreciated!
2. **Best-Effort Consistency**: While we aim to maintain a level of consistency between the models implemented in vLLM and other frameworks like transformers, complete alignment is not always feasible. Factors like acceleration techniques and the use of low-precision computations can introduce discrepancies. Our commitment is to ensure that the implemented models are functional and produce sensible results.

826
827
828
```{tip}
When comparing the output of {code}`model.generate` from HuggingFace Transformers with the output of {code}`llm.generate` from vLLM, note that the former reads the model's generation config file (i.e., [generation_config.json](https://github.com/huggingface/transformers/blob/19dabe96362803fb0a9ae7073d03533966598b17/src/transformers/generation/utils.py#L1945)) and applies the default parameters for generation, while the latter only uses the parameters passed to the function. Ensure all sampling parameters are identical when comparing outputs.
```
829

830
831
832
833
834
835
836
837
838
839
3. **Issue Resolution and Model Updates**: Users are encouraged to report any bugs or issues they encounter with third-party models. Proposed fixes should be submitted via PRs, with a clear explanation of the problem and the rationale behind the proposed solution. If a fix for one model impacts another, we rely on the community to highlight and address these cross-model dependencies. Note: for bugfix PRs, it is good etiquette to inform the original author to seek their feedback.
4. **Monitoring and Updates**: Users interested in specific models should monitor the commit history for those models (e.g., by tracking changes in the main/vllm/model_executor/models directory). This proactive approach helps users stay informed about updates and changes that may affect the models they use.
5. **Selective Focus**: Our resources are primarily directed towards models with significant user interest and impact. Models that are less frequently used may receive less attention, and we rely on the community to play a more active role in their upkeep and improvement.

Through this approach, vLLM fosters a collaborative environment where both the core development team and the broader community contribute to the robustness and diversity of the third-party models supported in our ecosystem.

Note that, as an inference engine, vLLM does not introduce new models. Therefore, all models supported by vLLM are third-party models in this regard.

We have the following levels of testing for models:

840
1. **Strict Consistency**: We compare the output of the model with the output of the model in the HuggingFace Transformers library under greedy decoding. This is the most stringent test. Please refer to [models tests](https://github.com/vllm-project/vllm/blob/main/tests/models) for the models that have passed this test.
841
2. **Output Sensibility**: We check if the output of the model is sensible and coherent, by measuring the perplexity of the output and checking for any obvious errors. This is a less stringent test.
842
3. **Runtime Functionality**: We check if the model can be loaded and run without errors. This is the least stringent test. Please refer to [functionality tests](gh-dir:tests) and [examples](gh-dir:main/examples) for the models that have passed this test.
843
4. **Community Feedback**: We rely on the community to provide feedback on the models. If a model is broken or not working as expected, we encourage users to raise issues to report it or open pull requests to fix it. The rest of the models fall under this category.