supported_models.rst 27.4 KB
Newer Older
Woosuk Kwon's avatar
Woosuk Kwon committed
1
2
3
4
5
.. _supported_models:

Supported Models
================

6
7
8
9
vLLM supports generative and pooling models across various tasks.
If a model supports more than one task, you can set the task via the :code:`--task` argument.

For each task, we list the model architectures that have been implemented in vLLM.
Woosuk Kwon's avatar
Woosuk Kwon committed
10
11
Alongside each architecture, we include some popular models that use it.

12
13
14
15
16
17
18
19
20
Loading a Model
^^^^^^^^^^^^^^^

HuggingFace Hub
+++++++++++++++

By default, vLLM loads models from `HuggingFace (HF) Hub <https://huggingface.co/models>`_.

To determine whether a given model is supported, you can check the :code:`config.json` file inside the HF repository.
21
22
23
24
25
26
27
28
29
If the :code:`"architectures"` field contains a model architecture listed below, then it should be supported in theory.

.. tip::
    The easiest way to check if your model is really supported at runtime is to run the program below:

    .. code-block:: python

        from vllm import LLM

30
31
        # For generative models (task=generate) only
        llm = LLM(model=..., task="generate")  # Name or path of your model
32
33
34
        output = llm.generate("Hello, my name is")
        print(output)

35
36
37
38
39
40
        # For pooling models (task={embed,classify,reward}) only
        llm = LLM(model=..., task="embed")  # Name or path of your model
        output = llm.encode("Hello, my name is")
        print(output)

    If vLLM successfully returns text (for generative models) or hidden states (for pooling models), it indicates that your model is supported.
41
42
43
44
45

Otherwise, please refer to :ref:`Adding a New Model <adding_a_new_model>` and :ref:`Enabling Multimodal Inputs <enabling_multimodal_inputs>` 
for instructions on how to implement your model in vLLM.
Alternatively, you can `open an issue on GitHub <https://github.com/vllm-project/vllm/issues/new/choose>`_ to request vLLM support.

46
47
ModelScope
++++++++++
48

49
To use models from `ModelScope <https://www.modelscope.cn>`_ instead of HuggingFace Hub, set an environment variable:
50

51
.. code-block:: shell
52

53
    $ export VLLM_USE_MODELSCOPE=True
54

55
And use with :code:`trust_remote_code=True`.
56

57
.. code-block:: python
58

59
60
61
    from vllm import LLM

    llm = LLM(model=..., revision=..., task=..., trust_remote_code=True)
62

63
64
65
    # For generative models (task=generate) only
    output = llm.generate("Hello, my name is")
    print(output)
66

67
68
69
70
71
72
73
74
75
76
77
78
79
80
    # For pooling models (task={embed,classify,reward}) only
    output = llm.encode("Hello, my name is")
    print(output)

List of Text-only Language Models
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Generative Models
+++++++++++++++++

See :ref:`this page <generative_models>` for more information on how to use generative models.

Text Generation (``--task generate``)
-------------------------------------
81

Woosuk Kwon's avatar
Woosuk Kwon committed
82
.. list-table::
83
  :widths: 25 25 50 5 5
Woosuk Kwon's avatar
Woosuk Kwon committed
84
85
86
87
  :header-rows: 1

  * - Architecture
    - Models
88
    - Example HF Models
89
    - :ref:`LoRA <lora>`
90
    - :ref:`PP <distributed_serving>`
91
  * - :code:`AquilaForCausalLM`
92
    - Aquila, Aquila2
93
    - :code:`BAAI/Aquila-7B`, :code:`BAAI/AquilaChat-7B`, etc.
94
    - ✅︎
95
    - ✅︎
96
97
98
99
  * - :code:`ArcticForCausalLM`
    - Arctic
    - :code:`Snowflake/snowflake-arctic-base`, :code:`Snowflake/snowflake-arctic-instruct`, etc.
    -
100
    - ✅︎
Zhuohan Li's avatar
Zhuohan Li committed
101
  * - :code:`BaiChuanForCausalLM`
102
    - Baichuan2, Baichuan
103
    - :code:`baichuan-inc/Baichuan2-13B-Chat`, :code:`baichuan-inc/Baichuan-7B`, etc.
Jee Li's avatar
Jee Li committed
104
    - ✅︎
105
    - ✅︎
106
107
108
109
  * - :code:`BloomForCausalLM`
    - BLOOM, BLOOMZ, BLOOMChat
    - :code:`bigscience/bloom`, :code:`bigscience/bloomz`, etc.
    -
110
    - ✅︎
111
112
113
114
115
  * - :code:`BartForConditionalGeneration`
    - BART
    - :code:`facebook/bart-base`, :code:`facebook/bart-large-cnn`, etc.
    - 
    - 
116
117
118
  * - :code:`ChatGLMModel`
    - ChatGLM
    - :code:`THUDM/chatglm2-6b`, :code:`THUDM/chatglm3-6b`, etc.
Jee Li's avatar
Jee Li committed
119
    - ✅︎
120
    - ✅︎
121
122
123
  * - :code:`CohereForCausalLM`
    - Command-R
    - :code:`CohereForAI/c4ai-command-r-v01`, etc.
124
125
    - ✅︎
    - ✅︎
126
127
128
  * - :code:`DbrxForCausalLM`
    - DBRX
    - :code:`databricks/dbrx-base`, :code:`databricks/dbrx-instruct`, etc.
129
    -
130
    - ✅︎
131
132
133
  * - :code:`DeciLMForCausalLM`
    - DeciLM
    - :code:`Deci/DeciLM-7B`, :code:`Deci/DeciLM-7B-instruct`, etc.
134
    -
135
    - ✅︎
136
137
138
139
  * - :code:`DeepseekForCausalLM`
    - DeepSeek
    - :code:`deepseek-ai/deepseek-llm-67b-base`, :code:`deepseek-ai/deepseek-llm-7b-chat` etc.
    - 
140
    - ✅︎
141
142
143
144
  * - :code:`DeepseekV2ForCausalLM`
    - DeepSeek-V2
    - :code:`deepseek-ai/DeepSeek-V2`, :code:`deepseek-ai/DeepSeek-V2-Chat` etc.
    - 
145
    - ✅︎
146
147
148
149
  * - :code:`ExaoneForCausalLM`
    - EXAONE-3
    - :code:`LGAI-EXAONE/EXAONE-3.0-7.8B-Instruct`, etc.
    - ✅︎
150
    - ✅︎
Zhuohan Li's avatar
Zhuohan Li committed
151
152
  * - :code:`FalconForCausalLM`
    - Falcon
153
    - :code:`tiiuae/falcon-7b`, :code:`tiiuae/falcon-40b`, :code:`tiiuae/falcon-rw-7b`, etc.
154
    -
155
    - ✅︎
156
157
158
159
  * - :code:`FalconMambaForCausalLM`
    - FalconMamba
    - :code:`tiiuae/falcon-mamba-7b`, :code:`tiiuae/falcon-mamba-7b-instruct`, etc.
    - ✅︎
160
    - ✅︎
161
162
163
  * - :code:`GemmaForCausalLM`
    - Gemma
    - :code:`google/gemma-2b`, :code:`google/gemma-7b`, etc.
164
    - ✅︎
165
    - ✅︎
Woosuk Kwon's avatar
Woosuk Kwon committed
166
167
168
169
  * - :code:`Gemma2ForCausalLM`
    - Gemma2
    - :code:`google/gemma-2-9b`, :code:`google/gemma-2-27b`, etc.
    - ✅︎
170
    - ✅︎
171
172
173
174
175
  * - :code:`GlmForCausalLM`
    - GLM-4
    - :code:`THUDM/glm-4-9b-chat-hf`, etc.
    - ✅︎
    - ✅︎
Woosuk Kwon's avatar
Woosuk Kwon committed
176
177
  * - :code:`GPT2LMHeadModel`
    - GPT-2
178
    - :code:`gpt2`, :code:`gpt2-xl`, etc.
179
    -
180
    - ✅︎
181
182
183
  * - :code:`GPTBigCodeForCausalLM`
    - StarCoder, SantaCoder, WizardCoder
    - :code:`bigcode/starcoder`, :code:`bigcode/gpt_bigcode-santacoder`, :code:`WizardLM/WizardCoder-15B-V1.0`, etc.
184
    - ✅︎
185
    - ✅︎
186
187
188
  * - :code:`GPTJForCausalLM`
    - GPT-J
    - :code:`EleutherAI/gpt-j-6b`, :code:`nomic-ai/gpt4all-j`, etc.
189
    -
190
    - ✅︎
Woosuk Kwon's avatar
Woosuk Kwon committed
191
192
  * - :code:`GPTNeoXForCausalLM`
    - GPT-NeoX, Pythia, OpenAssistant, Dolly V2, StableLM
193
    - :code:`EleutherAI/gpt-neox-20b`, :code:`EleutherAI/pythia-12b`, :code:`OpenAssistant/oasst-sft-4-pythia-12b-epoch-3.5`, :code:`databricks/dolly-v2-12b`, :code:`stabilityai/stablelm-tuned-alpha-7b`, etc.
194
    -
195
    - ✅︎
196
  * - :code:`GraniteForCausalLM`
197
198
    - Granite 3.0, PowerLM
    - :code:`ibm-granite/granite-3.0-2b-base`, :code:`ibm-granite/granite-3.0-8b-instruct`, :code:`ibm/PowerLM-3b`, etc.
199
    - ✅︎
200
    - ✅︎
201
  * - :code:`GraniteMoeForCausalLM`
202
203
    - Granite 3.0 MoE, PowerMoE
    - :code:`ibm-granite/granite-3.0-1b-a400m-base`, :code:`ibm-granite/granite-3.0-3b-a800m-instruct`, :code:`ibm/PowerMoE-3b`, etc.
204
    - ✅︎
205
    - ✅︎
206
207
208
  * - :code:`InternLMForCausalLM`
    - InternLM
    - :code:`internlm/internlm-7b`, :code:`internlm/internlm-chat-7b`, etc.
209
    - ✅︎
210
    - ✅︎
Fengzhe Zhou's avatar
Fengzhe Zhou committed
211
212
213
  * - :code:`InternLM2ForCausalLM`
    - InternLM2
    - :code:`internlm/internlm2-7b`, :code:`internlm/internlm2-chat-7b`, etc.
214
    - ✅︎
215
    - ✅︎
216
217
  * - :code:`JAISLMHeadModel`
    - Jais
218
    - :code:`inceptionai/jais-13b`, :code:`inceptionai/jais-13b-chat`, :code:`inceptionai/jais-30b-v3`, :code:`inceptionai/jais-30b-chat-v3`, etc.
219
    -
220
    - ✅︎
Mor Zusman's avatar
Mor Zusman committed
221
222
  * - :code:`JambaForCausalLM`
    - Jamba
223
    - :code:`ai21labs/AI21-Jamba-1.5-Large`, :code:`ai21labs/AI21-Jamba-1.5-Mini`, :code:`ai21labs/Jamba-v0.1`, etc.
Mor Zusman's avatar
Mor Zusman committed
224
    - ✅︎
225
    - ✅︎
Woosuk Kwon's avatar
Woosuk Kwon committed
226
  * - :code:`LlamaForCausalLM`
227
228
    - Llama 3.1, Llama 3, Llama 2, LLaMA, Yi
    - :code:`meta-llama/Meta-Llama-3.1-405B-Instruct`, :code:`meta-llama/Meta-Llama-3.1-70B`, :code:`meta-llama/Meta-Llama-3-70B-Instruct`, :code:`meta-llama/Llama-2-70b-hf`, :code:`01-ai/Yi-34B`, etc.
229
    - ✅︎
230
    - ✅︎
231
232
233
  * - :code:`MambaForCausalLM`
    - Mamba
    - :code:`state-spaces/mamba-130m-hf`, :code:`state-spaces/mamba-790m-hf`, :code:`state-spaces/mamba-2.8b-hf`, etc.
234
    -
235
    - ✅︎
ywfang's avatar
ywfang committed
236
237
  * - :code:`MiniCPMForCausalLM`
    - MiniCPM
238
    - :code:`openbmb/MiniCPM-2B-sft-bf16`, :code:`openbmb/MiniCPM-2B-dpo-bf16`, :code:`openbmb/MiniCPM-S-1B-sft`, etc.
239
240
    - ✅︎
    - ✅︎
ywfang's avatar
ywfang committed
241
242
243
  * - :code:`MiniCPM3ForCausalLM`
    - MiniCPM3
    - :code:`openbmb/MiniCPM3-4B`, etc.
244
245
    - ✅︎
    - ✅︎
246
247
248
  * - :code:`MistralForCausalLM`
    - Mistral, Mistral-Instruct
    - :code:`mistralai/Mistral-7B-v0.1`, :code:`mistralai/Mistral-7B-Instruct-v0.1`, etc.
249
    - ✅︎
250
    - ✅︎
Woosuk Kwon's avatar
Woosuk Kwon committed
251
252
  * - :code:`MixtralForCausalLM`
    - Mixtral-8x7B, Mixtral-8x7B-Instruct
253
    - :code:`mistralai/Mixtral-8x7B-v0.1`, :code:`mistralai/Mixtral-8x7B-Instruct-v0.1`, :code:`mistral-community/Mixtral-8x22B-v0.1`, etc.
254
    - ✅︎
255
    - ✅︎
Woosuk Kwon's avatar
Woosuk Kwon committed
256
  * - :code:`MPTForCausalLM`
257
258
    - MPT, MPT-Instruct, MPT-Chat, MPT-StoryWriter
    - :code:`mosaicml/mpt-7b`, :code:`mosaicml/mpt-7b-storywriter`, :code:`mosaicml/mpt-30b`, etc.
259
    -
260
    - ✅︎
261
262
263
264
  * - :code:`NemotronForCausalLM`
    - Nemotron-3, Nemotron-4, Minitron
    - :code:`nvidia/Minitron-8B-Base`, :code:`mgoin/Nemotron-4-340B-Base-hf-FP8`, etc.
    - ✅︎
265
    - ✅︎
Isotr0py's avatar
Isotr0py committed
266
267
  * - :code:`OLMoForCausalLM`
    - OLMo
268
    - :code:`allenai/OLMo-1B-hf`, :code:`allenai/OLMo-7B-hf`, etc.
269
    -
270
    - ✅︎
271
272
273
274
275
  * - :code:`OLMo2ForCausalLM`
    - OLMo2
    - :code:`allenai/OLMo2-7B-1124`, etc.
    -
    - ✅︎
276
277
278
279
280
  * - :code:`OLMoEForCausalLM`
    - OLMoE
    - :code:`allenai/OLMoE-1B-7B-0924`, :code:`allenai/OLMoE-1B-7B-0924-Instruct`, etc.
    - ✅︎
    - ✅︎
Woosuk Kwon's avatar
Woosuk Kwon committed
281
282
  * - :code:`OPTForCausalLM`
    - OPT, OPT-IML
283
    - :code:`facebook/opt-66b`, :code:`facebook/opt-iml-max-30b`, etc.
284
    -
285
    - ✅︎
张大成's avatar
张大成 committed
286
287
288
  * - :code:`OrionForCausalLM`
    - Orion
    - :code:`OrionStarAI/Orion-14B-Base`, :code:`OrionStarAI/Orion-14B-Chat`, etc.
289
    -
290
    - ✅︎
291
  * - :code:`PhiForCausalLM`
292
293
    - Phi
    - :code:`microsoft/phi-1_5`, :code:`microsoft/phi-2`, etc.
294
    - ✅︎
295
    - ✅︎
296
297
  * - :code:`Phi3ForCausalLM`
    - Phi-3
298
    - :code:`microsoft/Phi-3-mini-4k-instruct`, :code:`microsoft/Phi-3-mini-128k-instruct`, :code:`microsoft/Phi-3-medium-128k-instruct`, etc.
299
300
    - ✅︎
    - ✅︎
301
302
303
  * - :code:`Phi3SmallForCausalLM`
    - Phi-3-Small
    - :code:`microsoft/Phi-3-small-8k-instruct`, :code:`microsoft/Phi-3-small-128k-instruct`, etc.
304
    -
305
    - ✅︎
306
307
308
  * - :code:`PhiMoEForCausalLM`
    - Phi-3.5-MoE
    - :code:`microsoft/Phi-3.5-MoE-instruct`, etc.
309
310
    - ✅︎
    - ✅︎
311
312
313
314
  * - :code:`PersimmonForCausalLM`
    - Persimmon
    - :code:`adept/persimmon-8b-base`, :code:`adept/persimmon-8b-chat`, etc.
    - 
315
    - ✅︎
316
  * - :code:`QWenLMHeadModel`
317
318
    - Qwen
    - :code:`Qwen/Qwen-7B`, :code:`Qwen/Qwen-7B-Chat`, etc.
319
    - ✅︎
320
    - ✅︎
Junyang Lin's avatar
Junyang Lin committed
321
322
  * - :code:`Qwen2ForCausalLM`
    - Qwen2
323
    - :code:`Qwen/Qwen2-7B-Instruct`, :code:`Qwen/Qwen2-7B`, etc.
324
    - ✅︎
325
    - ✅︎
326
327
328
  * - :code:`Qwen2MoeForCausalLM`
    - Qwen2MoE
    - :code:`Qwen/Qwen1.5-MoE-A2.7B`, :code:`Qwen/Qwen1.5-MoE-A2.7B-Chat`, etc.
329
    -
330
    - ✅︎
331
  * - :code:`StableLmForCausalLM`
Hyunsung Lee's avatar
Hyunsung Lee committed
332
    - StableLM
333
    - :code:`stabilityai/stablelm-3b-4e1t`, :code:`stabilityai/stablelm-base-alpha-7b-v2`, etc.
334
    -
335
    - ✅︎
336
337
338
339
  * - :code:`Starcoder2ForCausalLM`
    - Starcoder2
    - :code:`bigcode/starcoder2-3b`, :code:`bigcode/starcoder2-7b`, :code:`bigcode/starcoder2-15b`, etc.
    -
340
    - ✅︎
341
  * - :code:`SolarForCausalLM`
342
    - Solar Pro
343
    - :code:`upstage/solar-pro-preview-instruct`, etc.
344
345
    - ✅︎
    - ✅︎
346
347
348
349
350
  * - :code:`TeleChat2ForCausalLM`
    - TeleChat2
    - :code:`TeleAI/TeleChat2-3B`, :code:`TeleAI/TeleChat2-7B`, :code:`TeleAI/TeleChat2-35B`, etc.
    - ✅︎
    - ✅︎
351
  * - :code:`XverseForCausalLM`
352
    - XVERSE
353
    - :code:`xverse/XVERSE-7B-Chat`, :code:`xverse/XVERSE-13B-Chat`, :code:`xverse/XVERSE-65B-Chat`, etc.
354
355
    - ✅︎
    - ✅︎
356

357
358
359
.. note::
    Currently, the ROCm version of vLLM supports Mistral and Mixtral only for context lengths up to 4096.

360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
Pooling Models
++++++++++++++

See :ref:`this page <pooling_models>` for more information on how to use pooling models.

.. important::
    Since some model architectures support both generative and pooling tasks,
    you should explicitly specify the task type to ensure that the model is used in pooling mode instead of generative mode.

Text Embedding (``--task embed``)
---------------------------------

Any text generation model can be converted into an embedding model by passing :code:`--task embed`.

.. note::
    To get the best results, you should use pooling models that are specifically trained as such.

The following table lists those that are tested in vLLM.
378
379
380
381
382
383
384

.. list-table::
  :widths: 25 25 50 5 5
  :header-rows: 1

  * - Architecture
    - Models
385
    - Example HF Models
386
387
    - :ref:`LoRA <lora>`
    - :ref:`PP <distributed_serving>`
388
389
390
391
392
  * - :code:`BertModel`
    - BERT-based
    - :code:`BAAI/bge-base-en-v1.5`, etc.
    - 
    - 
393
394
395
396
397
  * - :code:`Gemma2Model`
    - Gemma2-based
    - :code:`BAAI/bge-multilingual-gemma2`, etc.
    - 
    - ✅︎
398
399
  * - :code:`LlamaModel`, :code:`LlamaForCausalLM`, :code:`MistralModel`, etc.
    - Llama-based
400
    - :code:`intfloat/e5-mistral-7b-instruct`, etc.
401
    - ✅︎
402
    - ✅︎
403
404
  * - :code:`Qwen2Model`, :code:`Qwen2ForCausalLM`
    - Qwen2-based
405
    - :code:`ssmits/Qwen2-7B-Instruct-embed-base` (see note), :code:`Alibaba-NLP/gte-Qwen2-7B-instruct` (see note), etc.
406
407
    - ✅︎
    - ✅︎
408
409
410
411
412
413
414
415
416
417
  * - :code:`RobertaModel`, :code:`RobertaForMaskedLM`
    - RoBERTa-based
    - :code:`sentence-transformers/all-roberta-large-v1`, :code:`sentence-transformers/all-roberta-large-v1`, etc.
    - 
    - 
  * - :code:`XLMRobertaModel`
    - XLM-RoBERTa-based
    - :code:`intfloat/multilingual-e5-large`, etc.
    - 
    - 
418

419
420
421
422
.. note::
  :code:`ssmits/Qwen2-7B-Instruct-embed-base` has an improperly defined Sentence Transformers config.
  You should manually set mean pooling by passing :code:`--override-pooler-config '{"pooling_type": "MEAN"}'`.

423
424
.. note::
  Unlike base Qwen2, :code:`Alibaba-NLP/gte-Qwen2-7B-instruct` uses bi-directional attention.
Cyrus Leung's avatar
Cyrus Leung committed
425
  You can set :code:`--hf-overrides '{"is_causal": false}'` to change the attention mask accordingly.
426
427
428
429

  On the other hand, its 1.5B variant (:code:`Alibaba-NLP/gte-Qwen2-1.5B-instruct`) uses causal attention
  despite being described otherwise on its model card.

430
431
Reward Modeling (``--task reward``)
-----------------------------------
432
433
434
435
436
437
438

.. list-table::
  :widths: 25 25 50 5 5
  :header-rows: 1

  * - Architecture
    - Models
439
    - Example HF Models
440
441
    - :ref:`LoRA <lora>`
    - :ref:`PP <distributed_serving>`
442
443
444
445
446
  * - :code:`LlamaForCausalLM`
    - Llama-based
    - :code:`peiyi9979/math-shepherd-mistral-7b-prm`, etc.
    - ✅︎
    - ✅︎
447
448
449
  * - :code:`Qwen2ForRewardModel`
    - Qwen2-based
    - :code:`Qwen/Qwen2.5-Math-RM-72B`, etc.
450
    - ✅︎
451
452
    - ✅︎

453
454
455
456
.. important::
  For process-supervised reward models such as :code:`peiyi9979/math-shepherd-mistral-7b-prm`, the pooling config should be set explicitly,
  e.g.: :code:`--override-pooler-config '{"pooling_type": "STEP", "step_tag_id": 123, "returned_token_ids": [456, 789]}'`.

457
458
Classification (``--task classify``)
------------------------------------
459
460
461
462
463
464
465
466
467
468
469
470
471

.. list-table::
  :widths: 25 25 50 5 5
  :header-rows: 1

  * - Architecture
    - Models
    - Example HF Models
    - :ref:`LoRA <lora>`
    - :ref:`PP <distributed_serving>`
  * - :code:`Qwen2ForSequenceClassification`
    - Qwen2-based
    - :code:`jason9693/Qwen2.5-1.5B-apeach`, etc.
472
    - ✅︎
473
474
    - ✅︎

475
476
Sentence Pair Scoring (``--task score``)
----------------------------------------
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502

.. list-table::
  :widths: 25 25 50 5 5
  :header-rows: 1

  * - Architecture
    - Models
    - Example HF Models
    - :ref:`LoRA <lora>`
    - :ref:`PP <distributed_serving>`
  * - :code:`BertForSequenceClassification`
    - BERT-based
    - :code:`cross-encoder/ms-marco-MiniLM-L-6-v2`, etc.
    - 
    - 
  * - :code:`RobertaForSequenceClassification`
    - RoBERTa-based
    - :code:`cross-encoder/quora-roberta-base`, etc.
    - 
    - 
  * - :code:`XLMRobertaForSequenceClassification`
    - XLM-RoBERTa-based
    - :code:`BAAI/bge-reranker-v2-m3`, etc.
    - 
    - 

503
504
.. _supported_mm_models:

505
506
List of Multimodal Language Models
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
507
508
509
510
511
512
513

The following modalities are supported depending on the model:

- **T**\ ext
- **I**\ mage
- **V**\ ideo
- **A**\ udio
514

Cyrus Leung's avatar
Cyrus Leung committed
515
516
517
518
519
520
521
522
Any combination of modalities joined by :code:`+` are supported.

- e.g.: :code:`T + I` means that the model supports text-only, image-only, and text-with-image inputs.

On the other hand, modalities separated by :code:`/` are mutually exclusive.

- e.g.: :code:`T / I` means that the model supports text-only and image-only inputs, but not text-with-image inputs.

523
524
525
526
527
528
529
530
531
See :ref:`this page <multimodal_inputs>` on how to pass multi-modal inputs to the model.

Generative Models
+++++++++++++++++

See :ref:`this page <generative_models>` for more information on how to use generative models.

Text Generation (``--task generate``)
-------------------------------------
532

533
.. list-table::
534
  :widths: 25 25 15 20 5 5 5
535
536
537
538
  :header-rows: 1

  * - Architecture
    - Models
539
540
    - Inputs
    - Example HF Models
541
    - :ref:`LoRA <lora>`
542
    - :ref:`PP <distributed_serving>`
543
    - V1
544
545
546
547
548
549
  * - :code:`AriaForConditionalGeneration`
    - Aria
    - T + I
    - :code:`rhymes-ai/Aria`
    - 
    - ✅︎
550
    - 
551
552
  * - :code:`Blip2ForConditionalGeneration`
    - BLIP-2
553
    - T + I\ :sup:`E`
554
555
    - :code:`Salesforce/blip2-opt-2.7b`, :code:`Salesforce/blip2-opt-6.7b`, etc.
    -
556
    - ✅︎
557
    - 
558
559
  * - :code:`ChameleonForConditionalGeneration`
    - Chameleon
560
    - T + I
561
562
    - :code:`facebook/chameleon-7b` etc.
    - 
563
    - ✅︎
564
    - 
565
566
  * - :code:`FuyuForCausalLM`
    - Fuyu
567
    - T + I
568
569
    - :code:`adept/fuyu-8b` etc.
    - 
570
    - ✅︎
571
    - 
572
573
  * - :code:`ChatGLMModel`
    - GLM-4V
574
    - T + I
575
    - :code:`THUDM/glm-4v-9b` etc.
576
    - ✅︎
577
    - ✅︎
578
    - 
579
580
581
582
583
584
  * - :code:`H2OVLChatModel`
    - H2OVL
    - T + I\ :sup:`E+`
    - :code:`h2oai/h2ovl-mississippi-800m`, :code:`h2oai/h2ovl-mississippi-2b`, etc.
    - 
    - ✅︎
585
    - 
586
587
588
589
  * - :code:`Idefics3ForConditionalGeneration`
    - Idefics3
    - T + I
    - :code:`HuggingFaceM4/Idefics3-8B-Llama3` etc.
590
    - ✅︎
591
    -
592
    - 
593
  * - :code:`InternVLChatModel`
594
    - InternVL 2.5, Mono-InternVL, InternVL 2.0
595
    - T + I\ :sup:`E+`
596
    - :code:`OpenGVLab/InternVL2_5-4B`, :code:`OpenGVLab/Mono-InternVL-2B`, :code:`OpenGVLab/InternVL2-4B`, etc.
597
    - 
598
    - ✅︎
599
    - ✅︎
600
601
  * - :code:`LlavaForConditionalGeneration`
    - LLaVA-1.5
602
    - T + I\ :sup:`E+`
603
    - :code:`llava-hf/llava-1.5-7b-hf`, :code:`TIGER-Lab/Mantis-8B-siglip-llama3` (see note), etc.
604
    -
605
    - ✅︎
606
    - ✅︎
607
608
  * - :code:`LlavaNextForConditionalGeneration`
    - LLaVA-NeXT
609
    - T + I\ :sup:`E+`
610
611
    - :code:`llava-hf/llava-v1.6-mistral-7b-hf`, :code:`llava-hf/llava-v1.6-vicuna-7b-hf`, etc.
    -
612
    - ✅︎
613
    - 
614
615
  * - :code:`LlavaNextVideoForConditionalGeneration`
    - LLaVA-NeXT-Video
616
    - T + V
617
    - :code:`llava-hf/LLaVA-NeXT-Video-7B-hf`, etc.
618
    -
619
    - ✅︎
620
    - 
621
622
  * - :code:`LlavaOnevisionForConditionalGeneration`
    - LLaVA-Onevision
623
    - T + I\ :sup:`+` + V\ :sup:`+`
624
    - :code:`llava-hf/llava-onevision-qwen2-7b-ov-hf`, :code:`llava-hf/llava-onevision-qwen2-0.5b-ov-hf`, etc.
625
    -
626
    - ✅︎
627
    - 
628
629
  * - :code:`MiniCPMV`
    - MiniCPM-V
630
    - T + I\ :sup:`E+`
631
    - :code:`openbmb/MiniCPM-V-2` (see note), :code:`openbmb/MiniCPM-Llama3-V-2_5`, :code:`openbmb/MiniCPM-V-2_6`, etc.
632
633
    - ✅︎
    - ✅︎
634
    - 
635
636
  * - :code:`MllamaForConditionalGeneration`
    - Llama 3.2
637
    - T + I\ :sup:`+`
638
639
    - :code:`meta-llama/Llama-3.2-90B-Vision-Instruct`, :code:`meta-llama/Llama-3.2-11B-Vision`, etc.
    -
640
    -
641
    -
642
643
  * - :code:`MolmoForCausalLM`
    - Molmo
644
    - T + I
645
646
647
    - :code:`allenai/Molmo-7B-D-0924`, :code:`allenai/Molmo-72B-0924`, etc.
    -
    - ✅︎
648
    - ✅︎
649
650
  * - :code:`NVLM_D_Model`
    - NVLM-D 1.0
651
    - T + I\ :sup:`E+`
652
    - :code:`nvidia/NVLM-D-72B`, etc.
653
    - 
654
    - ✅︎
655
    - ✅︎
Roger Wang's avatar
Roger Wang committed
656
657
  * - :code:`PaliGemmaForConditionalGeneration`
    - PaliGemma
658
    - T + I\ :sup:`E`
Roger Wang's avatar
Roger Wang committed
659
660
    - :code:`google/paligemma-3b-pt-224`, :code:`google/paligemma-3b-mix-224`, etc.
    - 
661
    - ✅︎
662
    - 
663
  * - :code:`Phi3VForCausalLM`
664
    - Phi-3-Vision, Phi-3.5-Vision
665
    - T + I\ :sup:`E+`
666
    - :code:`microsoft/Phi-3-vision-128k-instruct`, :code:`microsoft/Phi-3.5-vision-instruct` etc.
667
    -
668
    - ✅︎
669
    - ✅︎
Patrick von Platen's avatar
Patrick von Platen committed
670
671
  * - :code:`PixtralForConditionalGeneration`
    - Pixtral
672
    - T + I\ :sup:`+`
673
    - :code:`mistralai/Pixtral-12B-2409`, :code:`mistral-community/pixtral-12b` etc.
Patrick von Platen's avatar
Patrick von Platen committed
674
    -
675
    - ✅︎
676
    - ✅︎
677
  * - :code:`QWenLMHeadModel`
678
    - Qwen-VL
679
    - T + I\ :sup:`E+`
680
    - :code:`Qwen/Qwen-VL`, :code:`Qwen/Qwen-VL-Chat`, etc.
681
    - ✅︎
682
    - ✅︎
683
    -
684
685
686
687
688
689
  * - :code:`Qwen2AudioForConditionalGeneration`
    - Qwen2-Audio
    - T + A\ :sup:`+`
    - :code:`Qwen/Qwen2-Audio-7B-Instruct`
    -
    - ✅︎
690
    - 
691
  * - :code:`Qwen2VLForConditionalGeneration`
692
    - Qwen2-VL
693
    - T + I\ :sup:`E+` + V\ :sup:`E+`
694
    - :code:`Qwen/Qwen2-VL-2B-Instruct`, :code:`Qwen/Qwen2-VL-7B-Instruct`, :code:`Qwen/Qwen2-VL-72B-Instruct`, etc.
695
    - ✅︎
696
    - ✅︎
697
    - 
698
  * - :code:`UltravoxModel`
699
    - Ultravox
700
    - T + A\ :sup:`E+`
701
    - :code:`fixie-ai/ultravox-v0_3`
702
    -
703
    - ✅︎
704
    - 
Woosuk Kwon's avatar
Woosuk Kwon committed
705

706
707
708
| :sup:`E` Pre-computed embeddings can be inputted for this modality.
| :sup:`+` Multiple items can be inputted per text prompt for this modality.

709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
.. important::
    To enable multiple multi-modal items per text prompt, you have to set :code:`limit_mm_per_prompt` (offline inference)
    or :code:`--limit-mm-per-prompt` (online inference). For example, to enable passing up to 4 images per text prompt:

    .. code-block:: python

        llm = LLM(
            model="Qwen/Qwen2-VL-7B-Instruct",
            limit_mm_per_prompt={"image": 4},
        )

    .. code-block:: bash

        vllm serve Qwen/Qwen2-VL-7B-Instruct --limit-mm-per-prompt image=4

724
.. note::
725
  vLLM currently only supports adding LoRA to the language backbone of multimodal models.
726

727
728
729
730
.. note::
  To use :code:`TIGER-Lab/Mantis-8B-siglip-llama3`, you have to install their GitHub repo (:code:`pip install git+https://github.com/TIGER-AI-Lab/Mantis.git`)
  and pass :code:`--hf_overrides '{"architectures": ["MantisForConditionalGeneration"]}'` when running vLLM.

Alphi's avatar
Alphi committed
731
.. note::
732
  The official :code:`openbmb/MiniCPM-V-2` doesn't work yet, so we need to use a fork (:code:`HwwwH/MiniCPM-V-2`) for now.
Alphi's avatar
Alphi committed
733
734
  For more details, please see: https://github.com/vllm-project/vllm/pull/4087#issuecomment-2250397630

735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
Pooling Models
++++++++++++++

See :ref:`this page <pooling_models>` for more information on how to use pooling models.

.. important::
    Since some model architectures support both generative and pooling tasks,
    you should explicitly specify the task type to ensure that the model is used in pooling mode instead of generative mode.

Text Embedding (``--task embed``)
---------------------------------

Any text generation model can be converted into an embedding model by passing :code:`--task embed`.

.. note::
    To get the best results, you should use pooling models that are specifically trained as such.

The following table lists those that are tested in vLLM.
753
754
755
756
757
758
759
760
761
762
763

.. list-table::
  :widths: 25 25 15 25 5 5
  :header-rows: 1

  * - Architecture
    - Models
    - Inputs
    - Example HF Models
    - :ref:`LoRA <lora>`
    - :ref:`PP <distributed_serving>`
Cyrus Leung's avatar
Cyrus Leung committed
764
765
766
767
768
769
  * - :code:`LlavaNextForConditionalGeneration`
    - LLaVA-NeXT-based
    - T / I
    - :code:`royokong/e5-v`
    - 
    - ✅︎
770
771
772
773
774
775
  * - :code:`Phi3VForCausalLM`
    - Phi-3-Vision-based
    - T + I
    - :code:`TIGER-Lab/VLM2Vec-Full`
    - 🚧
    - ✅︎
776
777
778
779
780
781
  * - :code:`Qwen2VLForConditionalGeneration`
    - Qwen2-VL-based
    - T + I
    - :code:`MrLight/dse-qwen2-2b-mrl-v1`
    - 
    - ✅︎
782

783
----
784

785
Model Support Policy
786
=====================
787
788
789
790
791
792
793

At vLLM, we are committed to facilitating the integration and support of third-party models within our ecosystem. Our approach is designed to balance the need for robustness and the practical limitations of supporting a wide range of models. Here’s how we manage third-party model support:

1. **Community-Driven Support**: We encourage community contributions for adding new models. When a user requests support for a new model, we welcome pull requests (PRs) from the community. These contributions are evaluated primarily on the sensibility of the output they generate, rather than strict consistency with existing implementations such as those in transformers. **Call for contribution:** PRs coming directly from model vendors are greatly appreciated!

2. **Best-Effort Consistency**: While we aim to maintain a level of consistency between the models implemented in vLLM and other frameworks like transformers, complete alignment is not always feasible. Factors like acceleration techniques and the use of low-precision computations can introduce discrepancies. Our commitment is to ensure that the implemented models are functional and produce sensible results.

794
795
796
.. tip::
  When comparing the output of :code:`model.generate` from HuggingFace Transformers with the output of :code:`llm.generate` from vLLM, note that the former reads the model's generation config file (i.e., `generation_config.json <https://github.com/huggingface/transformers/blob/19dabe96362803fb0a9ae7073d03533966598b17/src/transformers/generation/utils.py#L1945>`__) and applies the default parameters for generation, while the latter only uses the parameters passed to the function. Ensure all sampling parameters are identical when comparing outputs.

797
798
799
800
801
802
803
804
805
806
807
808
3. **Issue Resolution and Model Updates**: Users are encouraged to report any bugs or issues they encounter with third-party models. Proposed fixes should be submitted via PRs, with a clear explanation of the problem and the rationale behind the proposed solution. If a fix for one model impacts another, we rely on the community to highlight and address these cross-model dependencies. Note: for bugfix PRs, it is good etiquette to inform the original author to seek their feedback.

4. **Monitoring and Updates**: Users interested in specific models should monitor the commit history for those models (e.g., by tracking changes in the main/vllm/model_executor/models directory). This proactive approach helps users stay informed about updates and changes that may affect the models they use.

5. **Selective Focus**: Our resources are primarily directed towards models with significant user interest and impact. Models that are less frequently used may receive less attention, and we rely on the community to play a more active role in their upkeep and improvement.

Through this approach, vLLM fosters a collaborative environment where both the core development team and the broader community contribute to the robustness and diversity of the third-party models supported in our ecosystem.

Note that, as an inference engine, vLLM does not introduce new models. Therefore, all models supported by vLLM are third-party models in this regard.

We have the following levels of testing for models:

809
1. **Strict Consistency**: We compare the output of the model with the output of the model in the HuggingFace Transformers library under greedy decoding. This is the most stringent test. Please refer to `models tests <https://github.com/vllm-project/vllm/blob/main/tests/models>`_ for the models that have passed this test.
810
811
812
2. **Output Sensibility**: We check if the output of the model is sensible and coherent, by measuring the perplexity of the output and checking for any obvious errors. This is a less stringent test.
3. **Runtime Functionality**: We check if the model can be loaded and run without errors. This is the least stringent test. Please refer to `functionality tests <https://github.com/vllm-project/vllm/tree/main/tests>`_ and `examples <https://github.com/vllm-project/vllm/tree/main/examples>`_ for the models that have passed this test.
4. **Community Feedback**: We rely on the community to provide feedback on the models. If a model is broken or not working as expected, we encourage users to raise issues to report it or open pull requests to fix it. The rest of the models fall under this category.