supported_models.rst 21.5 KB
Newer Older
Woosuk Kwon's avatar
Woosuk Kwon committed
1
2
3
4
5
.. _supported_models:

Supported Models
================

6
7
vLLM supports a variety of generative and embedding models from `HuggingFace (HF) Transformers <https://huggingface.co/models>`_.
This page lists the model architectures that are currently supported by vLLM.
Woosuk Kwon's avatar
Woosuk Kwon committed
8
9
Alongside each architecture, we include some popular models that use it.

10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
For other models, you can check the :code:`config.json` file inside the model repository.
If the :code:`"architectures"` field contains a model architecture listed below, then it should be supported in theory.

.. tip::
    The easiest way to check if your model is really supported at runtime is to run the program below:

    .. code-block:: python

        from vllm import LLM

        llm = LLM(model=...)  # Name or path of your model
        output = llm.generate("Hello, my name is")
        print(output)

    If vLLM successfully generates text, it indicates that your model is supported.

Otherwise, please refer to :ref:`Adding a New Model <adding_a_new_model>` and :ref:`Enabling Multimodal Inputs <enabling_multimodal_inputs>` 
for instructions on how to implement your model in vLLM.
Alternatively, you can `open an issue on GitHub <https://github.com/vllm-project/vllm/issues/new/choose>`_ to request vLLM support.

.. note::
    To use models from `ModelScope <https://www.modelscope.cn>`_ instead of HuggingFace Hub, set an environment variable:

    .. code-block:: shell

       $ export VLLM_USE_MODELSCOPE=True

    And use with :code:`trust_remote_code=True`.

    .. code-block:: python

        from vllm import LLM

        llm = LLM(model=..., revision=..., trust_remote_code=True)  # Name or path of your model
        output = llm.generate("Hello, my name is")
        print(output)

47
48
49
50
51
Text-only Language Models
^^^^^^^^^^^^^^^^^^^^^^^^^

Text Generation
---------------
52

Woosuk Kwon's avatar
Woosuk Kwon committed
53
.. list-table::
54
  :widths: 25 25 50 5 5
Woosuk Kwon's avatar
Woosuk Kwon committed
55
56
57
58
  :header-rows: 1

  * - Architecture
    - Models
59
    - Example HF Models
60
    - :ref:`LoRA <lora>`
61
    - :ref:`PP <distributed_serving>`
62
  * - :code:`AquilaForCausalLM`
63
    - Aquila, Aquila2
64
    - :code:`BAAI/Aquila-7B`, :code:`BAAI/AquilaChat-7B`, etc.
65
    - ✅︎
66
    - ✅︎
67
68
69
70
  * - :code:`ArcticForCausalLM`
    - Arctic
    - :code:`Snowflake/snowflake-arctic-base`, :code:`Snowflake/snowflake-arctic-instruct`, etc.
    -
71
    - ✅︎
Zhuohan Li's avatar
Zhuohan Li committed
72
  * - :code:`BaiChuanForCausalLM`
73
    - Baichuan2, Baichuan
74
    - :code:`baichuan-inc/Baichuan2-13B-Chat`, :code:`baichuan-inc/Baichuan-7B`, etc.
Jee Li's avatar
Jee Li committed
75
    - ✅︎
76
    - ✅︎
77
78
79
80
  * - :code:`BloomForCausalLM`
    - BLOOM, BLOOMZ, BLOOMChat
    - :code:`bigscience/bloom`, :code:`bigscience/bloomz`, etc.
    -
81
    - ✅︎
82
83
84
85
86
  * - :code:`BartForConditionalGeneration`
    - BART
    - :code:`facebook/bart-base`, :code:`facebook/bart-large-cnn`, etc.
    - 
    - 
87
88
89
  * - :code:`ChatGLMModel`
    - ChatGLM
    - :code:`THUDM/chatglm2-6b`, :code:`THUDM/chatglm3-6b`, etc.
Jee Li's avatar
Jee Li committed
90
    - ✅︎
91
    - ✅︎
92
93
94
  * - :code:`CohereForCausalLM`
    - Command-R
    - :code:`CohereForAI/c4ai-command-r-v01`, etc.
95
96
    - ✅︎
    - ✅︎
97
98
99
  * - :code:`DbrxForCausalLM`
    - DBRX
    - :code:`databricks/dbrx-base`, :code:`databricks/dbrx-instruct`, etc.
100
    -
101
    - ✅︎
102
103
104
  * - :code:`DeciLMForCausalLM`
    - DeciLM
    - :code:`Deci/DeciLM-7B`, :code:`Deci/DeciLM-7B-instruct`, etc.
105
    -
106
    - ✅︎
107
108
109
110
  * - :code:`DeepseekForCausalLM`
    - DeepSeek
    - :code:`deepseek-ai/deepseek-llm-67b-base`, :code:`deepseek-ai/deepseek-llm-7b-chat` etc.
    - 
111
    - ✅︎
112
113
114
115
  * - :code:`DeepseekV2ForCausalLM`
    - DeepSeek-V2
    - :code:`deepseek-ai/DeepSeek-V2`, :code:`deepseek-ai/DeepSeek-V2-Chat` etc.
    - 
116
    - ✅︎
117
118
119
120
  * - :code:`ExaoneForCausalLM`
    - EXAONE-3
    - :code:`LGAI-EXAONE/EXAONE-3.0-7.8B-Instruct`, etc.
    - ✅︎
121
    - ✅︎
Zhuohan Li's avatar
Zhuohan Li committed
122
123
  * - :code:`FalconForCausalLM`
    - Falcon
124
    - :code:`tiiuae/falcon-7b`, :code:`tiiuae/falcon-40b`, :code:`tiiuae/falcon-rw-7b`, etc.
125
    -
126
    - ✅︎
127
128
129
130
131
  * - :code:`FalconMambaForCausalLM`
    - FalconMamba
    - :code:`tiiuae/falcon-mamba-7b`, :code:`tiiuae/falcon-mamba-7b-instruct`, etc.
    - ✅︎
    -  
132
133
134
  * - :code:`GemmaForCausalLM`
    - Gemma
    - :code:`google/gemma-2b`, :code:`google/gemma-7b`, etc.
135
    - ✅︎
136
    - ✅︎
Woosuk Kwon's avatar
Woosuk Kwon committed
137
138
139
140
  * - :code:`Gemma2ForCausalLM`
    - Gemma2
    - :code:`google/gemma-2-9b`, :code:`google/gemma-2-27b`, etc.
    - ✅︎
141
    - ✅︎
Woosuk Kwon's avatar
Woosuk Kwon committed
142
143
  * - :code:`GPT2LMHeadModel`
    - GPT-2
144
    - :code:`gpt2`, :code:`gpt2-xl`, etc.
145
    -
146
    - ✅︎
147
148
149
  * - :code:`GPTBigCodeForCausalLM`
    - StarCoder, SantaCoder, WizardCoder
    - :code:`bigcode/starcoder`, :code:`bigcode/gpt_bigcode-santacoder`, :code:`WizardLM/WizardCoder-15B-V1.0`, etc.
150
    - ✅︎
151
    - ✅︎
152
153
154
  * - :code:`GPTJForCausalLM`
    - GPT-J
    - :code:`EleutherAI/gpt-j-6b`, :code:`nomic-ai/gpt4all-j`, etc.
155
    -
156
    - ✅︎
Woosuk Kwon's avatar
Woosuk Kwon committed
157
158
  * - :code:`GPTNeoXForCausalLM`
    - GPT-NeoX, Pythia, OpenAssistant, Dolly V2, StableLM
159
    - :code:`EleutherAI/gpt-neox-20b`, :code:`EleutherAI/pythia-12b`, :code:`OpenAssistant/oasst-sft-4-pythia-12b-epoch-3.5`, :code:`databricks/dolly-v2-12b`, :code:`stabilityai/stablelm-tuned-alpha-7b`, etc.
160
    -
161
    - ✅︎
162
  * - :code:`GraniteForCausalLM`
163
164
    - Granite 3.0, PowerLM
    - :code:`ibm-granite/granite-3.0-2b-base`, :code:`ibm-granite/granite-3.0-8b-instruct`, :code:`ibm/PowerLM-3b`, etc.
165
    - ✅︎
166
    - ✅︎
167
  * - :code:`GraniteMoeForCausalLM`
168
169
    - Granite 3.0 MoE, PowerMoE
    - :code:`ibm-granite/granite-3.0-1b-a400m-base`, :code:`ibm-granite/granite-3.0-3b-a800m-instruct`, :code:`ibm/PowerMoE-3b`, etc.
170
    - ✅︎
171
    - ✅︎
172
173
174
  * - :code:`InternLMForCausalLM`
    - InternLM
    - :code:`internlm/internlm-7b`, :code:`internlm/internlm-chat-7b`, etc.
175
    - ✅︎
176
    - ✅︎
Fengzhe Zhou's avatar
Fengzhe Zhou committed
177
178
179
  * - :code:`InternLM2ForCausalLM`
    - InternLM2
    - :code:`internlm/internlm2-7b`, :code:`internlm/internlm2-chat-7b`, etc.
180
    -
181
    - ✅︎
182
183
  * - :code:`JAISLMHeadModel`
    - Jais
184
    - :code:`inceptionai/jais-13b`, :code:`inceptionai/jais-13b-chat`, :code:`inceptionai/jais-30b-v3`, :code:`inceptionai/jais-30b-chat-v3`, etc.
185
    -
186
    - ✅︎
Mor Zusman's avatar
Mor Zusman committed
187
188
  * - :code:`JambaForCausalLM`
    - Jamba
189
    - :code:`ai21labs/AI21-Jamba-1.5-Large`, :code:`ai21labs/AI21-Jamba-1.5-Mini`, :code:`ai21labs/Jamba-v0.1`, etc.
Mor Zusman's avatar
Mor Zusman committed
190
    - ✅︎
191
    - 
Woosuk Kwon's avatar
Woosuk Kwon committed
192
  * - :code:`LlamaForCausalLM`
193
194
    - Llama 3.1, Llama 3, Llama 2, LLaMA, Yi
    - :code:`meta-llama/Meta-Llama-3.1-405B-Instruct`, :code:`meta-llama/Meta-Llama-3.1-70B`, :code:`meta-llama/Meta-Llama-3-70B-Instruct`, :code:`meta-llama/Llama-2-70b-hf`, :code:`01-ai/Yi-34B`, etc.
195
    - ✅︎
196
    - ✅︎
197
198
199
  * - :code:`MambaForCausalLM`
    - Mamba
    - :code:`state-spaces/mamba-130m-hf`, :code:`state-spaces/mamba-790m-hf`, :code:`state-spaces/mamba-2.8b-hf`, etc.
200
    -
201
    -
ywfang's avatar
ywfang committed
202
203
  * - :code:`MiniCPMForCausalLM`
    - MiniCPM
204
    - :code:`openbmb/MiniCPM-2B-sft-bf16`, :code:`openbmb/MiniCPM-2B-dpo-bf16`, :code:`openbmb/MiniCPM-S-1B-sft`, etc.
205
206
    - ✅︎
    - ✅︎
ywfang's avatar
ywfang committed
207
208
209
  * - :code:`MiniCPM3ForCausalLM`
    - MiniCPM3
    - :code:`openbmb/MiniCPM3-4B`, etc.
210
211
    - ✅︎
    - ✅︎
212
213
214
  * - :code:`MistralForCausalLM`
    - Mistral, Mistral-Instruct
    - :code:`mistralai/Mistral-7B-v0.1`, :code:`mistralai/Mistral-7B-Instruct-v0.1`, etc.
215
    - ✅︎
216
    - ✅︎
Woosuk Kwon's avatar
Woosuk Kwon committed
217
218
  * - :code:`MixtralForCausalLM`
    - Mixtral-8x7B, Mixtral-8x7B-Instruct
219
    - :code:`mistralai/Mixtral-8x7B-v0.1`, :code:`mistralai/Mixtral-8x7B-Instruct-v0.1`, :code:`mistral-community/Mixtral-8x22B-v0.1`, etc.
220
    - ✅︎
221
    - ✅︎
Woosuk Kwon's avatar
Woosuk Kwon committed
222
  * - :code:`MPTForCausalLM`
223
224
    - MPT, MPT-Instruct, MPT-Chat, MPT-StoryWriter
    - :code:`mosaicml/mpt-7b`, :code:`mosaicml/mpt-7b-storywriter`, :code:`mosaicml/mpt-30b`, etc.
225
    -
226
    - ✅︎
227
228
229
230
  * - :code:`NemotronForCausalLM`
    - Nemotron-3, Nemotron-4, Minitron
    - :code:`nvidia/Minitron-8B-Base`, :code:`mgoin/Nemotron-4-340B-Base-hf-FP8`, etc.
    - ✅︎
231
    - ✅︎
Isotr0py's avatar
Isotr0py committed
232
233
  * - :code:`OLMoForCausalLM`
    - OLMo
234
    - :code:`allenai/OLMo-1B-hf`, :code:`allenai/OLMo-7B-hf`, etc.
235
    -
236
237
238
239
240
241
    - ✅︎
  * - :code:`OLMoEForCausalLM`
    - OLMoE
    - :code:`allenai/OLMoE-1B-7B-0924`, :code:`allenai/OLMoE-1B-7B-0924-Instruct`, etc.
    - ✅︎
    - ✅︎
Woosuk Kwon's avatar
Woosuk Kwon committed
242
243
  * - :code:`OPTForCausalLM`
    - OPT, OPT-IML
244
    - :code:`facebook/opt-66b`, :code:`facebook/opt-iml-max-30b`, etc.
245
    -
246
    - ✅︎
张大成's avatar
张大成 committed
247
248
249
  * - :code:`OrionForCausalLM`
    - Orion
    - :code:`OrionStarAI/Orion-14B-Base`, :code:`OrionStarAI/Orion-14B-Chat`, etc.
250
    -
251
    - ✅︎
252
  * - :code:`PhiForCausalLM`
253
254
    - Phi
    - :code:`microsoft/phi-1_5`, :code:`microsoft/phi-2`, etc.
255
    - ✅︎
256
    - ✅︎
257
258
  * - :code:`Phi3ForCausalLM`
    - Phi-3
259
    - :code:`microsoft/Phi-3-mini-4k-instruct`, :code:`microsoft/Phi-3-mini-128k-instruct`, :code:`microsoft/Phi-3-medium-128k-instruct`, etc.
260
261
    - ✅︎
    - ✅︎
262
263
264
  * - :code:`Phi3SmallForCausalLM`
    - Phi-3-Small
    - :code:`microsoft/Phi-3-small-8k-instruct`, :code:`microsoft/Phi-3-small-128k-instruct`, etc.
265
    -
266
    - ✅︎
267
268
269
  * - :code:`PhiMoEForCausalLM`
    - Phi-3.5-MoE
    - :code:`microsoft/Phi-3.5-MoE-instruct`, etc.
270
271
    - ✅︎
    - ✅︎
272
273
274
275
  * - :code:`PersimmonForCausalLM`
    - Persimmon
    - :code:`adept/persimmon-8b-base`, :code:`adept/persimmon-8b-chat`, etc.
    - 
276
    - ✅︎
277
  * - :code:`QWenLMHeadModel`
278
279
    - Qwen
    - :code:`Qwen/Qwen-7B`, :code:`Qwen/Qwen-7B-Chat`, etc.
280
    - ✅︎
281
    - ✅︎
Junyang Lin's avatar
Junyang Lin committed
282
283
  * - :code:`Qwen2ForCausalLM`
    - Qwen2
284
    - :code:`Qwen/Qwen2-7B-Instruct`, :code:`Qwen/Qwen2-7B`, etc.
285
    - ✅︎
286
    - ✅︎
287
288
289
  * - :code:`Qwen2MoeForCausalLM`
    - Qwen2MoE
    - :code:`Qwen/Qwen1.5-MoE-A2.7B`, :code:`Qwen/Qwen1.5-MoE-A2.7B-Chat`, etc.
290
    -
291
    - ✅︎
292
  * - :code:`StableLmForCausalLM`
Hyunsung Lee's avatar
Hyunsung Lee committed
293
    - StableLM
294
    - :code:`stabilityai/stablelm-3b-4e1t`, :code:`stabilityai/stablelm-base-alpha-7b-v2`, etc.
295
    -
296
    - ✅︎
297
298
299
300
  * - :code:`Starcoder2ForCausalLM`
    - Starcoder2
    - :code:`bigcode/starcoder2-3b`, :code:`bigcode/starcoder2-7b`, :code:`bigcode/starcoder2-15b`, etc.
    -
301
    - ✅︎
302
  * - :code:`SolarForCausalLM`
303
    - Solar Pro
304
    - :code:`upstage/solar-pro-preview-instruct`, etc.
305
306
    - ✅︎
    - ✅︎
307
  * - :code:`XverseForCausalLM`
308
    - XVERSE
309
    - :code:`xverse/XVERSE-7B-Chat`, :code:`xverse/XVERSE-13B-Chat`, :code:`xverse/XVERSE-65B-Chat`, etc.
310
311
    - ✅︎
    - ✅︎
312

313
314
315
.. note::
    Currently, the ROCm version of vLLM supports Mistral and Mixtral only for context lengths up to 4096.

316
317
318
319
320
321
322
323
324
Text Embedding
--------------

.. list-table::
  :widths: 25 25 50 5 5
  :header-rows: 1

  * - Architecture
    - Models
325
    - Example HF Models
326
327
328
329
330
331
332
    - :ref:`LoRA <lora>`
    - :ref:`PP <distributed_serving>`
  * - :code:`Gemma2Model`
    - Gemma2-based
    - :code:`BAAI/bge-multilingual-gemma2`, etc.
    - 
    - ✅︎
333
334
  * - :code:`LlamaModel`, :code:`LlamaForCausalLM`, :code:`MistralModel`, etc.
    - Llama-based
335
    - :code:`intfloat/e5-mistral-7b-instruct`, etc.
336
    - ✅︎
337
    - ✅︎
338
339
340
341
342
  * - :code:`Qwen2Model`, :code:`Qwen2ForCausalLM`
    - Qwen2-based
    - :code:`ssmits/Qwen2-7B-Instruct-embed-base`, :code:`Alibaba-NLP/gte-Qwen2-1.5B-instruct`, etc.
    - ✅︎
    - ✅︎
343

344
345
346
347
.. important::
  Some model architectures support both generation and embedding tasks.
  In this case, you have to pass :code:`--task embedding` to run the model in embedding mode.

348
349
350
351
352
353
354
355
356
Reward Modeling
---------------

.. list-table::
  :widths: 25 25 50 5 5
  :header-rows: 1

  * - Architecture
    - Models
357
    - Example HF Models
358
359
360
361
362
    - :ref:`LoRA <lora>`
    - :ref:`PP <distributed_serving>`
  * - :code:`Qwen2ForRewardModel`
    - Qwen2-based
    - :code:`Qwen/Qwen2.5-Math-RM-72B`, etc.
363
    - ✅︎
364
365
366
367
    - ✅︎

.. note::
    As an interim measure, these models are supported via Embeddings API. See `this RFC <https://github.com/vllm-project/vllm/issues/8967>`_ for upcoming changes.
368

369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
Classification
---------------

.. list-table::
  :widths: 25 25 50 5 5
  :header-rows: 1

  * - Architecture
    - Models
    - Example HF Models
    - :ref:`LoRA <lora>`
    - :ref:`PP <distributed_serving>`
  * - :code:`Qwen2ForSequenceClassification`
    - Qwen2-based
    - :code:`jason9693/Qwen2.5-1.5B-apeach`, etc.
384
    - ✅︎
385
386
387
388
389
390
    - ✅︎

.. note::
    As an interim measure, these models are supported via Embeddings API. It will be supported via Classification API in the future (no reference APIs exist now).


391
Multimodal Language Models
392
393
394
395
396
397
398
399
^^^^^^^^^^^^^^^^^^^^^^^^^^

The following modalities are supported depending on the model:

- **T**\ ext
- **I**\ mage
- **V**\ ideo
- **A**\ udio
400

Cyrus Leung's avatar
Cyrus Leung committed
401
402
403
404
405
406
407
408
Any combination of modalities joined by :code:`+` are supported.

- e.g.: :code:`T + I` means that the model supports text-only, image-only, and text-with-image inputs.

On the other hand, modalities separated by :code:`/` are mutually exclusive.

- e.g.: :code:`T / I` means that the model supports text-only and image-only inputs, but not text-with-image inputs.

409
410
.. _supported_vlms:

411
412
413
Text Generation
---------------

414
.. list-table::
415
  :widths: 25 25 15 25 5 5
416
417
418
419
  :header-rows: 1

  * - Architecture
    - Models
420
421
    - Inputs
    - Example HF Models
422
    - :ref:`LoRA <lora>`
423
    - :ref:`PP <distributed_serving>`
424
425
  * - :code:`Blip2ForConditionalGeneration`
    - BLIP-2
426
    - T + I\ :sup:`E`
427
428
    - :code:`Salesforce/blip2-opt-2.7b`, :code:`Salesforce/blip2-opt-6.7b`, etc.
    -
429
    - ✅︎
430
431
  * - :code:`ChameleonForConditionalGeneration`
    - Chameleon
432
    - T + I
433
434
    - :code:`facebook/chameleon-7b` etc.
    - 
435
    - ✅︎
436
437
  * - :code:`FuyuForCausalLM`
    - Fuyu
438
    - T + I
439
440
    - :code:`adept/fuyu-8b` etc.
    - 
441
    - ✅︎
442
443
  * - :code:`ChatGLMModel`
    - GLM-4V
444
    - T + I
445
446
447
    - :code:`THUDM/glm-4v-9b` etc.
    - 
    - ✅︎
448
449
450
451
452
453
  * - :code:`H2OVLChatModel`
    - H2OVL
    - T + I\ :sup:`E+`
    - :code:`h2oai/h2ovl-mississippi-800m`, :code:`h2oai/h2ovl-mississippi-2b`, etc.
    - 
    - ✅︎
454
455
456
457
  * - :code:`Idefics3ForConditionalGeneration`
    - Idefics3
    - T + I
    - :code:`HuggingFaceM4/Idefics3-8B-Llama3` etc.
458
    - ✅︎
459
    - 
460
461
  * - :code:`InternVLChatModel`
    - InternVL2
462
    - T + I\ :sup:`E+`
463
    - :code:`OpenGVLab/Mono-InternVL-2B`, :code:`OpenGVLab/InternVL2-4B`, :code:`OpenGVLab/InternVL2-8B`, etc.
464
    - 
465
    - ✅︎
466
467
  * - :code:`LlavaForConditionalGeneration`
    - LLaVA-1.5
468
    - T + I\ :sup:`E+`
469
470
    - :code:`llava-hf/llava-1.5-7b-hf`, :code:`llava-hf/llava-1.5-13b-hf`, etc.
    -
471
    - ✅︎
472
473
  * - :code:`LlavaNextForConditionalGeneration`
    - LLaVA-NeXT
474
    - T + I\ :sup:`E+`
475
476
    - :code:`llava-hf/llava-v1.6-mistral-7b-hf`, :code:`llava-hf/llava-v1.6-vicuna-7b-hf`, etc.
    -
477
    - ✅︎
478
479
  * - :code:`LlavaNextVideoForConditionalGeneration`
    - LLaVA-NeXT-Video
480
    - T + V
481
    - :code:`llava-hf/LLaVA-NeXT-Video-7B-hf`, etc.
482
    -
483
    - ✅︎
484
485
  * - :code:`LlavaOnevisionForConditionalGeneration`
    - LLaVA-Onevision
486
    - T + I\ :sup:`+` + V\ :sup:`+`
487
    - :code:`llava-hf/llava-onevision-qwen2-7b-ov-hf`, :code:`llava-hf/llava-onevision-qwen2-0.5b-ov-hf`, etc.
488
    -
489
    - ✅︎
490
491
  * - :code:`MiniCPMV`
    - MiniCPM-V
492
    - T + I\ :sup:`E+`
493
    - :code:`openbmb/MiniCPM-V-2` (see note), :code:`openbmb/MiniCPM-Llama3-V-2_5`, :code:`openbmb/MiniCPM-V-2_6`, etc.
494
495
    - ✅︎
    - ✅︎
496
497
  * - :code:`MllamaForConditionalGeneration`
    - Llama 3.2
498
    - T + I\ :sup:`+`
499
500
    - :code:`meta-llama/Llama-3.2-90B-Vision-Instruct`, :code:`meta-llama/Llama-3.2-11B-Vision`, etc.
    -
501
    -
502
503
  * - :code:`MolmoForCausalLM`
    - Molmo
504
    - T + I
505
506
507
    - :code:`allenai/Molmo-7B-D-0924`, :code:`allenai/Molmo-72B-0924`, etc.
    -
    - ✅︎
508
509
  * - :code:`NVLM_D_Model`
    - NVLM-D 1.0
510
    - T + I\ :sup:`E+`
511
    - :code:`nvidia/NVLM-D-72B`, etc.
512
    - 
513
    - ✅︎
Roger Wang's avatar
Roger Wang committed
514
515
  * - :code:`PaliGemmaForConditionalGeneration`
    - PaliGemma
516
    - T + I\ :sup:`E`
Roger Wang's avatar
Roger Wang committed
517
518
    - :code:`google/paligemma-3b-pt-224`, :code:`google/paligemma-3b-mix-224`, etc.
    - 
519
    - ✅︎
520
  * - :code:`Phi3VForCausalLM`
521
    - Phi-3-Vision, Phi-3.5-Vision
522
    - T + I\ :sup:`E+`
523
    - :code:`microsoft/Phi-3-vision-128k-instruct`, :code:`microsoft/Phi-3.5-vision-instruct` etc.
524
    -
525
    - ✅︎
Patrick von Platen's avatar
Patrick von Platen committed
526
527
  * - :code:`PixtralForConditionalGeneration`
    - Pixtral
528
    - T + I\ :sup:`+`
529
    - :code:`mistralai/Pixtral-12B-2409`, :code:`mistral-community/pixtral-12b` etc.
Patrick von Platen's avatar
Patrick von Platen committed
530
    -
531
    - ✅︎
532
  * - :code:`QWenLMHeadModel`
533
    - Qwen-VL
534
    - T + I\ :sup:`E+`
535
    - :code:`Qwen/Qwen-VL`, :code:`Qwen/Qwen-VL-Chat`, etc.
536
    - ✅︎
537
    - ✅︎
538
539
540
541
542
543
  * - :code:`Qwen2AudioForConditionalGeneration`
    - Qwen2-Audio
    - T + A\ :sup:`+`
    - :code:`Qwen/Qwen2-Audio-7B-Instruct`
    -
    - ✅︎
544
  * - :code:`Qwen2VLForConditionalGeneration`
545
    - Qwen2-VL
546
    - T + I\ :sup:`E+` + V\ :sup:`E+`
547
    - :code:`Qwen/Qwen2-VL-2B-Instruct`, :code:`Qwen/Qwen2-VL-7B-Instruct`, :code:`Qwen/Qwen2-VL-72B-Instruct`, etc.
548
    - ✅︎
549
    - ✅︎
550
  * - :code:`UltravoxModel`
551
    - Ultravox
552
    - T + A\ :sup:`E+`
553
    - :code:`fixie-ai/ultravox-v0_3`
554
    -
555
    - ✅︎
Woosuk Kwon's avatar
Woosuk Kwon committed
556

557
558
559
| :sup:`E` Pre-computed embeddings can be inputted for this modality.
| :sup:`+` Multiple items can be inputted per text prompt for this modality.

560
561
562
.. note::
  vLLM currently only supports adding LoRA to the language backbone of multimodal models.               

Alphi's avatar
Alphi committed
563
564
565
566
.. note::
  For :code:`openbmb/MiniCPM-V-2`, the official repo doesn't work yet, so we need to use a fork (:code:`HwwwH/MiniCPM-V-2`) for now.
  For more details, please see: https://github.com/vllm-project/vllm/pull/4087#issuecomment-2250397630

567
568
569
570
571
572
573
574
575
576
577
578
579
Multimodal Embedding
--------------------

.. list-table::
  :widths: 25 25 15 25 5 5
  :header-rows: 1

  * - Architecture
    - Models
    - Inputs
    - Example HF Models
    - :ref:`LoRA <lora>`
    - :ref:`PP <distributed_serving>`
Cyrus Leung's avatar
Cyrus Leung committed
580
581
582
583
584
585
  * - :code:`LlavaNextForConditionalGeneration`
    - LLaVA-NeXT-based
    - T / I
    - :code:`royokong/e5-v`
    - 
    - ✅︎
586
587
588
589
590
591
  * - :code:`Phi3VForCausalLM`
    - Phi-3-Vision-based
    - T + I
    - :code:`TIGER-Lab/VLM2Vec-Full`
    - 🚧
    - ✅︎
592
593
594
595
596
597
  * - :code:`Qwen2VLForConditionalGeneration`
    - Qwen2-VL-based
    - T + I
    - :code:`MrLight/dse-qwen2-2b-mrl-v1`
    - 
    - ✅︎
598

599
600
601
602
.. important::
  Some model architectures support both generation and embedding tasks.
  In this case, you have to pass :code:`--task embedding` to run the model in embedding mode.

603
Model Support Policy
604
=====================
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623

At vLLM, we are committed to facilitating the integration and support of third-party models within our ecosystem. Our approach is designed to balance the need for robustness and the practical limitations of supporting a wide range of models. Here’s how we manage third-party model support:

1. **Community-Driven Support**: We encourage community contributions for adding new models. When a user requests support for a new model, we welcome pull requests (PRs) from the community. These contributions are evaluated primarily on the sensibility of the output they generate, rather than strict consistency with existing implementations such as those in transformers. **Call for contribution:** PRs coming directly from model vendors are greatly appreciated!

2. **Best-Effort Consistency**: While we aim to maintain a level of consistency between the models implemented in vLLM and other frameworks like transformers, complete alignment is not always feasible. Factors like acceleration techniques and the use of low-precision computations can introduce discrepancies. Our commitment is to ensure that the implemented models are functional and produce sensible results.

3. **Issue Resolution and Model Updates**: Users are encouraged to report any bugs or issues they encounter with third-party models. Proposed fixes should be submitted via PRs, with a clear explanation of the problem and the rationale behind the proposed solution. If a fix for one model impacts another, we rely on the community to highlight and address these cross-model dependencies. Note: for bugfix PRs, it is good etiquette to inform the original author to seek their feedback.

4. **Monitoring and Updates**: Users interested in specific models should monitor the commit history for those models (e.g., by tracking changes in the main/vllm/model_executor/models directory). This proactive approach helps users stay informed about updates and changes that may affect the models they use.

5. **Selective Focus**: Our resources are primarily directed towards models with significant user interest and impact. Models that are less frequently used may receive less attention, and we rely on the community to play a more active role in their upkeep and improvement.

Through this approach, vLLM fosters a collaborative environment where both the core development team and the broader community contribute to the robustness and diversity of the third-party models supported in our ecosystem.

Note that, as an inference engine, vLLM does not introduce new models. Therefore, all models supported by vLLM are third-party models in this regard.

We have the following levels of testing for models:

624
1. **Strict Consistency**: We compare the output of the model with the output of the model in the HuggingFace Transformers library under greedy decoding. This is the most stringent test. Please refer to `models tests <https://github.com/vllm-project/vllm/blob/main/tests/models>`_ for the models that have passed this test.
625
626
627
2. **Output Sensibility**: We check if the output of the model is sensible and coherent, by measuring the perplexity of the output and checking for any obvious errors. This is a less stringent test.
3. **Runtime Functionality**: We check if the model can be loaded and run without errors. This is the least stringent test. Please refer to `functionality tests <https://github.com/vllm-project/vllm/tree/main/tests>`_ and `examples <https://github.com/vllm-project/vllm/tree/main/examples>`_ for the models that have passed this test.
4. **Community Feedback**: We rely on the community to provide feedback on the models. If a model is broken or not working as expected, we encourage users to raise issues to report it or open pull requests to fix it. The rest of the models fall under this category.