vLLM is a fast and easy-to-use library for LLM inference and serving.
pip install torch*(下载的torch的whl包)
pip install setuptools wheel
Originally developed in the [Sky Computing Lab](https://sky.cs.berkeley.edu) at UC Berkeley, vLLM has evolved into a community-driven project with contributions from both academia and industry.
vLLM is fast with:
- State-of-the-art serving throughput
- Efficient management of attention key and value memory with [**PagedAttention**](https://blog.vllm.ai/2023/06/20/vllm.html)
- Continuous batching of incoming requests
- Fast model execution with CUDA/HIP graph
- Quantizations: [GPTQ](https://arxiv.org/abs/2210.17323), [AWQ](https://arxiv.org/abs/2306.00978), [AutoRound](https://arxiv.org/abs/2309.05516), INT4, INT8, and FP8
- Optimized CUDA kernels, including integration with FlashAttention and FlashInfer
- Speculative decoding
- Chunked prefill
vLLM is flexible and easy to use with:
- Seamless integration with popular Hugging Face models
- High-throughput serving with various decoding algorithms, including *parallel sampling*, *beam search*, and more
- Tensor, pipeline, data and expert parallelism support for distributed inference
- Streaming outputs
- OpenAI-compatible API server
- Support for NVIDIA GPUs, AMD CPUs and GPUs, Intel CPUs and GPUs, PowerPC CPUs, Arm CPUs, and TPU. Additionally, support for diverse hardware plugins such as Intel Gaudi, IBM Spyre and Huawei Ascend.
- Prefix caching support
- Multi-LoRA support
vLLM seamlessly supports most popular open-source models on HuggingFace, including:
- Transformer-like LLMs (e.g., Llama)
- Mixture-of-Expert LLMs (e.g., Mixtral, Deepseek-V2 and V3)
- Embedding Models (e.g., E5-Mistral)
- Multi-modal LLMs (e.g., LLaVA)
Find the full list of supported models [here](https://docs.vllm.ai/en/latest/models/supported_models.html).
## Getting Started
Install vLLM with `pip` or [from source](https://docs.vllm.ai/en/latest/getting_started/installation/gpu/index.html#build-wheel-from-source):
```bash
pip install vllm
```
```
Visit our [documentation](https://docs.vllm.ai/en/latest/) to learn more.
title={Efficient Memory Management for Large Language Model Serving with PagedAttention},
author={Woosuk Kwon and Zhuohan Li and Siyuan Zhuang and Ying Sheng and Lianmin Zheng and Cody Hao Yu and Joseph E. Gonzalez and Hao Zhang and Ion Stoica},
booktitle={Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles},
@@ -47,10 +47,6 @@ You can tune the performance by adjusting `max_num_batched_tokens`:
...
@@ -47,10 +47,6 @@ You can tune the performance by adjusting `max_num_batched_tokens`:
- For optimal throughput, we recommend setting `max_num_batched_tokens > 8192` especially for smaller models on large GPUs.
- For optimal throughput, we recommend setting `max_num_batched_tokens > 8192` especially for smaller models on large GPUs.
- If `max_num_batched_tokens` is the same as `max_model_len`, that's almost the equivalent to the V0 default scheduling policy (except that it still prioritizes decodes).
- If `max_num_batched_tokens` is the same as `max_model_len`, that's almost the equivalent to the V0 default scheduling policy (except that it still prioritizes decodes).
!!! warning
When chunked prefill is disabled, `max_num_batched_tokens` must be greater than `max_model_len`.
In that case, if `max_num_batched_tokens < max_model_len`, vLLM may crash at server start‑up.
@@ -43,54 +43,9 @@ Further update the model as follows:
...
@@ -43,54 +43,9 @@ Further update the model as follows:
)
)
```
```
- Remove the embedding part from the [forward][torch.nn.Module.forward] method:
- Implement [embed_multimodal][vllm.model_executor.models.interfaces.SupportsMultiModal.embed_multimodal] that returns the embeddings from running the multimodal inputs through the multimodal tokenizer of the model. Below we provide a boilerplate of a typical implementation pattern, but feel free to adjust it to your own needs.
- Move the multi-modal embedding to [embed_multimodal][vllm.model_executor.models.interfaces.SupportsMultiModal.embed_multimodal].
- The text embedding and embedding merge are handled automatically by a default implementation of [embed_input_ids][vllm.model_executor.models.interfaces.SupportsMultiModal.embed_input_ids]. It does not need to be overridden in most cases.
Below we provide a boilerplate of a typical implementation pattern of [embed_multimodal][vllm.model_executor.models.interfaces.SupportsMultiModal.embed_multimodal], but feel free to adjust it to your own needs.
@@ -10,7 +10,7 @@ receives a request for a LoRA adapter that hasn't been loaded yet, the resolver
...
@@ -10,7 +10,7 @@ receives a request for a LoRA adapter that hasn't been loaded yet, the resolver
to locate and load the adapter from their configured storage locations. This enables:
to locate and load the adapter from their configured storage locations. This enables:
-**Dynamic LoRA Loading**: Load adapters on-demand without server restarts
-**Dynamic LoRA Loading**: Load adapters on-demand without server restarts
-**Multiple Storage Backends**: Support for filesystem, S3, and custom backends. The built-in `lora_filesystem_resolver` requires a local storage path, while the built-in `hf_hub_resolver` will pull LoRA adapters from Huggingface Hub and proceed in an identical manner. In general, custom resolvers can be implemented to fetch from any source.
-**Multiple Storage Backends**: Support for filesystem, S3, and custom backends. The built-in `lora_filesystem_resolver` requires a local storage path, but custom resolvers can be implemented to fetch from any source.
-**Automatic Discovery**: Seamless integration with existing LoRA workflows
-**Automatic Discovery**: Seamless integration with existing LoRA workflows
-**Scalable Deployment**: Centralized adapter management across multiple vLLM instances
-**Scalable Deployment**: Centralized adapter management across multiple vLLM instances
| pplx | batched | fp8,int8 | G,A,T | Y | Y | [`PplxPrepareAndFinalize`][vllm.model_executor.layers.fused_moe.pplx_prepare_finalize.PplxPrepareAndFinalize] |
| pplx | batched | fp8,int8 | G,A,T | Y | Y | [`PplxPrepareAndFinalize`][vllm.model_executor.layers.fused_moe.pplx_prepare_finalize.PplxPrepareAndFinalize] |
| deepep_high_throughput | standard | fp8 | G(128),A,T<sup>2</sup> | Y | Y | [`DeepEPLLPrepareAndFinalize`][vllm.model_executor.layers.fused_moe.deepep_ll_prepare_finalize.DeepEPLLPrepareAndFinalize] |
| deepep_high_throughput | standard | fp8 | G(128),A,T<sup>2</sup> | Y | Y | [`DeepEPLLPrepareAndFinalize`][vllm.model_executor.layers.fused_moe.deepep_ll_prepare_finalize.DeepEPLLPrepareAndFinalize] |
| deepep_low_latency | batched | fp8 | G(128),A,T<sup>3</sup> | Y | Y | [`DeepEPHTPrepareAndFinalize`][vllm.model_executor.layers.fused_moe.deepep_ht_prepare_finalize.DeepEPHTPrepareAndFinalize] |
| deepep_low_latency | batched | fp8 | G(128),A,T<sup>3</sup> | Y | Y | [`DeepEPHTPrepareAndFinalize`][vllm.model_executor.layers.fused_moe.deepep_ht_prepare_finalize.DeepEPHTPrepareAndFinalize] |
| flashinfer_all2allv | standard | nvfp4,fp8 | G,A,T | N | N | [`FlashInferA2APrepareAndFinalize`][vllm.model_executor.layers.fused_moe.flashinfer_a2a_prepare_finalize.FlashInferA2APrepareAndFinalize] |
| flashinfer_all2allv | standard | nvfp4,fp8 | G,A,T | N | N | [`FlashInferAllToAllMoEPrepareAndFinalize`][vllm.model_executor.layers.fused_moe.flashinfer_cutlass_prepare_finalize.FlashInferAllToAllMoEPrepareAndFinalize] |
| flashinfer<sup>4</sup> | standard | nvfp4,fp8 | G,A,T | N | N | [`FlashInferCutlassMoEPrepareAndFinalize`][vllm.model_executor.layers.fused_moe.flashinfer_cutlass_prepare_finalize.FlashInferCutlassMoEPrepareAndFinalize] |
| MoEPrepareAndFinalizeNoEP<sup>5</sup> | standard | fp8,int8 | G,A,T | N | Y | [`MoEPrepareAndFinalizeNoEP`][vllm.model_executor.layers.fused_moe.prepare_finalize.MoEPrepareAndFinalizeNoEP] |
| MoEPrepareAndFinalizeNoEP<sup>5</sup> | standard | fp8,int8 | G,A,T | N | Y | [`MoEPrepareAndFinalizeNoEP`][vllm.model_executor.layers.fused_moe.prepare_finalize.MoEPrepareAndFinalizeNoEP] |
| BatchedPrepareAndFinalize<sup>5</sup> | batched | fp8,int8 | G,A,T | N | Y | [`BatchedPrepareAndFinalize`][vllm.model_executor.layers.fused_moe.fused_batched_moe.BatchedPrepareAndFinalize] |
| BatchedPrepareAndFinalize<sup>5</sup> | batched | fp8,int8 | G,A,T | N | Y | [`BatchedPrepareAndFinalize`][vllm.model_executor.layers.fused_moe.fused_batched_moe.BatchedPrepareAndFinalize] |
@@ -159,12 +159,10 @@ Alternatively, you can use the LoRAResolver plugin to dynamically load LoRA adap
...
@@ -159,12 +159,10 @@ Alternatively, you can use the LoRAResolver plugin to dynamically load LoRA adap
You can set up multiple LoRAResolver plugins if you want to load LoRA adapters from different sources. For example, you might have one resolver for local files and another for S3 storage. vLLM will load the first LoRA adapter that it finds.
You can set up multiple LoRAResolver plugins if you want to load LoRA adapters from different sources. For example, you might have one resolver for local files and another for S3 storage. vLLM will load the first LoRA adapter that it finds.
You can either install existing plugins or implement your own. By default, vLLM comes with a [resolver plugin to load LoRA adapters from a local directory, as well as a resolver plugin to load LoRA adapters from repositories on Hugging Face Hub](https://github.com/vllm-project/vllm/tree/main/vllm/plugins/lora_resolvers)
You can either install existing plugins or implement your own. By default, vLLM comes with a [resolver plugin to load LoRA adapters from a local directory.](https://github.com/vllm-project/vllm/tree/main/vllm/plugins/lora_resolvers)
To enable either of these resolvers, you must `set VLLM_ALLOW_RUNTIME_LORA_UPDATING` to True.
To enable this resolver, set `VLLM_ALLOW_RUNTIME_LORA_UPDATING` to True, set `VLLM_PLUGINS` to include `lora_filesystem_resolver`, and then set `VLLM_LORA_RESOLVER_CACHE_DIR` to a local directory. When vLLM receives a request using a LoRA adapter `foobar`,
it will first look in the local directory for a directory `foobar`, and attempt to load the contents of that directory as a LoRA adapter. If successful, the request will complete as normal and
- To leverage a local directory, set `VLLM_PLUGINS` to include `lora_filesystem_resolver` and set `VLLM_LORA_RESOLVER_CACHE_DIR` to a local directory. When vLLM receives a request using a LoRA adapter `foobar`,
that adapter will then be available for normal use on the server.
it will first look in the local directory for a directory `foobar`, and attempt to load the contents of that directory as a LoRA adapter. If successful, the request will complete as normal and that adapter will then be available for normal use on the server.
- To leverage repositories on Hugging Face Hub, set `VLLM_PLUGINS` to include `lora_hf_hub_resolver` and set `VLLM_LORA_RESOLVER_HF_REPO_LIST` to a comma separated list of repository IDs on Hugging Face Hub. When vLLM receives a request for the LoRA adapter `my/repo/subpath`, it will download the adapter at the `subpath` of `my/repo` if it exists and contains an `adapter_config.json`, then build a request to the cached dir for the adapter, similar to the `lora_filesystem_resolver`. Please note that enabling remote downloads is insecure and not intended for use in production environments.
Alternatively, follow these example steps to implement your own plugin:
Alternatively, follow these example steps to implement your own plugin:
vLLM supports [generative](generative-models) and [pooling](pooling-models) models across various tasks.
If a model supports more than one task, you can set the task via the `--task` argument.
For each task, we list the model architectures that have been implemented in vLLM.
Alongside each architecture, we include some popular models that use it.
## Model Implementation
### vLLM
If vLLM natively supports a model, its implementation can be found in <gh-file:vllm/model_executor/models>.
These models are what we list in <project:#supported-text-models> and <project:#supported-mm-models>.
(transformers-backend)=
### Transformers
vLLM also supports model implementations that are available in Transformers. This does not currently work for all models, but most decoder language models are supported, and vision language model support is planned!
To check if the modeling backend is Transformers, you can simply do this:
```python
fromvllmimportLLM
llm=LLM(model=...,task="generate")# Name or path of your model
llm.apply_model(lambdamodel:print(type(model)))
```
If it is `TransformersForCausalLM` then it means it's based on Transformers!
:::{tip}
You can force the use of `TransformersForCausalLM` by setting `model_impl="transformers"` for <project:#offline-inference> or `--model-impl transformers` for the <project:#openai-compatible-server>.
:::
:::{note}
vLLM may not fully optimise the Transformers implementation so you may see degraded performance if comparing a native model to a Transformers model in vLLM.
:::
#### Custom models
If a model is neither supported natively by vLLM or Transformers, it can still be used in vLLM!
For a model to be compatible with the Transformers backend for vLLM it must:
- be a Transformers compatible custom model (see [Transformers - Customizing models](https://huggingface.co/docs/transformers/en/custom_models)):
* The model directory must have the correct structure (e.g. `config.json` is present).
*`config.json` must contain `auto_map.AutoModel`.
- be a Transformers backend for vLLM compatible model (see <project:#writing-custom-models>):
* Customisation should be done in the base model (e.g. in `MyModel`, not `MyModelForCausalLM`).
If the compatible model is:
- on the Hugging Face Model Hub, simply set `trust_remote_code=True` for <project:#offline-inference> or `--trust-remote-code` for the <project:#openai-compatible-server>.
- in a local directory, simply pass directory path to `model=<MODEL_DIR>` for <project:#offline-inference> or `vllm serve <MODEL_DIR>` for the <project:#openai-compatible-server>.
This means that, with the Transformers backend for vLLM, new models can be used before they are officially supported in Transformers or vLLM!
(writing-custom-models)=
#### Writing custom models
This section details the necessary modifications to make to a Transformers compatible custom model that make it compatible with the Transformers backend for vLLM. (We assume that a Transformers compatible custom model has already been created, see [Transformers - Customizing models](https://huggingface.co/docs/transformers/en/custom_models)).
To make your model compatible with the Transformers backend, it needs:
1.`kwargs` passed down through all modules from `MyModel` to `MyAttention`.
2.`MyAttention` must use `ALL_ATTENTION_FUNCTIONS` to call attention.
3.`MyModel` must contain `_supports_attention_backend = True`.
Here is what happens in the background when this model is loaded:
1. The config is loaded.
2.`MyModel` Python class is loaded from the `auto_map` in config, and we check that the model `is_backend_compatible()`.
3.`MyModel` is loaded into `TransformersForCausalLM` (see <gh-file:vllm/model_executor/models/transformers.py>) which sets `self.config._attn_implementation = "vllm"` so that vLLM's attention layer is used.
That's it!
For your model to be compatible with vLLM's tensor parallel and/or pipeline parallel features, you must add `base_model_tp_plan` and/or `base_model_pp_plan` to your model's config class:
-`base_model_tp_plan` is a `dict` that maps fully qualified layer name patterns to tensor parallel styles (currently only `"colwise"` and `"rowwise"` are supported).
-`base_model_pp_plan` is a `dict` that maps direct child layer names to `tuple`s of `list`s of `str`s:
* You only need to do this for layers which are not present on all pipeline stages
* vLLM assumes that there will be only one `nn.ModuleList`, which is distributed across the pipeline stages
* The `list` in the first element of the `tuple` contains the names of the input arguments
* The `list` in the last element of the `tuple` contains the names of the variables the layer outputs to in your modeling code
## Loading a Model
### Hugging Face Hub
By default, vLLM loads models from [Hugging Face (HF) Hub](https://huggingface.co/models). To change the download path for models, you can set the `HF_HOME` environment variable; for more details, refer to [their official documentation](https://huggingface.co/docs/huggingface_hub/package_reference/environment_variables#hfhome).
To determine whether a given model is natively supported, you can check the `config.json` file inside the HF repository.
If the `"architectures"` field contains a model architecture listed below, then it should be natively supported.
Models do not _need_ to be natively supported to be used in vLLM.
The [Transformers backend](#transformers-backend) enables you to run models directly using their Transformers implementation (or even remote code on the Hugging Face Model Hub!).
:::{tip}
The easiest way to check if your model is really supported at runtime is to run the program below:
```python
fromvllmimportLLM
# For generative models (task=generate) only
llm=LLM(model=...,task="generate")# Name or path of your model
output=llm.generate("Hello, my name is")
print(output)
# For pooling models (task={embed,classify,reward,score}) only
llm=LLM(model=...,task="embed")# Name or path of your model
output=llm.encode("Hello, my name is")
print(output)
```
If vLLM successfully returns text (for generative models) or hidden states (for pooling models), it indicates that your model is supported.
:::
Otherwise, please refer to [Adding a New Model](#new-model) for instructions on how to implement your model in vLLM.
Alternatively, you can [open an issue on GitHub](https://github.com/vllm-project/vllm/issues/new/choose) to request vLLM support.
#### Download a model
If you prefer, you can use the Hugging Face CLI to [download a model](https://huggingface.co/docs/huggingface_hub/guides/cli#huggingface-cli-download) or specific files from a model repository:
Use the Hugging Face CLI to interactively [delete downloaded model](https://huggingface.co/docs/huggingface_hub/guides/manage-cache#clean-your-cache) from the cache:
```console
#The `delete-cache`command requires extra dependencies to work with the TUI.
#Please run `pip install huggingface_hub[cli]` to install them.
#Launch the interactive TUI to select models to delete
$huggingface-cli delete-cache
? Select revisions to delete: 1 revisions selected counting for 438.9M.
○ None of the following (if selected, nothing will be deleted).
Model BAAI/bge-base-en-v1.5 (438.9M, used 1 week ago)
❯ ◉ a5beb1e3: main #modified 1 week ago
Model BAAI/bge-large-en-v1.5 (1.3G, used 1 week ago)
○ d4aa6901: main #modified 1 week ago
Model BAAI/bge-reranker-base (1.1G, used 4 weeks ago)
○ 2cfc18c9: main #modified 4 weeks ago
Press <space>to select, <enter> to validate and <ctrl+c> to quit without modification.
#Need to confirm after selected
? Select revisions to delete: 1 revision(s) selected.
*`internlm/internlm-7b`, `internlm/internlm-chat-7b`, etc.
* ✅︎
* ✅︎
-*`InternLM2ForCausalLM`
* InternLM2
*`internlm/internlm2-7b`, `internlm/internlm2-chat-7b`, etc.
* ✅︎
* ✅︎
-*`InternLM3ForCausalLM`
* InternLM3
*`internlm/internlm3-8b-instruct`, etc.
* ✅︎
* ✅︎
-*`JAISLMHeadModel`
* Jais
*`inceptionai/jais-13b`, `inceptionai/jais-13b-chat`, `inceptionai/jais-30b-v3`, `inceptionai/jais-30b-chat-v3`, etc.
*
* ✅︎
-*`JambaForCausalLM`
* Jamba
*`ai21labs/AI21-Jamba-1.5-Large`, `ai21labs/AI21-Jamba-1.5-Mini`, `ai21labs/Jamba-v0.1`, etc.
* ✅︎
* ✅︎
-*`LlamaForCausalLM`
* Llama 3.1, Llama 3, Llama 2, LLaMA, Yi
*`meta-llama/Meta-Llama-3.1-405B-Instruct`, `meta-llama/Meta-Llama-3.1-70B`, `meta-llama/Meta-Llama-3-70B-Instruct`, `meta-llama/Llama-2-70b-hf`, `01-ai/Yi-34B`, etc.
* ✅︎
* ✅︎
-*`MambaForCausalLM`
* Mamba
*`state-spaces/mamba-130m-hf`, `state-spaces/mamba-790m-hf`, `state-spaces/mamba-2.8b-hf`, etc.
*
* ✅︎
-*`MiniCPMForCausalLM`
* MiniCPM
*`openbmb/MiniCPM-2B-sft-bf16`, `openbmb/MiniCPM-2B-dpo-bf16`, `openbmb/MiniCPM-S-1B-sft`, etc.
* ✅︎
* ✅︎
-*`MiniCPM3ForCausalLM`
* MiniCPM3
*`openbmb/MiniCPM3-4B`, etc.
* ✅︎
* ✅︎
-*`MistralForCausalLM`
* Mistral, Mistral-Instruct
*`mistralai/Mistral-7B-v0.1`, `mistralai/Mistral-7B-Instruct-v0.1`, etc.
* ✅︎
* ✅︎
-*`MixtralForCausalLM`
* Mixtral-8x7B, Mixtral-8x7B-Instruct
*`mistralai/Mixtral-8x7B-v0.1`, `mistralai/Mixtral-8x7B-Instruct-v0.1`, `mistral-community/Mixtral-8x22B-v0.1`, etc.
* ✅︎
* ✅︎
-*`MPTForCausalLM`
* MPT, MPT-Instruct, MPT-Chat, MPT-StoryWriter
*`mosaicml/mpt-7b`, `mosaicml/mpt-7b-storywriter`, `mosaicml/mpt-30b`, etc.
*
* ✅︎
-*`NemotronForCausalLM`
* Nemotron-3, Nemotron-4, Minitron
*`nvidia/Minitron-8B-Base`, `mgoin/Nemotron-4-340B-Base-hf-FP8`, etc.
* ✅︎
* ✅︎
-*`OLMoForCausalLM`
* OLMo
*`allenai/OLMo-1B-hf`, `allenai/OLMo-7B-hf`, etc.
*
* ✅︎
-*`OLMo2ForCausalLM`
* OLMo2
*`allenai/OLMo-2-0425-1B`, etc.
*
* ✅︎
-*`OLMoEForCausalLM`
* OLMoE
*`allenai/OLMoE-1B-7B-0924`, `allenai/OLMoE-1B-7B-0924-Instruct`, etc.
* ✅︎
* ✅︎
-*`OPTForCausalLM`
* OPT, OPT-IML
*`facebook/opt-66b`, `facebook/opt-iml-max-30b`, etc.
*
* ✅︎
-*`OrionForCausalLM`
* Orion
*`OrionStarAI/Orion-14B-Base`, `OrionStarAI/Orion-14B-Chat`, etc.
*
* ✅︎
-*`PhiForCausalLM`
* Phi
*`microsoft/phi-1_5`, `microsoft/phi-2`, etc.
* ✅︎
* ✅︎
-*`Phi3ForCausalLM`
* Phi-4, Phi-3
*`microsoft/Phi-4-mini-instruct`, `microsoft/Phi-4`, `microsoft/Phi-3-mini-4k-instruct`, `microsoft/Phi-3-mini-128k-instruct`, `microsoft/Phi-3-medium-128k-instruct`, etc.
* ✅︎
* ✅︎
-*`Phi3SmallForCausalLM`
* Phi-3-Small
*`microsoft/Phi-3-small-8k-instruct`, `microsoft/Phi-3-small-128k-instruct`, etc.
*
* ✅︎
-*`PhiMoEForCausalLM`
* Phi-3.5-MoE
*`microsoft/Phi-3.5-MoE-instruct`, etc.
* ✅︎
* ✅︎
-*`PersimmonForCausalLM`
* Persimmon
*`adept/persimmon-8b-base`, `adept/persimmon-8b-chat`, etc.
*
* ✅︎
-*`Plamo2ForCausalLM`
* PLaMo2
*`pfnet/plamo-2-1b`, `pfnet/plamo-2-8b`, etc.
*
*
-*`QWenLMHeadModel`
* Qwen
*`Qwen/Qwen-7B`, `Qwen/Qwen-7B-Chat`, etc.
* ✅︎
* ✅︎
-*`Qwen2ForCausalLM`
* QwQ, Qwen2
*`Qwen/QwQ-32B-Preview`, `Qwen/Qwen2-7B-Instruct`, `Qwen/Qwen2-7B`, etc.
* ✅︎
* ✅︎
-*`Qwen2MoeForCausalLM`
* Qwen2MoE
*`Qwen/Qwen1.5-MoE-A2.7B`, `Qwen/Qwen1.5-MoE-A2.7B-Chat`, etc.
*
* ✅︎
-*`Qwen3ForCausalLM`
* Qwen3
*`Qwen/Qwen3-8B`, etc.
* ✅︎
* ✅︎
-*`Qwen3MoeForCausalLM`
* Qwen3MoE
*`Qwen/Qwen3-30B-A3B`, etc.
*
* ✅︎
-*`StableLmForCausalLM`
* StableLM
*`stabilityai/stablelm-3b-4e1t`, `stabilityai/stablelm-base-alpha-7b-v2`, etc.
*
* ✅︎
-*`Starcoder2ForCausalLM`
* Starcoder2
*`bigcode/starcoder2-3b`, `bigcode/starcoder2-7b`, `bigcode/starcoder2-15b`, etc.
*
* ✅︎
-*`SolarForCausalLM`
* Solar Pro
*`upstage/solar-pro-preview-instruct`, etc.
* ✅︎
* ✅︎
-*`TeleChat2ForCausalLM`
* TeleChat2
*`Tele-AI/TeleChat2-3B`, `Tele-AI/TeleChat2-7B`, `Tele-AI/TeleChat2-35B`, etc.
* ✅︎
* ✅︎
-*`TeleFLMForCausalLM`
* TeleFLM
*`CofeAI/FLM-2-52B-Instruct-2407`, `CofeAI/Tele-FLM`, etc.
* ✅︎
* ✅︎
-*`XverseForCausalLM`
* XVERSE
*`xverse/XVERSE-7B-Chat`, `xverse/XVERSE-13B-Chat`, `xverse/XVERSE-65B-Chat`, etc.
* ✅︎
* ✅︎
-*`MiniMaxText01ForCausalLM`
* MiniMax-Text
*`MiniMaxAI/MiniMax-Text-01`, etc.
*
* ✅︎
-*`Zamba2ForCausalLM`
* Zamba2
*`Zyphra/Zamba2-7B-instruct`, `Zyphra/Zamba2-2.7B-instruct`, `Zyphra/Zamba2-1.2B-instruct`, etc.
*
*
-*`MiMoForCausalLM`
* MiMo
*`XiaomiMiMo/MiMo-7B-RL`, etc.
*
*
:::
:::{note}
Currently, the ROCm version of vLLM supports Mistral and Mixtral only for context lengths up to 4096.
:::
### Pooling Models
See [this page](pooling-models) for more information on how to use pooling models.
:::{important}
Since some model architectures support both generative and pooling tasks,
you should explicitly specify the task type to ensure that the model is used in pooling mode instead of generative mode.
:::
#### Text Embedding
Specified using `--task embed`.
:::{list-table}
:widths: 25 25 50 5 5
:header-rows: 1
-* Architecture
* Models
* Example HF Models
*[LoRA](#lora-adapter)
*[PP](#distributed-serving)
-*`BertModel`
* BERT-based
*`BAAI/bge-base-en-v1.5`, `Snowflake/snowflake-arctic-embed-xs`, etc.
*
*
-*`Gemma2Model`
* Gemma 2-based
*`BAAI/bge-multilingual-gemma2`, etc.
*
* ✅︎
-*`GritLM`
* GritLM
*`parasail-ai/GritLM-7B-vllm`.
* ✅︎
* ✅︎
-*`GteModel`
* Arctic-Embed-2.0-M
*`Snowflake/snowflake-arctic-embed-m-v2.0`.
*
* ︎
-*`GteNewModel`
* mGTE-TRM (see note)
*`Alibaba-NLP/gte-multilingual-base`, etc.
* ︎
* ︎
-*`ModernBertModel`
* ModernBERT-based
*`Alibaba-NLP/gte-modernbert-base`, etc.
* ︎
* ︎
-*`NomicBertModel`
* Nomic BERT
*`nomic-ai/nomic-embed-text-v1`, `nomic-ai/nomic-embed-text-v2-moe`, `Snowflake/snowflake-arctic-embed-m-long`, etc.
* ︎
* ︎
-*`LlamaModel`, `LlamaForCausalLM`, `MistralModel`, etc.
* Llama-based
*`intfloat/e5-mistral-7b-instruct`, etc.
* ✅︎
* ✅︎
-*`Qwen2Model`, `Qwen2ForCausalLM`
* Qwen2-based
*`ssmits/Qwen2-7B-Instruct-embed-base` (see note), `Alibaba-NLP/gte-Qwen2-7B-instruct` (see note), etc.
* ✅︎
* ✅︎
-*`RobertaModel`, `RobertaForMaskedLM`
* RoBERTa-based
*`sentence-transformers/all-roberta-large-v1`, etc.
*
*
-*`XLMRobertaModel`
* XLM-RoBERTa-based
*`intfloat/multilingual-e5-large`, `jinaai/jina-reranker-v2-base-multilingual`, `Snowflake/snowflake-arctic-embed-l-v2.0`, `jinaai/jina-embeddings-v3`(see note), etc.
*
*
:::
:::{note}
`ssmits/Qwen2-7B-Instruct-embed-base` has an improperly defined Sentence Transformers config.
You should manually set mean pooling by passing `--override-pooler-config '{"pooling_type": "MEAN"}'`.
:::
:::{note}
The HF implementation of `Alibaba-NLP/gte-Qwen2-1.5B-instruct` is hardcoded to use causal attention despite what is shown in `config.json`. To compare vLLM vs HF results,
you should set `--hf-overrides '{"is_causal": true}'` in vLLM so that the two implementations are consistent with each other.
For both the 1.5B and 7B variants, you also need to enable `--trust-remote-code` for the correct tokenizer to be loaded.
See [relevant issue on HF Transformers](https://github.com/huggingface/transformers/issues/34882).
:::
:::{note}
`jinaai/jina-embeddings-v3` supports multiple tasks through lora, while vllm temporarily only supports text-matching tasks by merging lora weights.
:::
:::{note}
The second-generation GTE model (mGTE-TRM) is named `NewModel`. The name `NewModel` is too generic, you should set `--hf-overrides '{"architectures": ["GteNewModel"]}'` to specify the use of the `GteNewModel` architecture.
:::
If your model is not in the above list, we will try to automatically convert the model using
{func}`~vllm.model_executor.models.adapters.as_embedding_model`. By default, the embeddings
of the whole prompt are extracted from the normalized hidden state corresponding to the last token.
#### Reward Modeling
Specified using `--task reward`.
:::{list-table}
:widths: 25 25 50 5 5
:header-rows: 1
-* Architecture
* Models
* Example HF Models
*[LoRA](#lora-adapter)
*[PP](#distributed-serving)
-*`InternLM2ForRewardModel`
* InternLM2-based
*`internlm/internlm2-1_8b-reward`, `internlm/internlm2-7b-reward`, etc.
* ✅︎
* ✅︎
-*`LlamaForCausalLM`
* Llama-based
*`peiyi9979/math-shepherd-mistral-7b-prm`, etc.
* ✅︎
* ✅︎
-*`Qwen2ForRewardModel`
* Qwen2-based
*`Qwen/Qwen2.5-Math-RM-72B`, etc.
* ✅︎
* ✅︎
-*`Qwen2ForProcessRewardModel`
* Qwen2-based
*`Qwen/Qwen2.5-Math-PRM-7B`, `Qwen/Qwen2.5-Math-PRM-72B`, etc.
* ✅︎
* ✅︎
:::
If your model is not in the above list, we will try to automatically convert the model using
{func}`~vllm.model_executor.models.adapters.as_reward_model`. By default, we return the hidden states of each token directly.
:::{important}
For process-supervised reward models such as `peiyi9979/math-shepherd-mistral-7b-prm`, the pooling config should be set explicitly,
If your model is not in the above list, we will try to automatically convert the model using
{func}`~vllm.model_executor.models.adapters.as_classification_model`. By default, the class probabilities are extracted from the softmaxed hidden state corresponding to the last token.
#### Sentence Pair Scoring
Specified using `--task score`.
:::{list-table}
:widths: 25 25 50 5 5
:header-rows: 1
-* Architecture
* Models
* Example HF Models
*[LoRA](#lora-adapter)
*[PP](#distributed-serving)
-*`BertForSequenceClassification`
* BERT-based
*`cross-encoder/ms-marco-MiniLM-L-6-v2`, etc.
*
*
-*`RobertaForSequenceClassification`
* RoBERTa-based
*`cross-encoder/quora-roberta-base`, etc.
*
*
-*`XLMRobertaForSequenceClassification`
* XLM-RoBERTa-based
*`BAAI/bge-reranker-v2-m3`, etc.
*
*
-*`ModernBertForSequenceClassification`
* ModernBert-based
*`Alibaba-NLP/gte-reranker-modernbert-base`, etc.
*
*
:::
(supported-mm-models)=
## List of Multimodal Language Models
The following modalities are supported depending on the model:
-**T**ext
-**I**mage
-**V**ideo
-**A**udio
Any combination of modalities joined by `+` are supported.
- e.g.: `T + I` means that the model supports text-only, image-only, and text-with-image inputs.
On the other hand, modalities separated by `/` are mutually exclusive.
- e.g.: `T / I` means that the model supports text-only and image-only inputs, but not text-with-image inputs.
See [this page](#multimodal-inputs) on how to pass multi-modal inputs to the model.
:::{important}
**To enable multiple multi-modal items per text prompt in vLLM V0**, you have to set `limit_mm_per_prompt` (offline inference)
or `--limit-mm-per-prompt` (online serving). For example, to enable passing up to 4 images per text prompt:
*`meta-llama/Llama-4-Scout-17B-16E-Instruct`, `meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8`, `meta-llama/Llama-4-Maverick-17B-128E-Instruct`, etc.
*
* ✅︎
* ✅︎
-*`Llama4ForConditionalGeneration`
* Llama-4-17B-Omni-Instruct
* T + I<sup>+</sup>
*`meta-llama/Llama-4-Scout-17B-16E-Instruct`, `meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8`, `meta-llama/Llama-4-Maverick-17B-128E-Instruct`, etc.
*
*
* ✅︎
-*`LlavaForConditionalGeneration`
* LLaVA-1.5
* T + I<sup>E+</sup>
*`llava-hf/llava-1.5-7b-hf`, `TIGER-Lab/Mantis-8B-siglip-llama3` (see note), etc.
*
* ✅︎
* ✅︎
-*`LlavaNextForConditionalGeneration`
* LLaVA-NeXT
* T + I<sup>E+</sup>
*`llava-hf/llava-v1.6-mistral-7b-hf`, `llava-hf/llava-v1.6-vicuna-7b-hf`, etc.
*
* ✅︎
* ✅︎
-*`LlavaNextVideoForConditionalGeneration`
* LLaVA-NeXT-Video
* T + V
*`llava-hf/LLaVA-NeXT-Video-7B-hf`, etc.
*
* ✅︎
* ✅︎
-*`LlavaOnevisionForConditionalGeneration`
* LLaVA-Onevision
* T + I<sup>+</sup> + V<sup>+</sup>
*`llava-hf/llava-onevision-qwen2-7b-ov-hf`, `llava-hf/llava-onevision-qwen2-0.5b-ov-hf`, etc.
*
* ✅︎
* ✅︎
-*`MiniCPMO`
* MiniCPM-O
* T + I<sup>E+</sup> + V<sup>E+</sup> + A<sup>E+</sup>
*`openbmb/MiniCPM-o-2_6`, etc.
* ✅︎
* ✅︎
* ✅︎
-*`MiniCPMV`
* MiniCPM-V
* T + I<sup>E+</sup> + V<sup>E+</sup>
*`openbmb/MiniCPM-V-2` (see note), `openbmb/MiniCPM-Llama3-V-2_5`, `openbmb/MiniCPM-V-2_6`, etc.
* ✅︎
* ✅︎
* ✅︎
-*`MiniMaxVL01ForConditionalGeneration`
* MiniMax-VL
* T + I<sup>E+</sup>
*`MiniMaxAI/MiniMax-VL-01`, etc.
*
* ✅︎
* ✅︎
-*`Mistral3ForConditionalGeneration`
* Mistral3
* T + I<sup>+</sup>
*`mistralai/Mistral-Small-3.1-24B-Instruct-2503`, etc.
* ✅︎
* ✅︎
* ✅︎
-*`MllamaForConditionalGeneration`
* Llama 3.2
* T + I<sup>+</sup>
*`meta-llama/Llama-3.2-90B-Vision-Instruct`, `meta-llama/Llama-3.2-11B-Vision`, etc.
*
*
*
-*`MolmoForCausalLM`
* Molmo
* T + I<sup>+</sup>
*`allenai/Molmo-7B-D-0924`, `allenai/Molmo-7B-O-0924`, etc.
* ✅︎
* ✅︎
* ✅︎
-*`NVLM_D_Model`
* NVLM-D 1.0
* T + I<sup>+</sup>
*`nvidia/NVLM-D-72B`, etc.
*
* ✅︎
* ✅︎
-*`Ovis`
* Ovis2, Ovis1.6
* T + I<sup>+</sup>
*`AIDC-AI/Ovis2-1B`, `AIDC-AI/Ovis1.6-Llama3.2-3B`, etc.
*
*
* ✅︎
-*`PaliGemmaForConditionalGeneration`
* PaliGemma, PaliGemma 2
* T + I<sup>E</sup>
*`google/paligemma-3b-pt-224`, `google/paligemma-3b-mix-224`, `google/paligemma2-3b-ft-docci-448`, etc.
*
* ✅︎
* ⚠️
-*`Phi3VForCausalLM`
* Phi-3-Vision, Phi-3.5-Vision
* T + I<sup>E+</sup>
*`microsoft/Phi-3-vision-128k-instruct`, `microsoft/Phi-3.5-vision-instruct`, etc.
*
* ✅︎
* ✅︎
-*`Phi4MMForCausalLM`
* Phi-4-multimodal
* T + I<sup>+</sup> / T + A<sup>+</sup> / I<sup>+</sup> + A<sup>+</sup>
*`microsoft/Phi-4-multimodal-instruct`, etc.
* ✅︎
*
* ✅︎
-*`PixtralForConditionalGeneration`
* Pixtral
* T + I<sup>+</sup>
*`mistralai/Mistral-Small-3.1-24B-Instruct-2503`, `mistral-community/pixtral-12b`, etc.
*
* ✅︎
* ✅︎
-*`QwenVLForConditionalGeneration`<sup>^</sup>
* Qwen-VL
* T + I<sup>E+</sup>
*`Qwen/Qwen-VL`, `Qwen/Qwen-VL-Chat`, etc.
* ✅︎
* ✅︎
* ✅︎
-*`Qwen2AudioForConditionalGeneration`
* Qwen2-Audio
* T + A<sup>+</sup>
*`Qwen/Qwen2-Audio-7B-Instruct`
*
* ✅︎
* ✅︎
-*`Qwen2VLForConditionalGeneration`
* QVQ, Qwen2-VL
* T + I<sup>E+</sup> + V<sup>E+</sup>
*`Qwen/QVQ-72B-Preview`, `Qwen/Qwen2-VL-7B-Instruct`, `Qwen/Qwen2-VL-72B-Instruct`, etc.
* ✅︎
* ✅︎
* ✅︎
-*`Qwen2_5_VLForConditionalGeneration`
* Qwen2.5-VL
* T + I<sup>E+</sup> + V<sup>E+</sup>
*`Qwen/Qwen2.5-VL-3B-Instruct`, `Qwen/Qwen2.5-VL-72B-Instruct`, etc.
* ✅︎
* ✅︎
* ✅︎
-*`Qwen2_5OmniThinkerForConditionalGeneration`
* Qwen2.5-Omni
* T + I<sup>E+</sup> + V<sup>E+</sup> + A<sup>+</sup>
*`Qwen/Qwen2.5-Omni-7B`
*
* ✅︎
* ✅︎\*
-*`SkyworkR1VChatModel`
* Skywork-R1V-38B
* T + I
*`Skywork/Skywork-R1V-38B`
*
* ✅︎
* ✅︎
-*`SmolVLMForConditionalGeneration`
* SmolVLM2
* T + I
*`SmolVLM2-2.2B-Instruct`
*
* ✅︎
* ✅︎
-*`UltravoxModel`
* Ultravox
* T + A<sup>E+</sup>
*`fixie-ai/ultravox-v0_5-llama-3_2-1b`
* ✅︎
* ✅︎
* ✅︎
:::
<sup>^</sup> You need to set the architecture name via `--hf-overrides` to match the one in vLLM.
• For example, to use DeepSeek-VL2 series models:
<sup>E</sup> Pre-computed embeddings can be inputted for this modality.
<sup>+</sup> Multiple items can be inputted per text prompt for this modality.
:::{warning}
Both V0 and V1 support `Gemma3ForConditionalGeneration` for text-only inputs.
However, there are differences in how they handle text + image inputs:
V0 correctly implements the model's attention pattern:
- Uses bidirectional attention between the image tokens corresponding to the same image
- Uses causal attention for other tokens
- Implemented via (naive) PyTorch SDPA with masking tensors
- Note: May use significant memory for long prompts with image
V1 currently uses a simplified attention pattern:
- Uses causal attention for all tokens, including image tokens
- Generates reasonable outputs but does not match the original model's attention for text + image inputs, especially when `{"do_pan_and_scan": true}`
- Will be updated in the future to support the correct behavior
This limitation exists because the model's mixed attention pattern (bidirectional for images, causal otherwise) is not yet supported by vLLM's attention backends.
:::
:::{note}
`h2oai/h2ovl-mississippi-2b` will be available in V1 once we support head size 80.
:::
:::{note}
To use `TIGER-Lab/Mantis-8B-siglip-llama3`, you have to pass `--hf_overrides '{"architectures": ["MantisForConditionalGeneration"]}'` when running vLLM.
:::
:::{warning}
The output quality of `AllenAI/Molmo-7B-D-0924` (especially in object localization tasks) has deteriorated in recent updates.
For the best results, we recommend using the following dependency versions (tested on A10 and L40):
```text
# Core vLLM-compatible dependencies with Molmo accuracy setup (tested on L40)
torch==2.5.1
torchvision==0.20.1
transformers==4.48.1
tokenizers==0.21.0
tiktoken==0.7.0
vllm==0.7.0
# Optional but recommended for improved performance and stability
triton==3.1.0
xformers==0.0.28.post3
uvloop==0.21.0
protobuf==5.29.3
openai==1.60.2
opencv-python-headless==4.11.0.86
pillow==10.4.0
# Installed FlashAttention (for float16 only)
flash-attn>=2.5.6 # Not used in float32, but should be documented
```
**Note:** Make sure you understand the security implications of using outdated packages.
:::
:::{note}
The official `openbmb/MiniCPM-V-2` doesn't work yet, so we need to use a fork (`HwwwH/MiniCPM-V-2`) for now.
For more details, please see: <gh-pr:4087#issuecomment-2250397630>
:::
:::{warning}
Our PaliGemma implementations have the same problem as Gemma 3 (see above) for both V0 and V1.
:::
:::{note}
To use Qwen2.5-Omni, you have to install Hugging Face Transformers library from source via
See [this page](pooling-models) for more information on how to use pooling models.
:::{important}
Since some model architectures support both generative and pooling tasks,
you should explicitly specify the task type to ensure that the model is used in pooling mode instead of generative mode.
:::
#### Text Embedding
Specified using `--task embed`.
Any text generation model can be converted into an embedding model by passing `--task embed`.
:::{note}
To get the best results, you should use pooling models that are specifically trained as such.
:::
The following table lists those that are tested in vLLM.
:::{list-table}
:widths: 25 25 15 25 5 5
:header-rows: 1
-* Architecture
* Models
* Inputs
* Example HF Models
*[LoRA](#lora-adapter)
*[PP](#distributed-serving)
-*`LlavaNextForConditionalGeneration`
* LLaVA-NeXT-based
* T / I
*`royokong/e5-v`
*
* ✅︎
-*`Phi3VForCausalLM`
* Phi-3-Vision-based
* T + I
*`TIGER-Lab/VLM2Vec-Full`
* 🚧
* ✅︎
-*`Qwen2VLForConditionalGeneration`
* Qwen2-VL-based
* T + I
*`MrLight/dse-qwen2-2b-mrl-v1`
*
* ✅︎
:::
#### Transcription
Specified using `--task transcription`.
Speech2Text models trained specifically for Automatic Speech Recognition.
:::{list-table}
:widths: 25 25 25 5 5
:header-rows: 1
-* Architecture
* Models
* Example HF Models
*[LoRA](#lora-adapter)
*[PP](#distributed-serving)
-*`Whisper`
* Whisper-based
*`openai/whisper-large-v3-turbo`
* 🚧
* 🚧
:::
_________________
## Model Support Policy
At vLLM, we are committed to facilitating the integration and support of third-party models within our ecosystem. Our approach is designed to balance the need for robustness and the practical limitations of supporting a wide range of models. Here’s how we manage third-party model support:
1.**Community-Driven Support**: We encourage community contributions for adding new models. When a user requests support for a new model, we welcome pull requests (PRs) from the community. These contributions are evaluated primarily on the sensibility of the output they generate, rather than strict consistency with existing implementations such as those in transformers. **Call for contribution:** PRs coming directly from model vendors are greatly appreciated!
2.**Best-Effort Consistency**: While we aim to maintain a level of consistency between the models implemented in vLLM and other frameworks like transformers, complete alignment is not always feasible. Factors like acceleration techniques and the use of low-precision computations can introduce discrepancies. Our commitment is to ensure that the implemented models are functional and produce sensible results.
:::{tip}
When comparing the output of `model.generate` from Hugging Face Transformers with the output of `llm.generate` from vLLM, note that the former reads the model's generation config file (i.e., [generation_config.json](https://github.com/huggingface/transformers/blob/19dabe96362803fb0a9ae7073d03533966598b17/src/transformers/generation/utils.py#L1945)) and applies the default parameters for generation, while the latter only uses the parameters passed to the function. Ensure all sampling parameters are identical when comparing outputs.
:::
3.**Issue Resolution and Model Updates**: Users are encouraged to report any bugs or issues they encounter with third-party models. Proposed fixes should be submitted via PRs, with a clear explanation of the problem and the rationale behind the proposed solution. If a fix for one model impacts another, we rely on the community to highlight and address these cross-model dependencies. Note: for bugfix PRs, it is good etiquette to inform the original author to seek their feedback.
4.**Monitoring and Updates**: Users interested in specific models should monitor the commit history for those models (e.g., by tracking changes in the main/vllm/model_executor/models directory). This proactive approach helps users stay informed about updates and changes that may affect the models they use.
5.**Selective Focus**: Our resources are primarily directed towards models with significant user interest and impact. Models that are less frequently used may receive less attention, and we rely on the community to play a more active role in their upkeep and improvement.
Through this approach, vLLM fosters a collaborative environment where both the core development team and the broader community contribute to the robustness and diversity of the third-party models supported in our ecosystem.
Note that, as an inference engine, vLLM does not introduce new models. Therefore, all models supported by vLLM are third-party models in this regard.
We have the following levels of testing for models:
1.**Strict Consistency**: We compare the output of the model with the output of the model in the HuggingFace Transformers library under greedy decoding. This is the most stringent test. Please refer to [models tests](https://github.com/vllm-project/vllm/blob/main/tests/models) for the models that have passed this test.
2.**Output Sensibility**: We check if the output of the model is sensible and coherent, by measuring the perplexity of the output and checking for any obvious errors. This is a less stringent test.
3.**Runtime Functionality**: We check if the model can be loaded and run without errors. This is the least stringent test. Please refer to [functionality tests](gh-dir:tests) and [examples](gh-dir:examples) for the models that have passed this test.
4.**Community Feedback**: We rely on the community to provide feedback on the models. If a model is broken or not working as expected, we encourage users to raise issues to report it or open pull requests to fix it. The rest of the models fall under this category.
