supported_models.rst 4.65 KB
Newer Older
Woosuk Kwon's avatar
Woosuk Kwon committed
1
2
3
4
5
.. _supported_models:

Supported Models
================

6
vLLM supports a variety of generative Transformer models in `HuggingFace Transformers <https://huggingface.co/models>`_.
Woosuk Kwon's avatar
Woosuk Kwon committed
7
The following is the list of model architectures that are currently supported by vLLM.
Woosuk Kwon's avatar
Woosuk Kwon committed
8
9
10
Alongside each architecture, we include some popular models that use it.

.. list-table::
11
  :widths: 25 25 50
Woosuk Kwon's avatar
Woosuk Kwon committed
12
13
14
15
  :header-rows: 1

  * - Architecture
    - Models
16
    - Example HuggingFace Models
17
  * - :code:`AquilaForCausalLM`
18
    - Aquila
19
    - :code:`BAAI/Aquila-7B`, :code:`BAAI/AquilaChat-7B`, etc.
Zhuohan Li's avatar
Zhuohan Li committed
20
  * - :code:`BaiChuanForCausalLM`
21
    - Baichuan
22
    - :code:`baichuan-inc/Baichuan2-13B-Chat`, :code:`baichuan-inc/Baichuan-7B`, etc.
23
24
25
  * - :code:`ChatGLMModel`
    - ChatGLM
    - :code:`THUDM/chatglm2-6b`, :code:`THUDM/chatglm3-6b`, etc.
26
27
28
  * - :code:`DeciLMForCausalLM`
    - DeciLM
    - :code:`Deci/DeciLM-7B`, :code:`Deci/DeciLM-7B-instruct`, etc.
Woosuk Kwon's avatar
Woosuk Kwon committed
29
30
31
  * - :code:`BloomForCausalLM`
    - BLOOM, BLOOMZ, BLOOMChat
    - :code:`bigscience/bloom`, :code:`bigscience/bloomz`, etc.
Zhuohan Li's avatar
Zhuohan Li committed
32
33
  * - :code:`FalconForCausalLM`
    - Falcon
34
    - :code:`tiiuae/falcon-7b`, :code:`tiiuae/falcon-40b`, :code:`tiiuae/falcon-rw-7b`, etc.
Woosuk Kwon's avatar
Woosuk Kwon committed
35
36
  * - :code:`GPT2LMHeadModel`
    - GPT-2
37
    - :code:`gpt2`, :code:`gpt2-xl`, etc.
38
39
40
  * - :code:`GPTBigCodeForCausalLM`
    - StarCoder, SantaCoder, WizardCoder
    - :code:`bigcode/starcoder`, :code:`bigcode/gpt_bigcode-santacoder`, :code:`WizardLM/WizardCoder-15B-V1.0`, etc.
41
42
43
  * - :code:`GPTJForCausalLM`
    - GPT-J
    - :code:`EleutherAI/gpt-j-6b`, :code:`nomic-ai/gpt4all-j`, etc.
Woosuk Kwon's avatar
Woosuk Kwon committed
44
45
  * - :code:`GPTNeoXForCausalLM`
    - GPT-NeoX, Pythia, OpenAssistant, Dolly V2, StableLM
46
    - :code:`EleutherAI/gpt-neox-20b`, :code:`EleutherAI/pythia-12b`, :code:`OpenAssistant/oasst-sft-4-pythia-12b-epoch-3.5`, :code:`databricks/dolly-v2-12b`, :code:`stabilityai/stablelm-tuned-alpha-7b`, etc.
47
48
49
  * - :code:`InternLMForCausalLM`
    - InternLM
    - :code:`internlm/internlm-7b`, :code:`internlm/internlm-chat-7b`, etc.
Woosuk Kwon's avatar
Woosuk Kwon committed
50
  * - :code:`LlamaForCausalLM`
Zhuohan Li's avatar
Zhuohan Li committed
51
    - LLaMA, LLaMA-2, Vicuna, Alpaca, Koala, Guanaco
52
    - :code:`meta-llama/Llama-2-13b-hf`, :code:`meta-llama/Llama-2-70b-hf`, :code:`openlm-research/open_llama_13b`, :code:`lmsys/vicuna-13b-v1.3`, :code:`young-geng/koala`, etc.
53
54
55
  * - :code:`MistralForCausalLM`
    - Mistral, Mistral-Instruct
    - :code:`mistralai/Mistral-7B-v0.1`, :code:`mistralai/Mistral-7B-Instruct-v0.1`, etc.
Woosuk Kwon's avatar
Woosuk Kwon committed
56
57
58
  * - :code:`MixtralForCausalLM`
    - Mixtral-8x7B, Mixtral-8x7B-Instruct
    - :code:`mistralai/Mixtral-8x7B-v0.1`, :code:`mistralai/Mixtral-8x7B-Instruct-v0.1`, etc.
Woosuk Kwon's avatar
Woosuk Kwon committed
59
  * - :code:`MPTForCausalLM`
60
61
    - MPT, MPT-Instruct, MPT-Chat, MPT-StoryWriter
    - :code:`mosaicml/mpt-7b`, :code:`mosaicml/mpt-7b-storywriter`, :code:`mosaicml/mpt-30b`, etc.
Woosuk Kwon's avatar
Woosuk Kwon committed
62
63
  * - :code:`OPTForCausalLM`
    - OPT, OPT-IML
64
    - :code:`facebook/opt-66b`, :code:`facebook/opt-iml-max-30b`, etc.
65
  * - :code:`PhiForCausalLM`
66
67
    - Phi
    - :code:`microsoft/phi-1_5`, :code:`microsoft/phi-2`, etc.
68
  * - :code:`QWenLMHeadModel`
69
70
    - Qwen
    - :code:`Qwen/Qwen-7B`, :code:`Qwen/Qwen-7B-Chat`, etc.
Junyang Lin's avatar
Junyang Lin committed
71
72
  * - :code:`Qwen2ForCausalLM`
    - Qwen2
73
    - :code:`Qwen/Qwen2-beta-7B`, :code:`Qwen/Qwen2-beta-7B-Chat`, etc.
74
  * - :code:`StableLMEpochForCausalLM`
Hyunsung Lee's avatar
Hyunsung Lee committed
75
76
    - StableLM
    - :code:`stabilityai/stablelm-3b-4e1t/` , :code:`stabilityai/stablelm-base-alpha-7b-v2`, etc.
77
78
79
  * - :code:`YiForCausalLM`
    - Yi
    - :code:`01-ai/Yi-6B`, :code:`01-ai/Yi-34B`, etc.
Woosuk Kwon's avatar
Woosuk Kwon committed
80

Woosuk Kwon's avatar
Woosuk Kwon committed
81
If your model uses one of the above model architectures, you can seamlessly run your model with vLLM.
Woosuk Kwon's avatar
Woosuk Kwon committed
82
Otherwise, please refer to :ref:`Adding a New Model <adding_a_new_model>` for instructions on how to implement support for your model.
83
Alternatively, you can raise an issue on our `GitHub <https://github.com/vllm-project/vllm/issues>`_ project.
Woosuk Kwon's avatar
Woosuk Kwon committed
84

85
.. note::
86
    Currently, the ROCm version of vLLM supports Mistral and Mixtral only for context lengths up to 4096.
87

Woosuk Kwon's avatar
Woosuk Kwon committed
88
89
90
91
92
.. tip::
    The easiest way to check if your model is supported is to run the program below:

    .. code-block:: python

Woosuk Kwon's avatar
Woosuk Kwon committed
93
        from vllm import LLM
Woosuk Kwon's avatar
Woosuk Kwon committed
94
95
96
97
98

        llm = LLM(model=...)  # Name or path of your model
        output = llm.generate("Hello, my name is")
        print(output)

99
100
101
    If vLLM successfully generates text, it indicates that your model is supported.

.. tip::
102
    To use models from `ModelScope <https://www.modelscope.cn>`_ instead of HuggingFace Hub, set an environment variable:
103
104
105
106
107

    .. code-block:: shell

       $ export VLLM_USE_MODELSCOPE=True

108
109
    And use with :code:`trust_remote_code=True`.

110
111
112
113
114
115
116
    .. code-block:: python

        from vllm import LLM

        llm = LLM(model=..., revision=..., trust_remote_code=True)  # Name or path of your model
        output = llm.generate("Hello, my name is")
        print(output)