vlm.rst 8.59 KB
Newer Older
1
2
3
4
5
.. _vlm:

Using VLMs
==========

6
7
vLLM provides experimental support for Vision Language Models (VLMs). See the :ref:`list of supported VLMs here <supported_vlms>`.
This document shows you how to run and serve these models using vLLM.
8

9
10
11
.. important::
    We are actively iterating on VLM support. Expect breaking changes to VLM usage and development in upcoming releases without prior deprecation.

Cyrus Leung's avatar
Cyrus Leung committed
12
    We are continuously improving user & developer experience for VLMs. Please `open an issue on GitHub <https://github.com/vllm-project/vllm/issues/new/choose>`_ if you have any feedback or feature requests.
13

14
15
16
17
18
Offline Inference
-----------------

Single-image input
^^^^^^^^^^^^^^^^^^
19

20
The :class:`~vllm.LLM` class can be instantiated in much the same way as language-only models.
21
22
23

.. code-block:: python

24
    llm = LLM(model="llava-hf/llava-1.5-7b-hf")
25

26
.. note::
27
    We have removed all vision language related CLI args in the ``0.5.1`` release. **This is a breaking change**, so please update your code to follow
28
    the above snippet. Specifically, ``image_feature_size`` can no longer be specified as we now calculate that internally for each model.
29

30
To pass an image to the model, note the following in :class:`vllm.inputs.PromptType`:
31

32
* ``prompt``: The prompt should follow the format that is documented on HuggingFace.
33
34
* ``multi_modal_data``: This is a dictionary that follows the schema defined in :class:`vllm.multimodal.MultiModalDataDict`. 

35
36
.. code-block:: python

37
38
    # Refer to the HuggingFace repo for the correct format to use
    prompt = "USER: <image>\nWhat is the content of this image?\nASSISTANT:"
39
40

    # Load the image using PIL.Image
41
42
43
    image = PIL.Image.open(...)
    
    # Single prompt inference
44
45
    outputs = llm.generate({
        "prompt": prompt,
46
        "multi_modal_data": {"image": image},
47
48
    })

49
50
51
52
53
54
55
56
57
58
59
    for o in outputs:
        generated_text = o.outputs[0].text
        print(generated_text)

    # Inference with image embeddings as input
    image_embeds = torch.load(...) # torch.Tensor of shape (1, image_feature_size, hidden_size of LM)
    outputs = llm.generate({
        "prompt": prompt,
        "multi_modal_data": {"image": image_embeds},
    })

60
61
62
    for o in outputs:
        generated_text = o.outputs[0].text
        print(generated_text)
63
64
65
66
67
68
69
70
71
72
73
74
75

    # Inference with image embeddings as input with additional parameters
    # Specifically, we are conducting a trial run of Qwen2VL with the new input format, as the model utilizes additional parameters for calculating positional encoding.
    image_embeds = torch.load(...) # torch.Tensor of shape (1, image_feature_size, hidden_size of LM)
    image_grid_thw = torch.load(...) # torch.Tensor of shape (1, 3)
    mm_data['image'] = {
        "image_embeds": image_embeds,
        "image_grid_thw":  image_grid_thw,
    }
    outputs = llm.generate({
        "prompt": prompt,
        "multi_modal_data": mm_data,
    })
76
    
77
78
79
80
    for o in outputs:
        generated_text = o.outputs[0].text
        print(generated_text)

81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
    # Batch inference
    image_1 = PIL.Image.open(...)
    image_2 = PIL.Image.open(...)
    outputs = llm.generate(
        [
            {
                "prompt": "USER: <image>\nWhat is the content of this image?\nASSISTANT:",
                "multi_modal_data": {"image": image_1},
            },
            {
                "prompt": "USER: <image>\nWhat's the color of this image?\nASSISTANT:",
                "multi_modal_data": {"image": image_2},
            }
        ]
    )

    for o in outputs:
        generated_text = o.outputs[0].text
        print(generated_text)
100

101
A code example can be found in `examples/offline_inference_vision_language.py <https://github.com/vllm-project/vllm/blob/main/examples/offline_inference_vision_language.py>`_.
102

103
104
Multi-image input
^^^^^^^^^^^^^^^^^
105

106
Multi-image input is only supported for a subset of VLMs, as shown :ref:`here <supported_vlms>`.
107

108
To enable multiple multi-modal items per text prompt, you have to set ``limit_mm_per_prompt`` for the :class:`~vllm.LLM` class.
109

110
.. code-block:: python
111

112
113
114
115
116
117
    llm = LLM(
        model="microsoft/Phi-3.5-vision-instruct",
        trust_remote_code=True,  # Required to load Phi-3.5-vision
        max_model_len=4096,  # Otherwise, it may not fit in smaller GPUs
        limit_mm_per_prompt={"image": 2},  # The maximum number to accept
    )
118

119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
Instead of passing in a single image, you can pass in a list of images.

.. code-block:: python

    # Refer to the HuggingFace repo for the correct format to use
    prompt = "<|user|>\n<image_1>\n<image_2>\nWhat is the content of each image?<|end|>\n<|assistant|>\n"

    # Load the images using PIL.Image
    image1 = PIL.Image.open(...)
    image2 = PIL.Image.open(...)

    outputs = llm.generate({
        "prompt": prompt,
        "multi_modal_data": {
            "image": [image1, image2]
        },
    })

    for o in outputs:
        generated_text = o.outputs[0].text
        print(generated_text)

A code example can be found in `examples/offline_inference_vision_language_multi_image.py <https://github.com/vllm-project/vllm/blob/main/examples/offline_inference_vision_language_multi_image.py>`_.

Online Inference
----------------

OpenAI Vision API
^^^^^^^^^^^^^^^^^

You can serve vision language models with vLLM's HTTP server that is compatible with `OpenAI Vision API <https://platform.openai.com/docs/guides/vision>`_.

Below is an example on how to launch the same ``microsoft/Phi-3.5-vision-instruct`` with vLLM's OpenAI-compatible API server.
152
153
154

.. code-block:: bash

155
156
    vllm serve microsoft/Phi-3.5-vision-instruct --max-model-len 4096 \
      --trust-remote-code --limit-mm-per-prompt image=2
157

158
.. important::
159
160
161
162
163
164
    Since OpenAI Vision API is based on `Chat Completions <https://platform.openai.com/docs/api-reference/chat>`_ API,
    a chat template is **required** to launch the API server.

    Although Phi-3.5-Vision comes with a chat template, for other models you may have to provide one if the model's tokenizer does not come with it.
    The chat template can be inferred based on the documentation on the model's HuggingFace repo.
    For example, LLaVA-1.5 (``llava-hf/llava-1.5-7b-hf``) requires a chat template that can be found `here <https://github.com/vllm-project/vllm/blob/main/examples/template_llava.jinja>`_.
165

166
167
168
169
170
To consume the server, you can use the OpenAI client like in the example below:

.. code-block:: python

    from openai import OpenAI
171

172
173
    openai_api_key = "EMPTY"
    openai_api_base = "http://localhost:8000/v1"
174

175
176
177
178
    client = OpenAI(
        api_key=openai_api_key,
        base_url=openai_api_base,
    )
179
180
181
182

    # Single-image input inference
    image_url = "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg"

183
    chat_response = client.chat.completions.create(
184
        model="microsoft/Phi-3.5-vision-instruct",
185
186
187
        messages=[{
            "role": "user",
            "content": [
188
189
                # NOTE: The prompt formatting with the image token `<image>` is not needed
                # since the prompt will be processed automatically by the API server.
190
191
                {"type": "text", "text": "What’s in this image?"},
                {"type": "image_url", "image_url": {"url": image_url}},
192
193
194
            ],
        }],
    )
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
    print("Chat completion output:", chat_response.choices[0].message.content)

    # Multi-image input inference
    image_url_duck = "https://upload.wikimedia.org/wikipedia/commons/d/da/2015_Kaczka_krzy%C5%BCowka_w_wodzie_%28samiec%29.jpg"
    image_url_lion = "https://upload.wikimedia.org/wikipedia/commons/7/77/002_The_lion_king_Snyggve_in_the_Serengeti_National_Park_Photo_by_Giles_Laurent.jpg"

    chat_response = client.chat.completions.create(
        model="microsoft/Phi-3.5-vision-instruct",
        messages=[{
            "role": "user",
            "content": [
                {"type": "text", "text": "What are the animals in these images?"},
                {"type": "image_url", "image_url": {"url": image_url_duck}},
                {"type": "image_url", "image_url": {"url": image_url_lion}},
            ],
        }],
    )
    print("Chat completion output:", chat_response.choices[0].message.content)

214

215
216
A full code example can be found in `examples/openai_vision_api_client.py <https://github.com/vllm-project/vllm/blob/main/examples/openai_vision_api_client.py>`_.

217
218
219
220
221
222
223
224
225
.. note::

    By default, the timeout for fetching images through http url is ``5`` seconds. You can override this by setting the environment variable:

    .. code-block:: shell

        export VLLM_IMAGE_FETCH_TIMEOUT=<timeout>

.. note::
226
    There is no need to format the prompt in the API request since it will be handled by the server.