multimodal_inputs.md 30.7 KB
Newer Older
1
# Multimodal Inputs
2

3
This page teaches you how to pass multi-modal inputs to [multi-modal models](../models/supported_models.md#list-of-multimodal-language-models) in vLLM.
4

5
!!! note
6
    We are actively iterating on multi-modal support. See [this RFC](https://github.com/vllm-project/vllm/issues/4194) for upcoming changes,
7
    and [open an issue on GitHub](https://github.com/vllm-project/vllm/issues/new/choose) if you have any feedback or feature requests.
8

9
10
!!! tip
    When serving multi-modal models, consider setting `--allowed-media-domains` to restrict domain that vLLM can access to prevent it from accessing arbitrary endpoints that can potentially be vulnerable to Server-Side Request Forgery (SSRF) attacks. You can provide a list of domains for this arg. For example: `--allowed-media-domains upload.wikimedia.org github.com www.bogotobogo.com`
11
12
13

    Also, consider setting `VLLM_MEDIA_URL_ALLOW_REDIRECTS=0` to prevent HTTP redirects from being followed to bypass domain restrictions.

14
15
    This restriction is especially important if you run vLLM in a containerized environment where the vLLM pods may have unrestricted access to internal networks.

16
17
## Offline Inference

18
To input multi-modal data, follow this schema in [vllm.inputs.PromptType][]:
19
20

- `prompt`: The prompt should follow the format that is documented on HuggingFace.
21
- `multi_modal_data`: This is a dictionary that follows the schema defined in [vllm.multimodal.inputs.MultiModalDataDict][].
22

23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
### Stable UUIDs for Caching (multi_modal_uuids)

When using multi-modal inputs, vLLM normally hashes each media item by content to enable caching across requests. You can optionally pass `multi_modal_uuids` to provide your own stable IDs for each item so caching can reuse work across requests without rehashing the raw content.

??? code

    ```python
    from vllm import LLM
    from PIL import Image

    # Qwen2.5-VL example with two images
    llm = LLM(model="Qwen/Qwen2.5-VL-3B-Instruct")

    prompt = "USER: <image><image>\nDescribe the differences.\nASSISTANT:"
    img_a = Image.open("/path/to/a.jpg")
    img_b = Image.open("/path/to/b.jpg")

    outputs = llm.generate({
        "prompt": prompt,
        "multi_modal_data": {"image": [img_a, img_b]},
        # Provide stable IDs for caching.
        # Requirements (matched by this example):
        #  - Include every modality present in multi_modal_data.
        #  - For lists, provide the same number of entries.
        #  - Use None to fall back to content hashing for that item.
        "multi_modal_uuids": {"image": ["sku-1234-a", None]},
    })

    for o in outputs:
        print(o.outputs[0].text)
    ```

55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
Using UUIDs, you can also skip sending media data entirely if you expect cache hits for respective items. Note that the request will fail if the skipped media doesn't have a corresponding UUID, or if the UUID fails to hit the cache.

??? code

    ```python
    from vllm import LLM
    from PIL import Image

    # Qwen2.5-VL example with two images
    llm = LLM(model="Qwen/Qwen2.5-VL-3B-Instruct")

    prompt = "USER: <image><image>\nDescribe the differences.\nASSISTANT:"
    img_b = Image.open("/path/to/b.jpg")

    outputs = llm.generate({
        "prompt": prompt,
        "multi_modal_data": {"image": [None, img_b]},
        # Since img_a is expected to be cached, we can skip sending the actual
        # image entirely.
        "multi_modal_uuids": {"image": ["sku-1234-a", None]},
    })

    for o in outputs:
        print(o.outputs[0].text)
    ```

81
82
83
!!! warning
    If both multimodal processor caching and prefix caching are disabled, user-provided `multi_modal_uuids` are ignored.

84
### Image Inputs
85

86
You can pass a single image to the `'image'` field of the multi-modal dictionary, as shown in the following examples:
87

88
??? code
89

90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
    ```python
    from vllm import LLM

    llm = LLM(model="llava-hf/llava-1.5-7b-hf")

    # Refer to the HuggingFace repo for the correct format to use
    prompt = "USER: <image>\nWhat is the content of this image?\nASSISTANT:"

    # Load the image using PIL.Image
    image = PIL.Image.open(...)

    # Single prompt inference
    outputs = llm.generate({
        "prompt": prompt,
        "multi_modal_data": {"image": image},
    })

    for o in outputs:
        generated_text = o.outputs[0].text
        print(generated_text)

    # Batch inference
    image_1 = PIL.Image.open(...)
    image_2 = PIL.Image.open(...)
    outputs = llm.generate(
        [
            {
                "prompt": "USER: <image>\nWhat is the content of this image?\nASSISTANT:",
                "multi_modal_data": {"image": image_1},
            },
            {
                "prompt": "USER: <image>\nWhat's the color of this image?\nASSISTANT:",
                "multi_modal_data": {"image": image_2},
            }
        ]
    )

    for o in outputs:
        generated_text = o.outputs[0].text
        print(generated_text)
    ```
131

132
Full example: [examples/offline_inference/vision_language.py](../../examples/offline_inference/vision_language.py)
133
134
135

To substitute multiple images inside the same text prompt, you can pass in a list of images instead:

136
??? code
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156

    ```python
    from vllm import LLM

    llm = LLM(
        model="microsoft/Phi-3.5-vision-instruct",
        trust_remote_code=True,  # Required to load Phi-3.5-vision
        max_model_len=4096,  # Otherwise, it may not fit in smaller GPUs
        limit_mm_per_prompt={"image": 2},  # The maximum number to accept
    )

    # Refer to the HuggingFace repo for the correct format to use
    prompt = "<|user|>\n<|image_1|>\n<|image_2|>\nWhat is the content of each image?<|end|>\n<|assistant|>\n"

    # Load the images using PIL.Image
    image1 = PIL.Image.open(...)
    image2 = PIL.Image.open(...)

    outputs = llm.generate({
        "prompt": prompt,
157
        "multi_modal_data": {"image": [image1, image2]},
158
159
160
161
162
163
    })

    for o in outputs:
        generated_text = o.outputs[0].text
        print(generated_text)
    ```
164

165
Full example: [examples/offline_inference/vision_language_multi_image.py](../../examples/offline_inference/vision_language_multi_image.py)
166

167
If using the [LLM.chat](../models/generative_models.md#llmchat) method, you can pass images directly in the message content using various formats: image URLs, PIL Image objects, or pre-computed embeddings:
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183

```python
from vllm import LLM
from vllm.assets.image import ImageAsset

llm = LLM(model="llava-hf/llava-1.5-7b-hf")
image_url = "https://picsum.photos/id/32/512/512"
image_pil = ImageAsset('cherry_blossom').pil_image
image_embeds = torch.load(...)

conversation = [
    {"role": "system", "content": "You are a helpful assistant"},
    {"role": "user", "content": "Hello"},
    {"role": "assistant", "content": "Hello! How can I assist you today?"},
    {
        "role": "user",
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
        "content": [
            {
                "type": "image_url",
                "image_url": {"url": image_url},
            },
            {
                "type": "image_pil",
                "image_pil": image_pil,
            },
            {
                "type": "image_embeds",
                "image_embeds": image_embeds,
            },
            {
                "type": "text",
                "text": "What's in these images?",
            },
        ],
202
203
204
205
206
207
208
209
210
211
212
    },
]

# Perform inference and log output.
outputs = llm.chat(conversation)

for o in outputs:
    generated_text = o.outputs[0].text
    print(generated_text)
```

213
214
Multi-image input can be extended to perform video captioning. We show this with [Qwen2-VL](https://huggingface.co/Qwen/Qwen2-VL-2B-Instruct) as it supports videos:

215
??? code
Reid's avatar
Reid committed
216

217
218
    ```python
    from vllm import LLM
219

220
221
222
223
224
225
226
227
    # Specify the maximum number of frames per video to be 4. This can be changed.
    llm = LLM("Qwen/Qwen2-VL-2B-Instruct", limit_mm_per_prompt={"image": 4})

    # Create the request payload.
    video_frames = ... # load your video making sure it only has the number of frames specified earlier.
    message = {
        "role": "user",
        "content": [
228
229
230
231
            {
                "type": "text",
                "text": "Describe this set of frames. Consider the frames to be a part of the same video.",
            },
232
233
234
235
236
237
238
239
240
241
242
243
244
245
        ],
    }
    for i in range(len(video_frames)):
        base64_image = encode_image(video_frames[i]) # base64 encoding.
        new_image = {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{base64_image}"}}
        message["content"].append(new_image)

    # Perform inference and log output.
    outputs = llm.chat([message])

    for o in outputs:
        generated_text = o.outputs[0].text
        print(generated_text)
    ```
246

247
248
249
250
251
252
253
254
#### Custom RGBA Background Color

When loading RGBA images (images with transparency), vLLM converts them to RGB format. By default, transparent pixels are replaced with white background. You can customize this background color using the `rgba_background_color` parameter in `media_io_kwargs`.

??? code

    ```python
    from vllm import LLM
255

256
257
    # Default white background (no configuration needed)
    llm = LLM(model="llava-hf/llava-1.5-7b-hf")
258

259
260
261
    # Custom black background for dark theme
    llm = LLM(
        model="llava-hf/llava-1.5-7b-hf",
262
        media_io_kwargs={"image": {"rgba_background_color": [0, 0, 0]}},
263
    )
264

265
266
    # Custom brand color background (e.g., blue)
    llm = LLM(
267
        model="llava-hf/llava-1.5-7b-hf",
268
        media_io_kwargs={"image": {"rgba_background_color": [0, 0, 255]}},
269
270
271
272
273
274
275
276
    )
    ```

!!! note
    - The `rgba_background_color` accepts RGB values as a list `[R, G, B]` or tuple `(R, G, B)` where each value is 0-255
    - This setting only affects RGBA images with transparency; RGB images are unchanged
    - If not specified, the default white background `(255, 255, 255)` is used for backward compatibility

277
### Video Inputs
278

279
You can pass a list of NumPy arrays directly to the `'video'` field of the multi-modal dictionary
280
281
instead of using multi-image input.

282
283
284
285
286
287
288
289
290
Instead of NumPy arrays, you can also pass `'torch.Tensor'` instances, as shown in this example using Qwen2.5-VL:

??? code

    ```python
    from transformers import AutoProcessor
    from vllm import LLM, SamplingParams
    from qwen_vl_utils import process_vision_info

291
    model_path = "Qwen/Qwen2.5-VL-3B-Instruct"
292
293
294
295
296
297
298
299
300
    video_path = "https://content.pexels.com/videos/free-videos.mp4"

    llm = LLM(
        model=model_path,
        gpu_memory_utilization=0.8,
        enforce_eager=True,
        limit_mm_per_prompt={"video": 1},
    )

301
    sampling_params = SamplingParams(max_tokens=1024)
302
303

    video_messages = [
304
305
306
307
308
309
310
        {
            "role": "system",
            "content": "You are a helpful assistant.",
        },
        {
            "role": "user",
            "content": [
311
312
313
314
315
                {"type": "text", "text": "describe this video."},
                {
                    "type": "video",
                    "video": video_path,
                    "total_pixels": 20480 * 28 * 28,
316
317
                    "min_pixels": 16 * 28 * 28,
                },
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
            ]
        },
    ]

    messages = video_messages
    processor = AutoProcessor.from_pretrained(model_path)
    prompt = processor.apply_chat_template(
        messages,
        tokenize=False,
        add_generation_prompt=True,
    )

    image_inputs, video_inputs = process_vision_info(messages)
    mm_data = {}
    if video_inputs is not None:
        mm_data["video"] = video_inputs

    llm_inputs = {
        "prompt": prompt,
        "multi_modal_data": mm_data,
    }

    outputs = llm.generate([llm_inputs], sampling_params=sampling_params)
    for o in outputs:
        generated_text = o.outputs[0].text
        print(generated_text)
    ```

    !!! note
        'process_vision_info' is only applicable to Qwen2.5-VL and similar models.

349
Full example: [examples/offline_inference/vision_language.py](../../examples/offline_inference/vision_language.py)
350

351
### Audio Inputs
352

353
You can pass a tuple `(array, sampling_rate)` to the `'audio'` field of the multi-modal dictionary.
354

355
Full example: [examples/offline_inference/audio_language.py](../../examples/offline_inference/audio_language.py)
356

357
### Embedding Inputs
358
359

To input pre-computed embeddings belonging to a data type (i.e. image, video, or audio) directly to the language model,
360
pass a tensor of shape `(num_items, feature_size, hidden_size of LM)` to the corresponding field of the multi-modal dictionary.
361

362
363
364
365
366
367
You must enable this feature via `enable_mm_embeds=True`.

!!! warning
    The vLLM engine may crash if incorrect shape of embeddings is passed.
    Only enable this flag for trusted users!

368
369
#### Image Embeddings

370
??? code
Reid's avatar
Reid committed
371

372
373
    ```python
    from vllm import LLM
374

375
    # Inference with image embeddings as input
376
    llm = LLM(model="llava-hf/llava-1.5-7b-hf", enable_mm_embeds=True)
377

378
379
    # Refer to the HuggingFace repo for the correct format to use
    prompt = "USER: <image>\nWhat is the content of this image?\nASSISTANT:"
380

381
382
383
    # Embeddings for single image
    # torch.Tensor of shape (1, image_feature_size, hidden_size of LM)
    image_embeds = torch.load(...)
384

385
386
387
388
389
390
391
392
393
    outputs = llm.generate({
        "prompt": prompt,
        "multi_modal_data": {"image": image_embeds},
    })

    for o in outputs:
        generated_text = o.outputs[0].text
        print(generated_text)
    ```
394
395
396

For Qwen2-VL and MiniCPM-V, we accept additional parameters alongside the embeddings:

397
??? code
398
399
400
401
402
403
404
405
406
407

    ```python
    # Construct the prompt based on your model
    prompt = ...

    # Embeddings for multiple images
    # torch.Tensor of shape (num_images, image_feature_size, hidden_size of LM)
    image_embeds = torch.load(...)

    # Qwen2-VL
408
409
410
411
412
    llm = LLM(
        "Qwen/Qwen2-VL-2B-Instruct",
        limit_mm_per_prompt={"image": 4},
        enable_mm_embeds=True,
    )
413
414
415
416
417
418
    mm_data = {
        "image": {
            "image_embeds": image_embeds,
            # image_grid_thw is needed to calculate positional encoding.
            "image_grid_thw": torch.load(...),  # torch.Tensor of shape (1, 3),
        }
419
    }
420
421

    # MiniCPM-V
422
423
424
425
426
427
    llm = LLM(
        "openbmb/MiniCPM-V-2_6",
        trust_remote_code=True,
        limit_mm_per_prompt={"image": 4},
        enable_mm_embeds=True,
    )
428
429
430
431
432
433
    mm_data = {
        "image": {
            "image_embeds": image_embeds,
            # image_sizes is needed to calculate details of the sliced image.
            "image_sizes": [image.size for image in images],  # list of image sizes
        }
434
435
    }

436
437
438
439
    outputs = llm.generate({
        "prompt": prompt,
        "multi_modal_data": mm_data,
    })
440

441
442
443
444
    for o in outputs:
        generated_text = o.outputs[0].text
        print(generated_text)
    ```
445

446
447
For Qwen3-VL, the `image_embeds` should contain both the base image embedding and deepstack features.

448
#### Audio Embedding Inputs
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477

You can pass pre-computed audio embeddings similar to image embeddings:

??? code

    ```python
    from vllm import LLM
    import torch

    # Enable audio embeddings support
    llm = LLM(model="fixie-ai/ultravox-v0_5-llama-3_2-1b", enable_mm_embeds=True)

    # Refer to the HuggingFace repo for the correct format to use
    prompt = "USER: <audio>\nWhat is in this audio?\nASSISTANT:"

    # Load pre-computed audio embeddings
    # torch.Tensor of shape (1, audio_feature_size, hidden_size of LM)
    audio_embeds = torch.load(...)

    outputs = llm.generate({
        "prompt": prompt,
        "multi_modal_data": {"audio": audio_embeds},
    })

    for o in outputs:
        generated_text = o.outputs[0].text
        print(generated_text)
    ```

478
## Online Serving
479

480
Our OpenAI-compatible server accepts multi-modal data via the [Chat Completions API](https://platform.openai.com/docs/api-reference/chat). Media inputs also support optional UUIDs users can provide to uniquely identify each media, which is used to cache the media results across requests.
481

482
!!! important
483
484
    A chat template is **required** to use Chat Completions API.
    For HF format models, the default chat template is defined inside `chat_template.json` or `tokenizer_config.json`.
485

486
    If no default chat template is available, we will first look for a built-in fallback in [vllm/transformers_utils/chat_templates/registry.py](../../vllm/transformers_utils/chat_templates/registry.py).
487
    If no fallback is available, an error is raised and you have to provide the chat template manually via the `--chat-template` argument.
488

489
490
    For certain models, we provide alternative chat templates inside [examples](../../examples).
    For example, VLM2Vec uses [examples/template_vlm2vec_phi3v.jinja](../../examples/template_vlm2vec_phi3v.jinja) which is different from the default one for Phi-3-Vision.
491

492
### Image Inputs
493
494
495
496
497
498
499

Image input is supported according to [OpenAI Vision API](https://platform.openai.com/docs/guides/vision).
Here is a simple example using Phi-3.5-Vision.

First, launch the OpenAI-compatible server:

```bash
500
vllm serve microsoft/Phi-3.5-vision-instruct --runner generate \
501
  --trust-remote-code --max-model-len 4096 --limit-mm-per-prompt '{"image":2}'
502
503
504
505
```

Then, you can use the OpenAI client as follows:

506
??? code
507
508

    ```python
509
    import os
510
511
512
513
514
515
516
517
518
519
520
    from openai import OpenAI

    openai_api_key = "EMPTY"
    openai_api_base = "http://localhost:8000/v1"

    client = OpenAI(
        api_key=openai_api_key,
        base_url=openai_api_base,
    )

    # Single-image input inference
521
522

    # Public image URL for testing remote image processing
523
    image_url = "https://vllm-public-assets.s3.us-west-2.amazonaws.com/vision_model_images/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg"
524

525
    # Create chat completion with remote image
526
527
    chat_response = client.chat.completions.create(
        model="microsoft/Phi-3.5-vision-instruct",
528
529
530
531
532
533
534
535
536
        messages=[
            {
                "role": "user",
                "content": [
                    # NOTE: The prompt formatting with the image token `<image>` is not needed
                    # since the prompt will be processed automatically by the API server.
                    {
                        "type": "text",
                        "text": "What’s in this image?",
537
                    },
538
539
540
541
542
543
544
545
                    {
                        "type": "image_url",
                        "image_url": {"url": image_url},
                        "uuid": image_url,  # Optional
                    },
                ],
            }
        ],
546
547
548
    )
    print("Chat completion output:", chat_response.choices[0].message.content)

549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
    # Local image file path (update this to point to your actual image file)
    image_file = "/path/to/image.jpg"

    # Create chat completion with local image file
    # Launch the API server/engine with the --allowed-local-media-path argument.
    if os.path.exists(image_file):
        chat_completion_from_local_image_url = client.chat.completions.create(
            model="microsoft/Phi-3.5-vision-instruct",
            messages=[
                {
                    "role": "user",
                    "content": [
                        {
                            "type": "text",
                            "text": "What’s in this image?",
                        },
                        {
                            "type": "image_url",
                            "image_url": {"url": f"file://{image_file}"},
                        },
                    ],
                }
            ],
        )
        result = chat_completion_from_local_image_url.choices[0].message.content
        print("Chat completion output from local image file:\n", result)
    else:
        print(f"Local image file not found at {image_file}, skipping local file test.")

578
    # Multi-image input inference
579
580
    image_url_duck = "https://vllm-public-assets.s3.us-west-2.amazonaws.com/multimodal_asset/duck.jpg"
    image_url_lion = "https://vllm-public-assets.s3.us-west-2.amazonaws.com/multimodal_asset/lion.jpg"
581
582
583

    chat_response = client.chat.completions.create(
        model="microsoft/Phi-3.5-vision-instruct",
584
585
586
587
588
589
590
        messages=[
            {
                "role": "user",
                "content": [
                    {
                        "type": "text",
                        "text": "What are the animals in these images?",
591
                    },
592
593
594
595
                    {
                        "type": "image_url",
                        "image_url": {"url": image_url_duck},
                        "uuid": image_url_duck,  # Optional
596
                    },
597
598
599
600
601
602
603
604
                    {
                        "type": "image_url",
                        "image_url": {"url": image_url_lion},
                        "uuid": image_url_lion,  # Optional
                    },
                ],
            }
        ],
605
606
607
    )
    print("Chat completion output:", chat_response.choices[0].message.content)
    ```
608

609
Full example: [examples/online_serving/openai_chat_completion_client_for_multimodal.py](../../examples/online_serving/openai_chat_completion_client_for_multimodal.py)
610

611
612
613
!!! tip
    Loading from local file paths is also supported on vLLM: You can specify the allowed local media path via `--allowed-local-media-path` when launching the API server/engine,
    and pass the file path as `url` in the API request.
614

615
616
617
!!! tip
    There is no need to place image placeholders in the text content of the API request - they are already represented by the image content.
    In fact, you can place image placeholders in the middle of the text by interleaving text and image content.
618

619
620
621
!!! note
    By default, the timeout for fetching images through HTTP URL is `5` seconds.
    You can override this by setting the environment variable:
622

623
    ```bash
624
625
    export VLLM_IMAGE_FETCH_TIMEOUT=<timeout>
    ```
626

627
### Video Inputs
628

629
Instead of `image_url`, you can pass a video file via `video_url`. Here is a simple example using [LLaVA-OneVision](https://huggingface.co/llava-hf/llava-onevision-qwen2-0.5b-ov-hf).
630

631
632
633
First, launch the OpenAI-compatible server:

```bash
634
vllm serve llava-hf/llava-onevision-qwen2-0.5b-ov-hf --runner generate --max-model-len 8192
635
636
637
```

Then, you can use the OpenAI client as follows:
638

639
??? code
640

641
642
    ```python
    from openai import OpenAI
643

644
645
    openai_api_key = "EMPTY"
    openai_api_base = "http://localhost:8000/v1"
646

647
648
649
650
    client = OpenAI(
        api_key=openai_api_key,
        base_url=openai_api_base,
    )
651

652
    video_url = "http://commondatastorage.googleapis.com/gtv-videos-bucket/sample/ForBiggerFun.mp4"
653

654
655
    ## Use video url in the payload
    chat_completion_from_url = client.chat.completions.create(
656
657
658
659
660
661
662
        messages=[
            {
                "role": "user",
                "content": [
                    {
                        "type": "text",
                        "text": "What's in this video?",
663
                    },
664
665
666
667
668
669
670
671
                    {
                        "type": "video_url",
                        "video_url": {"url": video_url},
                        "uuid": video_url,  # Optional
                    },
                ],
            }
        ],
672
673
674
675
676
677
678
        model=model,
        max_completion_tokens=64,
    )

    result = chat_completion_from_url.choices[0].message.content
    print("Chat completion output from image url:", result)
    ```
679

680
Full example: [examples/online_serving/openai_chat_completion_client_for_multimodal.py](../../examples/online_serving/openai_chat_completion_client_for_multimodal.py)
681

682
683
684
!!! note
    By default, the timeout for fetching videos through HTTP URL is `30` seconds.
    You can override this by setting the environment variable:
685

686
    ```bash
687
688
    export VLLM_VIDEO_FETCH_TIMEOUT=<timeout>
    ```
689

690
691
692
693
694
695
696
697
698
699
700
701
702
703
#### Custom RGBA Background Color

To use a custom background color for RGBA images, pass the `rgba_background_color` parameter via `--media-io-kwargs`:

```bash
# Example: Black background for dark theme
vllm serve llava-hf/llava-1.5-7b-hf \
  --media-io-kwargs '{"image": {"rgba_background_color": [0, 0, 0]}}'

# Example: Custom gray background
vllm serve llava-hf/llava-1.5-7b-hf \
  --media-io-kwargs '{"image": {"rgba_background_color": [128, 128, 128]}}'
```

704
### Audio Inputs
705
706

Audio input is supported according to [OpenAI Audio API](https://platform.openai.com/docs/guides/audio?audio-generation-quickstart-example=audio-in).
707
Here is a simple example using Ultravox-v0.5-1B.
708
709
710
711

First, launch the OpenAI-compatible server:

```bash
712
vllm serve fixie-ai/ultravox-v0_5-llama-3_2-1b
713
714
715
716
```

Then, you can use the OpenAI client as follows:

717
??? code
718

719
720
721
722
723
    ```python
    import base64
    import requests
    from openai import OpenAI
    from vllm.assets.audio import AudioAsset
724

725
726
    def encode_base64_content_from_url(content_url: str) -> str:
        """Encode a content retrieved from a remote url to base64 format."""
727

728
729
730
        with requests.get(content_url) as response:
            response.raise_for_status()
            result = base64.b64encode(response.content).decode('utf-8')
731

732
        return result
733

734
735
    openai_api_key = "EMPTY"
    openai_api_base = "http://localhost:8000/v1"
736

737
738
739
740
    client = OpenAI(
        api_key=openai_api_key,
        base_url=openai_api_base,
    )
741

742
743
744
    # Any format supported by librosa is supported
    audio_url = AudioAsset("winning_call").url
    audio_base64 = encode_base64_content_from_url(audio_url)
745

746
    chat_completion_from_base64 = client.chat.completions.create(
747
748
749
750
751
752
753
        messages=[
            {
                "role": "user",
                "content": [
                    {
                        "type": "text",
                        "text": "What's in this audio?",
754
                    },
755
756
757
758
759
760
761
762
763
764
765
                    {
                        "type": "input_audio",
                        "input_audio": {
                            "data": audio_base64,
                            "format": "wav",
                        },
                        "uuid": audio_url,  # Optional
                    },
                ],
            },
        ],
766
767
768
769
770
771
772
        model=model,
        max_completion_tokens=64,
    )

    result = chat_completion_from_base64.choices[0].message.content
    print("Chat completion output from input audio:", result)
    ```
773

774
Alternatively, you can pass `audio_url`, which is the audio counterpart of `image_url` for image input:
775

776
??? code
777

778
779
    ```python
    chat_completion_from_url = client.chat.completions.create(
780
781
782
783
784
785
786
        messages=[
            {
                "role": "user",
                "content": [
                    {
                        "type": "text",
                        "text": "What's in this audio?",
787
                    },
788
789
790
791
792
793
794
795
                    {
                        "type": "audio_url",
                        "audio_url": {"url": audio_url},
                        "uuid": audio_url,  # Optional
                    },
                ],
            }
        ],
796
797
798
799
800
801
802
        model=model,
        max_completion_tokens=64,
    )

    result = chat_completion_from_url.choices[0].message.content
    print("Chat completion output from audio url:", result)
    ```
803

804
Full example: [examples/online_serving/openai_chat_completion_client_for_multimodal.py](../../examples/online_serving/openai_chat_completion_client_for_multimodal.py)
805

806
807
808
!!! note
    By default, the timeout for fetching audios through HTTP URL is `10` seconds.
    You can override this by setting the environment variable:
809

810
    ```bash
811
812
    export VLLM_AUDIO_FETCH_TIMEOUT=<timeout>
    ```
813

814
### Embedding Inputs
815

816
To input pre-computed embeddings belonging to a data type (i.e. image, video, or audio) directly to the language model,
817
818
819
820
821
822
823
pass a tensor of shape `(num_items, feature_size, hidden_size of LM)` to the corresponding field of the multi-modal dictionary.

You must enable this feature via the `--enable-mm-embeds` flag in `vllm serve`.

!!! warning
    The vLLM engine may crash if incorrect shape of embeddings is passed.
    Only enable this flag for trusted users!
824

825
#### Image Embedding Inputs
826

827
828
829
For image embeddings, you can pass the base64-encoded tensor to the `image_embeds` field.
The following example demonstrates how to pass image embeddings to the OpenAI server:

830
??? code
831
832

    ```python
833
834
    from vllm.utils.serial_utils import tensor2base64

835
836
837
    image_embedding = torch.load(...)
    grid_thw = torch.load(...) # Required by Qwen/Qwen2-VL-2B-Instruct

838
    base64_image_embedding = tensor2base64(image_embedding)
839
840
841
842
843
844
845
846
847

    client = OpenAI(
        # defaults to os.environ.get("OPENAI_API_KEY")
        api_key=openai_api_key,
        base_url=openai_api_base,
    )

    # Basic usage - this is equivalent to the LLaVA example for offline inference
    model = "llava-hf/llava-1.5-7b-hf"
848
    embeds = {
849
        "type": "image_embeds",
850
        "image_embeds": f"{base64_image_embedding}",
851
        "uuid": image_url,  # Optional
852
853
854
855
    }

    # Pass additional parameters (available to Qwen2-VL and MiniCPM-V)
    model = "Qwen/Qwen2-VL-2B-Instruct"
856
    embeds = {
857
858
        "type": "image_embeds",
        "image_embeds": {
859
860
            "image_embeds": f"{base64_image_embedding}",  # Required
            "image_grid_thw": f"{base64_image_grid_thw}",  # Required by Qwen/Qwen2-VL-2B-Instruct
861
        },
862
        "uuid": image_url,  # Optional
863
864
    }
    model = "openbmb/MiniCPM-V-2_6"
865
    embeds = {
866
867
        "type": "image_embeds",
        "image_embeds": {
868
869
            "image_embeds": f"{base64_image_embedding}",  # Required
            "image_sizes": f"{base64_image_sizes}",  # Required by openbmb/MiniCPM-V-2_6
870
        },
871
        "uuid": image_url,  # Optional
872
873
874
875
    }
    chat_completion = client.chat.completions.create(
        messages=[
            {
876
877
                "role": "system",
                "content": "You are a helpful assistant.",
878
            },
879
880
881
882
883
884
885
886
887
888
889
            {
                "role": "user",
                "content": [
                    {
                        "type": "text",
                        "text": "What's in this image?",
                    },
                    embeds,
                ],
            },
        ],
890
891
892
        model=model,
    )
    ```
893

894
895
896
897
898
899
900
901
902
903
904
905
906
907
For Online Serving, you can also skip sending media if you expect cache hits with provided UUIDs. You can do so by sending media like this:

    ```python
        # Image/video/audio URL:
        {
            "type": "image_url",
            "image_url": None,
            "uuid": image_uuid,
        },

        # image_embeds
        {
            "type": "image_embeds",
            "image_embeds": None,
908
            "uuid": image_uuid,
909
910
911
912
913
914
        },

        # input_audio:
        {
            "type": "input_audio",
            "input_audio": None,
915
            "uuid": audio_uuid,
916
917
918
919
920
        },

        # PIL Image:
        {
            "type": "image_pil",
921
922
923
            "image_pil": None,
            "uuid": image_uuid,
        },
924
925
926

    ```

927
!!! note
928
929
930
931
932
933
934
    Multiple messages can now contain `{"type": "image_embeds"}`, enabling you to pass multiple image embeddings in a single request (similar to regular images). The number of embeddings is limited by `--limit-mm-per-prompt`.

    **Important**: The embedding shape format differs based on the number of embeddings:

    - **Single embedding**: 3D tensor of shape `(1, feature_size, hidden_size)`
    - **Multiple embeddings**: List of 2D tensors, each of shape `(feature_size, hidden_size)`

935
    If used with a model that requires additional parameters, you must also provide a tensor for each of them, e.g. `image_grid_thw`, `image_sizes`, etc.