multimodal_inputs.md 35.4 KB
Newer Older
1
# Multimodal Inputs
2

3
This page teaches you how to pass multi-modal inputs to [multi-modal models](../models/supported_models.md#list-of-multimodal-language-models) in vLLM.
4

5
!!! note
6
    We are actively iterating on multi-modal support. See [this RFC](https://github.com/vllm-project/vllm/issues/4194) for upcoming changes,
7
    and [open an issue on GitHub](https://github.com/vllm-project/vllm/issues/new/choose) if you have any feedback or feature requests.
8

9
10
!!! tip
    When serving multi-modal models, consider setting `--allowed-media-domains` to restrict domain that vLLM can access to prevent it from accessing arbitrary endpoints that can potentially be vulnerable to Server-Side Request Forgery (SSRF) attacks. You can provide a list of domains for this arg. For example: `--allowed-media-domains upload.wikimedia.org github.com www.bogotobogo.com`
11
12
13

    Also, consider setting `VLLM_MEDIA_URL_ALLOW_REDIRECTS=0` to prevent HTTP redirects from being followed to bypass domain restrictions.

14
15
    This restriction is especially important if you run vLLM in a containerized environment where the vLLM pods may have unrestricted access to internal networks.

16
17
## Offline Inference

18
To input multi-modal data, follow this schema in [vllm.inputs.PromptType][]:
19
20

- `prompt`: The prompt should follow the format that is documented on HuggingFace.
21
- `multi_modal_data`: This is a dictionary that follows the schema defined in [vllm.inputs.MultiModalDataDict][].
22

23
### Image Inputs
24

25
You can pass a single image to the `'image'` field of the multi-modal dictionary, as shown in the following examples:
26

27
??? code
28

29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
    ```python
    from vllm import LLM

    llm = LLM(model="llava-hf/llava-1.5-7b-hf")

    # Refer to the HuggingFace repo for the correct format to use
    prompt = "USER: <image>\nWhat is the content of this image?\nASSISTANT:"

    # Load the image using PIL.Image
    image = PIL.Image.open(...)

    # Single prompt inference
    outputs = llm.generate({
        "prompt": prompt,
        "multi_modal_data": {"image": image},
    })

    for o in outputs:
        generated_text = o.outputs[0].text
        print(generated_text)

    # Batch inference
    image_1 = PIL.Image.open(...)
    image_2 = PIL.Image.open(...)
    outputs = llm.generate(
        [
            {
                "prompt": "USER: <image>\nWhat is the content of this image?\nASSISTANT:",
                "multi_modal_data": {"image": image_1},
            },
            {
                "prompt": "USER: <image>\nWhat's the color of this image?\nASSISTANT:",
                "multi_modal_data": {"image": image_2},
            }
        ]
    )

    for o in outputs:
        generated_text = o.outputs[0].text
        print(generated_text)
    ```
70

71
Full example: [examples/offline_inference/vision_language.py](../../examples/offline_inference/vision_language.py)
72
73
74

To substitute multiple images inside the same text prompt, you can pass in a list of images instead:

75
??? code
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95

    ```python
    from vllm import LLM

    llm = LLM(
        model="microsoft/Phi-3.5-vision-instruct",
        trust_remote_code=True,  # Required to load Phi-3.5-vision
        max_model_len=4096,  # Otherwise, it may not fit in smaller GPUs
        limit_mm_per_prompt={"image": 2},  # The maximum number to accept
    )

    # Refer to the HuggingFace repo for the correct format to use
    prompt = "<|user|>\n<|image_1|>\n<|image_2|>\nWhat is the content of each image?<|end|>\n<|assistant|>\n"

    # Load the images using PIL.Image
    image1 = PIL.Image.open(...)
    image2 = PIL.Image.open(...)

    outputs = llm.generate({
        "prompt": prompt,
96
        "multi_modal_data": {"image": [image1, image2]},
97
98
99
100
101
102
    })

    for o in outputs:
        generated_text = o.outputs[0].text
        print(generated_text)
    ```
103

104
Full example: [examples/offline_inference/vision_language_multi_image.py](../../examples/offline_inference/vision_language_multi_image.py)
105

106
If using the [LLM.chat](../models/generative_models.md#llmchat) method, you can pass images directly in the message content using various formats: image URLs, PIL Image objects, or pre-computed embeddings:
107

108
??? code
109

110
111
112
    ```python
    from vllm import LLM
    from vllm.assets.image import ImageAsset
113

114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
    llm = LLM(model="llava-hf/llava-1.5-7b-hf")
    image_url = "https://picsum.photos/id/32/512/512"
    image_pil = ImageAsset('cherry_blossom').pil_image
    image_embeds = torch.load(...)

    conversation = [
        {"role": "system", "content": "You are a helpful assistant"},
        {"role": "user", "content": "Hello"},
        {"role": "assistant", "content": "Hello! How can I assist you today?"},
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {"url": image_url},
                },
                {
                    "type": "image_pil",
                    "image_pil": image_pil,
                },
                {
                    "type": "image_embeds",
                    "image_embeds": image_embeds,
                },
                {
                    "type": "text",
                    "text": "What's in these images?",
                },
            ],
        },
    ]

    # Perform inference and log output.
    outputs = llm.chat(conversation)

    for o in outputs:
        generated_text = o.outputs[0].text
        print(generated_text)
    ```
153

154
155
Multi-image input can be extended to perform video captioning. We show this with [Qwen2-VL](https://huggingface.co/Qwen/Qwen2-VL-2B-Instruct) as it supports videos:

156
??? code
Reid's avatar
Reid committed
157

158
159
    ```python
    from vllm import LLM
160

161
162
163
164
165
166
167
168
    # Specify the maximum number of frames per video to be 4. This can be changed.
    llm = LLM("Qwen/Qwen2-VL-2B-Instruct", limit_mm_per_prompt={"image": 4})

    # Create the request payload.
    video_frames = ... # load your video making sure it only has the number of frames specified earlier.
    message = {
        "role": "user",
        "content": [
169
170
171
172
            {
                "type": "text",
                "text": "Describe this set of frames. Consider the frames to be a part of the same video.",
            },
173
174
175
176
177
178
179
180
181
182
183
184
185
186
        ],
    }
    for i in range(len(video_frames)):
        base64_image = encode_image(video_frames[i]) # base64 encoding.
        new_image = {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{base64_image}"}}
        message["content"].append(new_image)

    # Perform inference and log output.
    outputs = llm.chat([message])

    for o in outputs:
        generated_text = o.outputs[0].text
        print(generated_text)
    ```
187

188
189
190
191
192
193
194
195
#### Custom RGBA Background Color

When loading RGBA images (images with transparency), vLLM converts them to RGB format. By default, transparent pixels are replaced with white background. You can customize this background color using the `rgba_background_color` parameter in `media_io_kwargs`.

??? code

    ```python
    from vllm import LLM
196

197
198
    # Default white background (no configuration needed)
    llm = LLM(model="llava-hf/llava-1.5-7b-hf")
199

200
201
202
    # Custom black background for dark theme
    llm = LLM(
        model="llava-hf/llava-1.5-7b-hf",
203
        media_io_kwargs={"image": {"rgba_background_color": [0, 0, 0]}},
204
    )
205

206
207
    # Custom brand color background (e.g., blue)
    llm = LLM(
208
        model="llava-hf/llava-1.5-7b-hf",
209
        media_io_kwargs={"image": {"rgba_background_color": [0, 0, 255]}},
210
211
212
213
214
215
216
217
    )
    ```

!!! note
    - The `rgba_background_color` accepts RGB values as a list `[R, G, B]` or tuple `(R, G, B)` where each value is 0-255
    - This setting only affects RGBA images with transparency; RGB images are unchanged
    - If not specified, the default white background `(255, 255, 255)` is used for backward compatibility

218
### Video Inputs
219

220
You can pass a list of NumPy arrays directly to the `'video'` field of the multi-modal dictionary
221
222
instead of using multi-image input.

223
224
225
226
227
228
229
230
231
Instead of NumPy arrays, you can also pass `'torch.Tensor'` instances, as shown in this example using Qwen2.5-VL:

??? code

    ```python
    from transformers import AutoProcessor
    from vllm import LLM, SamplingParams
    from qwen_vl_utils import process_vision_info

232
    model_path = "Qwen/Qwen2.5-VL-3B-Instruct"
233
234
235
236
237
238
239
240
241
    video_path = "https://content.pexels.com/videos/free-videos.mp4"

    llm = LLM(
        model=model_path,
        gpu_memory_utilization=0.8,
        enforce_eager=True,
        limit_mm_per_prompt={"video": 1},
    )

242
    sampling_params = SamplingParams(max_tokens=1024)
243
244

    video_messages = [
245
246
247
248
249
250
251
        {
            "role": "system",
            "content": "You are a helpful assistant.",
        },
        {
            "role": "user",
            "content": [
252
253
254
255
256
                {"type": "text", "text": "describe this video."},
                {
                    "type": "video",
                    "video": video_path,
                    "total_pixels": 20480 * 28 * 28,
257
258
                    "min_pixels": 16 * 28 * 28,
                },
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
            ]
        },
    ]

    messages = video_messages
    processor = AutoProcessor.from_pretrained(model_path)
    prompt = processor.apply_chat_template(
        messages,
        tokenize=False,
        add_generation_prompt=True,
    )

    image_inputs, video_inputs = process_vision_info(messages)
    mm_data = {}
    if video_inputs is not None:
        mm_data["video"] = video_inputs

    llm_inputs = {
        "prompt": prompt,
        "multi_modal_data": mm_data,
    }

    outputs = llm.generate([llm_inputs], sampling_params=sampling_params)
    for o in outputs:
        generated_text = o.outputs[0].text
        print(generated_text)
    ```

    !!! note
        'process_vision_info' is only applicable to Qwen2.5-VL and similar models.

290
Full example: [examples/offline_inference/vision_language.py](../../examples/offline_inference/vision_language.py)
291

292
### Audio Inputs
293

294
You can pass a tuple `(array, sampling_rate)` to the `'audio'` field of the multi-modal dictionary.
295

296
Full example: [examples/offline_inference/audio_language.py](../../examples/offline_inference/audio_language.py)
297

298
299
300
301
302
303
304
#### Chunking Long Audio for Transcription

Speech-to-text models like Whisper have a maximum audio length they can process (typically 30 seconds). For longer audio files, vLLM provides a utility to intelligently split audio into chunks at quiet points to minimize cutting through speech.

```python
from vllm import LLM, SamplingParams
from vllm.multimodal.audio import split_audio
305
from vllm.multimodal.media.audio import load_audio
306
307

# Load long audio file
308
audio, sr = load_audio("long_audio.wav", sr=16000)
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342

# Split into chunks at low-energy (quiet) regions
chunks = split_audio(
    audio_data=audio,
    sample_rate=sr,
    max_clip_duration_s=30.0,      # Maximum chunk length in seconds
    overlap_duration_s=1.0,         # Search window for finding quiet split points
    min_energy_window_size=1600,    # Window size for energy calculation (~100ms at 16kHz)
)

# Initialize Whisper model
llm = LLM(model="openai/whisper-large-v3-turbo")
sampling_params = SamplingParams(temperature=0, max_tokens=256)

# Transcribe each chunk
transcriptions = []
for chunk in chunks:
    outputs = llm.generate({
        "prompt": "<|startoftranscript|><|en|><|transcribe|><|notimestamps|>",
        "multi_modal_data": {"audio": (chunk, sr)},
    }, sampling_params)
    transcriptions.append(outputs[0].outputs[0].text)

# Combine results
full_transcription = " ".join(transcriptions)
```

The `split_audio` function:

- Splits audio at quiet points to avoid cutting through speech
- Uses RMS energy to find low-amplitude regions within the overlap window
- Preserves all audio samples (no data loss)
- Supports any sample rate

343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
#### Automatic Audio Channel Normalization

vLLM automatically normalizes audio channels for models that require specific audio formats. When loading audio with libraries like `torchaudio`, stereo files return shape `[channels, time]`, but many audio models (particularly Whisper-based models) expect mono audio with shape `[time]`.

**Supported models with automatic mono conversion:**

- **Whisper** and all Whisper-based models
- **Qwen2-Audio**
- **Qwen2.5-Omni** / **Qwen3-Omni** (inherits from Qwen2.5-Omni)
- **Ultravox**

For these models, vLLM automatically:

1. Detects if the model requires mono audio via the feature extractor
2. Converts multi-channel audio to mono using channel averaging
3. Handles both `(channels, time)` format (torchaudio) and `(time, channels)` format (soundfile)

**Example with stereo audio:**

```python
import torchaudio
from vllm import LLM

# Load stereo audio file - returns (channels, time) shape
audio, sr = torchaudio.load("stereo_audio.wav")
print(f"Original shape: {audio.shape}")  # e.g., torch.Size([2, 16000])

# vLLM automatically converts to mono for Whisper-based models
llm = LLM(model="openai/whisper-large-v3")

outputs = llm.generate({
    "prompt": "",
    "multi_modal_data": {"audio": (audio.numpy(), sr)},
})
```

No manual conversion is needed - vLLM handles the channel normalization automatically based on the model's requirements.

381
### Embedding Inputs
382
383

To input pre-computed embeddings belonging to a data type (i.e. image, video, or audio) directly to the language model,
384
385
pass a tensor of shape `(..., hidden_size of LM)` to the corresponding field of the multi-modal dictionary.
The exact shape depends on the model being used.
386

387
388
389
390
391
392
You must enable this feature via `enable_mm_embeds=True`.

!!! warning
    The vLLM engine may crash if incorrect shape of embeddings is passed.
    Only enable this flag for trusted users!

393
394
#### Image Embeddings

395
??? code
Reid's avatar
Reid committed
396

397
398
    ```python
    from vllm import LLM
399

400
    # Inference with image embeddings as input
401
    llm = LLM(model="llava-hf/llava-1.5-7b-hf", enable_mm_embeds=True)
402

403
404
    # Refer to the HuggingFace repo for the correct format to use
    prompt = "USER: <image>\nWhat is the content of this image?\nASSISTANT:"
405

406
    # For most models, `image_embeds` has shape: (num_images, image_feature_size, hidden_size)
407
    image_embeds = torch.load(...)
408

409
410
411
412
413
414
415
416
417
    outputs = llm.generate({
        "prompt": prompt,
        "multi_modal_data": {"image": image_embeds},
    })

    for o in outputs:
        generated_text = o.outputs[0].text
        print(generated_text)

418
    # Additional examples for models that require extra fields
419
420
421
422
423
    llm = LLM(
        "Qwen/Qwen2-VL-2B-Instruct",
        limit_mm_per_prompt={"image": 4},
        enable_mm_embeds=True,
    )
424
425
    mm_data = {
        "image": {
426
427
428
429
            # Shape: (total_feature_size, hidden_size)
            # total_feature_size = sum(image_feature_size for image in images)
            "image_embeds": torch.load(...),
            # Shape: (num_images, 3)
430
            # image_grid_thw is needed to calculate positional encoding.
431
            "image_grid_thw": torch.load(...),
432
        }
433
    }
434

435
436
437
438
439
440
    llm = LLM(
        "openbmb/MiniCPM-V-2_6",
        trust_remote_code=True,
        limit_mm_per_prompt={"image": 4},
        enable_mm_embeds=True,
    )
441
442
    mm_data = {
        "image": {
443
444
445
446
            # Shape: (num_images, num_slices, hidden_size)
            # num_slices can differ for each image
            "image_embeds": [torch.load(...) for image in images],  
            # Shape: (num_images, 2)
447
            # image_sizes is needed to calculate details of the sliced image.
448
            "image_sizes": [image.size for image in images],
449
        }
450
    }
451
    ```
452

453
454
For Qwen3-VL, the `image_embeds` should contain both the base image embedding and deepstack features.

455
#### Audio Embedding Inputs
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470

You can pass pre-computed audio embeddings similar to image embeddings:

??? code

    ```python
    from vllm import LLM
    import torch

    # Enable audio embeddings support
    llm = LLM(model="fixie-ai/ultravox-v0_5-llama-3_2-1b", enable_mm_embeds=True)

    # Refer to the HuggingFace repo for the correct format to use
    prompt = "USER: <audio>\nWhat is in this audio?\nASSISTANT:"

471
472
    # Load pre-computed audio embeddings, usually with shape:
    # (num_audios, audio_feature_size, hidden_size of LM)
473
474
475
476
477
478
479
480
481
482
483
484
    audio_embeds = torch.load(...)

    outputs = llm.generate({
        "prompt": prompt,
        "multi_modal_data": {"audio": audio_embeds},
    })

    for o in outputs:
        generated_text = o.outputs[0].text
        print(generated_text)
    ```

485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
### Cached Inputs

When using multi-modal inputs, vLLM normally hashes each media item by content to enable caching across requests. You can optionally pass `multi_modal_uuids` to provide your own stable IDs for each item so caching can reuse work across requests without rehashing the raw content.

??? code

    ```python
    from vllm import LLM
    from PIL import Image

    # Qwen2.5-VL example with two images
    llm = LLM(model="Qwen/Qwen2.5-VL-3B-Instruct")

    prompt = "USER: <image><image>\nDescribe the differences.\nASSISTANT:"
    img_a = Image.open("/path/to/a.jpg")
    img_b = Image.open("/path/to/b.jpg")

    outputs = llm.generate({
        "prompt": prompt,
        "multi_modal_data": {"image": [img_a, img_b]},
        # Provide stable IDs for caching.
        # Requirements (matched by this example):
        #  - Include every modality present in multi_modal_data.
        #  - For lists, provide the same number of entries.
        #  - Use None to fall back to content hashing for that item.
        "multi_modal_uuids": {"image": ["sku-1234-a", None]},
    })

    for o in outputs:
        print(o.outputs[0].text)
    ```

Using UUIDs, you can also skip sending media data entirely if you expect cache hits for respective items. Note that the request will fail if the skipped media doesn't have a corresponding UUID, or if the UUID fails to hit the cache.

??? code

    ```python
    from vllm import LLM
    from PIL import Image

    # Qwen2.5-VL example with two images
    llm = LLM(model="Qwen/Qwen2.5-VL-3B-Instruct")

    prompt = "USER: <image><image>\nDescribe the differences.\nASSISTANT:"
    img_b = Image.open("/path/to/b.jpg")

    outputs = llm.generate({
        "prompt": prompt,
        "multi_modal_data": {"image": [None, img_b]},
        # Since img_a is expected to be cached, we can skip sending the actual
        # image entirely.
        "multi_modal_uuids": {"image": ["sku-1234-a", None]},
    })

    for o in outputs:
        print(o.outputs[0].text)
    ```

!!! warning
    If both multimodal processor caching and prefix caching are disabled, user-provided `multi_modal_uuids` are ignored.

546
## Online Serving
547

548
Our OpenAI-compatible server accepts multi-modal data via the [Chat Completions API](https://platform.openai.com/docs/api-reference/chat). Media inputs also support optional UUIDs users can provide to uniquely identify each media, which is used to cache the media results across requests.
549

550
!!! important
551
552
    A chat template is **required** to use Chat Completions API.
    For HF format models, the default chat template is defined inside `chat_template.json` or `tokenizer_config.json`.
553

554
    If no default chat template is available, we will first look for a built-in fallback in [vllm/transformers_utils/chat_templates/registry.py](../../vllm/transformers_utils/chat_templates/registry.py).
555
    If no fallback is available, an error is raised and you have to provide the chat template manually via the `--chat-template` argument.
556

557
    For certain models, we provide alternative chat templates inside [examples](../../examples).
558
    For example, VLM2Vec uses [examples/pooling/embed/template/vlm2vec_phi3v.jinja](../../examples/pooling/embed/template/vlm2vec_phi3v.jinja) which is different from the default one for Phi-3-Vision.
559

560
### Image Inputs
561
562
563
564
565
566
567

Image input is supported according to [OpenAI Vision API](https://platform.openai.com/docs/guides/vision).
Here is a simple example using Phi-3.5-Vision.

First, launch the OpenAI-compatible server:

```bash
568
vllm serve microsoft/Phi-3.5-vision-instruct --runner generate \
569
  --trust-remote-code --max-model-len 4096 --limit-mm-per-prompt.image 2
570
571
572
573
```

Then, you can use the OpenAI client as follows:

574
??? code
575
576

    ```python
577
    import os
578
579
580
581
582
583
584
585
586
587
588
    from openai import OpenAI

    openai_api_key = "EMPTY"
    openai_api_base = "http://localhost:8000/v1"

    client = OpenAI(
        api_key=openai_api_key,
        base_url=openai_api_base,
    )

    # Single-image input inference
589
590

    # Public image URL for testing remote image processing
591
    image_url = "https://vllm-public-assets.s3.us-west-2.amazonaws.com/vision_model_images/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg"
592

593
    # Create chat completion with remote image
594
595
    chat_response = client.chat.completions.create(
        model="microsoft/Phi-3.5-vision-instruct",
596
597
598
599
600
601
602
603
604
        messages=[
            {
                "role": "user",
                "content": [
                    # NOTE: The prompt formatting with the image token `<image>` is not needed
                    # since the prompt will be processed automatically by the API server.
                    {
                        "type": "text",
                        "text": "What’s in this image?",
605
                    },
606
607
608
609
610
611
612
613
                    {
                        "type": "image_url",
                        "image_url": {"url": image_url},
                        "uuid": image_url,  # Optional
                    },
                ],
            }
        ],
614
615
616
    )
    print("Chat completion output:", chat_response.choices[0].message.content)

617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
    # Local image file path (update this to point to your actual image file)
    image_file = "/path/to/image.jpg"

    # Create chat completion with local image file
    # Launch the API server/engine with the --allowed-local-media-path argument.
    if os.path.exists(image_file):
        chat_completion_from_local_image_url = client.chat.completions.create(
            model="microsoft/Phi-3.5-vision-instruct",
            messages=[
                {
                    "role": "user",
                    "content": [
                        {
                            "type": "text",
                            "text": "What’s in this image?",
                        },
                        {
                            "type": "image_url",
                            "image_url": {"url": f"file://{image_file}"},
                        },
                    ],
                }
            ],
        )
        result = chat_completion_from_local_image_url.choices[0].message.content
        print("Chat completion output from local image file:\n", result)
    else:
        print(f"Local image file not found at {image_file}, skipping local file test.")

646
    # Multi-image input inference
647
648
    image_url_duck = "https://vllm-public-assets.s3.us-west-2.amazonaws.com/multimodal_asset/duck.jpg"
    image_url_lion = "https://vllm-public-assets.s3.us-west-2.amazonaws.com/multimodal_asset/lion.jpg"
649
650
651

    chat_response = client.chat.completions.create(
        model="microsoft/Phi-3.5-vision-instruct",
652
653
654
655
656
657
658
        messages=[
            {
                "role": "user",
                "content": [
                    {
                        "type": "text",
                        "text": "What are the animals in these images?",
659
                    },
660
661
662
663
                    {
                        "type": "image_url",
                        "image_url": {"url": image_url_duck},
                        "uuid": image_url_duck,  # Optional
664
                    },
665
666
667
668
669
670
671
672
                    {
                        "type": "image_url",
                        "image_url": {"url": image_url_lion},
                        "uuid": image_url_lion,  # Optional
                    },
                ],
            }
        ],
673
674
675
    )
    print("Chat completion output:", chat_response.choices[0].message.content)
    ```
676

677
Full example: [examples/online_serving/openai_chat_completion_client_for_multimodal.py](../../examples/online_serving/openai_chat_completion_client_for_multimodal.py)
678

679
680
681
!!! tip
    Loading from local file paths is also supported on vLLM: You can specify the allowed local media path via `--allowed-local-media-path` when launching the API server/engine,
    and pass the file path as `url` in the API request.
682

683
684
685
!!! tip
    There is no need to place image placeholders in the text content of the API request - they are already represented by the image content.
    In fact, you can place image placeholders in the middle of the text by interleaving text and image content.
686

687
688
689
!!! note
    By default, the timeout for fetching images through HTTP URL is `5` seconds.
    You can override this by setting the environment variable:
690

691
    ```bash
692
693
    export VLLM_IMAGE_FETCH_TIMEOUT=<timeout>
    ```
694

695
### Video Inputs
696

697
Instead of `image_url`, you can pass a video file via `video_url`. Here is a simple example using [LLaVA-OneVision](https://huggingface.co/llava-hf/llava-onevision-qwen2-0.5b-ov-hf).
698

699
700
701
First, launch the OpenAI-compatible server:

```bash
702
vllm serve llava-hf/llava-onevision-qwen2-0.5b-ov-hf --runner generate --max-model-len 8192
703
704
705
```

Then, you can use the OpenAI client as follows:
706

707
??? code
708

709
710
    ```python
    from openai import OpenAI
711

712
713
    openai_api_key = "EMPTY"
    openai_api_base = "http://localhost:8000/v1"
714

715
716
717
718
    client = OpenAI(
        api_key=openai_api_key,
        base_url=openai_api_base,
    )
719

720
    video_url = "http://commondatastorage.googleapis.com/gtv-videos-bucket/sample/ForBiggerFun.mp4"
721

722
723
    ## Use video url in the payload
    chat_completion_from_url = client.chat.completions.create(
724
725
726
727
728
729
730
        messages=[
            {
                "role": "user",
                "content": [
                    {
                        "type": "text",
                        "text": "What's in this video?",
731
                    },
732
733
734
735
736
737
738
739
                    {
                        "type": "video_url",
                        "video_url": {"url": video_url},
                        "uuid": video_url,  # Optional
                    },
                ],
            }
        ],
740
741
742
743
744
745
746
        model=model,
        max_completion_tokens=64,
    )

    result = chat_completion_from_url.choices[0].message.content
    print("Chat completion output from image url:", result)
    ```
747

748
Full example: [examples/online_serving/openai_chat_completion_client_for_multimodal.py](../../examples/online_serving/openai_chat_completion_client_for_multimodal.py)
749

750
751
752
!!! note
    By default, the timeout for fetching videos through HTTP URL is `30` seconds.
    You can override this by setting the environment variable:
753

754
    ```bash
755
756
    export VLLM_VIDEO_FETCH_TIMEOUT=<timeout>
    ```
757

758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
#### Video Frame Recovery

For improved robustness when processing potentially corrupted or truncated video files, vLLM supports optional frame recovery using a dynamic window forward-scan approach. When enabled, if a target frame fails to load during sequential reading, the next successfully grabbed frame (before the next target frame) will be used in its place.

To enable video frame recovery, pass the `frame_recovery` parameter via `--media-io-kwargs`:

```bash
# Example: Enable frame recovery
vllm serve Qwen/Qwen3-VL-30B-A3B-Instruct \
  --media-io-kwargs '{"video": {"frame_recovery": true}}'
```

**Parameters:**

- `frame_recovery`: Boolean flag to enable forward-scan recovery. When `true`, failed frames are recovered using the next available frame within the dynamic window (up to the next target frame). Default is `false`.

**How it works:**

1. The system reads frames sequentially
2. If a target frame fails to grab, it's marked as "failed"
3. The next successfully grabbed frame (before reaching the next target) is used to recover the failed frame
4. This approach handles both mid-video corruption and end-of-video truncation

Works with common video formats like MP4 when using OpenCV backends.

783
784
785
786
787
788
789
790
791
792
793
794
795
796
#### Custom RGBA Background Color

To use a custom background color for RGBA images, pass the `rgba_background_color` parameter via `--media-io-kwargs`:

```bash
# Example: Black background for dark theme
vllm serve llava-hf/llava-1.5-7b-hf \
  --media-io-kwargs '{"image": {"rgba_background_color": [0, 0, 0]}}'

# Example: Custom gray background
vllm serve llava-hf/llava-1.5-7b-hf \
  --media-io-kwargs '{"image": {"rgba_background_color": [128, 128, 128]}}'
```

797
### Audio Inputs
798
799

Audio input is supported according to [OpenAI Audio API](https://platform.openai.com/docs/guides/audio?audio-generation-quickstart-example=audio-in).
800
Here is a simple example using Ultravox-v0.5-1B.
801
802
803
804

First, launch the OpenAI-compatible server:

```bash
805
vllm serve fixie-ai/ultravox-v0_5-llama-3_2-1b
806
807
808
809
```

Then, you can use the OpenAI client as follows:

810
??? code
811

812
813
814
815
816
    ```python
    import base64
    import requests
    from openai import OpenAI
    from vllm.assets.audio import AudioAsset
817

818
819
    def encode_base64_content_from_url(content_url: str) -> str:
        """Encode a content retrieved from a remote url to base64 format."""
820

821
822
823
        with requests.get(content_url) as response:
            response.raise_for_status()
            result = base64.b64encode(response.content).decode('utf-8')
824

825
        return result
826

827
828
    openai_api_key = "EMPTY"
    openai_api_base = "http://localhost:8000/v1"
829

830
831
832
833
    client = OpenAI(
        api_key=openai_api_key,
        base_url=openai_api_base,
    )
834

835
    # Any format supported by soundfile/PyAV is supported
836
837
    audio_url = AudioAsset("winning_call").url
    audio_base64 = encode_base64_content_from_url(audio_url)
838

839
    chat_completion_from_base64 = client.chat.completions.create(
840
841
842
843
844
845
846
        messages=[
            {
                "role": "user",
                "content": [
                    {
                        "type": "text",
                        "text": "What's in this audio?",
847
                    },
848
849
850
851
852
853
854
855
856
857
858
                    {
                        "type": "input_audio",
                        "input_audio": {
                            "data": audio_base64,
                            "format": "wav",
                        },
                        "uuid": audio_url,  # Optional
                    },
                ],
            },
        ],
859
860
861
862
863
864
865
        model=model,
        max_completion_tokens=64,
    )

    result = chat_completion_from_base64.choices[0].message.content
    print("Chat completion output from input audio:", result)
    ```
866

867
Alternatively, you can pass `audio_url`, which is the audio counterpart of `image_url` for image input:
868

869
??? code
870

871
872
    ```python
    chat_completion_from_url = client.chat.completions.create(
873
874
875
876
877
878
879
        messages=[
            {
                "role": "user",
                "content": [
                    {
                        "type": "text",
                        "text": "What's in this audio?",
880
                    },
881
882
883
884
885
886
887
888
                    {
                        "type": "audio_url",
                        "audio_url": {"url": audio_url},
                        "uuid": audio_url,  # Optional
                    },
                ],
            }
        ],
889
890
891
892
893
894
895
        model=model,
        max_completion_tokens=64,
    )

    result = chat_completion_from_url.choices[0].message.content
    print("Chat completion output from audio url:", result)
    ```
896

897
Full example: [examples/online_serving/openai_chat_completion_client_for_multimodal.py](../../examples/online_serving/openai_chat_completion_client_for_multimodal.py)
898

899
900
901
!!! note
    By default, the timeout for fetching audios through HTTP URL is `10` seconds.
    You can override this by setting the environment variable:
902

903
    ```bash
904
905
    export VLLM_AUDIO_FETCH_TIMEOUT=<timeout>
    ```
906

907
### Embedding Inputs
908

909
To input pre-computed embeddings belonging to a data type (i.e. image, video, or audio) directly to the language model,
910
911
912
913
914
pass a tensor of shape `(..., hidden_size of LM)` for each item to the corresponding field of the multi-modal dictionary.

!!! important
    Unlike offline inference, the embeddings for each item must be passed separately
    in order for placeholder tokens to be applied correctly by the chat template.
915
916
917
918
919
920

You must enable this feature via the `--enable-mm-embeds` flag in `vllm serve`.

!!! warning
    The vLLM engine may crash if incorrect shape of embeddings is passed.
    Only enable this flag for trusted users!
921

922
#### Image Embedding Inputs
923

924
925
926
For image embeddings, you can pass the base64-encoded tensor to the `image_embeds` field.
The following example demonstrates how to pass image embeddings to the OpenAI server:

927
??? code
928
929

    ```python
930
931
    from vllm.utils.serial_utils import tensor2base64

932
933
934
935
936
937
938
939
    client = OpenAI(
        # defaults to os.environ.get("OPENAI_API_KEY")
        api_key=openai_api_key,
        base_url=openai_api_base,
    )

    # Basic usage - this is equivalent to the LLaVA example for offline inference
    model = "llava-hf/llava-1.5-7b-hf"
940
    embeds = {
941
        "type": "image_embeds",
942
        "image_embeds": tensor2base64(torch.load(...)),  # Shape: (image_feature_size, hidden_size)
943
        "uuid": image_url,  # Optional
944
945
    }

946
947

    # Additional examples for models that require extra fields
948
    model = "Qwen/Qwen2-VL-2B-Instruct"
949
    embeds = {
950
951
        "type": "image_embeds",
        "image_embeds": {
952
953
            "image_embeds": tensor2base64(torch.load(...)),  # Shape: (image_feature_size, hidden_size)
            "image_grid_thw": tensor2base64(torch.load(...)),  # Shape: (3,)
954
        },
955
        "uuid": image_url,  # Optional
956
    }
957

958
    model = "openbmb/MiniCPM-V-2_6"
959
    embeds = {
960
961
        "type": "image_embeds",
        "image_embeds": {
962
963
            "image_embeds": tensor2base64(torch.load(...)),  # Shape: (num_slices, hidden_size)
            "image_sizes": tensor2base64(torch.load(...)),  # Shape: (2,)
964
        },
965
        "uuid": image_url,  # Optional
966
    }
967
968

    # Single image input
969
970
971
    chat_completion = client.chat.completions.create(
        messages=[
            {
972
973
                "role": "system",
                "content": "You are a helpful assistant.",
974
            },
975
976
977
978
979
980
981
982
983
984
985
            {
                "role": "user",
                "content": [
                    {
                        "type": "text",
                        "text": "What's in this image?",
                    },
                    embeds,
                ],
            },
        ],
986
987
        model=model,
    )
988
989
990
991
992
993
994
995
996
997
998
999
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012
1013
1014
1015
1016
1017
1018
1019
1020
1021
1022
1023
1024
1025
1026
1027
1028
1029
1030
1031

    # Multi image input
    chat_completion = client.chat.completions.create(
        messages=[
            {
                "role": "system",
                "content": "You are a helpful assistant.",
            },
            {
                "role": "user",
                "content": [
                    {
                        "type": "text",
                        "text": "What's in this image?",
                    },
                    embeds,
                    embeds,
                ],
            },
        ],
        model=model,
    )

    # Multi image input (interleaved)
    chat_completion = client.chat.completions.create(
        messages=[
            {
                "role": "system",
                "content": "You are a helpful assistant.",
            },
            {
                "role": "user",
                "content": [
                    embeds,
                    {
                        "type": "text",
                        "text": "What's in this image?",
                    },
                    embeds,
                ],
            },
        ],
        model=model,
    )
1032
    ```
1033

1034
1035
1036
### Cached Inputs

Just like with offline inference, you can skip sending media if you expect cache hits with provided UUIDs. You can do so by sending media like this:
1037

1038
1039
??? code

1040
1041
1042
1043
1044
1045
1046
1047
1048
1049
1050
1051
    ```python
        # Image/video/audio URL:
        {
            "type": "image_url",
            "image_url": None,
            "uuid": image_uuid,
        },

        # image_embeds
        {
            "type": "image_embeds",
            "image_embeds": None,
1052
            "uuid": image_uuid,
1053
1054
1055
1056
1057
1058
        },

        # input_audio:
        {
            "type": "input_audio",
            "input_audio": None,
1059
            "uuid": audio_uuid,
1060
1061
1062
1063
1064
        },

        # PIL Image:
        {
            "type": "image_pil",
1065
1066
1067
            "image_pil": None,
            "uuid": image_uuid,
        },
1068
1069

    ```