multimodal_inputs.md 38 KB
Newer Older
1
# Multimodal Inputs
2

3
This page teaches you how to pass multi-modal inputs to [multi-modal models](../models/supported_models.md#list-of-multimodal-language-models) in vLLM.
4

5
!!! note
6
    We are actively iterating on multi-modal support. See [this RFC](https://github.com/vllm-project/vllm/issues/4194) for upcoming changes,
7
    and [open an issue on GitHub](https://github.com/vllm-project/vllm/issues/new/choose) if you have any feedback or feature requests.
8

9
10
!!! tip
    When serving multi-modal models, consider setting `--allowed-media-domains` to restrict domain that vLLM can access to prevent it from accessing arbitrary endpoints that can potentially be vulnerable to Server-Side Request Forgery (SSRF) attacks. You can provide a list of domains for this arg. For example: `--allowed-media-domains upload.wikimedia.org github.com www.bogotobogo.com`
11
12
13

    Also, consider setting `VLLM_MEDIA_URL_ALLOW_REDIRECTS=0` to prevent HTTP redirects from being followed to bypass domain restrictions.

14
15
    This restriction is especially important if you run vLLM in a containerized environment where the vLLM pods may have unrestricted access to internal networks.

16
17
## Offline Inference

18
To input multi-modal data, follow this schema in [vllm.inputs.PromptType][]:
19
20

- `prompt`: The prompt should follow the format that is documented on HuggingFace.
21
- `multi_modal_data`: This is a dictionary that follows the schema defined in [vllm.inputs.MultiModalDataDict][].
22

23
### Image Inputs
24

25
You can pass a single image to the `'image'` field of the multi-modal dictionary, as shown in the following examples:
26

27
??? code
28

29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
    ```python
    from vllm import LLM

    llm = LLM(model="llava-hf/llava-1.5-7b-hf")

    # Refer to the HuggingFace repo for the correct format to use
    prompt = "USER: <image>\nWhat is the content of this image?\nASSISTANT:"

    # Load the image using PIL.Image
    image = PIL.Image.open(...)

    # Single prompt inference
    outputs = llm.generate({
        "prompt": prompt,
        "multi_modal_data": {"image": image},
    })

    for o in outputs:
        generated_text = o.outputs[0].text
        print(generated_text)

    # Batch inference
    image_1 = PIL.Image.open(...)
    image_2 = PIL.Image.open(...)
    outputs = llm.generate(
        [
            {
                "prompt": "USER: <image>\nWhat is the content of this image?\nASSISTANT:",
                "multi_modal_data": {"image": image_1},
            },
            {
                "prompt": "USER: <image>\nWhat's the color of this image?\nASSISTANT:",
                "multi_modal_data": {"image": image_2},
            }
        ]
    )

    for o in outputs:
        generated_text = o.outputs[0].text
        print(generated_text)
    ```
70

71
Full example: [examples/offline_inference/vision_language.py](../../examples/offline_inference/vision_language.py)
72
73
74

To substitute multiple images inside the same text prompt, you can pass in a list of images instead:

75
??? code
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95

    ```python
    from vllm import LLM

    llm = LLM(
        model="microsoft/Phi-3.5-vision-instruct",
        trust_remote_code=True,  # Required to load Phi-3.5-vision
        max_model_len=4096,  # Otherwise, it may not fit in smaller GPUs
        limit_mm_per_prompt={"image": 2},  # The maximum number to accept
    )

    # Refer to the HuggingFace repo for the correct format to use
    prompt = "<|user|>\n<|image_1|>\n<|image_2|>\nWhat is the content of each image?<|end|>\n<|assistant|>\n"

    # Load the images using PIL.Image
    image1 = PIL.Image.open(...)
    image2 = PIL.Image.open(...)

    outputs = llm.generate({
        "prompt": prompt,
96
        "multi_modal_data": {"image": [image1, image2]},
97
98
99
100
101
102
    })

    for o in outputs:
        generated_text = o.outputs[0].text
        print(generated_text)
    ```
103

104
Full example: [examples/offline_inference/vision_language_multi_image.py](../../examples/offline_inference/vision_language_multi_image.py)
105

106
If using the [LLM.chat](../models/generative_models.md#llmchat) method, you can pass images directly in the message content using various formats: image URLs, PIL Image objects, or pre-computed embeddings:
107

108
??? code
109

110
111
112
    ```python
    from vllm import LLM
    from vllm.assets.image import ImageAsset
113

114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
    llm = LLM(model="llava-hf/llava-1.5-7b-hf")
    image_url = "https://picsum.photos/id/32/512/512"
    image_pil = ImageAsset('cherry_blossom').pil_image
    image_embeds = torch.load(...)

    conversation = [
        {"role": "system", "content": "You are a helpful assistant"},
        {"role": "user", "content": "Hello"},
        {"role": "assistant", "content": "Hello! How can I assist you today?"},
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {"url": image_url},
                },
                {
                    "type": "image_pil",
                    "image_pil": image_pil,
                },
                {
                    "type": "image_embeds",
                    "image_embeds": image_embeds,
                },
                {
                    "type": "text",
                    "text": "What's in these images?",
                },
            ],
        },
    ]

    # Perform inference and log output.
    outputs = llm.chat(conversation)

    for o in outputs:
        generated_text = o.outputs[0].text
        print(generated_text)
    ```
153

154
155
Multi-image input can be extended to perform video captioning. We show this with [Qwen2-VL](https://huggingface.co/Qwen/Qwen2-VL-2B-Instruct) as it supports videos:

156
??? code
Reid's avatar
Reid committed
157

158
159
    ```python
    from vllm import LLM
160

161
162
163
164
165
166
167
168
    # Specify the maximum number of frames per video to be 4. This can be changed.
    llm = LLM("Qwen/Qwen2-VL-2B-Instruct", limit_mm_per_prompt={"image": 4})

    # Create the request payload.
    video_frames = ... # load your video making sure it only has the number of frames specified earlier.
    message = {
        "role": "user",
        "content": [
169
170
171
172
            {
                "type": "text",
                "text": "Describe this set of frames. Consider the frames to be a part of the same video.",
            },
173
174
175
176
177
178
179
180
181
182
183
184
185
186
        ],
    }
    for i in range(len(video_frames)):
        base64_image = encode_image(video_frames[i]) # base64 encoding.
        new_image = {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{base64_image}"}}
        message["content"].append(new_image)

    # Perform inference and log output.
    outputs = llm.chat([message])

    for o in outputs:
        generated_text = o.outputs[0].text
        print(generated_text)
    ```
187

188
189
190
191
192
193
194
195
#### Custom RGBA Background Color

When loading RGBA images (images with transparency), vLLM converts them to RGB format. By default, transparent pixels are replaced with white background. You can customize this background color using the `rgba_background_color` parameter in `media_io_kwargs`.

??? code

    ```python
    from vllm import LLM
196

197
198
    # Default white background (no configuration needed)
    llm = LLM(model="llava-hf/llava-1.5-7b-hf")
199

200
201
202
    # Custom black background for dark theme
    llm = LLM(
        model="llava-hf/llava-1.5-7b-hf",
203
        media_io_kwargs={"image": {"rgba_background_color": [0, 0, 0]}},
204
    )
205

206
207
    # Custom brand color background (e.g., blue)
    llm = LLM(
208
        model="llava-hf/llava-1.5-7b-hf",
209
        media_io_kwargs={"image": {"rgba_background_color": [0, 0, 255]}},
210
211
212
213
214
215
216
217
    )
    ```

!!! note
    - The `rgba_background_color` accepts RGB values as a list `[R, G, B]` or tuple `(R, G, B)` where each value is 0-255
    - This setting only affects RGBA images with transparency; RGB images are unchanged
    - If not specified, the default white background `(255, 255, 255)` is used for backward compatibility

218
### Video Inputs
219

220
You can pass a list of NumPy arrays directly to the `'video'` field of the multi-modal dictionary
221
222
instead of using multi-image input.

223
224
225
226
227
228
229
230
231
Instead of NumPy arrays, you can also pass `'torch.Tensor'` instances, as shown in this example using Qwen2.5-VL:

??? code

    ```python
    from transformers import AutoProcessor
    from vllm import LLM, SamplingParams
    from qwen_vl_utils import process_vision_info

232
    model_path = "Qwen/Qwen2.5-VL-3B-Instruct"
233
234
235
236
237
238
239
240
241
    video_path = "https://content.pexels.com/videos/free-videos.mp4"

    llm = LLM(
        model=model_path,
        gpu_memory_utilization=0.8,
        enforce_eager=True,
        limit_mm_per_prompt={"video": 1},
    )

242
    sampling_params = SamplingParams(max_tokens=1024)
243
244

    video_messages = [
245
246
247
248
249
250
251
        {
            "role": "system",
            "content": "You are a helpful assistant.",
        },
        {
            "role": "user",
            "content": [
252
253
254
255
256
                {"type": "text", "text": "describe this video."},
                {
                    "type": "video",
                    "video": video_path,
                    "total_pixels": 20480 * 28 * 28,
257
258
                    "min_pixels": 16 * 28 * 28,
                },
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
            ]
        },
    ]

    messages = video_messages
    processor = AutoProcessor.from_pretrained(model_path)
    prompt = processor.apply_chat_template(
        messages,
        tokenize=False,
        add_generation_prompt=True,
    )

    image_inputs, video_inputs = process_vision_info(messages)
    mm_data = {}
    if video_inputs is not None:
        mm_data["video"] = video_inputs

    llm_inputs = {
        "prompt": prompt,
        "multi_modal_data": mm_data,
    }

    outputs = llm.generate([llm_inputs], sampling_params=sampling_params)
    for o in outputs:
        generated_text = o.outputs[0].text
        print(generated_text)
    ```

    !!! note
        'process_vision_info' is only applicable to Qwen2.5-VL and similar models.

290
Full example: [examples/offline_inference/vision_language.py](../../examples/offline_inference/vision_language.py)
291

292
### Audio Inputs
293

294
You can pass a tuple `(array, sampling_rate)` to the `'audio'` field of the multi-modal dictionary.
295

296
Full example: [examples/offline_inference/audio_language.py](../../examples/offline_inference/audio_language.py)
297

298
299
300
301
302
303
304
#### Chunking Long Audio for Transcription

Speech-to-text models like Whisper have a maximum audio length they can process (typically 30 seconds). For longer audio files, vLLM provides a utility to intelligently split audio into chunks at quiet points to minimize cutting through speech.

```python
from vllm import LLM, SamplingParams
from vllm.multimodal.audio import split_audio
305
from vllm.multimodal.media.audio import load_audio
306
307

# Load long audio file
308
audio, sr = load_audio("long_audio.wav", sr=16000)
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342

# Split into chunks at low-energy (quiet) regions
chunks = split_audio(
    audio_data=audio,
    sample_rate=sr,
    max_clip_duration_s=30.0,      # Maximum chunk length in seconds
    overlap_duration_s=1.0,         # Search window for finding quiet split points
    min_energy_window_size=1600,    # Window size for energy calculation (~100ms at 16kHz)
)

# Initialize Whisper model
llm = LLM(model="openai/whisper-large-v3-turbo")
sampling_params = SamplingParams(temperature=0, max_tokens=256)

# Transcribe each chunk
transcriptions = []
for chunk in chunks:
    outputs = llm.generate({
        "prompt": "<|startoftranscript|><|en|><|transcribe|><|notimestamps|>",
        "multi_modal_data": {"audio": (chunk, sr)},
    }, sampling_params)
    transcriptions.append(outputs[0].outputs[0].text)

# Combine results
full_transcription = " ".join(transcriptions)
```

The `split_audio` function:

- Splits audio at quiet points to avoid cutting through speech
- Uses RMS energy to find low-amplitude regions within the overlap window
- Preserves all audio samples (no data loss)
- Supports any sample rate

343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
#### Automatic Audio Channel Normalization

vLLM automatically normalizes audio channels for models that require specific audio formats. When loading audio with libraries like `torchaudio`, stereo files return shape `[channels, time]`, but many audio models (particularly Whisper-based models) expect mono audio with shape `[time]`.

**Supported models with automatic mono conversion:**

- **Whisper** and all Whisper-based models
- **Qwen2-Audio**
- **Qwen2.5-Omni** / **Qwen3-Omni** (inherits from Qwen2.5-Omni)
- **Ultravox**

For these models, vLLM automatically:

1. Detects if the model requires mono audio via the feature extractor
2. Converts multi-channel audio to mono using channel averaging
3. Handles both `(channels, time)` format (torchaudio) and `(time, channels)` format (soundfile)

**Example with stereo audio:**

```python
import torchaudio
from vllm import LLM

# Load stereo audio file - returns (channels, time) shape
audio, sr = torchaudio.load("stereo_audio.wav")
print(f"Original shape: {audio.shape}")  # e.g., torch.Size([2, 16000])

# vLLM automatically converts to mono for Whisper-based models
llm = LLM(model="openai/whisper-large-v3")

outputs = llm.generate({
    "prompt": "",
    "multi_modal_data": {"audio": (audio.numpy(), sr)},
})
```

No manual conversion is needed - vLLM handles the channel normalization automatically based on the model's requirements.

381
### Embedding Inputs
382
383

To input pre-computed embeddings belonging to a data type (i.e. image, video, or audio) directly to the language model,
384
385
pass a tensor of shape `(..., hidden_size of LM)` to the corresponding field of the multi-modal dictionary.
The exact shape depends on the model being used.
386

387
388
389
390
391
392
You must enable this feature via `enable_mm_embeds=True`.

!!! warning
    The vLLM engine may crash if incorrect shape of embeddings is passed.
    Only enable this flag for trusted users!

393
394
#### Image Embeddings

395
??? code
Reid's avatar
Reid committed
396

397
398
    ```python
    from vllm import LLM
399

400
    # Inference with image embeddings as input
401
    llm = LLM(model="llava-hf/llava-1.5-7b-hf", enable_mm_embeds=True)
402

403
404
    # Refer to the HuggingFace repo for the correct format to use
    prompt = "USER: <image>\nWhat is the content of this image?\nASSISTANT:"
405

406
    # For most models, `image_embeds` has shape: (num_images, image_feature_size, hidden_size)
407
    image_embeds = torch.load(...)
408

409
410
411
412
413
414
415
416
417
    outputs = llm.generate({
        "prompt": prompt,
        "multi_modal_data": {"image": image_embeds},
    })

    for o in outputs:
        generated_text = o.outputs[0].text
        print(generated_text)

418
    # Additional examples for models that require extra fields
419
420
421
422
423
    llm = LLM(
        "Qwen/Qwen2-VL-2B-Instruct",
        limit_mm_per_prompt={"image": 4},
        enable_mm_embeds=True,
    )
424
425
    mm_data = {
        "image": {
426
427
428
429
            # Shape: (total_feature_size, hidden_size)
            # total_feature_size = sum(image_feature_size for image in images)
            "image_embeds": torch.load(...),
            # Shape: (num_images, 3)
430
            # image_grid_thw is needed to calculate positional encoding.
431
            "image_grid_thw": torch.load(...),
432
        }
433
    }
434

435
436
437
438
439
440
    llm = LLM(
        "openbmb/MiniCPM-V-2_6",
        trust_remote_code=True,
        limit_mm_per_prompt={"image": 4},
        enable_mm_embeds=True,
    )
441
442
    mm_data = {
        "image": {
443
444
445
446
            # Shape: (num_images, num_slices, hidden_size)
            # num_slices can differ for each image
            "image_embeds": [torch.load(...) for image in images],  
            # Shape: (num_images, 2)
447
            # image_sizes is needed to calculate details of the sliced image.
448
            "image_sizes": [image.size for image in images],
449
        }
450
    }
451
    ```
452

453
454
For Qwen3-VL, the `image_embeds` should contain both the base image embedding and deepstack features.

455
#### Audio Embedding Inputs
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470

You can pass pre-computed audio embeddings similar to image embeddings:

??? code

    ```python
    from vllm import LLM
    import torch

    # Enable audio embeddings support
    llm = LLM(model="fixie-ai/ultravox-v0_5-llama-3_2-1b", enable_mm_embeds=True)

    # Refer to the HuggingFace repo for the correct format to use
    prompt = "USER: <audio>\nWhat is in this audio?\nASSISTANT:"

471
472
    # Load pre-computed audio embeddings, usually with shape:
    # (num_audios, audio_feature_size, hidden_size of LM)
473
474
475
476
477
478
479
480
481
482
483
484
    audio_embeds = torch.load(...)

    outputs = llm.generate({
        "prompt": prompt,
        "multi_modal_data": {"audio": audio_embeds},
    })

    for o in outputs:
        generated_text = o.outputs[0].text
        print(generated_text)
    ```

485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
### Cached Inputs

When using multi-modal inputs, vLLM normally hashes each media item by content to enable caching across requests. You can optionally pass `multi_modal_uuids` to provide your own stable IDs for each item so caching can reuse work across requests without rehashing the raw content.

??? code

    ```python
    from vllm import LLM
    from PIL import Image

    # Qwen2.5-VL example with two images
    llm = LLM(model="Qwen/Qwen2.5-VL-3B-Instruct")

    prompt = "USER: <image><image>\nDescribe the differences.\nASSISTANT:"
    img_a = Image.open("/path/to/a.jpg")
    img_b = Image.open("/path/to/b.jpg")

    outputs = llm.generate({
        "prompt": prompt,
        "multi_modal_data": {"image": [img_a, img_b]},
        # Provide stable IDs for caching.
        # Requirements (matched by this example):
        #  - Include every modality present in multi_modal_data.
        #  - For lists, provide the same number of entries.
        #  - Use None to fall back to content hashing for that item.
        "multi_modal_uuids": {"image": ["sku-1234-a", None]},
    })

    for o in outputs:
        print(o.outputs[0].text)
    ```

Using UUIDs, you can also skip sending media data entirely if you expect cache hits for respective items. Note that the request will fail if the skipped media doesn't have a corresponding UUID, or if the UUID fails to hit the cache.

??? code

    ```python
    from vllm import LLM
    from PIL import Image

    # Qwen2.5-VL example with two images
    llm = LLM(model="Qwen/Qwen2.5-VL-3B-Instruct")

    prompt = "USER: <image><image>\nDescribe the differences.\nASSISTANT:"
    img_b = Image.open("/path/to/b.jpg")

    outputs = llm.generate({
        "prompt": prompt,
        "multi_modal_data": {"image": [None, img_b]},
        # Since img_a is expected to be cached, we can skip sending the actual
        # image entirely.
        "multi_modal_uuids": {"image": ["sku-1234-a", None]},
    })

    for o in outputs:
        print(o.outputs[0].text)
    ```

!!! warning
    If both multimodal processor caching and prefix caching are disabled, user-provided `multi_modal_uuids` are ignored.

546
## Online Serving
547

548
Our OpenAI-compatible server accepts multi-modal data via the [Chat Completions API](https://platform.openai.com/docs/api-reference/chat). Media inputs also support optional UUIDs users can provide to uniquely identify each media, which is used to cache the media results across requests.
549

550
!!! important
551
552
    A chat template is **required** to use Chat Completions API.
    For HF format models, the default chat template is defined inside `chat_template.json` or `tokenizer_config.json`.
553

554
    If no default chat template is available, we will first look for a built-in fallback in [vllm/transformers_utils/chat_templates/registry.py](../../vllm/transformers_utils/chat_templates/registry.py).
555
    If no fallback is available, an error is raised and you have to provide the chat template manually via the `--chat-template` argument.
556

557
    For certain models, we provide alternative chat templates inside [examples](../../examples).
558
    For example, VLM2Vec uses [examples/pooling/embed/template/vlm2vec_phi3v.jinja](../../examples/pooling/embed/template/vlm2vec_phi3v.jinja) which is different from the default one for Phi-3-Vision.
559

560
### Image Inputs
561
562
563
564
565
566
567

Image input is supported according to [OpenAI Vision API](https://platform.openai.com/docs/guides/vision).
Here is a simple example using Phi-3.5-Vision.

First, launch the OpenAI-compatible server:

```bash
568
vllm serve microsoft/Phi-3.5-vision-instruct --runner generate \
569
  --trust-remote-code --max-model-len 4096 --limit-mm-per-prompt.image 2
570
571
572
573
```

Then, you can use the OpenAI client as follows:

574
??? code
575
576

    ```python
577
    import os
578
579
580
581
582
583
584
585
586
587
588
    from openai import OpenAI

    openai_api_key = "EMPTY"
    openai_api_base = "http://localhost:8000/v1"

    client = OpenAI(
        api_key=openai_api_key,
        base_url=openai_api_base,
    )

    # Single-image input inference
589
590

    # Public image URL for testing remote image processing
591
    image_url = "https://vllm-public-assets.s3.us-west-2.amazonaws.com/vision_model_images/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg"
592

593
    # Create chat completion with remote image
594
595
    chat_response = client.chat.completions.create(
        model="microsoft/Phi-3.5-vision-instruct",
596
597
598
599
600
601
602
603
604
        messages=[
            {
                "role": "user",
                "content": [
                    # NOTE: The prompt formatting with the image token `<image>` is not needed
                    # since the prompt will be processed automatically by the API server.
                    {
                        "type": "text",
                        "text": "What’s in this image?",
605
                    },
606
607
608
609
610
611
612
613
                    {
                        "type": "image_url",
                        "image_url": {"url": image_url},
                        "uuid": image_url,  # Optional
                    },
                ],
            }
        ],
614
615
616
    )
    print("Chat completion output:", chat_response.choices[0].message.content)

617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
    # Local image file path (update this to point to your actual image file)
    image_file = "/path/to/image.jpg"

    # Create chat completion with local image file
    # Launch the API server/engine with the --allowed-local-media-path argument.
    if os.path.exists(image_file):
        chat_completion_from_local_image_url = client.chat.completions.create(
            model="microsoft/Phi-3.5-vision-instruct",
            messages=[
                {
                    "role": "user",
                    "content": [
                        {
                            "type": "text",
                            "text": "What’s in this image?",
                        },
                        {
                            "type": "image_url",
                            "image_url": {"url": f"file://{image_file}"},
                        },
                    ],
                }
            ],
        )
        result = chat_completion_from_local_image_url.choices[0].message.content
        print("Chat completion output from local image file:\n", result)
    else:
        print(f"Local image file not found at {image_file}, skipping local file test.")

646
    # Multi-image input inference
647
648
    image_url_duck = "https://vllm-public-assets.s3.us-west-2.amazonaws.com/multimodal_asset/duck.jpg"
    image_url_lion = "https://vllm-public-assets.s3.us-west-2.amazonaws.com/multimodal_asset/lion.jpg"
649
650
651

    chat_response = client.chat.completions.create(
        model="microsoft/Phi-3.5-vision-instruct",
652
653
654
655
656
657
658
        messages=[
            {
                "role": "user",
                "content": [
                    {
                        "type": "text",
                        "text": "What are the animals in these images?",
659
                    },
660
661
662
663
                    {
                        "type": "image_url",
                        "image_url": {"url": image_url_duck},
                        "uuid": image_url_duck,  # Optional
664
                    },
665
666
667
668
669
670
671
672
                    {
                        "type": "image_url",
                        "image_url": {"url": image_url_lion},
                        "uuid": image_url_lion,  # Optional
                    },
                ],
            }
        ],
673
674
675
    )
    print("Chat completion output:", chat_response.choices[0].message.content)
    ```
676

677
Full example: [examples/online_serving/openai_chat_completion_client_for_multimodal.py](../../examples/online_serving/openai_chat_completion_client_for_multimodal.py)
678

679
680
681
!!! tip
    Loading from local file paths is also supported on vLLM: You can specify the allowed local media path via `--allowed-local-media-path` when launching the API server/engine,
    and pass the file path as `url` in the API request.
682

683
684
685
!!! tip
    There is no need to place image placeholders in the text content of the API request - they are already represented by the image content.
    In fact, you can place image placeholders in the middle of the text by interleaving text and image content.
686

687
688
689
!!! note
    By default, the timeout for fetching images through HTTP URL is `5` seconds.
    You can override this by setting the environment variable:
690

691
    ```bash
692
693
    export VLLM_IMAGE_FETCH_TIMEOUT=<timeout>
    ```
694

695
### Video Inputs
696

697
Instead of `image_url`, you can pass a video file via `video_url`. Here is a simple example using [LLaVA-OneVision](https://huggingface.co/llava-hf/llava-onevision-qwen2-0.5b-ov-hf).
698

699
700
701
First, launch the OpenAI-compatible server:

```bash
702
vllm serve llava-hf/llava-onevision-qwen2-0.5b-ov-hf --runner generate --max-model-len 8192
703
704
705
```

Then, you can use the OpenAI client as follows:
706

707
??? code
708

709
710
    ```python
    from openai import OpenAI
711

712
713
    openai_api_key = "EMPTY"
    openai_api_base = "http://localhost:8000/v1"
714

715
716
717
718
    client = OpenAI(
        api_key=openai_api_key,
        base_url=openai_api_base,
    )
719

720
    video_url = "http://commondatastorage.googleapis.com/gtv-videos-bucket/sample/ForBiggerFun.mp4"
721

722
723
    ## Use video url in the payload
    chat_completion_from_url = client.chat.completions.create(
724
725
726
727
728
729
730
        messages=[
            {
                "role": "user",
                "content": [
                    {
                        "type": "text",
                        "text": "What's in this video?",
731
                    },
732
733
734
735
736
737
738
739
                    {
                        "type": "video_url",
                        "video_url": {"url": video_url},
                        "uuid": video_url,  # Optional
                    },
                ],
            }
        ],
740
741
742
743
744
745
746
        model=model,
        max_completion_tokens=64,
    )

    result = chat_completion_from_url.choices[0].message.content
    print("Chat completion output from image url:", result)
    ```
747

748
Full example: [examples/online_serving/openai_chat_completion_client_for_multimodal.py](../../examples/online_serving/openai_chat_completion_client_for_multimodal.py)
749

750
751
752
!!! note
    By default, the timeout for fetching videos through HTTP URL is `30` seconds.
    You can override this by setting the environment variable:
753

754
    ```bash
755
756
    export VLLM_VIDEO_FETCH_TIMEOUT=<timeout>
    ```
757

758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
#### Video Frame Recovery

For improved robustness when processing potentially corrupted or truncated video files, vLLM supports optional frame recovery using a dynamic window forward-scan approach. When enabled, if a target frame fails to load during sequential reading, the next successfully grabbed frame (before the next target frame) will be used in its place.

To enable video frame recovery, pass the `frame_recovery` parameter via `--media-io-kwargs`:

```bash
# Example: Enable frame recovery
vllm serve Qwen/Qwen3-VL-30B-A3B-Instruct \
  --media-io-kwargs '{"video": {"frame_recovery": true}}'
```

**Parameters:**

- `frame_recovery`: Boolean flag to enable forward-scan recovery. When `true`, failed frames are recovered using the next available frame within the dynamic window (up to the next target frame). Default is `false`.

**How it works:**

1. The system reads frames sequentially
2. If a target frame fails to grab, it's marked as "failed"
3. The next successfully grabbed frame (before reaching the next target) is used to recover the failed frame
4. This approach handles both mid-video corruption and end-of-video truncation

Works with common video formats like MP4 when using OpenCV backends.

783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
#### Pre-extracted Frame Sequences with `media_io_kwargs`

When you extract video frames on the client side and send them as `video/jpeg` (base64-concatenated JPEG frames), you can preserve the original video metadata by using `media_io_kwargs` in your request. This enables more accurate video understanding by preserving temporal information that would otherwise be lost during client-side frame extraction.

**Supported Parameters:**

| Parameter | Type | Description |
| --------- | ---- | ----------- |
| `fps` | float | Frame rate of the original video |
| `frames_indices` | list[int] | Indices of the actually sampled frames |
| `total_num_frames` | int | Total frame count of the original video |
| `duration` | float | Duration of the original video in seconds |
| `do_sample_frames` | bool | Whether to perform frame sampling |

??? code

    ```python
    from openai import OpenAI

    client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

    # Client-side frame extraction
    frames = extract_frames(video_path, num_frames=32)
    frames_b64 = ",".join([encode_image(f) for f in frames])
    video_url = f"data:video/jpeg;base64,{frames_b64}"

    # Pass video metadata via media_io_kwargs
    response = client.chat.completions.create(
        model="your-multimodal-model",
        messages=[{
            "role": "user",
            "content": [
                {"type": "video_url", "video_url": {"url": video_url}},
                {"type": "text", "text": "Describe what happens in this video."}
            ]
        }],
        extra_body={
            "media_io_kwargs": {
                "video": {
                    "fps": 30.0,
                    "frames_indices": [0, 10, 20, 30, 40, 50, 60, 70, 80, 90,
                                       100, 110, 120, 130, 140, 150, 160, 170,
                                       180, 190, 200, 210, 220, 230, 240, 250,
                                       260, 270, 280, 290, 300, 310],
                    "total_num_frames": 900,
                    "duration": 30.0,
                }
            }
        },
    )

    print(response.choices[0].message.content)
    ```

**Why use `media_io_kwargs`?**

When extracting frames client-side, the server loses important context about the original video:

- **Temporal information**: Which frames were sampled and their positions in the original timeline
- **Video duration**: How long the original video was
- **Frame rate**: The original playback speed

By passing this metadata, the model can better understand the temporal distribution of the sampled frames and whether important moments might have been skipped.

847
848
849
850
851
852
853
854
855
856
857
858
859
860
#### Custom RGBA Background Color

To use a custom background color for RGBA images, pass the `rgba_background_color` parameter via `--media-io-kwargs`:

```bash
# Example: Black background for dark theme
vllm serve llava-hf/llava-1.5-7b-hf \
  --media-io-kwargs '{"image": {"rgba_background_color": [0, 0, 0]}}'

# Example: Custom gray background
vllm serve llava-hf/llava-1.5-7b-hf \
  --media-io-kwargs '{"image": {"rgba_background_color": [128, 128, 128]}}'
```

861
### Audio Inputs
862
863

Audio input is supported according to [OpenAI Audio API](https://platform.openai.com/docs/guides/audio?audio-generation-quickstart-example=audio-in).
864
Here is a simple example using Ultravox-v0.5-1B.
865
866
867
868

First, launch the OpenAI-compatible server:

```bash
869
vllm serve fixie-ai/ultravox-v0_5-llama-3_2-1b
870
871
872
873
```

Then, you can use the OpenAI client as follows:

874
??? code
875

876
877
878
879
880
    ```python
    import base64
    import requests
    from openai import OpenAI
    from vllm.assets.audio import AudioAsset
881

882
883
    def encode_base64_content_from_url(content_url: str) -> str:
        """Encode a content retrieved from a remote url to base64 format."""
884

885
886
887
        with requests.get(content_url) as response:
            response.raise_for_status()
            result = base64.b64encode(response.content).decode('utf-8')
888

889
        return result
890

891
892
    openai_api_key = "EMPTY"
    openai_api_base = "http://localhost:8000/v1"
893

894
895
896
897
    client = OpenAI(
        api_key=openai_api_key,
        base_url=openai_api_base,
    )
898

899
    # Any format supported by soundfile/PyAV is supported
900
901
    audio_url = AudioAsset("winning_call").url
    audio_base64 = encode_base64_content_from_url(audio_url)
902

903
    chat_completion_from_base64 = client.chat.completions.create(
904
905
906
907
908
909
910
        messages=[
            {
                "role": "user",
                "content": [
                    {
                        "type": "text",
                        "text": "What's in this audio?",
911
                    },
912
913
914
915
916
917
918
919
920
921
922
                    {
                        "type": "input_audio",
                        "input_audio": {
                            "data": audio_base64,
                            "format": "wav",
                        },
                        "uuid": audio_url,  # Optional
                    },
                ],
            },
        ],
923
924
925
926
927
928
929
        model=model,
        max_completion_tokens=64,
    )

    result = chat_completion_from_base64.choices[0].message.content
    print("Chat completion output from input audio:", result)
    ```
930

931
Alternatively, you can pass `audio_url`, which is the audio counterpart of `image_url` for image input:
932

933
??? code
934

935
936
    ```python
    chat_completion_from_url = client.chat.completions.create(
937
938
939
940
941
942
943
        messages=[
            {
                "role": "user",
                "content": [
                    {
                        "type": "text",
                        "text": "What's in this audio?",
944
                    },
945
946
947
948
949
950
951
952
                    {
                        "type": "audio_url",
                        "audio_url": {"url": audio_url},
                        "uuid": audio_url,  # Optional
                    },
                ],
            }
        ],
953
954
955
956
957
958
959
        model=model,
        max_completion_tokens=64,
    )

    result = chat_completion_from_url.choices[0].message.content
    print("Chat completion output from audio url:", result)
    ```
960

961
Full example: [examples/online_serving/openai_chat_completion_client_for_multimodal.py](../../examples/online_serving/openai_chat_completion_client_for_multimodal.py)
962

963
964
965
!!! note
    By default, the timeout for fetching audios through HTTP URL is `10` seconds.
    You can override this by setting the environment variable:
966

967
    ```bash
968
969
    export VLLM_AUDIO_FETCH_TIMEOUT=<timeout>
    ```
970

971
### Embedding Inputs
972

973
To input pre-computed embeddings belonging to a data type (i.e. image, video, or audio) directly to the language model,
974
975
976
977
978
pass a tensor of shape `(..., hidden_size of LM)` for each item to the corresponding field of the multi-modal dictionary.

!!! important
    Unlike offline inference, the embeddings for each item must be passed separately
    in order for placeholder tokens to be applied correctly by the chat template.
979
980
981
982
983
984

You must enable this feature via the `--enable-mm-embeds` flag in `vllm serve`.

!!! warning
    The vLLM engine may crash if incorrect shape of embeddings is passed.
    Only enable this flag for trusted users!
985

986
#### Image Embedding Inputs
987

988
989
990
For image embeddings, you can pass the base64-encoded tensor to the `image_embeds` field.
The following example demonstrates how to pass image embeddings to the OpenAI server:

991
??? code
992
993

    ```python
994
995
    from vllm.utils.serial_utils import tensor2base64

996
997
998
999
1000
1001
1002
1003
    client = OpenAI(
        # defaults to os.environ.get("OPENAI_API_KEY")
        api_key=openai_api_key,
        base_url=openai_api_base,
    )

    # Basic usage - this is equivalent to the LLaVA example for offline inference
    model = "llava-hf/llava-1.5-7b-hf"
1004
    embeds = {
1005
        "type": "image_embeds",
1006
        "image_embeds": tensor2base64(torch.load(...)),  # Shape: (image_feature_size, hidden_size)
1007
        "uuid": image_url,  # Optional
1008
1009
    }

1010
1011

    # Additional examples for models that require extra fields
1012
    model = "Qwen/Qwen2-VL-2B-Instruct"
1013
    embeds = {
1014
1015
        "type": "image_embeds",
        "image_embeds": {
1016
1017
            "image_embeds": tensor2base64(torch.load(...)),  # Shape: (image_feature_size, hidden_size)
            "image_grid_thw": tensor2base64(torch.load(...)),  # Shape: (3,)
1018
        },
1019
        "uuid": image_url,  # Optional
1020
    }
1021

1022
    model = "openbmb/MiniCPM-V-2_6"
1023
    embeds = {
1024
1025
        "type": "image_embeds",
        "image_embeds": {
1026
1027
            "image_embeds": tensor2base64(torch.load(...)),  # Shape: (num_slices, hidden_size)
            "image_sizes": tensor2base64(torch.load(...)),  # Shape: (2,)
1028
        },
1029
        "uuid": image_url,  # Optional
1030
    }
1031
1032

    # Single image input
1033
1034
1035
    chat_completion = client.chat.completions.create(
        messages=[
            {
1036
1037
                "role": "system",
                "content": "You are a helpful assistant.",
1038
            },
1039
1040
1041
1042
1043
1044
1045
1046
1047
1048
1049
            {
                "role": "user",
                "content": [
                    {
                        "type": "text",
                        "text": "What's in this image?",
                    },
                    embeds,
                ],
            },
        ],
1050
1051
        model=model,
    )
1052
1053
1054
1055
1056
1057
1058
1059
1060
1061
1062
1063
1064
1065
1066
1067
1068
1069
1070
1071
1072
1073
1074
1075
1076
1077
1078
1079
1080
1081
1082
1083
1084
1085
1086
1087
1088
1089
1090
1091
1092
1093
1094
1095

    # Multi image input
    chat_completion = client.chat.completions.create(
        messages=[
            {
                "role": "system",
                "content": "You are a helpful assistant.",
            },
            {
                "role": "user",
                "content": [
                    {
                        "type": "text",
                        "text": "What's in this image?",
                    },
                    embeds,
                    embeds,
                ],
            },
        ],
        model=model,
    )

    # Multi image input (interleaved)
    chat_completion = client.chat.completions.create(
        messages=[
            {
                "role": "system",
                "content": "You are a helpful assistant.",
            },
            {
                "role": "user",
                "content": [
                    embeds,
                    {
                        "type": "text",
                        "text": "What's in this image?",
                    },
                    embeds,
                ],
            },
        ],
        model=model,
    )
1096
    ```
1097

1098
1099
1100
### Cached Inputs

Just like with offline inference, you can skip sending media if you expect cache hits with provided UUIDs. You can do so by sending media like this:
1101

1102
1103
??? code

1104
1105
1106
1107
1108
1109
1110
1111
1112
1113
1114
1115
    ```python
        # Image/video/audio URL:
        {
            "type": "image_url",
            "image_url": None,
            "uuid": image_uuid,
        },

        # image_embeds
        {
            "type": "image_embeds",
            "image_embeds": None,
1116
            "uuid": image_uuid,
1117
1118
1119
1120
1121
1122
        },

        # input_audio:
        {
            "type": "input_audio",
            "input_audio": None,
1123
            "uuid": audio_uuid,
1124
1125
1126
1127
1128
        },

        # PIL Image:
        {
            "type": "image_pil",
1129
1130
1131
            "image_pil": None,
            "uuid": image_uuid,
        },
1132
1133

    ```