multimodal_index.rst 1.52 KB
Newer Older
1
2
.. _multi_modality:

3
4
5
6
7
8
9
Multi-Modality
==============

.. currentmodule:: vllm.multimodal
    
vLLM provides experimental support for multi-modal models through the :mod:`vllm.multimodal` package.

10
Multi-modal inputs can be passed alongside text and token prompts to :ref:`supported models <supported_vlms>`
11
via the ``multi_modal_data`` field in :class:`vllm.inputs.PromptInputs`.
12

13
14
Currently, vLLM only has built-in support for image data. You can extend vLLM to process additional modalities
by following :ref:`this guide <adding_multimodal_plugin>`.
15

16
Looking to add your own multi-modal model? Please follow the instructions listed :ref:`here <enabling_multimodal_inputs>`.
17

18
19
20
21
22
23
24
Guides
++++++

.. toctree::
   :maxdepth: 1

   adding_multimodal_plugin
25

26
27
28
29
30
31
32
33
Module Contents
+++++++++++++++

.. automodule:: vllm.multimodal

Registry
--------

34
.. autodata:: vllm.multimodal.MULTIMODAL_REGISTRY
35
36
37
38
39
40
41
42

.. autoclass:: vllm.multimodal.MultiModalRegistry
    :members:
    :show-inheritance:

Base Classes
------------

43
44
.. autodata:: vllm.multimodal.NestedTensors

45
46
.. autodata:: vllm.multimodal.BatchedTensors

47
48
.. autodata:: vllm.multimodal.BatchedTensorInputs

49
.. autoclass:: vllm.multimodal.MultiModalDataBuiltins
50
51
52
    :members:
    :show-inheritance:

53
54
.. autodata:: vllm.multimodal.MultiModalDataDict

55
56
57
58
.. autoclass:: vllm.multimodal.MultiModalInputs
    :members:
    :show-inheritance:

59
60
61
62
63
64
65
66
67
68
.. autoclass:: vllm.multimodal.MultiModalPlugin
    :members:
    :show-inheritance:

Image Classes
-------------

.. automodule:: vllm.multimodal.image
    :members:
    :show-inheritance: