installation.rst 8.62 KB
Newer Older
Zhuohan Li's avatar
Zhuohan Li committed
1
2
.. _installation:

Woosuk Kwon's avatar
Woosuk Kwon committed
3
4
5
Installation
============

6
vLLM is a Python library that also contains pre-compiled C++ and CUDA (12.1) binaries.
7
8
9

Requirements
------------
10
11

* OS: Linux
12
* Python: 3.8 -- 3.12
13
* GPU: compute capability 7.0 or higher (e.g., V100, T4, RTX20xx, A100, L4, H100, etc.)
14
15
16
17

Install with pip
----------------

Woosuk Kwon's avatar
Woosuk Kwon committed
18
You can install vLLM using pip:
19
20
21

.. code-block:: console

22
    $ # (Recommended) Create a new conda environment.
23
    $ conda create -n myenv python=3.10 -y
24
25
    $ conda activate myenv

26
    $ # Install vLLM with CUDA 12.1.
27
    $ pip install vllm
28

29
30
31
32
.. note::

    Although we recommend using ``conda`` to create and manage Python environments, it is highly recommended to use ``pip`` to install vLLM. This is because ``pip`` can install ``torch`` with separate library packages like ``NCCL``, while ``conda`` installs ``torch`` with statically linked ``NCCL``. This can cause issues when vLLM tries to use ``NCCL``. See `this issue <https://github.com/vllm-project/vllm/issues/8420>`_ for more details.

33
34
.. note::

35
36
    As of now, vLLM's binaries are compiled with CUDA 12.1 and public PyTorch release versions by default.
    We also provide vLLM binaries compiled with CUDA 11.8 and public PyTorch release versions:
37
38
39
40

    .. code-block:: console

        $ # Install vLLM with CUDA 11.8.
41
        $ export VLLM_VERSION=0.6.1.post1
42
        $ export PYTHON_VERSION=310
43
        $ pip install https://github.com/vllm-project/vllm/releases/download/v${VLLM_VERSION}/vllm-${VLLM_VERSION}+cu118-cp${PYTHON_VERSION}-cp${PYTHON_VERSION}-manylinux1_x86_64.whl --extra-index-url https://download.pytorch.org/whl/cu118
44

45
    In order to be performant, vLLM has to compile many cuda kernels. The compilation unfortunately introduces binary incompatibility with other CUDA versions and PyTorch versions, even for the same PyTorch version with different building configurations.
46

47
    Therefore, it is recommended to install vLLM with a **fresh new** conda environment. If either you have a different CUDA version or you want to use an existing PyTorch installation, you need to build vLLM from source. See below for instructions.
48

49
50
.. note::

youkaichao's avatar
youkaichao committed
51
    vLLM also publishes wheels for Linux running on x86 platform with cuda 12 for every commit since v0.5.3. You can download and install them with the following command:
52
53
54

    .. code-block:: console

youkaichao's avatar
youkaichao committed
55
56
57
58
59
60
61
62
63
64
        $ export VLLM_COMMIT=33f460b17a54acb3b6cc0b03f4a17876cff5eafd # use full commit hash from the main branch
        $ pip install https://vllm-wheels.s3.us-west-2.amazonaws.com/${VLLM_COMMIT}/vllm-1.0.0.dev-cp38-abi3-manylinux1_x86_64.whl

    You can also just download the latest wheel by running:

    .. code-block:: console

        $ pip install https://vllm-wheels.s3.us-west-2.amazonaws.com/nightly/vllm-1.0.0.dev-cp38-abi3-manylinux1_x86_64.whl

    Note that the wheels are built with Python 3.8 abi (see `PEP 425 <https://peps.python.org/pep-0425/>`_ for more details about abi), so they are compatible with Python 3.8 and later. The version string in the wheel file name (``1.0.0.dev``) is just a placeholder to have a unified URL for the wheels. The actual version of wheels is contained in the wheel metadata.
65

66
67
68
69
70
71
72
73
74
Build from source (without compilation)
---------------------------------------

If you want to develop vLLM, and you only need to change the Python code, you can build vLLM without compilation.

The first step is to follow the previous instructions to install the latest vLLM wheel:

.. code-block:: console

youkaichao's avatar
youkaichao committed
75
    $ pip install https://vllm-wheels.s3.us-west-2.amazonaws.com/nightly/vllm-1.0.0.dev-cp38-abi3-manylinux1_x86_64.whl
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92

After verifying that the installation is successful, we have a script for you to copy and link directories, so that you can edit the Python code directly:

.. code-block:: console

    $ git clone https://github.com/vllm-project/vllm.git
    $ cd vllm
    $ python python_only_dev.py

It will:

- Find the installed vLLM in the current environment.
- Copy built files to the current directory.
- Rename the installed vLLM
- Symbolically link the current directory to the installed vLLM.

This way, you can edit the Python code in the current directory, and the changes will be reflected in the installed vLLM.
93

94
95
.. _build_from_source:

96
97
Build from source (with compilation)
------------------------------------
Woosuk Kwon's avatar
Woosuk Kwon committed
98

99
If you need to touch the C++ or CUDA code, you need to build vLLM from source:
100

Woosuk Kwon's avatar
Woosuk Kwon committed
101
102
.. code-block:: console

103
    $ git clone https://github.com/vllm-project/vllm.git
Woosuk Kwon's avatar
Woosuk Kwon committed
104
    $ cd vllm
105
    $ pip install -e .  # This may take 5-10 minutes.
106

107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
.. note::

    This will uninstall existing PyTorch, and install the version required by vLLM. If you want to use an existing PyTorch installation, there need to be some changes:

    .. code-block:: console

        $ git clone https://github.com/vllm-project/vllm.git
        $ cd vllm
        $ python use_existing_torch.py
        $ pip install -r requirements-build.txt
        $ pip install -e . --no-build-isolation

    The differences are:

    - ``python use_existing_torch.py``: This script will remove all the PyTorch versions in the requirements files, so that the existing PyTorch installation will be used.
    - ``pip install -r requirements-build.txt``: You need to manually install the requirements for building vLLM.
    - ``pip install -e . --no-build-isolation``: You need to disable build isolation, so that the build system can use the existing PyTorch installation.

    This is especially useful when the PyTorch dependency cannot be easily installed via pip, e.g.:

    - build vLLM with PyTorch nightly or a custom PyTorch build.
    - build vLLM with aarch64 and cuda (GH200), where the PyTorch wheels are not available on PyPI. Currently, only PyTorch nightly has wheels for aarch64 with CUDA. You can run ``pip3 install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cu124`` to install PyTorch nightly, and then build vLLM on top of it.

130
131
132
133
134
135
136
137
138
139
.. note::

    vLLM can fully run only on Linux, but you can still build it on other systems (for example, macOS). This build is only for development purposes, allowing for imports and a more convenient dev environment. The binaries will not be compiled and not work on non-Linux systems. You can create such a build with the following commands:

    .. code-block:: console

        $ export VLLM_TARGET_DEVICE=empty
        $ pip install -e .


140
141
.. tip::

142
    Building from source requires quite a lot compilation. If you are building from source for multiple times, it is beneficial to cache the compilation results. For example, you can install `ccache <https://github.com/ccache/ccache>`_ via either ``conda install ccache`` or ``apt install ccache`` . As long as ``which ccache`` command can find the ``ccache`` binary, it will be used automatically by the build system. After the first build, the subsequent builds will be much faster.
143

144
145
.. tip::
    To avoid your system being overloaded, you can limit the number of compilation jobs
146
    to be run simultaneously, via the environment variable ``MAX_JOBS``. For example:
147
148
149
150
151
152

    .. code-block:: console

        $ export MAX_JOBS=6
        $ pip install -e .

153
154
    This is especially useful when you are building on less powerful machines. For example, when you use WSL, it only `gives you half of the memory by default <https://learn.microsoft.com/en-us/windows/wsl/wsl-config>`_, and you'd better use ``export MAX_JOBS=1`` to avoid compiling multiple files simultaneously and running out of memory. The side effect is that the build process will be much slower. If you only touch the Python code, slow compilation is okay, as you are building in an editable mode: you can just change the code and run the Python script without any re-compilation or re-installation.

155
156
157
158
159
.. tip::
    If you have trouble building vLLM, we recommend using the NVIDIA PyTorch Docker image.

    .. code-block:: console

160
        $ # Use `--ipc=host` to make sure the shared memory is large enough.
161
        $ docker run --gpus all -it --rm --ipc=host nvcr.io/nvidia/pytorch:23.10-py3
162

163
    If you don't want to use docker, it is recommended to have a full installation of CUDA Toolkit. You can download and install it from `the official website <https://developer.nvidia.com/cuda-toolkit-archive>`_. After installation, set the environment variable ``CUDA_HOME`` to the installation path of CUDA Toolkit, and make sure that the ``nvcc`` compiler is in your ``PATH``, e.g.:
164
165
166
167
168
169
170
171
172
173
174
175

    .. code-block:: console

        $ export CUDA_HOME=/usr/local/cuda
        $ export PATH="${CUDA_HOME}/bin:$PATH"

    Here is a sanity check to verify that the CUDA Toolkit is correctly installed:

    .. code-block:: console

        $ nvcc --version # verify that nvcc is in your PATH
        $ ${CUDA_HOME}/bin/nvcc --version # verify that nvcc is in your CUDA_HOME