gpu.cuda.inc.md 14.1 KB
Newer Older
1
# --8<-- [start:installation]
2

3
vLLM contains pre-compiled C++ and CUDA (12.8) binaries.
4

5
6
# --8<-- [end:installation]
# --8<-- [start:requirements]
7
8
9

- GPU: compute capability 7.0 or higher (e.g., V100, T4, RTX20xx, A100, L4, H100, etc.)

10
11
# --8<-- [end:requirements]
# --8<-- [start:set-up-using-python]
12

13
!!! note
14
    PyTorch installed via `conda` will statically link `NCCL` library, which can cause issues when vLLM tries to use `NCCL`. See <https://github.com/vllm-project/vllm/issues/8420> for more details.
15
16
17

In order to be performant, vLLM has to compile many cuda kernels. The compilation unfortunately introduces binary incompatibility with other CUDA versions and PyTorch versions, even for the same PyTorch version with different building configurations.

18
Therefore, it is recommended to install vLLM with a **fresh new** environment. If either you have a different CUDA version or you want to use an existing PyTorch installation, you need to build vLLM from source. See [below](#build-wheel-from-source) for more details.
19

20
21
# --8<-- [end:set-up-using-python]
# --8<-- [start:pre-built-wheels]
22

23
```bash
24
uv pip install vllm --torch-backend=auto
25
26
```

27
28
??? console "pip"
    ```bash
29
30
    # Install vLLM with CUDA 12.9.
    pip install vllm --extra-index-url https://download.pytorch.org/whl/cu129
31
32
    ```

33
We recommend leveraging `uv` to [automatically select the appropriate PyTorch index at runtime](https://docs.astral.sh/uv/guides/integration/pytorch/#automatic-backend-selection) by inspecting the installed CUDA driver version via `--torch-backend=auto` (or `UV_TORCH_BACKEND=auto`). To select a specific backend (e.g., `cu128`), set `--torch-backend=cu128` (or `UV_TORCH_BACKEND=cu128`). If this doesn't work, try running `uv self update` to update `uv` first.
34

35
36
!!! note
    NVIDIA Blackwell GPUs (B200, GB200) require a minimum of CUDA 12.8, so make sure you are installing PyTorch wheels with at least that version. PyTorch itself offers a [dedicated interface](https://pytorch.org/get-started/locally/) to determine the appropriate pip command to run for a given target configuration.
37

38
As of now, vLLM's binaries are compiled with CUDA 12.9 and public PyTorch release versions by default. We also provide vLLM binaries compiled with CUDA 12.8, 13.0, and public PyTorch release versions:
39

40
```bash
41
# Install vLLM with a specific CUDA version (e.g., 13.0).
42
export VLLM_VERSION=$(curl -s https://api.github.com/repos/vllm-project/vllm/releases/latest | jq -r .tag_name | sed 's/^v//')
43
export CUDA_VERSION=130 # or other
44
45
export CPU_ARCH=$(uname -m) # x86_64 or aarch64
uv pip install https://github.com/vllm-project/vllm/releases/download/v${VLLM_VERSION}/vllm-${VLLM_VERSION}+cu${CUDA_VERSION}-cp38-abi3-manylinux_2_35_${CPU_ARCH}.whl --extra-index-url https://download.pytorch.org/whl/cu${CUDA_VERSION}
46
47
```

48
#### Install the latest code
49

50
51
52
53
54
55
LLM inference is a fast-evolving field, and the latest code may contain bug fixes, performance improvements, and new features that are not released yet. To allow users to try the latest code without waiting for the next release, vLLM provides wheels for every commit since `v0.5.3` on <https://wheels.vllm.ai/nightly>. There are multiple indices that could be used:

* `https://wheels.vllm.ai/nightly`: the default variant (CUDA with version specified in `VLLM_MAIN_CUDA_VERSION`) built with the last commit on the `main` branch. Currently it is CUDA 12.9.
* `https://wheels.vllm.ai/nightly/<variant>`: all other variants. Now this includes `cu130`, and `cpu`. The default variant (`cu129`) also has a subdirectory to keep consistency.

To install from nightly index, run:
56

57
```bash
Reid's avatar
Reid committed
58
59
uv pip install -U vllm \
    --torch-backend=auto \
60
    --extra-index-url https://wheels.vllm.ai/nightly # add variant subdirectory here if needed
61
62
```

63
64
65
66
67
68
!!! warning "`pip` caveat"

    Using `pip` to install from nightly indices is _not supported_, because `pip` combines packages from `--extra-index-url` and the default index, choosing only the latest version, which makes it difficult to install a development version prior to the released version. In contrast, `uv` gives the extra index [higher priority than the default index](https://docs.astral.sh/uv/pip/compatibility/#packages-that-exist-on-multiple-indexes).

    If you insist on using `pip`, you have to specify the full URL of the wheel file (which can be obtained from the web page).

69
    ```bash
70
71
    pip install -U https://wheels.vllm.ai/nightly/vllm-0.11.2.dev399%2Bg3c7461c18-cp38-abi3-manylinux_2_31_x86_64.whl # current nightly build (the filename will change!)
    pip install -U https://wheels.vllm.ai/${VLLM_COMMIT}/vllm-0.11.2.dev399%2Bg3c7461c18-cp38-abi3-manylinux_2_31_x86_64.whl # from specific commit
72
    ```
73

74
##### Install specific revisions
75

76
77
If you want to access the wheels for previous commits (e.g. to bisect the behavior change, performance regression), you can specify the commit hash in the URL:

78
```bash
79
export VLLM_COMMIT=72d9c316d3f6ede485146fe5aabd4e61dbc59069 # use full commit hash from the main branch
Reid's avatar
Reid committed
80
81
uv pip install vllm \
    --torch-backend=auto \
82
    --extra-index-url https://wheels.vllm.ai/${VLLM_COMMIT} # add variant subdirectory here if needed
83
84
```

85
86
# --8<-- [end:pre-built-wheels]
# --8<-- [start:build-wheel-from-source]
87

88
#### Set up using Python-only build (without compilation) {#python-only-build}
89

90
If you only need to change Python code, you can build and install vLLM without compilation. Using `uv pip`'s [`--editable` flag](https://docs.astral.sh/uv/pip/packages/#editable-packages), changes you make to the code will be reflected when you run vLLM:
91

92
```bash
93
94
git clone https://github.com/vllm-project/vllm.git
cd vllm
95
VLLM_USE_PRECOMPILED=1 uv pip install --editable .
96
97
```

98
This command will do the following:
David Xia's avatar
David Xia committed
99

100
1. Look for the current branch in your vLLM clone.
David Xia's avatar
David Xia committed
101
102
103
1. Identify the corresponding base commit in the main branch.
1. Download the pre-built wheel of the base commit.
1. Use its compiled libraries in the installation.
104

105
106
107
!!! note
    1. If you change C++ or kernel code, you cannot use Python-only build; otherwise you will see an import error about library not found or undefined symbol.
    2. If you rebase your dev branch, it is recommended to uninstall vllm and re-run the above command to make sure your libraries are up to date.
108
109

In case you see an error about wheel not found when running the above command, it might be because the commit you based on in the main branch was just merged and the wheel is being built. In this case, you can wait for around an hour to try again, or manually assign the previous commit in the installation using the `VLLM_PRECOMPILED_WHEEL_LOCATION` environment variable.
110

111
```bash
112
113
export VLLM_PRECOMPILED_WHEEL_COMMIT=$(git rev-parse HEAD~1) # or earlier commit on main
export VLLM_USE_PRECOMPILED=1
114
uv pip install --editable .
115
116
```

117
118
119
120
There are more environment variables to control the behavior of Python-only build:

* `VLLM_PRECOMPILED_WHEEL_LOCATION`: specify the exact wheel URL or local file path of a pre-compiled wheel to use. All other logic to find the wheel will be skipped.
* `VLLM_PRECOMPILED_WHEEL_COMMIT`: override the commit hash to download the pre-compiled wheel. It can be `nightly` to use the last **already built** commit on the main branch.
121
* `VLLM_PRECOMPILED_WHEEL_VARIANT`: specify the variant subdirectory to use on the nightly index, e.g., `cu129`, `cu130`, `cpu`. If not specified, the variant is auto-detected based on your system's CUDA version (from PyTorch or nvidia-smi). You can also set `VLLM_MAIN_CUDA_VERSION` to override auto-detection.
122

123
You can find more information about vLLM's wheels in [Install the latest code](#install-the-latest-code).
124

125
126
!!! note
    There is a possibility that your source code may have a different commit ID compared to the latest vLLM wheel, which could potentially lead to unknown errors.
127
    It is recommended to use the same commit ID for the source code as the vLLM wheel you have installed. Please refer to [Install the latest code](#install-the-latest-code) for instructions on how to install a specified wheel.
128

129
#### Full build (with compilation) {#full-build}
130
131
132

If you want to modify C++ or CUDA code, you'll need to build vLLM from source. This can take several minutes:

133
```bash
134
135
git clone https://github.com/vllm-project/vllm.git
cd vllm
136
uv pip install -e .
137
138
```

139
140
!!! tip
    Building from source requires a lot of compilation. If you are building from source repeatedly, it's more efficient to cache the compilation results.
141

142
143
    For example, you can install [ccache](https://github.com/ccache/ccache) using `conda install ccache` or `apt install ccache` .
    As long as `which ccache` command can find the `ccache` binary, it will be used automatically by the build system. After the first build, subsequent builds will be much faster.
144

145
    When using `ccache` with `pip install -e .`, you should run `CCACHE_NOHASHDIR="true" pip install --no-build-isolation -e .`. This is because `pip` creates a new folder with a random name for each build, preventing `ccache` from recognizing that the same files are being built.
146

147
148
    [sccache](https://github.com/mozilla/sccache) works similarly to `ccache`, but has the capability to utilize caching in remote storage environments.
    The following environment variables can be set to configure the vLLM `sccache` remote: `SCCACHE_BUCKET=vllm-build-sccache SCCACHE_REGION=us-west-2 SCCACHE_S3_NO_CREDENTIALS=1`. We also recommend setting `SCCACHE_IDLE_TIMEOUT=0`.
149

150
!!! note "Faster Kernel Development"
151
    For frequent C++/CUDA kernel changes, after the initial `uv pip install -e .` setup, consider using the [Incremental Compilation Workflow](../../contributing/incremental_build.md) for significantly faster rebuilds of only the modified kernel code.
152

153
##### Use an existing PyTorch installation
154

155
There are scenarios where the PyTorch dependency cannot be easily installed with `uv`, for example, when building vLLM with non-default PyTorch builds (like nightly or a custom build).
156

157
158
159
160
161
162
163
164
165
166
167
168
169
To build vLLM using an existing PyTorch installation:

```bash
# install PyTorch first, either from PyPI or from source
git clone https://github.com/vllm-project/vllm.git
cd vllm
python use_existing_torch.py
uv pip install -r requirements/build.txt
uv pip install --no-build-isolation -e .
```

Alternatively: if you are exclusively using `uv` to create and manage virtual environments, it has [a unique mechanism](https://docs.astral.sh/uv/concepts/projects/config/#disabling-build-isolation)
for disabling build isolation for specific packages. vLLM can leverage this mechanism to specify `torch` as the package to disable build isolation for:
170

171
```bash
172
# install PyTorch first, either from PyPI or from source
173
174
git clone https://github.com/vllm-project/vllm.git
cd vllm
175
176
# pip install -e . does not work directly, only uv can do this
uv pip install -e .
177
178
```

179
##### Use the local cutlass for compilation
180
181
182
183

Currently, before starting the build process, vLLM fetches cutlass code from GitHub. However, there may be scenarios where you want to use a local version of cutlass instead.
To achieve this, you can set the environment variable VLLM_CUTLASS_SRC_DIR to point to your local cutlass directory.

184
```bash
185
186
git clone https://github.com/vllm-project/vllm.git
cd vllm
187
VLLM_CUTLASS_SRC_DIR=/path/to/cutlass uv pip install -e .
188
189
```

190
##### Troubleshooting
191
192
193
194

To avoid your system being overloaded, you can limit the number of compilation jobs
to be run simultaneously, via the environment variable `MAX_JOBS`. For example:

195
```bash
196
export MAX_JOBS=6
197
uv pip install -e .
198
199
200
201
202
203
204
```

This is especially useful when you are building on less powerful machines. For example, when you use WSL it only [assigns 50% of the total memory by default](https://learn.microsoft.com/en-us/windows/wsl/wsl-config#main-wsl-settings), so using `export MAX_JOBS=1` can avoid compiling multiple files simultaneously and running out of memory.
A side effect is a much slower build process.

Additionally, if you have trouble building vLLM, we recommend using the NVIDIA PyTorch Docker image.

205
```bash
206
# Use `--ipc=host` to make sure the shared memory is large enough.
Reid's avatar
Reid committed
207
208
209
210
211
docker run \
    --gpus all \
    -it \
    --rm \
    --ipc=host nvcr.io/nvidia/pytorch:23.10-py3
212
213
214
215
```

If you don't want to use docker, it is recommended to have a full installation of CUDA Toolkit. You can download and install it from [the official website](https://developer.nvidia.com/cuda-toolkit-archive). After installation, set the environment variable `CUDA_HOME` to the installation path of CUDA Toolkit, and make sure that the `nvcc` compiler is in your `PATH`, e.g.:

216
```bash
217
218
export CUDA_HOME=/usr/local/cuda
export PATH="${CUDA_HOME}/bin:$PATH"
219
220
221
222
```

Here is a sanity check to verify that the CUDA Toolkit is correctly installed:

223
```bash
224
225
nvcc --version # verify that nvcc is in your PATH
${CUDA_HOME}/bin/nvcc --version # verify that nvcc is in your CUDA_HOME
226
227
```

228
#### Unsupported OS build
229
230
231
232
233

vLLM can fully run only on Linux but for development purposes, you can still build it on other systems (for example, macOS), allowing for imports and a more convenient development environment. The binaries will not be compiled and won't work on non-Linux systems.

Simply disable the `VLLM_TARGET_DEVICE` environment variable before installing:

234
```bash
235
export VLLM_TARGET_DEVICE=empty
236
uv pip install -e .
237
```
238

239
240
# --8<-- [end:build-wheel-from-source]
# --8<-- [start:pre-built-images]
241

242
See [Using Docker](../../deployment/docker.md) for instructions on using the official Docker image.
243
244
245

Another way to access the latest code is to use the docker images:

246
```bash
247
248
249
250
251
252
253
254
export VLLM_COMMIT=33f460b17a54acb3b6cc0b03f4a17876cff5eafd # use full commit hash from the main branch
docker pull public.ecr.aws/q9t5s3a7/vllm-ci-postmerge-repo:${VLLM_COMMIT}
```

These docker images are used for CI and testing only, and they are not intended for production use. They will be expired after several days.

The latest code can contain bugs and may not be stable. Please use it with caution.

255
256
# --8<-- [end:pre-built-images]
# --8<-- [start:build-image-from-source]
257

258
See [Building vLLM's Docker Image from Source](../../deployment/docker.md#building-vllms-docker-image-from-source) for instructions on building the Docker image.
259

260
261
# --8<-- [end:build-image-from-source]
# --8<-- [start:supported-features]
262

263
See [Feature x Hardware](../../features/README.md#feature-x-hardware) compatibility matrix for feature support information.
264
265

# --8<-- [end:supported-features]