"vscode:/vscode.git/clone" did not exist on "a8f66cbde878d1ddca2288313041dbe3a556dbc4"
gpu.rocm.inc.md 13.9 KB
Newer Older
1
# --8<-- [start:installation]
2

3
vLLM supports AMD GPUs with ROCm 6.3 or above. Pre-built wheels are available for ROCm 7.0.
4

5
6
# --8<-- [end:installation]
# --8<-- [start:requirements]
7

8
- GPU: MI200s (gfx90a), MI300 (gfx942), MI350 (gfx950), Radeon RX 7900 series (gfx1100/1101), Radeon RX 9000 series (gfx1200/1201), Ryzen AI MAX / AI 300 Series (gfx1151/1150)
9
10
- ROCm 6.3 or above
    - MI350 requires ROCm 7.0 or above
11
    - Ryzen AI MAX / AI 300 Series requires ROCm 7.0.2 or above
12

13
14
# --8<-- [end:requirements]
# --8<-- [start:set-up-using-python]
15

16
17
The vLLM wheel bundles PyTorch and all required dependencies, and you should use the included PyTorch for compatibility. Because vLLM compiles many ROCm kernels to ensure a validated, high‑performance stack, the resulting binaries may not be compatible with other ROCm or PyTorch builds.
If you need a different ROCm version or want to use an existing PyTorch installation, you’ll need to build vLLM from source.  See [below](#build-wheel-from-source) for more details.
18

19
20
# --8<-- [end:set-up-using-python]
# --8<-- [start:pre-built-wheels]
21

22
23
24
25
26
27
28
29
30
31
32
33
To install the latest version of vLLM for Python 3.12, ROCm 7.0 and `glibc >= 2.35`.

```bash
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/
```

!!! tip
    You can find out about which ROCm version the latest vLLM supports by checking the index in extra-index-url [https://wheels.vllm.ai/rocm/](https://wheels.vllm.ai/rocm/) .

To install a specific version and ROCm variant of vLLM wheel.

```bash
34
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/0.14.1/rocm700
35
36
37
38
39
40
41
42
43
```

!!! warning "Caveats for using `pip`" 

    We recommend leveraging `uv` to install vLLM wheel. Using `pip` to install from custom indices is cumbersome, because `pip` combines packages from `--extra-index-url` and the default index, choosing only the latest version, which makes it difficult to install wheel from custom index if exact versions of all packages are specified exactly. In contrast, `uv` gives the extra index [higher priority than the default index](https://docs.astral.sh/uv/pip/compatibility/#packages-that-exist-on-multiple-indexes).

    If you insist on using `pip`, you have to specify the exact vLLM version and full URL of the wheel path `https://wheels.vllm.ai/rocm/<version>/<rocm-variant>` (which can be obtained from the web page).

    ```bash
44
    pip install vllm==0.14.1+rocm700 --extra-index-url https://wheels.vllm.ai/rocm/0.14.1/rocm700
45
    ```
46

47
48
# --8<-- [end:pre-built-wheels]
# --8<-- [start:build-wheel-from-source]
49

50
51
52
!!! tip
    - If you found that the following installation step does not work for you, please refer to [docker/Dockerfile.rocm_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_base). Dockerfile is a form of installation steps.

53
54
0. Install prerequisites (skip if you are already in an environment/docker with the following installed):

55
56
    - [ROCm](https://rocm.docs.amd.com/en/latest/deploy/linux/index.html)
    - [PyTorch](https://pytorch.org/)
57

58
    For installing PyTorch, you can start from a fresh docker image, e.g, `rocm/pytorch:rocm7.0_ubuntu22.04_py3.10_pytorch_release_2.8.0`, `rocm/pytorch-nightly`. If you are using docker image, you can skip to Step 3.
59

60
61
    Alternatively, you can install PyTorch using PyTorch wheels. You can check PyTorch installation guide in PyTorch [Getting Started](https://pytorch.org/get-started/locally/). Example:

62
    ```bash
63
    # Install PyTorch
64
    pip uninstall torch -y
65
    pip install --no-cache-dir torch torchvision --index-url https://download.pytorch.org/whl/nightly/rocm7.0
66
    ```
67

68
1. Install [Triton for ROCm](https://github.com/ROCm/triton.git)
69

70
    Install ROCm's Triton following the instructions from [ROCm/triton](https://github.com/ROCm/triton.git)
71

72
    ```bash
73
74
    python3 -m pip install ninja cmake wheel pybind11
    pip uninstall -y triton
75
    git clone https://github.com/ROCm/triton.git
76
    cd triton
77
78
    # git checkout $TRITON_BRANCH
    git checkout f9e5bf54
79
80
    if [ ! -f setup.py ]; then cd python; fi
    python3 setup.py install
81
82
    cd ../..
    ```
83

84
    !!! note
85
86
        - The validated `$TRITON_BRANCH` can be found in the [docker/Dockerfile.rocm_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_base).
        - If you see HTTP issue related to downloading packages during building triton, please try again as the HTTP error is intermittent.
87

88
2. Optionally, if you choose to use CK flash attention, you can install [flash attention for ROCm](https://github.com/Dao-AILab/flash-attention.git)
89

90
    Install ROCm's flash attention (v2.8.0) following the instructions from [ROCm/flash-attention](https://github.com/Dao-AILab/flash-attention#amd-rocm-support)
91

92
    For example, for ROCm 7.0, suppose your gfx arch is `gfx942`. To get your gfx architecture, run `rocminfo |grep gfx`.
93

94
    ```bash
95
    git clone https://github.com/Dao-AILab/flash-attention.git
96
    cd flash-attention
97
98
    # git checkout $FA_BRANCH
    git checkout 0e60e394
99
    git submodule update --init
100
    GPU_ARCHS="gfx942" python3 setup.py install
101
102
    cd ..
    ```
103

104
    !!! note
105
106
        - The validated `$FA_BRANCH` can be found in the [docker/Dockerfile.rocm_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_base).

107

108
3. Optionally, if you choose to build AITER yourself to use a certain branch or commit, you can build AITER using the following steps:
109

110
    ```bash
111
112
113
114
115
116
117
118
    python3 -m pip uninstall -y aiter
    git clone --recursive https://github.com/ROCm/aiter.git
    cd aiter
    git checkout $AITER_BRANCH_OR_COMMIT
    git submodule sync; git submodule update --init --recursive
    python3 setup.py develop
    ```

119
    !!! note
120
121
        - You will need to config the `$AITER_BRANCH_OR_COMMIT` for your purpose.
        - The validated `$AITER_BRANCH_OR_COMMIT` can be found in the [docker/Dockerfile.rocm_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_base).
122

123

124
4. Optionally, if you want to use MORI for EP or PD disaggregation, you can install [MORI](https://github.com/ROCm/mori) using the following steps:
125
126
127
128
129
130

    ```bash
    git clone https://github.com/ROCm/mori.git
    cd mori
    git checkout $MORI_BRANCH_OR_COMMIT
    git submodule sync; git submodule update --init --recursive
131
    MORI_GPU_ARCHS="gfx942;gfx950" python3 setup.py install
132
133
134
135
136
137
138
139
    ```

    !!! note
        - You will need to config the `$MORI_BRANCH_OR_COMMIT` for your purpose.
        - The validated `$MORI_BRANCH_OR_COMMIT` can be found in the [docker/Dockerfile.rocm_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_base).


5. Build vLLM. For example, vLLM on ROCM 7.0 can be built with the following steps:
140

141
    ???+ console "Commands"
142
143
144
145
146
147
148
149
150
151
152
153
154
155

        ```bash
        pip install --upgrade pip

        # Build & install AMD SMI
        pip install /opt/rocm/share/amd_smi

        # Install dependencies
        pip install --upgrade numba \
            scipy \
            huggingface-hub[cli,hf_transfer] \
            setuptools_scm
        pip install -r requirements/rocm.txt

156
157
158
159
160
161
        # To build for a single architecture (e.g., MI300) for faster installation (recommended):
        export PYTORCH_ROCM_ARCH="gfx942"

        # To build vLLM for multiple arch MI210/MI250/MI300, use this instead
        # export PYTORCH_ROCM_ARCH="gfx90a;gfx942"

162
163
        python3 setup.py develop
        ```
164

165
    This may take 5-10 minutes. Currently, `pip install .` does not work for ROCm when installing vLLM from source.
166

167
168
    !!! tip
        - The ROCm version of PyTorch, ideally, should match the ROCm driver version.
169

170
171
!!! tip
    - For MI300x (gfx942) users, to achieve optimal performance, please refer to [MI300x tuning guide](https://rocm.docs.amd.com/en/latest/how-to/tuning-guides/mi300x/index.html) for performance optimization and tuning tips on system and workflow level.
172
      For vLLM, please refer to [vLLM performance optimization](https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/inference-optimization/vllm-optimization.html).
173

174
# --8<-- [end:build-wheel-from-source]
175
# --8<-- [start:pre-built-images]
176

177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
#### Use vLLM's Official Docker Image

vLLM offers an official Docker image for deployment.
The image can be used to run OpenAI compatible server and is available on Docker Hub as [vllm/vllm-openai-rocm](https://hub.docker.com/r/vllm/vllm-openai-rocm/tags).

???+ console "Commands"
    ```bash
    docker run --rm \
        --group-add=video \
        --cap-add=SYS_PTRACE \
        --security-opt seccomp=unconfined \
        --device /dev/kfd \
        --device /dev/dri \
        -v ~/.cache/huggingface:/root/.cache/huggingface \
        --env "HF_TOKEN=$HF_TOKEN" \
        -p 8000:8000 \
        --ipc=host \
        vllm/vllm-openai-rocm:latest \
        --model Qwen/Qwen3-0.6B
    ```

To use the docker image as base for development, you can launch it in interactive session through overriding the entrypoint.

???+ console "Commands"
    ```bash
    docker run --rm -it \
        --group-add=video \
        --cap-add=SYS_PTRACE \
        --security-opt seccomp=unconfined \
        --device /dev/kfd \
        --device /dev/dri \
        -v ~/.cache/huggingface:/root/.cache/huggingface \
        --env "HF_TOKEN=$HF_TOKEN" \
        -p 8000:8000 \
        --ipc=host \
        --entrypoint bash \
        vllm/vllm-openai-rocm:latest
    ```


#### Use AMD's Docker Images

219
220
The [AMD Infinity hub for vLLM](https://hub.docker.com/r/rocm/vllm/tags) offers a prebuilt, optimized
docker image designed for validating inference performance on the AMD Instinct™ MI300X accelerator.
221
AMD also offers nightly prebuilt docker image from [Docker Hub](https://hub.docker.com/r/rocm/vllm-dev), which has vLLM and all its dependencies installed. The entrypoint of this docker image is `/bin/bash` (different from the vLLM's Official Docker Image).
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237

???+ console "Commands"
    ```bash
    docker pull rocm/vllm-dev:nightly # to get the latest image
    docker run -it --rm \
    --network=host \
    --group-add=video \
    --ipc=host \
    --cap-add=SYS_PTRACE \
    --security-opt seccomp=unconfined \
    --device /dev/kfd \
    --device /dev/dri \
    -v <path/to/your/models>:/app/models \
    -e HF_HOME="/app/models" \
    rocm/vllm-dev:nightly
    ```
238

239
240
241
!!! tip
    Please check [LLM inference performance validation on AMD Instinct MI300X](https://rocm.docs.amd.com/en/latest/how-to/performance-validation/mi300x/vllm-benchmark.html)
    for instructions on how to use this prebuilt docker image.
242

243
244
# --8<-- [end:pre-built-images]
# --8<-- [start:build-image-from-source]
245
246
247

Building the Docker image from source is the recommended way to use vLLM with ROCm.

248
??? info "(Optional) Build an image with ROCm software stack"
249

250
251
252
    Build a docker image from [docker/Dockerfile.rocm_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_base) which setup ROCm software stack needed by the vLLM.
    **This step is optional as this rocm_base image is usually prebuilt and store at [Docker Hub](https://hub.docker.com/r/rocm/vllm-dev) under tag `rocm/vllm-dev:base` to speed up user experience.**
    If you choose to build this rocm_base image yourself, the steps are as follows.
253

254
    It is important that the user kicks off the docker build using buildkit. Either the user put `DOCKER_BUILDKIT=1` as environment variable when calling docker build command, or the user needs to set up buildkit in the docker daemon configuration `/etc/docker/daemon.json` as follows and restart the daemon:
255

256
257
258
259
260
    ```json
    {
        "features": {
            "buildkit": true
        }
261
    }
262
    ```
263

264
    To build vllm on ROCm 7.0 for MI200 and MI300 series, you can use the default:
265

266
267
268
269
270
    ```bash
    DOCKER_BUILDKIT=1 docker build \
        -f docker/Dockerfile.rocm_base \
        -t rocm/vllm-dev:base .
    ```
271
272
273

#### Build an image with vLLM

274
First, build a docker image from [docker/Dockerfile.rocm](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm) and launch a docker container from the image.
275
It is important that the user kicks off the docker build using buildkit. Either the user put `DOCKER_BUILDKIT=1` as environment variable when calling docker build command, or the user needs to set up buildkit in the docker daemon configuration /etc/docker/daemon.json as follows and restart the daemon:
276

277
```json
278
279
280
281
282
283
284
{
    "features": {
        "buildkit": true
    }
}
```

285
[docker/Dockerfile.rocm](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm) uses ROCm 7.0 by default, but also supports ROCm 5.7, 6.0, 6.1, 6.2, 6.3, and 6.4, in older vLLM branches.
286
287
It provides flexibility to customize the build of docker image using the following arguments:

288
- `BASE_IMAGE`: specifies the base image used when running `docker build`. The default value `rocm/vllm-dev:base` is an image published and maintained by AMD. It is being built using [docker/Dockerfile.rocm_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_base)
289
- `ARG_PYTORCH_ROCM_ARCH`: Allows to override the gfx architecture values from the base docker image
290
291
292

Their values can be passed in when running `docker build` with `--build-arg` options.

293
To build vllm on ROCm 7.0 for MI200 and MI300 series, you can use the default (which build a docker image with `vllm serve` as entrypoint):
294

295
296
297
298
???+ console "Commands"
    ```bash
    DOCKER_BUILDKIT=1 docker build -f docker/Dockerfile.rocm -t vllm-rocm .
    ```
299
300
301

To run the above docker image `vllm-rocm`, use the below command:

302

303
???+ console "Commands"
304
    ```bash
305
306
307
308
309
310
311
312
313
    docker run -it \
    --network=host \
    --group-add=video \
    --ipc=host \
    --cap-add=SYS_PTRACE \
    --security-opt seccomp=unconfined \
    --device /dev/kfd \
    --device /dev/dri \
    -v <path/to/model>:/app/model \
314
315
    vllm-rocm \
    --model Qwen/Qwen3-0.6B
316
    ```
317
318
319

Where the `<path/to/model>` is the location where the model is stored, for example, the weights for llama2 or llama3 models.

320
321
# --8<-- [end:build-image-from-source]
# --8<-- [start:supported-features]
322

323
See [Feature x Hardware](../../features/README.md#feature-x-hardware) compatibility matrix for feature support information.
324
325

# --8<-- [end:supported-features]