installation.rst 1.68 KB
Newer Older
Zhuohan Li's avatar
Zhuohan Li committed
1
2
.. _installation:

Woosuk Kwon's avatar
Woosuk Kwon committed
3
4
5
Installation
============

6
vLLM is a Python library that also contains pre-compiled C++ and CUDA (12.1) binaries.
7
8
9

Requirements
------------
10
11

* OS: Linux
12
* Python: 3.8 -- 3.11
13
* GPU: compute capability 7.0 or higher (e.g., V100, T4, RTX20xx, A100, L4, H100, etc.)
14
15
16
17

Install with pip
----------------

Woosuk Kwon's avatar
Woosuk Kwon committed
18
You can install vLLM using pip:
19
20
21
22
23
24
25

.. code-block:: console

    $ # (Optional) Create a new conda environment.
    $ conda create -n myenv python=3.8 -y
    $ conda activate myenv

26
    $ # Install vLLM with CUDA 12.1.
27
    $ pip install vllm
28

29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
.. note::

    As of now, vLLM's binaries are compiled on CUDA 12.1 by default.
    However, you can install vLLM with CUDA 11.8 by running:

    .. code-block:: console

        $ # Install vLLM with CUDA 11.8.
        $ # Replace `cp310` with your Python version (e.g., `cp38`, `cp39`, `cp311`).
        $ pip install https://github.com/vllm-project/vllm/releases/download/v0.2.2/vllm-0.2.2+cu118-cp310-cp310-manylinux1_x86_64.whl

        $ # Re-install PyTorch with CUDA 11.8.
        $ pip uninstall torch -y
        $ pip install torch --upgrade --index-url https://download.pytorch.org/whl/cu118

44
45
46

.. _build_from_source:

Woosuk Kwon's avatar
Woosuk Kwon committed
47
48
49
Build from source
-----------------

50
You can also build and install vLLM from source:
51

Woosuk Kwon's avatar
Woosuk Kwon committed
52
53
.. code-block:: console

54
    $ git clone https://github.com/vllm-project/vllm.git
Woosuk Kwon's avatar
Woosuk Kwon committed
55
    $ cd vllm
56
    $ pip install -e .  # This may take 5-10 minutes.
57
58
59
60
61
62

.. tip::
    If you have trouble building vLLM, we recommend using the NVIDIA PyTorch Docker image.

    .. code-block:: console

63
        $ # Use `--ipc=host` to make sure the shared memory is large enough.
64
        $ docker run --gpus all -it --rm --ipc=host nvcr.io/nvidia/pytorch:23.10-py3