install.md 5.19 KB
Newer Older
1
# Install SGLang
2

3
You can install SGLang using any of the methods below.
4

5
## Method 1: With pip
6
7
8
9
```
pip install --upgrade pip
pip install "sglang[all]"

10
# Install FlashInfer accelerated kernels (CUDA only for now)
11
12
13
pip install flashinfer -i https://flashinfer.ai/whl/cu121/torch2.4/
```

14
Note: Please check the [FlashInfer installation doc](https://docs.flashinfer.ai/installation.html) to install the proper version according to your PyTorch and CUDA versions.
Lianmin Zheng's avatar
Lianmin Zheng committed
15

16
## Method 2: From source
17
```
18
# Use the last release branch
Lianmin Zheng's avatar
Lianmin Zheng committed
19
git clone -b v0.3.5 https://github.com/sgl-project/sglang.git
20
cd sglang
21

22
23
pip install --upgrade pip
pip install -e "python[all]"
24

25
# Install FlashInfer accelerated kernels (CUDA only for now)
26
27
pip install flashinfer -i https://flashinfer.ai/whl/cu121/torch2.4/
```
28

29
Note: Please check the [FlashInfer installation doc](https://docs.flashinfer.ai/installation.html) to install the proper version according to your PyTorch and CUDA versions.
Lianmin Zheng's avatar
Lianmin Zheng committed
30

31
## Method 3: Using docker
32
33
The docker images are available on Docker Hub as [lmsysorg/sglang](https://hub.docker.com/r/lmsysorg/sglang/tags), built from [Dockerfile](https://github.com/sgl-project/sglang/tree/main/docker).
Replace `<secret>` below with your huggingface hub [token](https://huggingface.co/docs/hub/en/security-tokens).
34
35

```bash
36
37
docker run --gpus all \
    -p 30000:30000 \
38
    -v ~/.cache/huggingface:/root/.cache/huggingface \
39
40
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
41
    lmsysorg/sglang:latest \
42
    python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --host 0.0.0.0 --port 30000
43
44
```

45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
Note: To AMD ROCm system with Instinct/MI GPUs, it is recommended to use `docker/Dockerfile.rocm` to build images, example and usage as below:

```bash
docker build --build-arg SGL_BRANCH=v0.3.5 -t v0.3.5-rocm620 -f Dockerfile.rocm .

alias drun='docker run -it --rm --network=host --device=/dev/kfd --device=/dev/dri --ipc=host \
    --shm-size 16G --group-add video --cap-add=SYS_PTRACE --security-opt seccomp=unconfined \
    -v $HOME/dockerx:/dockerx -v /data:/data'

drun -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    v0.3.5-rocm620 \
    python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --host 0.0.0.0 --port 30000

# Till flashinfer backend available, --attention-backend triton --sampling-backend pytorch are set by default
drun v0.3.5-rocm620 python3 -m sglang.bench_latency --batch-size 32 --input 1024 --output 128 --model amd/Meta-Llama-3.1-8B-Instruct-FP8-KV --tp 8 --quantization fp8
```

64
## Method 4: Using docker compose
65
66
67

<details>
<summary>More</summary>
68

69
> This method is recommended if you plan to serve it as a service.
Lianmin Zheng's avatar
Lianmin Zheng committed
70
> A better approach is to use the [k8s-sglang-service.yaml](https://github.com/sgl-project/sglang/blob/main/docker/k8s-sglang-service.yaml).
71

Lianmin Zheng's avatar
Lianmin Zheng committed
72
1. Copy the [compose.yml](https://github.com/sgl-project/sglang/blob/main/docker/compose.yaml) to your local machine
73
2. Execute the command `docker compose up -d` in your terminal.
74
</details>
75

76
## Method 5: Run on Kubernetes or Clouds with SkyPilot
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100

<details>
<summary>More</summary>

To deploy on Kubernetes or 12+ clouds, you can use [SkyPilot](https://github.com/skypilot-org/skypilot).

1. Install SkyPilot and set up Kubernetes cluster or cloud access: see [SkyPilot's documentation](https://skypilot.readthedocs.io/en/latest/getting-started/installation.html).
2. Deploy on your own infra with a single command and get the HTTP API endpoint:
<details>
<summary>SkyPilot YAML: <code>sglang.yaml</code></summary>

```yaml
# sglang.yaml
envs:
  HF_TOKEN: null

resources:
  image_id: docker:lmsysorg/sglang:latest
  accelerators: A100
  ports: 30000

run: |
  conda deactivate
  python3 -m sglang.launch_server \
101
    --model-path meta-llama/Llama-3.1-8B-Instruct \
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
    --host 0.0.0.0 \
    --port 30000
```
</details>

```bash
# Deploy on any cloud or Kubernetes cluster. Use --cloud <cloud> to select a specific cloud provider.
HF_TOKEN=<secret> sky launch -c sglang --env HF_TOKEN sglang.yaml

# Get the HTTP API endpoint
sky status --endpoint 30000 sglang
```
3. To further scale up your deployment with autoscaling and failure recovery, check out the [SkyServe + SGLang guide](https://github.com/skypilot-org/skypilot/tree/master/llm/sglang#serving-llama-2-with-sglang-for-more-traffic-using-skyserve).
</details>

117
## Common Notes
Lianmin Zheng's avatar
Lianmin Zheng committed
118
- [FlashInfer](https://github.com/flashinfer-ai/flashinfer) is the default attention kernel backend. It only supports sm75 and above. If you encounter any FlashInfer-related issues on sm75+ devices (e.g., T4, A10, A100, L4, L40S, H100), please switch to other kernels by adding `--attention-backend triton --sampling-backend pytorch` and open an issue on GitHub.
Lianmin Zheng's avatar
Lianmin Zheng committed
119
- If you only need to use OpenAI models with the frontend language, you can avoid installing other dependencies by using `pip install "sglang[openai]"`.
Lianmin Zheng's avatar
Lianmin Zheng committed
120
- The language frontend operates independently of the backend runtime. You can install the frontend locally without needing a GPU, while the backend can be set up on a GPU-enabled machine. To install the frontend, run `pip install sglang`, and for the backend, use `pip install sglang[srt]`. This allows you to build SGLang programs locally and execute them by connecting to the remote backend.