README.md 9.17 KB
Newer Older
Byron Hsu's avatar
Byron Hsu committed
1
# SGLang Router
2

Simo Lin's avatar
Simo Lin committed
3
SGLang router is a standalone Rust module that enables data parallelism across SGLang instances, providing high-performance request routing and advanced load balancing. The router supports multiple load balancing algorithms including cache-aware, power of two, random, and round robin, and acts as a specialized load balancer for prefill-decode disaggregated serving architectures.
4

Simo Lin's avatar
Simo Lin committed
5
## Documentation
Byron Hsu's avatar
Byron Hsu committed
6

Simo Lin's avatar
Simo Lin committed
7
- **User Guide**: [docs.sglang.ai/router/router.html](https://docs.sglang.ai/router/router.html)
8

Simo Lin's avatar
Simo Lin committed
9
## Quick Start
10

11
### Prerequisites
12

Simo Lin's avatar
Simo Lin committed
13
**Rust and Cargo:**
14
15
16
17
18
19
20
21
22
23
24
25
```bash
# Install rustup (Rust installer and version manager)
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

# Follow the installation prompts, then reload your shell
source $HOME/.cargo/env

# Verify installation
rustc --version
cargo --version
```

Simo Lin's avatar
Simo Lin committed
26
**Python with pip installed**
27

Simo Lin's avatar
Simo Lin committed
28
### Installation
29

Simo Lin's avatar
Simo Lin committed
30
#### Option A: Build and Install Wheel (Recommended)
31
```bash
Simo Lin's avatar
Simo Lin committed
32
33
# Install build dependencies
pip install setuptools-rust wheel build
34

Simo Lin's avatar
Simo Lin committed
35
36
# Build the wheel package
python -m build
37

Simo Lin's avatar
Simo Lin committed
38
39
# Install the generated wheel
pip install dist/*.whl
40

Simo Lin's avatar
Simo Lin committed
41
42
# One-liner for development (rebuild + install)
python -m build && pip install --force-reinstall dist/*.whl
43
44
```

Simo Lin's avatar
Simo Lin committed
45
#### Option B: Development Mode
46
```bash
Simo Lin's avatar
Simo Lin committed
47
pip install -e .
48
49
```

Simo Lin's avatar
Simo Lin committed
50
⚠️ **Warning**: Editable installs may suffer performance degradation. Use wheel builds for performance testing.
51

Simo Lin's avatar
Simo Lin committed
52
### Basic Usage
53

54
```bash
Simo Lin's avatar
Simo Lin committed
55
56
# Build Rust components
cargo build
57
```
Simo Lin's avatar
Simo Lin committed
58

59
60
#### Launch Router with Worker URLs in regular mode
```bash
Simo Lin's avatar
Simo Lin committed
61
62
63
# Launch router with worker URLs
python -m sglang_router.launch_router \
    --worker-urls http://worker1:8000 http://worker2:8000
64
65
```

66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
#### Launch Router with Worker URLs in prefill-decode mode
```bash
# Note that the prefill and decode URLs must be provided in the following format:
# http://<ip>:<port> for  decode nodes
# http://<ip>:<port> bootstrap-port for  prefill nodes, where bootstrap-port is optional
# Launch router with worker URLs
python -m sglang_router.launch_router \
    --pd-disaggregation \
    --policy cache_aware \
    --prefill http://127.0.0.1:30001 9001 \
    --prefill http://127.0.0.2:30002 9002 \
    --prefill http://127.0.0.3:30003 9003 \
    --prefill http://127.0.0.4:30004 9004 \
    --decode http://127.0.0.5:30005 \
    --decode http://127.0.0.6:30006 \
    --decode http://127.0.0.7:30007 \
    --host 0.0.0.0 \
    --port 8080
````

Simo Lin's avatar
Simo Lin committed
86
## Configuration
87

88
89
### Logging

Simo Lin's avatar
Simo Lin committed
90
Enable structured logging with optional file output:
91
92

```python
Simo Lin's avatar
Simo Lin committed
93
94
95
96
97
98
from sglang_router import Router

# Console logging (default)
router = Router(worker_urls=["http://worker1:8000", "http://worker2:8000"])

# File logging enabled
99
100
router = Router(
    worker_urls=["http://worker1:8000", "http://worker2:8000"],
Simo Lin's avatar
Simo Lin committed
101
    log_dir="./logs"  # Daily log files created here
102
103
104
)
```

Simo Lin's avatar
Simo Lin committed
105
Set log level with `--log-level` flag ([documentation](https://docs.sglang.ai/backend/server_arguments.html#logging)).
106

107
108
### Metrics

Simo Lin's avatar
Simo Lin committed
109
Prometheus metrics endpoint available at `127.0.0.1:29000` by default.
110

Simo Lin's avatar
Simo Lin committed
111
112
```bash
# Custom metrics configuration
113
python -m sglang_router.launch_router \
Simo Lin's avatar
Simo Lin committed
114
115
116
    --worker-urls http://localhost:8080 http://localhost:8081 \
    --prometheus-host 0.0.0.0 \
    --prometheus-port 9000
117
118
```

119
120
121
122
123
124
125
126
127
128
129
130
131
### Request ID Tracking

Track requests across distributed systems with configurable headers:

```bash
# Use custom request ID headers
python -m sglang_router.launch_router \
    --worker-urls http://localhost:8080 \
    --request-id-headers x-trace-id x-request-id
```

Default headers: `x-request-id`, `x-correlation-id`, `x-trace-id`, `request-id`

Simo Lin's avatar
Simo Lin committed
132
## Advanced Features
133

Simo Lin's avatar
Simo Lin committed
134
### Kubernetes Service Discovery
135

Simo Lin's avatar
Simo Lin committed
136
Automatic worker discovery and management in Kubernetes environments.
137

Simo Lin's avatar
Simo Lin committed
138
#### Basic Service Discovery
139
140
141
142
143
144
145
146

```bash
python -m sglang_router.launch_router \
    --service-discovery \
    --selector app=sglang-worker role=inference \
    --service-discovery-namespace default
```

Simo Lin's avatar
Simo Lin committed
147
#### PD (Prefill-Decode) Mode
148

Simo Lin's avatar
Simo Lin committed
149
For disaggregated prefill/decode routing:
150
151
152
153
154
155
156
157
158

```bash
python -m sglang_router.launch_router \
    --pd-disaggregation \
    --policy cache_aware \
    --service-discovery \
    --prefill-selector app=sglang component=prefill \
    --decode-selector app=sglang component=decode \
    --service-discovery-namespace sglang-system
159
160
161
162
163
164
165
166
167
168

# With separate routing policies:
python -m sglang_router.launch_router \
    --pd-disaggregation \
    --prefill-policy cache_aware \
    --decode-policy power_of_two \
    --service-discovery \
    --prefill-selector app=sglang component=prefill \
    --decode-selector app=sglang component=decode \
    --service-discovery-namespace sglang-system
169
170
```

Simo Lin's avatar
Simo Lin committed
171
#### Kubernetes Pod Configuration
172
173
174
175
176
177
178
179
180
181
182

**Prefill Server Pod:**
```yaml
apiVersion: v1
kind: Pod
metadata:
  name: sglang-prefill-1
  labels:
    app: sglang
    component: prefill
  annotations:
Simo Lin's avatar
Simo Lin committed
183
    sglang.ai/bootstrap-port: "9001"  # Optional: Bootstrap port
184
185
186
187
188
189
spec:
  containers:
  - name: sglang
    image: lmsys/sglang:latest
    ports:
    - containerPort: 8000  # Main API port
Simo Lin's avatar
Simo Lin committed
190
    - containerPort: 9001  # Optional: Bootstrap port
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
```

**Decode Server Pod:**
```yaml
apiVersion: v1
kind: Pod
metadata:
  name: sglang-decode-1
  labels:
    app: sglang
    component: decode
spec:
  containers:
  - name: sglang
    image: lmsys/sglang:latest
    ports:
Simo Lin's avatar
Simo Lin committed
207
    - containerPort: 8000
208
209
```

Simo Lin's avatar
Simo Lin committed
210
#### RBAC Configuration
211

212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
**Namespace-scoped (recommended):**
```yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: sglang-router
  namespace: sglang-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: sglang-system
  name: sglang-router
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: sglang-router
  namespace: sglang-system
subjects:
- kind: ServiceAccount
  name: sglang-router
  namespace: sglang-system
roleRef:
  kind: Role
  name: sglang-router
  apiGroup: rbac.authorization.k8s.io
```

Simo Lin's avatar
Simo Lin committed
245
#### Complete PD Example
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260

```bash
python -m sglang_router.launch_router \
    --pd-disaggregation \
    --policy cache_aware \
    --service-discovery \
    --prefill-selector app=sglang component=prefill environment=production \
    --decode-selector app=sglang component=decode environment=production \
    --service-discovery-namespace production \
    --host 0.0.0.0 \
    --port 8080 \
    --prometheus-host 0.0.0.0 \
    --prometheus-port 9090
```

Simo Lin's avatar
Simo Lin committed
261
### Command Line Arguments Reference
262

Simo Lin's avatar
Simo Lin committed
263
264
265
266
267
#### Service Discovery
- `--service-discovery`: Enable Kubernetes service discovery
- `--service-discovery-port`: Port for worker URLs (default: 8000)
- `--service-discovery-namespace`: Kubernetes namespace to watch
- `--selector`: Label selectors for regular mode (format: `key1=value1 key2=value2`)
268

Simo Lin's avatar
Simo Lin committed
269
270
271
272
273
274
#### PD Mode
- `--pd-disaggregation`: Enable Prefill-Decode disaggregated mode
- `--prefill`: Initial prefill server (format: `URL BOOTSTRAP_PORT`)
- `--decode`: Initial decode server URL
- `--prefill-selector`: Label selector for prefill pods
- `--decode-selector`: Label selector for decode pods
275
276
277
- `--policy`: Routing policy (`cache_aware`, `random`, `power_of_two`, `round_robin`)
- `--prefill-policy`: Separate routing policy for prefill nodes (optional, overrides `--policy` for prefill)
- `--decode-policy`: Separate routing policy for decode nodes (optional, overrides `--policy` for decode)
278

Simo Lin's avatar
Simo Lin committed
279
## Development
280

Simo Lin's avatar
Simo Lin committed
281
### Build Process
282

Simo Lin's avatar
Simo Lin committed
283
284
285
286
287
288
```bash
# Build Rust project
cargo build

# Build Python binding (see Installation section above)
```
289

Simo Lin's avatar
Simo Lin committed
290
**Note**: When modifying Rust code, you must rebuild the wheel for changes to take effect.
291

Simo Lin's avatar
Simo Lin committed
292
### Troubleshooting
293

Simo Lin's avatar
Simo Lin committed
294
295
**VSCode Rust Analyzer Issues:**
Set `rust-analyzer.linkedProjects` to the absolute path of `Cargo.toml`:
296

Simo Lin's avatar
Simo Lin committed
297
298
299
300
301
```json
{
  "rust-analyzer.linkedProjects": ["/workspaces/sglang/sgl-router/Cargo.toml"]
}
```
302

Simo Lin's avatar
Simo Lin committed
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
### CI/CD Pipeline

The continuous integration pipeline includes comprehensive testing, benchmarking, and publishing:

#### Build & Test
1. **Build Wheels**: Uses `cibuildwheel` for manylinux x86_64 packages
2. **Build Source Distribution**: Creates source distribution for pip fallback
3. **Rust HTTP Server Benchmarking**: Performance testing of router overhead
4. **Basic Inference Testing**: End-to-end validation through the router
5. **PD Disaggregation Testing**: Benchmark and sanity checks for prefill-decode load balancing

#### Publishing
- **PyPI Publishing**: Wheels and source distributions are published only when the version changes in `pyproject.toml`
- **Container Images**: Docker images published using `/docker/Dockerfile.router`

## Features

- **High Performance**: Rust-based routing with connection pooling and optimized request handling
- **Advanced Load Balancing**: Multiple algorithms including:
  - **Cache-Aware**: Intelligent routing based on cache locality for optimal performance
  - **Power of Two**: Chooses the less loaded of two randomly selected workers
  - **Random**: Distributes requests randomly across available workers
  - **Round Robin**: Sequential distribution across workers in rotation
- **Prefill-Decode Disaggregation**: Specialized load balancing for separated prefill and decode servers
- **Service Discovery**: Automatic Kubernetes worker discovery and health management
- **Monitoring**: Comprehensive Prometheus metrics and structured logging
- **Scalability**: Handles thousands of concurrent connections with efficient resource utilization