README.md 11.1 KB
Newer Older
Byron Hsu's avatar
Byron Hsu committed
1
# SGLang Router
2

Simo Lin's avatar
Simo Lin committed
3
SGLang router is a standalone Rust module that enables data parallelism across SGLang instances, providing high-performance request routing and advanced load balancing. The router supports multiple load balancing algorithms including cache-aware, power of two, random, and round robin, and acts as a specialized load balancer for prefill-decode disaggregated serving architectures.
4

Simo Lin's avatar
Simo Lin committed
5
## Documentation
Byron Hsu's avatar
Byron Hsu committed
6

Simo Lin's avatar
Simo Lin committed
7
- **User Guide**: [docs.sglang.ai/router/router.html](https://docs.sglang.ai/router/router.html)
8

Simo Lin's avatar
Simo Lin committed
9
## Quick Start
10

11
### Prerequisites
12

Simo Lin's avatar
Simo Lin committed
13
**Rust and Cargo:**
14
15
16
17
18
19
20
21
22
23
24
25
```bash
# Install rustup (Rust installer and version manager)
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

# Follow the installation prompts, then reload your shell
source $HOME/.cargo/env

# Verify installation
rustc --version
cargo --version
```

Simo Lin's avatar
Simo Lin committed
26
**Python with pip installed**
27

Simo Lin's avatar
Simo Lin committed
28
### Installation
29

Simo Lin's avatar
Simo Lin committed
30
#### Option A: Build and Install Wheel (Recommended)
31
```bash
Simo Lin's avatar
Simo Lin committed
32
33
# Install build dependencies
pip install setuptools-rust wheel build
34

Simo Lin's avatar
Simo Lin committed
35
36
# Build the wheel package
python -m build
37

Simo Lin's avatar
Simo Lin committed
38
39
# Install the generated wheel
pip install dist/*.whl
40

Simo Lin's avatar
Simo Lin committed
41
42
# One-liner for development (rebuild + install)
python -m build && pip install --force-reinstall dist/*.whl
43
44
```

Simo Lin's avatar
Simo Lin committed
45
#### Option B: Development Mode
46
```bash
Simo Lin's avatar
Simo Lin committed
47
pip install -e .
48
49
```

Simo Lin's avatar
Simo Lin committed
50
⚠️ **Warning**: Editable installs may suffer performance degradation. Use wheel builds for performance testing.
51

Simo Lin's avatar
Simo Lin committed
52
### Basic Usage
53

54
```bash
Simo Lin's avatar
Simo Lin committed
55
56
# Build Rust components
cargo build
57
```
Simo Lin's avatar
Simo Lin committed
58

59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
#### Using the Rust Binary Directly (Alternative to Python)
```bash
# Build the Rust binary
cargo build --release

# Launch router with worker URLs in regular mode
./target/release/sglang-router \
    --worker-urls http://worker1:8000 http://worker2:8000

# Or use cargo run
cargo run --release -- \
    --worker-urls http://worker1:8000 http://worker2:8000
```

#### Launch Router with Python (Original Method)
74
```bash
Simo Lin's avatar
Simo Lin committed
75
76
77
# Launch router with worker URLs
python -m sglang_router.launch_router \
    --worker-urls http://worker1:8000 http://worker2:8000
78
79
```

80
81
82
83
84
#### Launch Router with Worker URLs in prefill-decode mode
```bash
# Note that the prefill and decode URLs must be provided in the following format:
# http://<ip>:<port> for  decode nodes
# http://<ip>:<port> bootstrap-port for  prefill nodes, where bootstrap-port is optional
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100

# Using Rust binary directly
./target/release/sglang-router \
    --pd-disaggregation \
    --policy cache_aware \
    --prefill http://127.0.0.1:30001 9001 \
    --prefill http://127.0.0.2:30002 9002 \
    --prefill http://127.0.0.3:30003 9003 \
    --prefill http://127.0.0.4:30004 9004 \
    --decode http://127.0.0.5:30005 \
    --decode http://127.0.0.6:30006 \
    --decode http://127.0.0.7:30007 \
    --host 0.0.0.0 \
    --port 8080

# Or using Python launcher
101
102
103
104
105
106
107
108
109
110
111
112
113
114
python -m sglang_router.launch_router \
    --pd-disaggregation \
    --policy cache_aware \
    --prefill http://127.0.0.1:30001 9001 \
    --prefill http://127.0.0.2:30002 9002 \
    --prefill http://127.0.0.3:30003 9003 \
    --prefill http://127.0.0.4:30004 9004 \
    --decode http://127.0.0.5:30005 \
    --decode http://127.0.0.6:30006 \
    --decode http://127.0.0.7:30007 \
    --host 0.0.0.0 \
    --port 8080
````

Simo Lin's avatar
Simo Lin committed
115
## Configuration
116

117
118
### Logging

Simo Lin's avatar
Simo Lin committed
119
Enable structured logging with optional file output:
120
121

```python
Simo Lin's avatar
Simo Lin committed
122
123
124
125
126
127
from sglang_router import Router

# Console logging (default)
router = Router(worker_urls=["http://worker1:8000", "http://worker2:8000"])

# File logging enabled
128
129
router = Router(
    worker_urls=["http://worker1:8000", "http://worker2:8000"],
Simo Lin's avatar
Simo Lin committed
130
    log_dir="./logs"  # Daily log files created here
131
132
133
)
```

Simo Lin's avatar
Simo Lin committed
134
Set log level with `--log-level` flag ([documentation](https://docs.sglang.ai/backend/server_arguments.html#logging)).
135

136
137
### Metrics

Simo Lin's avatar
Simo Lin committed
138
Prometheus metrics endpoint available at `127.0.0.1:29000` by default.
139

Simo Lin's avatar
Simo Lin committed
140
141
```bash
# Custom metrics configuration
142
python -m sglang_router.launch_router \
Simo Lin's avatar
Simo Lin committed
143
144
145
    --worker-urls http://localhost:8080 http://localhost:8081 \
    --prometheus-host 0.0.0.0 \
    --prometheus-port 9000
146
147
```

148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
### Retries and Circuit Breakers

- Retries (regular router) are enabled by default with exponential backoff and jitter. You can tune them via CLI:

```bash
python -m sglang_router.launch_router \
  --worker-urls http://localhost:8080 http://localhost:8081 \
  --retry-max-retries 3 \
  --retry-initial-backoff-ms 100 \
  --retry-max-backoff-ms 10000 \
  --retry-backoff-multiplier 2.0 \
  --retry-jitter-factor 0.1
```

- Circuit Breaker defaults protect workers and auto-recover. Tune thresholds/timeouts:

```bash
python -m sglang_router.launch_router \
  --worker-urls http://localhost:8080 http://localhost:8081 \
  --cb-failure-threshold 5 \
  --cb-success-threshold 2 \
  --cb-timeout-duration-secs 30 \
  --cb-window-duration-secs 60
```

Behavior summary:
- Closed → Open after N consecutive failures (failure-threshold)
- Open → HalfOpen after timeout (timeout-duration-secs)
- HalfOpen → Closed after M consecutive successes (success-threshold)
- Any failure in HalfOpen reopens immediately

Retry predicate (regular router): retry on 408/429/500/502/503/504, otherwise return immediately. Backoff/jitter observed between attempts.

181
182
183
184
185
186
187
188
189
190
191
192
193
### Request ID Tracking

Track requests across distributed systems with configurable headers:

```bash
# Use custom request ID headers
python -m sglang_router.launch_router \
    --worker-urls http://localhost:8080 \
    --request-id-headers x-trace-id x-request-id
```

Default headers: `x-request-id`, `x-correlation-id`, `x-trace-id`, `request-id`

Simo Lin's avatar
Simo Lin committed
194
## Advanced Features
195

Simo Lin's avatar
Simo Lin committed
196
### Kubernetes Service Discovery
197

Simo Lin's avatar
Simo Lin committed
198
Automatic worker discovery and management in Kubernetes environments.
199

Simo Lin's avatar
Simo Lin committed
200
#### Basic Service Discovery
201
202
203
204
205
206
207
208

```bash
python -m sglang_router.launch_router \
    --service-discovery \
    --selector app=sglang-worker role=inference \
    --service-discovery-namespace default
```

Simo Lin's avatar
Simo Lin committed
209
#### PD (Prefill-Decode) Mode
210

Simo Lin's avatar
Simo Lin committed
211
For disaggregated prefill/decode routing:
212
213
214
215
216
217
218
219
220

```bash
python -m sglang_router.launch_router \
    --pd-disaggregation \
    --policy cache_aware \
    --service-discovery \
    --prefill-selector app=sglang component=prefill \
    --decode-selector app=sglang component=decode \
    --service-discovery-namespace sglang-system
221
222
223
224
225
226
227
228
229
230

# With separate routing policies:
python -m sglang_router.launch_router \
    --pd-disaggregation \
    --prefill-policy cache_aware \
    --decode-policy power_of_two \
    --service-discovery \
    --prefill-selector app=sglang component=prefill \
    --decode-selector app=sglang component=decode \
    --service-discovery-namespace sglang-system
231
232
```

Simo Lin's avatar
Simo Lin committed
233
#### Kubernetes Pod Configuration
234
235
236
237
238
239
240
241
242
243
244

**Prefill Server Pod:**
```yaml
apiVersion: v1
kind: Pod
metadata:
  name: sglang-prefill-1
  labels:
    app: sglang
    component: prefill
  annotations:
Simo Lin's avatar
Simo Lin committed
245
    sglang.ai/bootstrap-port: "9001"  # Optional: Bootstrap port
246
247
248
249
250
251
spec:
  containers:
  - name: sglang
    image: lmsys/sglang:latest
    ports:
    - containerPort: 8000  # Main API port
Simo Lin's avatar
Simo Lin committed
252
    - containerPort: 9001  # Optional: Bootstrap port
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
```

**Decode Server Pod:**
```yaml
apiVersion: v1
kind: Pod
metadata:
  name: sglang-decode-1
  labels:
    app: sglang
    component: decode
spec:
  containers:
  - name: sglang
    image: lmsys/sglang:latest
    ports:
Simo Lin's avatar
Simo Lin committed
269
    - containerPort: 8000
270
271
```

Simo Lin's avatar
Simo Lin committed
272
#### RBAC Configuration
273

274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
**Namespace-scoped (recommended):**
```yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: sglang-router
  namespace: sglang-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: sglang-system
  name: sglang-router
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: sglang-router
  namespace: sglang-system
subjects:
- kind: ServiceAccount
  name: sglang-router
  namespace: sglang-system
roleRef:
  kind: Role
  name: sglang-router
  apiGroup: rbac.authorization.k8s.io
```

Simo Lin's avatar
Simo Lin committed
307
#### Complete PD Example
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322

```bash
python -m sglang_router.launch_router \
    --pd-disaggregation \
    --policy cache_aware \
    --service-discovery \
    --prefill-selector app=sglang component=prefill environment=production \
    --decode-selector app=sglang component=decode environment=production \
    --service-discovery-namespace production \
    --host 0.0.0.0 \
    --port 8080 \
    --prometheus-host 0.0.0.0 \
    --prometheus-port 9090
```

Simo Lin's avatar
Simo Lin committed
323
### Command Line Arguments Reference
324

Simo Lin's avatar
Simo Lin committed
325
326
327
328
329
#### Service Discovery
- `--service-discovery`: Enable Kubernetes service discovery
- `--service-discovery-port`: Port for worker URLs (default: 8000)
- `--service-discovery-namespace`: Kubernetes namespace to watch
- `--selector`: Label selectors for regular mode (format: `key1=value1 key2=value2`)
330

Simo Lin's avatar
Simo Lin committed
331
332
333
334
335
336
#### PD Mode
- `--pd-disaggregation`: Enable Prefill-Decode disaggregated mode
- `--prefill`: Initial prefill server (format: `URL BOOTSTRAP_PORT`)
- `--decode`: Initial decode server URL
- `--prefill-selector`: Label selector for prefill pods
- `--decode-selector`: Label selector for decode pods
337
338
339
- `--policy`: Routing policy (`cache_aware`, `random`, `power_of_two`, `round_robin`)
- `--prefill-policy`: Separate routing policy for prefill nodes (optional, overrides `--policy` for prefill)
- `--decode-policy`: Separate routing policy for decode nodes (optional, overrides `--policy` for decode)
340

Simo Lin's avatar
Simo Lin committed
341
## Development
342

Simo Lin's avatar
Simo Lin committed
343
### Build Process
344

Simo Lin's avatar
Simo Lin committed
345
346
347
348
349
350
```bash
# Build Rust project
cargo build

# Build Python binding (see Installation section above)
```
351

Simo Lin's avatar
Simo Lin committed
352
**Note**: When modifying Rust code, you must rebuild the wheel for changes to take effect.
353

Simo Lin's avatar
Simo Lin committed
354
### Troubleshooting
355

Simo Lin's avatar
Simo Lin committed
356
357
**VSCode Rust Analyzer Issues:**
Set `rust-analyzer.linkedProjects` to the absolute path of `Cargo.toml`:
358

Simo Lin's avatar
Simo Lin committed
359
360
361
362
363
```json
{
  "rust-analyzer.linkedProjects": ["/workspaces/sglang/sgl-router/Cargo.toml"]
}
```
364

Simo Lin's avatar
Simo Lin committed
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
### CI/CD Pipeline

The continuous integration pipeline includes comprehensive testing, benchmarking, and publishing:

#### Build & Test
1. **Build Wheels**: Uses `cibuildwheel` for manylinux x86_64 packages
2. **Build Source Distribution**: Creates source distribution for pip fallback
3. **Rust HTTP Server Benchmarking**: Performance testing of router overhead
4. **Basic Inference Testing**: End-to-end validation through the router
5. **PD Disaggregation Testing**: Benchmark and sanity checks for prefill-decode load balancing

#### Publishing
- **PyPI Publishing**: Wheels and source distributions are published only when the version changes in `pyproject.toml`
- **Container Images**: Docker images published using `/docker/Dockerfile.router`

## Features

- **High Performance**: Rust-based routing with connection pooling and optimized request handling
- **Advanced Load Balancing**: Multiple algorithms including:
  - **Cache-Aware**: Intelligent routing based on cache locality for optimal performance
  - **Power of Two**: Chooses the less loaded of two randomly selected workers
  - **Random**: Distributes requests randomly across available workers
  - **Round Robin**: Sequential distribution across workers in rotation
- **Prefill-Decode Disaggregation**: Specialized load balancing for separated prefill and decode servers
- **Service Discovery**: Automatic Kubernetes worker discovery and health management
- **Monitoring**: Comprehensive Prometheus metrics and structured logging
- **Scalability**: Handles thousands of concurrent connections with efficient resource utilization