README.md 21.3 KB
Newer Older
Lianmin Zheng's avatar
Lianmin Zheng committed
1
<div align="center">
2
<img src="https://raw.githubusercontent.com/sgl-project/sglang/main/assets/logo.png" alt="logo" width="400"></img>
Lianmin Zheng's avatar
Lianmin Zheng committed
3

Yineng Zhang's avatar
Yineng Zhang committed
4
5
6
7
8
9
[![PyPI](https://img.shields.io/pypi/v/sglang)](https://pypi.org/project/sglang)
![PyPI - Downloads](https://img.shields.io/pypi/dm/sglang)
[![license](https://img.shields.io/github/license/sgl-project/sglang.svg)](https://github.com/sgl-project/sglang/tree/main/LICENSE)
[![issue resolution](https://img.shields.io/github/issues-closed-raw/sgl-project/sglang)](https://github.com/sgl-project/sglang/issues)
[![open issues](https://img.shields.io/github/issues-raw/sgl-project/sglang)](https://github.com/sgl-project/sglang/issues)

Yineng Zhang's avatar
Yineng Zhang committed
10
11
</div>

Lianmin Zheng's avatar
Lianmin Zheng committed
12
13
--------------------------------------------------------------------------------

Ying Sheng's avatar
Ying Sheng committed
14
| [**Blog**](https://lmsys.org/blog/2024-07-25-sglang-llama3/) | [**Paper**](https://arxiv.org/abs/2312.07104) | [**Slack**](https://join.slack.com/t/sgl-fru7574/shared_invite/zt-2ngly9muu-t37XiH87qvD~6rVBTkTEHw) |
Lianmin Zheng's avatar
Lianmin Zheng committed
15

Ying Sheng's avatar
Ying Sheng committed
16
17
SGLang is a fast serving framework for large language models and vision language models.
It makes your interaction with models faster and more controllable by co-designing the backend runtime and frontend language.
Lianmin Zheng's avatar
Lianmin Zheng committed
18

19
The core features include:
Lianmin Zheng's avatar
Lianmin Zheng committed
20
- **Fast Backend Runtime**: Efficient serving with RadixAttention for prefix caching, jump-forward constrained decoding, continuous batching, token attention (paged attention), tensor parallelism, FlashInfer kernels, and quantization (AWQ/FP8/GPTQ/Marlin).
Lianmin Zheng's avatar
Lianmin Zheng committed
21
- **Flexible Frontend Language**: Enables easy programming of LLM applications with chained generation calls, advanced prompting, control flow, multiple modalities, parallelism, and external interactions.
Lianmin Zheng's avatar
Lianmin Zheng committed
22

Ying Sheng's avatar
Ying Sheng committed
23
## News
Ying Sheng's avatar
Ying Sheng committed
24
- [2024/07] 🔥 Faster Llama3 Serving with SGLang Runtime (vs. TensorRT-LLM, vLLM) ([blog](https://lmsys.org/blog/2024-07-25-sglang-llama3/)).
25
- [2024/08] 🔥 LLaVA-OneVision with single-image, multi-image and video are supported ([blog](https://llava-vl.github.io/blog/2024-08-05-llava-onevision/)).
Ying Sheng's avatar
Ying Sheng committed
26
- [2024/02] SGLang enables **3x faster JSON decoding** with compressed finite state machine ([blog](https://lmsys.org/blog/2024-02-05-compressed-fsm/)).
Ying Sheng's avatar
Ying Sheng committed
27

Ying Sheng's avatar
Ying Sheng committed
28
29
30
<details>
<summary>More</summary>

31
- [2024/04] SGLang is used by the official **LLaVA-NeXT (video)** release ([blog](https://llava-vl.github.io/blog/2024-04-30-llava-next-video/)).
Ying Sheng's avatar
Ying Sheng committed
32
- [2024/01] SGLang provides up to **5x faster inference** with RadixAttention ([blog](https://lmsys.org/blog/2024-01-17-sglang/)).
Ying Sheng's avatar
Ying Sheng committed
33
34
35
36
- [2024/01] SGLang powers the serving of the official **LLaVA v1.6** release demo ([usage](https://github.com/haotian-liu/LLaVA?tab=readme-ov-file#demo)).

</details>

Lianmin Zheng's avatar
Lianmin Zheng committed
37
38
39
## Contents
- [Install](#install)
- [Backend: SGLang Runtime (SRT)](#backend-sglang-runtime-srt)
Ying Sheng's avatar
Ying Sheng committed
40
- [Frontend: Structured Generation Language (SGLang)](#frontend-structured-generation-language-sglang)
Lianmin Zheng's avatar
Lianmin Zheng committed
41
42
43
44
45
46
- [Benchmark And Performance](#benchmark-and-performance)
- [Roadmap](#roadmap)
- [Citation And Acknowledgment](#citation-and-acknowledgment)

## Install

Lianmin Zheng's avatar
Lianmin Zheng committed
47
48
### Method 1: With pip
```
49
pip install --upgrade pip
Lianmin Zheng's avatar
Lianmin Zheng committed
50
pip install "sglang[all]"
Lianmin Zheng's avatar
Lianmin Zheng committed
51

Lianmin Zheng's avatar
Lianmin Zheng committed
52
# Install FlashInfer CUDA kernels
53
pip install flashinfer -i https://flashinfer.ai/whl/cu121/torch2.4/
Lianmin Zheng's avatar
Lianmin Zheng committed
54
```
55

Lianmin Zheng's avatar
Lianmin Zheng committed
56
### Method 2: From source
Lianmin Zheng's avatar
Lianmin Zheng committed
57
```
Ying Sheng's avatar
Ying Sheng committed
58
# Use the last release branch
Yineng Zhang's avatar
Yineng Zhang committed
59
git clone -b v0.2.13 https://github.com/sgl-project/sglang.git
Lianmin Zheng's avatar
Lianmin Zheng committed
60
61
cd sglang

Lianmin Zheng's avatar
Lianmin Zheng committed
62
pip install --upgrade pip
Lianmin Zheng's avatar
Lianmin Zheng committed
63
64
pip install -e "python[all]"

Lianmin Zheng's avatar
Lianmin Zheng committed
65
# Install FlashInfer CUDA kernels
66
pip install flashinfer -i https://flashinfer.ai/whl/cu121/torch2.4/
Lianmin Zheng's avatar
Lianmin Zheng committed
67
```
68

Lianmin Zheng's avatar
Lianmin Zheng committed
69
### Method 3: Using docker
Ying Sheng's avatar
Ying Sheng committed
70
The docker images are available on Docker Hub as [lmsysorg/sglang](https://hub.docker.com/r/lmsysorg/sglang/tags), built from [Dockerfile](docker).
71
Replace `<secret>` below with your huggingface hub [token](https://huggingface.co/docs/hub/en/security-tokens).
Ying Sheng's avatar
Ying Sheng committed
72

Liangsheng Yin's avatar
Liangsheng Yin committed
73
74
75
76
```bash
docker run --gpus all \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
77
    --env "HF_TOKEN=<secret>" \
Liangsheng Yin's avatar
Liangsheng Yin committed
78
79
    --ipc=host \
    lmsysorg/sglang:latest \
80
    python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct --host 0.0.0.0 --port 30000
Liangsheng Yin's avatar
Liangsheng Yin committed
81
82
```

83
84
### Method 4: Using docker compose

85
<details>
86

87
88
89
90
91
> This method is recommended if you plan to serve it as a service.
> A better approach is to use the [k8s-sglang-service.yaml](./docker/k8s-sglang-service.yaml).

1. Copy the [compose.yml](./docker/compose.yaml) to your local machine
2. Execute the command `docker compose up -d` in your terminal.
92
</details>
93

94
95
### Method 5: Run on Kubernetes or Clouds with SkyPilot

96
<details>
97

98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
To deploy on Kubernetes or 12+ clouds, you can use [SkyPilot](https://github.com/skypilot-org/skypilot).

1. Install SkyPilot and set up Kubernetes cluster or cloud access: see [SkyPilot's documentation](https://skypilot.readthedocs.io/en/latest/getting-started/installation.html).
2. Deploy on your own infra with a single command and get the HTTP API endpoint:
<details>
<summary>SkyPilot YAML: <code>sglang.yaml</code></summary>

```yaml
# sglang.yaml
envs:
  HF_TOKEN: null

resources:
  image_id: docker:lmsysorg/sglang:latest
  accelerators: A100
  ports: 30000

run: |
  conda deactivate
  python3 -m sglang.launch_server \
    --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
    --host 0.0.0.0 \
    --port 30000
```
122
</details>
123
124
125
126
127
128
129
130
131

```bash
# Deploy on any cloud or Kubernetes cluster. Use --cloud <cloud> to select a specific cloud provider.
HF_TOKEN=<secret> sky launch -c sglang --env HF_TOKEN sglang.yaml

# Get the HTTP API endpoint
sky status --endpoint 30000 sglang
```
3. To further scale up your deployment with autoscaling and failure recovery, check out the [SkyServe + SGLang guide](https://github.com/skypilot-org/skypilot/tree/master/llm/sglang#serving-llama-2-with-sglang-for-more-traffic-using-skyserve).
132
</details>
133
134


Lianmin Zheng's avatar
Lianmin Zheng committed
135
### Common Notes
Yineng Zhang's avatar
Yineng Zhang committed
136
- [FlashInfer](https://github.com/flashinfer-ai/flashinfer) is currently one of the dependencies that must be installed for SGLang. If you are using NVIDIA GPU devices below sm80, such as T4, you can't use SGLang for the time being. We expect to resolve this issue soon, so please stay tuned. If you encounter any FlashInfer-related issues on sm80+ devices (e.g., A100, L40S, H100), consider using Triton's kernel by `--disable-flashinfer --disable-flashinfer-sampling` and raise a issue.
Lianmin Zheng's avatar
Lianmin Zheng committed
137
- If you only need to use the OpenAI backend, you can avoid installing other dependencies by using `pip install "sglang[openai]"`.
Ying Sheng's avatar
Ying Sheng committed
138

Ying Sheng's avatar
Ying Sheng committed
139
140
141
## Backend: SGLang Runtime (SRT)
The SGLang Runtime (SRT) is an efficient serving engine.

Ying Sheng's avatar
Ying Sheng committed
142
### Quick Start
Ying Sheng's avatar
Ying Sheng committed
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
Launch a server
```
python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct --port 30000
```

Send a request
```
curl http://localhost:30000/generate \
  -H "Content-Type: application/json" \
  -d '{
    "text": "Once upon a time,",
    "sampling_params": {
      "max_new_tokens": 16,
      "temperature": 0
    }
  }'
```
160
Learn more about the argument format [here](docs/en/sampling_params.md).
Ying Sheng's avatar
Ying Sheng committed
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191

### OpenAI Compatible API
In addition, the server supports OpenAI-compatible APIs.

```python
import openai
client = openai.Client(
    base_url="http://127.0.0.1:30000/v1", api_key="EMPTY")

# Text completion
response = client.completions.create(
	model="default",
	prompt="The capital of France is",
	temperature=0,
	max_tokens=32,
)
print(response)

# Chat completion
response = client.chat.completions.create(
    model="default",
    messages=[
        {"role": "system", "content": "You are a helpful AI assistant"},
        {"role": "user", "content": "List 3 countries and their capitals."},
    ],
    temperature=0,
    max_tokens=64,
)
print(response)
```

Ying Sheng's avatar
Ying Sheng committed
192
It supports streaming, vision, and most features of the Chat/Completions/Models/Batch endpoints specified by the [OpenAI API Reference](https://platform.openai.com/docs/api-reference/).
Ying Sheng's avatar
Ying Sheng committed
193
194

### Additional Server Arguments
195
- Add `--tp 2` to enable multi-GPU tensor parallelism. If it reports the error "peer access is not supported between these two devices", add `--enable-p2p-check` to the server launch command.
Ying Sheng's avatar
Ying Sheng committed
196
197
198
```
python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct --port 30000 --tp 2
```
199
- Add `--dp 2` to enable multi-GPU data parallelism. It can also be used together with tensor parallelism. Data parallelism is better for throughput if there is enough memory.
Ying Sheng's avatar
Ying Sheng committed
200
201
202
```
python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct --port 30000 --dp 2 --tp 2
```
203
- If you see out-of-memory errors during serving, try to reduce the memory usage of the KV cache pool by setting a smaller value of `--mem-fraction-static`. The default value is `0.9`.
Ying Sheng's avatar
Ying Sheng committed
204
205
206
```
python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct --port 30000 --mem-fraction-static 0.7
```
207
208
- See [hyperparameter_tuning.md](docs/en/hyperparameter_tuning.md) on tuning hyperparameters for better performance.
- If you see out-of-memory errors during prefill for long prompts, try to set a smaller chunked prefill size.
Ying Sheng's avatar
Ying Sheng committed
209
```
210
python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct --port 30000 --chunked-prefill-size 4096
Ying Sheng's avatar
Ying Sheng committed
211
```
212
- Add `--nnodes 2` to run tensor parallelism on multiple nodes. If you have two nodes with two GPUs on each node and want to run TP=4, let `sgl-dev-0` be the hostname of the first node and `50000` be an available port.
Ying Sheng's avatar
Ying Sheng committed
213
214
```
# Node 0
215
python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct --tp 4 --nccl-init sgl-dev-0:50000 --nnodes 2 --node-rank 0
Ying Sheng's avatar
Ying Sheng committed
216
217

# Node 1
218
python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct --tp 4 --nccl-init sgl-dev-0:50000 --nnodes 2 --node-rank 1
Ying Sheng's avatar
Ying Sheng committed
219
```
220
- If the model does not have a template in the Hugging Face tokenizer, you can specify a [custom chat template](docs/en/custom_chat_template.md).
221
- To enable experimental torch.compile support, you can add `--enable-torch-compile`. It accelerates small models on small batch sizes.
222
- To enable fp8 quantization, you can add `--quantization fp8` on a fp16 checkpoint or directly load a fp8 checkpoint without specifying any arguments.
Lianmin Zheng's avatar
Lianmin Zheng committed
223
 
Ying Sheng's avatar
Ying Sheng committed
224
225
### Supported Models

226
- Llama / Llama 2 / Llama 3 / Llama 3.1
227
- Mistral / Mixtral / Mistral NeMo
Ying Sheng's avatar
Ying Sheng committed
228
229
- Gemma / Gemma 2
- Qwen / Qwen 2 / Qwen 2 MoE
230
- DeepSeek / DeepSeek 2
231
232
233
234
235
236
237
- [LLaVA-OneVision](https://llava-vl.github.io/blog/2024-08-05-llava-onevision/)
  - `python3 -m sglang.launch_server --model-path lmms-lab/llava-onevision-qwen2-72b-ov --port=30000 --tp-size=8 --chat-template=chatml-llava --chunked-prefill-size=16384`
  - Query the server with the [OpenAI Vision API](https://platform.openai.com/docs/guides/vision). See examples at [test/srt/test_vision_openai_server.py](test/srt/test_vision_openai_server.py)
- LLaVA 1.5 / 1.6 / NeXT
  - `python -m sglang.launch_server --model-path lmms-lab/llama3-llava-next-8b --port=30000 --tp-size=1 --chat-template=llava_llama_3`
  - `python -m sglang.launch_server --model-path lmms-lab/llava-next-72b --port=30000 --tp-size=8 --chat-template=chatml-llava`
  - Query the server with the [OpenAI Vision API](https://platform.openai.com/docs/guides/vision). See examples at [test/srt/test_vision_openai_server.py](test/srt/test_vision_openai_server.py)
Ying Sheng's avatar
Ying Sheng committed
238
239
240
241
242
243
244
245
- Yi-VL
- StableLM
- Command-R
- DBRX
- Grok
- ChatGLM
- InternLM 2

246
Instructions for supporting a new model are [here](https://github.com/sgl-project/sglang/blob/main/docs/en/model_support.md).
Ying Sheng's avatar
Ying Sheng committed
247

Lianmin Zheng's avatar
Lianmin Zheng committed
248
#### Use Models From ModelScope
249
250
<details>

Lianmin Zheng's avatar
Lianmin Zheng committed
251
To use a model from [ModelScope](https://www.modelscope.cn), set the environment variable SGLANG_USE_MODELSCOPE.
Lianmin Zheng's avatar
Lianmin Zheng committed
252
253
254
255
256
257
```
export SGLANG_USE_MODELSCOPE=true
```
Launch [Qwen2-7B-Instruct](https://www.modelscope.cn/models/qwen/qwen2-7b-instruct) Server
```
SGLANG_USE_MODELSCOPE=true python -m sglang.launch_server --model-path qwen/Qwen2-7B-Instruct --port 30000
Lianmin Zheng's avatar
Lianmin Zheng committed
258
259
```
  
260
</details>
Lianmin Zheng's avatar
Lianmin Zheng committed
261
262

#### Run Llama 3.1 405B
Lianmin Zheng's avatar
Lianmin Zheng committed
263
<details>
Ying Sheng's avatar
Ying Sheng committed
264
265

```bash
266
# Run 405B (fp8) on a single node
Ying Sheng's avatar
Ying Sheng committed
267
268
python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-405B-Instruct-FP8 --tp 8

269
270
271
# Run 405B (fp16) on two nodes
## on the first node, replace the `172.16.4.52:20000` with your own first node ip address and port
GLOO_SOCKET_IFNAME=eth0 python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-405B-Instruct --tp 16 --nccl-init-addr 172.16.4.52:20000 --nnodes 2 --node-rank 0 --disable-cuda-graph
Ying Sheng's avatar
Ying Sheng committed
272

273
274
## on the first node, replace the `172.16.4.52:20000` with your own first node ip address and port
GLOO_SOCKET_IFNAME=eth0 python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-405B-Instruct --tp 16 --nccl-init-addr 172.16.4.52:20000 --nnodes 2 --node-rank 1 --disable-cuda-graph
Ying Sheng's avatar
Ying Sheng committed
275
276
```

Lianmin Zheng's avatar
Lianmin Zheng committed
277
278
</details>

Ying Sheng's avatar
Ying Sheng committed
279
280
### Benchmark Performance

Ying Sheng's avatar
Ying Sheng committed
281
- Benchmark a single static batch by running the following command without launching a server. The arguments are the same as for `launch_server.py`. Note that this is not a dynamic batching server, so it may run out of memory for a batch size that a real server can handle. A real server truncates the prefill into several batches, while this unit test does not. For accurate large batch testing, consider using `sglang.bench_serving`.
Ying Sheng's avatar
Ying Sheng committed
282
  ```
283
  python -m sglang.bench_latency --model-path meta-llama/Meta-Llama-3-8B-Instruct --batch 32 --input-len 256 --output-len 32
Ying Sheng's avatar
Ying Sheng committed
284
285
286
  ```
- Benchmark online serving. Launch a server first and run the following command.
  ```
287
  python3 -m sglang.bench_serving --backend sglang --num-prompt 10
Ying Sheng's avatar
Ying Sheng committed
288
289
  ```

Ying Sheng's avatar
Ying Sheng committed
290
## Frontend: Structured Generation Language (SGLang)
291
The frontend language can be used with local models or API models. It is an alternative to the OpenAI API. You may found it easier to use for complex prompting workflow.
Ying Sheng's avatar
Ying Sheng committed
292
293

### Quick Start
Lianmin Zheng's avatar
Lianmin Zheng committed
294
295
The example below shows how to use sglang to answer a mulit-turn question.

Ying Sheng's avatar
Ying Sheng committed
296
#### Using Local Models
297
First, launch a server with
Lianmin Zheng's avatar
Lianmin Zheng committed
298
```
Ying Sheng's avatar
Ying Sheng committed
299
python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3-8B-Instruct --port 30000
Lianmin Zheng's avatar
Lianmin Zheng committed
300
301
```

302
303
Then, connect to the server and answer a multi-turn question.

Lianmin Zheng's avatar
Lianmin Zheng committed
304
```python
305
from sglang import function, system, user, assistant, gen, set_default_backend, RuntimeEndpoint
Lianmin Zheng's avatar
Lianmin Zheng committed
306
307
308
309
310
311
312
313
314

@function
def multi_turn_question(s, question_1, question_2):
    s += system("You are a helpful assistant.")
    s += user(question_1)
    s += assistant(gen("answer_1", max_tokens=256))
    s += user(question_2)
    s += assistant(gen("answer_2", max_tokens=256))

315
set_default_backend(RuntimeEndpoint("http://localhost:30000"))
Lianmin Zheng's avatar
Lianmin Zheng committed
316
317
318
319
320
321
322
323

state = multi_turn_question.run(
    question_1="What is the capital of the United States?",
    question_2="List two local attractions.",
)

for m in state.messages():
    print(m["role"], ":", m["content"])
324
325

print(state["answer_1"])
Lianmin Zheng's avatar
Lianmin Zheng committed
326
327
```

Ying Sheng's avatar
Ying Sheng committed
328
#### Using OpenAI Models
329
Set the OpenAI API Key
Lianmin Zheng's avatar
Lianmin Zheng committed
330
```
331
export OPENAI_API_KEY=sk-******
Lianmin Zheng's avatar
Lianmin Zheng committed
332
333
```

334
Then, answer a multi-turn question.
Lianmin Zheng's avatar
Lianmin Zheng committed
335
```python
336
from sglang import function, system, user, assistant, gen, set_default_backend, OpenAI
Lianmin Zheng's avatar
Lianmin Zheng committed
337
338
339
340
341
342
343
344
345

@function
def multi_turn_question(s, question_1, question_2):
    s += system("You are a helpful assistant.")
    s += user(question_1)
    s += assistant(gen("answer_1", max_tokens=256))
    s += user(question_2)
    s += assistant(gen("answer_2", max_tokens=256))

346
set_default_backend(OpenAI("gpt-3.5-turbo"))
Lianmin Zheng's avatar
Lianmin Zheng committed
347
348
349
350
351
352
353
354

state = multi_turn_question.run(
    question_1="What is the capital of the United States?",
    question_2="List two local attractions.",
)

for m in state.messages():
    print(m["role"], ":", m["content"])
355
356

print(state["answer_1"])
Lianmin Zheng's avatar
Lianmin Zheng committed
357
358
```

Ying Sheng's avatar
Ying Sheng committed
359
#### More Examples
Lianmin Zheng's avatar
Lianmin Zheng committed
360

361
Anthropic and VertexAI (Gemini) models are also supported.
Lianmin Zheng's avatar
Lianmin Zheng committed
362
363
You can find more examples at [examples/quick_start](examples/quick_start).

Ying Sheng's avatar
Ying Sheng committed
364
### Language Feature
Lianmin Zheng's avatar
Lianmin Zheng committed
365
366
367
368
369
To begin with, import sglang.
```python
import sglang as sgl
```

Lianmin Zheng's avatar
Lianmin Zheng committed
370
`sglang` provides some simple primitives such as `gen`, `select`, `fork`, `image`.
Lianmin Zheng's avatar
Lianmin Zheng committed
371
372
You can implement your prompt flow in a function decorated by `sgl.function`.
You can then invoke the function with `run` or `run_batch`.
373
The system will manage the state, chat template, parallelism and batching for you.
Lianmin Zheng's avatar
Lianmin Zheng committed
374

375
376
The complete code for the examples below can be found at [readme_examples.py](examples/usage/readme_examples.py)

Ying Sheng's avatar
Ying Sheng committed
377
#### Control Flow
Lianmin Zheng's avatar
Lianmin Zheng committed
378
379
You can use any Python code within the function body, including control flow, nested function calls, and external libraries.

Lianmin Zheng's avatar
Lianmin Zheng committed
380
381
```python
@sgl.function
382
383
384
def tool_use(s, question):
    s += "To answer this question: " + question + ". "
    s += "I need to use a " + sgl.gen("tool", choices=["calculator", "search engine"]) + ". "
Lianmin Zheng's avatar
Lianmin Zheng committed
385
386
387

    if s["tool"] == "calculator":
        s += "The math expression is" + sgl.gen("expression")
388
389
    elif s["tool"] == "search engine":
        s += "The key word to search is" + sgl.gen("word")
Lianmin Zheng's avatar
Lianmin Zheng committed
390
```
Lianmin Zheng's avatar
Lianmin Zheng committed
391

Ying Sheng's avatar
Ying Sheng committed
392
#### Parallelism
Lianmin Zheng's avatar
Lianmin Zheng committed
393
394
395
Use `fork` to launch parallel prompts.
Because `sgl.gen` is non-blocking, the for loop below issues two generation calls in parallel.

Lianmin Zheng's avatar
Lianmin Zheng committed
396
397
398
399
400
401
402
403
```python
@sgl.function
def tip_suggestion(s):
    s += (
        "Here are two tips for staying healthy: "
        "1. Balanced Diet. 2. Regular Exercise.\n\n"
    )

Lianmin Zheng's avatar
Lianmin Zheng committed
404
    forks = s.fork(2)
Lianmin Zheng's avatar
Lianmin Zheng committed
405
406
407
408
409
410
411
412
    for i, f in enumerate(forks):
        f += f"Now, expand tip {i+1} into a paragraph:\n"
        f += sgl.gen(f"detailed_tip", max_tokens=256, stop="\n\n")

    s += "Tip 1:" + forks[0]["detailed_tip"] + "\n"
    s += "Tip 2:" + forks[1]["detailed_tip"] + "\n"
    s += "In summary" + sgl.gen("summary")
```
Lianmin Zheng's avatar
Lianmin Zheng committed
413

Lianmin Zheng's avatar
Lianmin Zheng committed
414
#### Multi-Modality
Lianmin Zheng's avatar
Lianmin Zheng committed
415
416
Use `sgl.image` to pass an image as input.

Lianmin Zheng's avatar
Lianmin Zheng committed
417
418
```python
@sgl.function
Lianmin Zheng's avatar
Lianmin Zheng committed
419
def image_qa(s, image_file, question):
Lianmin Zheng's avatar
Lianmin Zheng committed
420
    s += sgl.user(sgl.image(image_file) + question)
Lianmin Zheng's avatar
Lianmin Zheng committed
421
    s += sgl.assistant(sgl.gen("answer", max_tokens=256)
Lianmin Zheng's avatar
Lianmin Zheng committed
422
423
```

424
425
See also [srt_example_llava.py](examples/quick_start/srt_example_llava.py).

Ying Sheng's avatar
Ying Sheng committed
426
#### Constrained Decoding
427
428
Use `regex` to specify a regular expression as a decoding constraint.
This is only supported for local models.
Lianmin Zheng's avatar
Lianmin Zheng committed
429

Lianmin Zheng's avatar
Lianmin Zheng committed
430
```python
Lianmin Zheng's avatar
Lianmin Zheng committed
431
@sgl.function
Lianmin Zheng's avatar
Lianmin Zheng committed
432
433
def regular_expression_gen(s):
    s += "Q: What is the IP address of the Google DNS servers?\n"
Lianmin Zheng's avatar
Lianmin Zheng committed
434
    s += "A: " + sgl.gen(
Lianmin Zheng's avatar
Lianmin Zheng committed
435
436
437
438
439
        "answer",
        temperature=0,
        regex=r"((25[0-5]|2[0-4]\d|[01]?\d\d?).){3}(25[0-5]|2[0-4]\d|[01]?\d\d?)",
    )
```
Lianmin Zheng's avatar
Lianmin Zheng committed
440

Ying Sheng's avatar
Ying Sheng committed
441
#### JSON Decoding
Lianmin Zheng's avatar
Lianmin Zheng committed
442
Use `regex` to specify a JSON schema with a regular expression.
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463

```python
character_regex = (
    r"""\{\n"""
    + r"""    "name": "[\w\d\s]{1,16}",\n"""
    + r"""    "house": "(Gryffindor|Slytherin|Ravenclaw|Hufflepuff)",\n"""
    + r"""    "blood status": "(Pure-blood|Half-blood|Muggle-born)",\n"""
    + r"""    "occupation": "(student|teacher|auror|ministry of magic|death eater|order of the phoenix)",\n"""
    + r"""    "wand": \{\n"""
    + r"""        "wood": "[\w\d\s]{1,16}",\n"""
    + r"""        "core": "[\w\d\s]{1,16}",\n"""
    + r"""        "length": [0-9]{1,2}\.[0-9]{0,2}\n"""
    + r"""    \},\n"""
    + r"""    "alive": "(Alive|Deceased)",\n"""
    + r"""    "patronus": "[\w\d\s]{1,16}",\n"""
    + r"""    "bogart": "[\w\d\s]{1,16}"\n"""
    + r"""\}"""
)

@sgl.function
def character_gen(s, name):
Lianmin Zheng's avatar
Lianmin Zheng committed
464
    s += name + " is a character in Harry Potter. Please fill in the following information about this character.\n"
465
466
467
    s += sgl.gen("json_output", max_tokens=256, regex=character_regex)
```

Lianmin Zheng's avatar
Lianmin Zheng committed
468
See also [json_decode.py](examples/usage/json_decode.py) for an additional example of specifying formats with Pydantic models.
469

Ying Sheng's avatar
Ying Sheng committed
470
#### Batching
Lianmin Zheng's avatar
Lianmin Zheng committed
471
472
Use `run_batch` to run a batch of requests with continuous batching.

Lianmin Zheng's avatar
Lianmin Zheng committed
473
474
475
476
477
478
479
480
481
482
483
484
```python
@sgl.function
def text_qa(s, question):
    s += "Q: " + question + "\n"
    s += "A:" + sgl.gen("answer", stop="\n")

states = text_qa.run_batch(
    [
        {"question": "What is the capital of the United Kingdom?"},
        {"question": "What is the capital of France?"},
        {"question": "What is the capital of Japan?"},
    ],
Lianmin Zheng's avatar
Lianmin Zheng committed
485
    progress_bar=True
Lianmin Zheng's avatar
Lianmin Zheng committed
486
487
)
```
Lianmin Zheng's avatar
Lianmin Zheng committed
488

Ying Sheng's avatar
Ying Sheng committed
489
#### Streaming
Lianmin Zheng's avatar
Lianmin Zheng committed
490
491
Add `stream=True` to enable streaming.

Lianmin Zheng's avatar
Lianmin Zheng committed
492
493
494
495
496
497
```python
@sgl.function
def text_qa(s, question):
    s += "Q: " + question + "\n"
    s += "A:" + sgl.gen("answer", stop="\n")

498
state = text_qa.run(
Lianmin Zheng's avatar
Lianmin Zheng committed
499
    question="What is the capital of France?",
Lianmin Zheng's avatar
Lianmin Zheng committed
500
501
502
    temperature=0.1,
    stream=True
)
Lianmin Zheng's avatar
Lianmin Zheng committed
503

Lianmin Zheng's avatar
Lianmin Zheng committed
504
505
506
for out in state.text_iter():
    print(out, end="", flush=True)
```
Lianmin Zheng's avatar
Lianmin Zheng committed
507

508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
#### Roles

Use `sgl.system``sgl.user` and `sgl.assistant` to set roles when using Chat models. You can also define more complex role prompts using begin and end tokens.

```python
@sgl.function
def chat_example(s):
    s += sgl.system("You are a helpful assistant.")
    # Same as: s += s.system("You are a helpful assistant.")

    with s.user():
        s += "Question: What is the capital of France?"

    s += sgl.assistant_begin()
    s += "Answer: " + sgl.gen(max_tokens=100, stop="\n")
    s += sgl.assistant_end()
```

Ying Sheng's avatar
Ying Sheng committed
526
#### Tips and Implementation Details
527
528
- The `choices` argument in `sgl.gen` is implemented by computing the [token-length normalized log probabilities](https://blog.eleuther.ai/multiple-choice-normalization/) of all choices and selecting the one with the highest probability.
- The `regex` argument in `sgl.gen` is implemented through autoregressive decoding with logit bias masking, according to the constraints set by the regex. It is compatible with `temperature=0` and `temperature != 0`.
Lianmin Zheng's avatar
Lianmin Zheng committed
529

Ying Sheng's avatar
Ying Sheng committed
530
531
532
## Benchmark And Performance
![8b_throughput](https://lmsys.org/images/blog/sglang_llama3/8b_throughput.svg)
![70b_fp8_throughput](https://lmsys.org/images/blog/sglang_llama3/70b_fp8_throughput.svg)
Lianmin Zheng's avatar
Lianmin Zheng committed
533

Ying Sheng's avatar
Ying Sheng committed
534
Learn more at this [blog](https://lmsys.org/blog/2024-07-25-sglang-llama3/).
Lianmin Zheng's avatar
Lianmin Zheng committed
535

Lianmin Zheng's avatar
Lianmin Zheng committed
536
## Roadmap
Ying Sheng's avatar
Ying Sheng committed
537
[Development Roadmap (2024 Q3)](https://github.com/sgl-project/sglang/issues/634)
Lianmin Zheng's avatar
Lianmin Zheng committed
538
539

## Citation And Acknowledgment
Ying Sheng's avatar
Ying Sheng committed
540
541
Please cite our paper, [SGLang: Efficient Execution of Structured Language Model Programs](https://arxiv.org/abs/2312.07104), if you find the project useful.
We also learned from the design and reused code from the following projects: [Guidance](https://github.com/guidance-ai/guidance), [vLLM](https://github.com/vllm-project/vllm), [LightLLM](https://github.com/ModelTC/lightllm), [FlashInfer](https://github.com/flashinfer-ai/flashinfer), [Outlines](https://github.com/outlines-dev/outlines), and [LMQL](https://github.com/eth-sri/lmql).