README.md 14.7 KB
Newer Older
Lianmin Zheng's avatar
Lianmin Zheng committed
1
2
3
4
5
6
<div align="center">
<img src="assets/logo.png" alt="logo" width="400"></img>
</div>

--------------------------------------------------------------------------------

7
| [**Blog**](https://lmsys.org/blog/2024-01-17-sglang/) | [**Paper**](https://arxiv.org/abs/2312.07104) |
Lianmin Zheng's avatar
Lianmin Zheng committed
8
9
10
11

SGLang is a structured generation language designed for large language models (LLMs).
It makes your interaction with LLMs faster and more controllable by co-designing the frontend language and the runtime system.

12
The core features include:
Lianmin Zheng's avatar
Lianmin Zheng committed
13
14
- **Flexible Frontend Language**: Enables easy programming of LLM applications with chained generation calls, advanced prompting, control flow, multiple modalities, parallelism, and external interactions.
- **High-Performance Backend Runtime**: Features RadixAttention for accelerating complex LLM programs by reusing the KV cache across multiple calls. It can also serve as a standalone engine with all common techniques implemented (e.g., continuous batching and tensor parallelism).
Lianmin Zheng's avatar
Lianmin Zheng committed
15

Ying Sheng's avatar
Ying Sheng committed
16
## News
Lianmin Zheng's avatar
Lianmin Zheng committed
17
- [2024/02] 🔥 SGLang enables **3x faster JSON decoding** with compressed finite state machine ([blog](https://lmsys.org/blog/2024-02-05-compressed-fsm/)).
18
- [2024/01] 🔥 SGLang powers the serving of the official **LLaVA v1.6** release demo ([usage](https://github.com/haotian-liu/LLaVA?tab=readme-ov-file#demo)).
Lianmin Zheng's avatar
Lianmin Zheng committed
19
- [2024/01] SGLang provides up to **5x faster inference** with RadixAttention ([blog](https://lmsys.org/blog/2024-01-17-sglang/)).
Ying Sheng's avatar
Ying Sheng committed
20

Lianmin Zheng's avatar
Lianmin Zheng committed
21
22
23
## Contents
- [Install](#install)
- [Quick Start](#quick-start)
24
- [Frontend: Structured Generation Language (SGLang)](#frontend-structured-generation-language-sglang)
Lianmin Zheng's avatar
Lianmin Zheng committed
25
26
27
28
29
30
31
- [Backend: SGLang Runtime (SRT)](#backend-sglang-runtime-srt)
- [Benchmark And Performance](#benchmark-and-performance)
- [Roadmap](#roadmap)
- [Citation And Acknowledgment](#citation-and-acknowledgment)

## Install

Lianmin Zheng's avatar
Lianmin Zheng committed
32
33
34
35
### Method 1: With pip
```
pip install "sglang[all]"
```
Lianmin Zheng's avatar
Lianmin Zheng committed
36

37
38
Next, [install FlashInfer](https://docs.flashinfer.ai/installation.html) for attention CUDA kernels.

Lianmin Zheng's avatar
Lianmin Zheng committed
39
### Method 2: From source
Lianmin Zheng's avatar
Lianmin Zheng committed
40
```
41
git clone https://github.com/sgl-project/sglang.git
Lianmin Zheng's avatar
Lianmin Zheng committed
42
43
44
45
46
47
cd sglang

pip install --upgrade pip
pip install -e "python[all]"
```

48
49
Next, [install FlashInfer](https://docs.flashinfer.ai/installation.html) for attention CUDA kernels.

Ying Sheng's avatar
Ying Sheng committed
50
### Notes
51
52
- If you see triton errors, please install the [Triton Nightly](https://triton-lang.org/main/getting-started/installation.html).
- If you cannot install FlashInfer, you can use the slower triton kernels by adding `--disable-flashinfer` when launching the server.
Lianmin Zheng's avatar
Lianmin Zheng committed
53
- If you only need to use the OpenAI backend, you can avoid installing other dependencies by using `pip install "sglang[openai]"`
Ying Sheng's avatar
Ying Sheng committed
54

Lianmin Zheng's avatar
Lianmin Zheng committed
55
56
57
## Quick Start
The example below shows how to use sglang to answer a mulit-turn question.

58
59
### Using Local Models
First, launch a server with
Lianmin Zheng's avatar
Lianmin Zheng committed
60
```
61
python -m sglang.launch_server --model-path meta-llama/Llama-2-7b-chat-hf --port 30000
Lianmin Zheng's avatar
Lianmin Zheng committed
62
63
```

64
65
Then, connect to the server and answer a multi-turn question.

Lianmin Zheng's avatar
Lianmin Zheng committed
66
```python
67
from sglang import function, system, user, assistant, gen, set_default_backend, RuntimeEndpoint
Lianmin Zheng's avatar
Lianmin Zheng committed
68
69
70
71
72
73
74
75
76

@function
def multi_turn_question(s, question_1, question_2):
    s += system("You are a helpful assistant.")
    s += user(question_1)
    s += assistant(gen("answer_1", max_tokens=256))
    s += user(question_2)
    s += assistant(gen("answer_2", max_tokens=256))

77
set_default_backend(RuntimeEndpoint("http://localhost:30000"))
Lianmin Zheng's avatar
Lianmin Zheng committed
78
79
80
81
82
83
84
85

state = multi_turn_question.run(
    question_1="What is the capital of the United States?",
    question_2="List two local attractions.",
)

for m in state.messages():
    print(m["role"], ":", m["content"])
86
87

print(state["answer_1"])
Lianmin Zheng's avatar
Lianmin Zheng committed
88
89
```

90
91
### Using OpenAI Models
Set the OpenAI API Key
Lianmin Zheng's avatar
Lianmin Zheng committed
92
```
93
export OPENAI_API_KEY=sk-******
Lianmin Zheng's avatar
Lianmin Zheng committed
94
95
```

96
Then, answer a multi-turn question.
Lianmin Zheng's avatar
Lianmin Zheng committed
97
```python
98
from sglang import function, system, user, assistant, gen, set_default_backend, OpenAI
Lianmin Zheng's avatar
Lianmin Zheng committed
99
100
101
102
103
104
105
106
107

@function
def multi_turn_question(s, question_1, question_2):
    s += system("You are a helpful assistant.")
    s += user(question_1)
    s += assistant(gen("answer_1", max_tokens=256))
    s += user(question_2)
    s += assistant(gen("answer_2", max_tokens=256))

108
set_default_backend(OpenAI("gpt-3.5-turbo"))
Lianmin Zheng's avatar
Lianmin Zheng committed
109
110
111
112
113
114
115
116

state = multi_turn_question.run(
    question_1="What is the capital of the United States?",
    question_2="List two local attractions.",
)

for m in state.messages():
    print(m["role"], ":", m["content"])
117
118

print(state["answer_1"])
Lianmin Zheng's avatar
Lianmin Zheng committed
119
120
121
122
```

### More Examples

123
Anthropic and VertexAI (Gemini) models are also supported.
Lianmin Zheng's avatar
Lianmin Zheng committed
124
125
You can find more examples at [examples/quick_start](examples/quick_start).

126
## Frontend: Structured Generation Language (SGLang)
Lianmin Zheng's avatar
Lianmin Zheng committed
127

Lianmin Zheng's avatar
Lianmin Zheng committed
128
129
130
131
132
To begin with, import sglang.
```python
import sglang as sgl
```

Lianmin Zheng's avatar
Lianmin Zheng committed
133
`sglang` provides some simple primitives such as `gen`, `select`, `fork`, `image`.
Lianmin Zheng's avatar
Lianmin Zheng committed
134
135
You can implement your prompt flow in a function decorated by `sgl.function`.
You can then invoke the function with `run` or `run_batch`.
136
The system will manage the state, chat template, parallelism and batching for you.
Lianmin Zheng's avatar
Lianmin Zheng committed
137

138
139
The complete code for the examples below can be found at [readme_examples.py](examples/usage/readme_examples.py)

Lianmin Zheng's avatar
Lianmin Zheng committed
140
### Control Flow
Lianmin Zheng's avatar
Lianmin Zheng committed
141
142
You can use any Python code within the function body, including control flow, nested function calls, and external libraries.

Lianmin Zheng's avatar
Lianmin Zheng committed
143
144
```python
@sgl.function
145
146
147
def tool_use(s, question):
    s += "To answer this question: " + question + ". "
    s += "I need to use a " + sgl.gen("tool", choices=["calculator", "search engine"]) + ". "
Lianmin Zheng's avatar
Lianmin Zheng committed
148
149
150

    if s["tool"] == "calculator":
        s += "The math expression is" + sgl.gen("expression")
151
152
    elif s["tool"] == "search engine":
        s += "The key word to search is" + sgl.gen("word")
Lianmin Zheng's avatar
Lianmin Zheng committed
153
```
Lianmin Zheng's avatar
Lianmin Zheng committed
154
155

### Parallelism
Lianmin Zheng's avatar
Lianmin Zheng committed
156
157
158
Use `fork` to launch parallel prompts.
Because `sgl.gen` is non-blocking, the for loop below issues two generation calls in parallel.

Lianmin Zheng's avatar
Lianmin Zheng committed
159
160
161
162
163
164
165
166
```python
@sgl.function
def tip_suggestion(s):
    s += (
        "Here are two tips for staying healthy: "
        "1. Balanced Diet. 2. Regular Exercise.\n\n"
    )

Lianmin Zheng's avatar
Lianmin Zheng committed
167
    forks = s.fork(2)
Lianmin Zheng's avatar
Lianmin Zheng committed
168
169
170
171
172
173
174
175
    for i, f in enumerate(forks):
        f += f"Now, expand tip {i+1} into a paragraph:\n"
        f += sgl.gen(f"detailed_tip", max_tokens=256, stop="\n\n")

    s += "Tip 1:" + forks[0]["detailed_tip"] + "\n"
    s += "Tip 2:" + forks[1]["detailed_tip"] + "\n"
    s += "In summary" + sgl.gen("summary")
```
Lianmin Zheng's avatar
Lianmin Zheng committed
176
177

### Multi Modality
Lianmin Zheng's avatar
Lianmin Zheng committed
178
179
Use `sgl.image` to pass an image as input.

Lianmin Zheng's avatar
Lianmin Zheng committed
180
181
```python
@sgl.function
Lianmin Zheng's avatar
Lianmin Zheng committed
182
def image_qa(s, image_file, question):
Lianmin Zheng's avatar
Lianmin Zheng committed
183
    s += sgl.user(sgl.image(image_file) + question)
Lianmin Zheng's avatar
Lianmin Zheng committed
184
    s += sgl.assistant(sgl.gen("answer", max_tokens=256)
Lianmin Zheng's avatar
Lianmin Zheng committed
185
186
```

187
188
See also [srt_example_llava.py](examples/quick_start/srt_example_llava.py).

Lianmin Zheng's avatar
Lianmin Zheng committed
189
### Constrained Decoding
190
191
Use `regex` to specify a regular expression as a decoding constraint.
This is only supported for local models.
Lianmin Zheng's avatar
Lianmin Zheng committed
192

Lianmin Zheng's avatar
Lianmin Zheng committed
193
```python
Lianmin Zheng's avatar
Lianmin Zheng committed
194
@sgl.function
Lianmin Zheng's avatar
Lianmin Zheng committed
195
196
def regular_expression_gen(s):
    s += "Q: What is the IP address of the Google DNS servers?\n"
Lianmin Zheng's avatar
Lianmin Zheng committed
197
    s += "A: " + sgl.gen(
Lianmin Zheng's avatar
Lianmin Zheng committed
198
199
200
201
202
        "answer",
        temperature=0,
        regex=r"((25[0-5]|2[0-4]\d|[01]?\d\d?).){3}(25[0-5]|2[0-4]\d|[01]?\d\d?)",
    )
```
Lianmin Zheng's avatar
Lianmin Zheng committed
203

204
### JSON Decoding
Lianmin Zheng's avatar
Lianmin Zheng committed
205
Use `regex` to specify a JSON schema with a regular expression.
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226

```python
character_regex = (
    r"""\{\n"""
    + r"""    "name": "[\w\d\s]{1,16}",\n"""
    + r"""    "house": "(Gryffindor|Slytherin|Ravenclaw|Hufflepuff)",\n"""
    + r"""    "blood status": "(Pure-blood|Half-blood|Muggle-born)",\n"""
    + r"""    "occupation": "(student|teacher|auror|ministry of magic|death eater|order of the phoenix)",\n"""
    + r"""    "wand": \{\n"""
    + r"""        "wood": "[\w\d\s]{1,16}",\n"""
    + r"""        "core": "[\w\d\s]{1,16}",\n"""
    + r"""        "length": [0-9]{1,2}\.[0-9]{0,2}\n"""
    + r"""    \},\n"""
    + r"""    "alive": "(Alive|Deceased)",\n"""
    + r"""    "patronus": "[\w\d\s]{1,16}",\n"""
    + r"""    "bogart": "[\w\d\s]{1,16}"\n"""
    + r"""\}"""
)

@sgl.function
def character_gen(s, name):
Lianmin Zheng's avatar
Lianmin Zheng committed
227
    s += name + " is a character in Harry Potter. Please fill in the following information about this character.\n"
228
229
230
    s += sgl.gen("json_output", max_tokens=256, regex=character_regex)
```

Lianmin Zheng's avatar
Lianmin Zheng committed
231
See also [json_decode.py](examples/usage/json_decode.py) for an additional example on specifying formats with Pydantic models.
232
233


Lianmin Zheng's avatar
Lianmin Zheng committed
234
### Batching
Lianmin Zheng's avatar
Lianmin Zheng committed
235
236
Use `run_batch` to run a batch of requests with continuous batching.

Lianmin Zheng's avatar
Lianmin Zheng committed
237
238
239
240
241
242
243
244
245
246
247
248
```python
@sgl.function
def text_qa(s, question):
    s += "Q: " + question + "\n"
    s += "A:" + sgl.gen("answer", stop="\n")

states = text_qa.run_batch(
    [
        {"question": "What is the capital of the United Kingdom?"},
        {"question": "What is the capital of France?"},
        {"question": "What is the capital of Japan?"},
    ],
Lianmin Zheng's avatar
Lianmin Zheng committed
249
    progress_bar=True
Lianmin Zheng's avatar
Lianmin Zheng committed
250
251
)
```
Lianmin Zheng's avatar
Lianmin Zheng committed
252
253

### Streaming
Lianmin Zheng's avatar
Lianmin Zheng committed
254
255
Add `stream=True` to enable streaming.

Lianmin Zheng's avatar
Lianmin Zheng committed
256
257
258
259
260
261
```python
@sgl.function
def text_qa(s, question):
    s += "Q: " + question + "\n"
    s += "A:" + sgl.gen("answer", stop="\n")

262
state = text_qa.run(
Lianmin Zheng's avatar
Lianmin Zheng committed
263
    question="What is the capital of France?",
Lianmin Zheng's avatar
Lianmin Zheng committed
264
265
266
    temperature=0.1,
    stream=True
)
Lianmin Zheng's avatar
Lianmin Zheng committed
267

Lianmin Zheng's avatar
Lianmin Zheng committed
268
269
270
for out in state.text_iter():
    print(out, end="", flush=True)
```
Lianmin Zheng's avatar
Lianmin Zheng committed
271

Lianmin Zheng's avatar
Lianmin Zheng committed
272
273
274
275
### Tips and Implementation Details
- The `choices` argument in `sgl.gen` is implemented by computing the normalized log probabilities of all choices and selecting the one with the highest probability.
- The `regex` argument in `sgl.gen` is implemented through autoregressive decoding with logit bias masking, according to the constraints set by the regex.

Lianmin Zheng's avatar
Lianmin Zheng committed
276
277
278
## Backend: SGLang Runtime (SRT)
The SGLang Runtime (SRT) is designed to work best with the SGLang frontend.
However, it can also be used as a standalone API server.
Ying Sheng's avatar
Ying Sheng committed
279
In this case, the [RadixAttention](https://arxiv.org/abs/2312.07104) can still greatly accelerate many use cases with automatic KV cache reuse.
Lianmin Zheng's avatar
Lianmin Zheng committed
280
281
282
283
284
285
286
287
288

### Usage
Launch a server
```
python -m sglang.launch_server --model-path meta-llama/Llama-2-7b-chat-hf --port 30000
```

Send a request
```
289
curl http://localhost:30000/generate \
Lianmin Zheng's avatar
Lianmin Zheng committed
290
291
  -H "Content-Type: application/json" \
  -d '{
292
    "text": "Once upon a time,",
293
    "sampling_params": {
294
295
296
      "max_new_tokens": 16,
      "temperature": 0
    }
Lianmin Zheng's avatar
Lianmin Zheng committed
297
298
  }'
```
299
300
Learn more about the argument format [here](docs/sampling_params.md).

301
302
303
304
305
306
307
### OpenAI Compatible API
In addition, the server supports an experimental OpenAI-compatible API.

```python
import openai
client = openai.Client(
    base_url="http://127.0.0.1:30000/v1", api_key="EMPTY")
Cody Yu's avatar
Cody Yu committed
308
309

# Text completion
310
311
312
313
314
315
316
response = client.completions.create(
	model="default",
	prompt="The capital of France is",
	temperature=0,
	max_tokens=32,
)
print(response)
Cody Yu's avatar
Cody Yu committed
317
318
319
320
321
322
323
324
325
326
327
328
329
330

# Chat completion
response = client.chat.completions.create(
    model="default",
    messages=[
        {"role": "system", "content": "You are a helpful AI assistant"},
        {"role": "user", "content": "List 3 countries and their capitals."},
    ],
    temperature=0,
    max_tokens=64,
)
print(response)
```

Lianmin Zheng's avatar
Lianmin Zheng committed
331
332
333
334

By default, the server uses the chat template specified in the model tokenizer from Hugging Face. It should just work for most official models such as Llama-2/Llama-3.

If needed, you can also override the chat template when launching the server:
Cody Yu's avatar
Cody Yu committed
335
336

```
337
python -m sglang.launch_server --model-path meta-llama/Llama-2-7b-chat-hf --port 30000 --chat-template llama-2
Cody Yu's avatar
Cody Yu committed
338
339
340
```

If the chat template you are looking for is missing, you are welcome to contribute it.
Lianmin Zheng's avatar
Lianmin Zheng committed
341
Meanwhile, you can also temporarily register your chat template as follows:
Cody Yu's avatar
Cody Yu committed
342
343
344
345
346
347
348
349
350
351
352
353
354
355

```json
{
  "name": "my_model",
  "system": "<|im_start|>system",
  "user": "<|im_start|>user",
  "assistant": "<|im_start|>assistant",
  "sep_style": "CHATML",
  "sep": "<|im_end|>",
  "stop_str": ["<|im_end|>", "<|im_start|>"]
}
```

```
356
python -m sglang.launch_server --model-path meta-llama/Llama-2-7b-chat-hf --port 30000 --chat-template ./my_model_template.json
357
358
```

Lianmin Zheng's avatar
Lianmin Zheng committed
359
360
361
362
363
### Additional Arguments
- Add `--tp 2` to enable tensor parallelism.
```
python -m sglang.launch_server --model-path meta-llama/Llama-2-7b-chat-hf --port 30000 --tp 2
```
Lianmin Zheng's avatar
Lianmin Zheng committed
364
365
366
367
- Add `--dp 2` to enable data parallelism. It can also be used together with tp. Data parallelism is better for throughput if there is enough memory.
```
python -m sglang.launch_server --model-path meta-llama/Llama-2-7b-chat-hf --port 30000 --dp 2 --tp 2
```
Ying Sheng's avatar
Ying Sheng committed
368
369
370
371
- If you see out-of-memory errors during serving, please try to reduce the memory usage of the KV cache pool by setting a smaller value of `--mem-fraction-static`. The default value is `0.9`
```
python -m sglang.launch_server --model-path meta-llama/Llama-2-7b-chat-hf --port 30000 --mem-fraction-static 0.7
```
Lianmin Zheng's avatar
Lianmin Zheng committed
372
- See [hyperparameter_tuning.md](docs/hyperparameter_tuning.md) on tuning hyperparameters for better performance.
Lianmin Zheng's avatar
Lianmin Zheng committed
373
374
375
376
377

### Supported Models
- Llama
- Mistral
- Mixtral
Lianmin Zheng's avatar
Lianmin Zheng committed
378
- Qwen / Qwen 2
379
380
381
- Gemma
  - Please add a new flag `--attention-reduce-in-fp32` to avoid some precision errors.
  - `python -m sglang.launch_server --model-path google/gemma-7b-it --port 30000 --attention-reduce-in-fp32`
Lianmin Zheng's avatar
Lianmin Zheng committed
382
- LLaVA
383
  - `python3 -m sglang.launch_server --model-path liuhaotian/llava-v1.5-7b --tokenizer-path llava-hf/llava-1.5-7b-hf --chat-template vicuna_v1.1 --port 30000`
384
385
  - `python3 -m sglang.launch_server --model-path liuhaotian/llava-v1.6-vicuna-7b --tokenizer-path llava-hf/llava-1.5-7b-hf --chat-template vicuna_v1.1 --port 30000`
  - `python3 -m sglang.launch_server --model-path liuhaotian/llava-v1.6-34b --tokenizer-path liuhaotian/llava-v1.6-34b-tokenizer --port 3000`
386
387
- LLaVA-NeXT-Video
  - see [srt_example_llava_v.sh](examples/usage/llava_video/srt_example_llava_v.sh)
Lianmin Zheng's avatar
Lianmin Zheng committed
388
389
- Yi-VL
  - see [srt_example_yi_vl.py](examples/quick_start/srt_example_yi_vl.py).
390
391
392
393
394
395
- StableLM
- Command-R
- DBRX
- AWQ/GPTQ/Marlin quantization

Instructions for supporting a new model are [here](https://github.com/sgl-project/sglang/blob/main/docs/model_support.md).
Lianmin Zheng's avatar
Lianmin Zheng committed
396
397

## Benchmark And Performance
Lianmin Zheng's avatar
Lianmin Zheng committed
398
399
400
401
402
403
- Llama-7B on NVIDIA A10G, FP16, Tensor Parallelism=1
![llama_7b](assets/llama_7b.jpg)

- Mixtral-8x7B on NVIDIA A10G, FP16, Tensor Parallelism=8
![mixtral_8x7b](assets/mixtral_8x7b.jpg)

Lianmin Zheng's avatar
Lianmin Zheng committed
404
405
- Learn more about the above [results](docs/benchmark_results.md).
- Synthetic latency and throughput benchmark [scripts](https://github.com/sgl-project/sglang/tree/main/benchmark/latency_throughput).
Lianmin Zheng's avatar
Lianmin Zheng committed
406

Lianmin Zheng's avatar
Lianmin Zheng committed
407
## Roadmap
Ying Sheng's avatar
Ying Sheng committed
408
https://github.com/sgl-project/sglang/issues/157
Lianmin Zheng's avatar
Lianmin Zheng committed
409
410
411

## Citation And Acknowledgment
```
Lianmin Zheng's avatar
Lianmin Zheng committed
412
413
414
415
@misc{zheng2024sglang,
      title={SGLang: Efficient Execution of Structured Language Model Programs},
      author={Lianmin Zheng and Liangsheng Yin and Zhiqiang Xie and Chuyue Sun and Jeff Huang and Cody Hao Yu and Shiyi Cao and Christos Kozyrakis and Ion Stoica and Joseph E. Gonzalez and Clark Barrett and Ying Sheng},
      year={2024},
Lianmin Zheng's avatar
Lianmin Zheng committed
416
417
418
419
420
421
      eprint={2312.07104},
      archivePrefix={arXiv},
      primaryClass={cs.AI}
}
```

422
We learned from the design and reused some code of the following projects: [Guidance](https://github.com/guidance-ai/guidance), [vLLM](https://github.com/vllm-project/vllm), [LightLLM](https://github.com/ModelTC/lightllm), [FlashInfer](https://github.com/flashinfer-ai/flashinfer), [Outlines](https://github.com/outlines-dev/outlines), [LMQL](https://github.com/eth-sri/lmql).