README.md 14.8 KB
Newer Older
Lianmin Zheng's avatar
Lianmin Zheng committed
1
2
3
4
5
6
<div align="center">
<img src="assets/logo.png" alt="logo" width="400"></img>
</div>

--------------------------------------------------------------------------------

7
| [**Blog**](https://lmsys.org/blog/2024-01-17-sglang/) | [**Paper**](https://arxiv.org/abs/2312.07104) |
Lianmin Zheng's avatar
Lianmin Zheng committed
8
9
10
11

SGLang is a structured generation language designed for large language models (LLMs).
It makes your interaction with LLMs faster and more controllable by co-designing the frontend language and the runtime system.

12
The core features include:
Lianmin Zheng's avatar
Lianmin Zheng committed
13
- **Flexible Frontend Language**: Enables easy programming of LLM applications with chained generation calls, advanced prompting, control flow, multiple modalities, parallelism, and external interactions.
Lianmin Zheng's avatar
Lianmin Zheng committed
14
- **High-Performance Backend Runtime**: Features RadixAttention for accelerating complex LLM programs by reusing the KV cache across multiple calls. It can also serve as a standalone inference engine with all common techniques implemented (e.g., continuous batching and tensor parallelism).
Lianmin Zheng's avatar
Lianmin Zheng committed
15

Ying Sheng's avatar
Ying Sheng committed
16
## News
Lianmin Zheng's avatar
Lianmin Zheng committed
17
- [2024/02] 🔥 SGLang enables **3x faster JSON decoding** with compressed finite state machine ([blog](https://lmsys.org/blog/2024-02-05-compressed-fsm/)).
18
- [2024/01] 🔥 SGLang powers the serving of the official **LLaVA v1.6** release demo ([usage](https://github.com/haotian-liu/LLaVA?tab=readme-ov-file#demo)).
Lianmin Zheng's avatar
Lianmin Zheng committed
19
- [2024/01] SGLang provides up to **5x faster inference** with RadixAttention ([blog](https://lmsys.org/blog/2024-01-17-sglang/)).
Ying Sheng's avatar
Ying Sheng committed
20

Lianmin Zheng's avatar
Lianmin Zheng committed
21
22
23
## Contents
- [Install](#install)
- [Quick Start](#quick-start)
24
- [Frontend: Structured Generation Language (SGLang)](#frontend-structured-generation-language-sglang)
Lianmin Zheng's avatar
Lianmin Zheng committed
25
26
27
28
29
30
31
- [Backend: SGLang Runtime (SRT)](#backend-sglang-runtime-srt)
- [Benchmark And Performance](#benchmark-and-performance)
- [Roadmap](#roadmap)
- [Citation And Acknowledgment](#citation-and-acknowledgment)

## Install

Lianmin Zheng's avatar
Lianmin Zheng committed
32
33
34
### Method 1: With pip
```
pip install "sglang[all]"
Lianmin Zheng's avatar
Lianmin Zheng committed
35

Lianmin Zheng's avatar
Lianmin Zheng committed
36
37
38
# Install FlashInfer CUDA kernels
pip install flashinfer -i https://flashinfer.ai/whl/cu121/torch2.3/
```
39

Lianmin Zheng's avatar
Lianmin Zheng committed
40
### Method 2: From source
Lianmin Zheng's avatar
Lianmin Zheng committed
41
```
42
git clone https://github.com/sgl-project/sglang.git
Lianmin Zheng's avatar
Lianmin Zheng committed
43
44
45
46
47
cd sglang

pip install --upgrade pip
pip install -e "python[all]"

Lianmin Zheng's avatar
Lianmin Zheng committed
48
49
50
# Install FlashInfer CUDA kernels
pip install flashinfer -i https://flashinfer.ai/whl/cu121/torch2.3/
```
51

Ying Sheng's avatar
Ying Sheng committed
52
### Notes
Lianmin Zheng's avatar
Lianmin Zheng committed
53
54
55
- If you see errors from the Triton compiler, please install the [Triton Nightly](https://triton-lang.org/main/getting-started/installation.html).
- If you cannot install FlashInfer, check out its [installation](https://docs.flashinfer.ai/installation.html#) page. If you still cannot install it, you can use the slower Triton kernels by adding `--disable-flashinfer` when launching the server.
- If you only need to use the OpenAI backend, you can avoid installing other dependencies by using `pip install "sglang[openai]"`.
Ying Sheng's avatar
Ying Sheng committed
56

Lianmin Zheng's avatar
Lianmin Zheng committed
57
58
59
## Quick Start
The example below shows how to use sglang to answer a mulit-turn question.

60
61
### Using Local Models
First, launch a server with
Lianmin Zheng's avatar
Lianmin Zheng committed
62
```
63
python -m sglang.launch_server --model-path meta-llama/Llama-2-7b-chat-hf --port 30000
Lianmin Zheng's avatar
Lianmin Zheng committed
64
65
```

66
67
Then, connect to the server and answer a multi-turn question.

Lianmin Zheng's avatar
Lianmin Zheng committed
68
```python
69
from sglang import function, system, user, assistant, gen, set_default_backend, RuntimeEndpoint
Lianmin Zheng's avatar
Lianmin Zheng committed
70
71
72
73
74
75
76
77
78

@function
def multi_turn_question(s, question_1, question_2):
    s += system("You are a helpful assistant.")
    s += user(question_1)
    s += assistant(gen("answer_1", max_tokens=256))
    s += user(question_2)
    s += assistant(gen("answer_2", max_tokens=256))

79
set_default_backend(RuntimeEndpoint("http://localhost:30000"))
Lianmin Zheng's avatar
Lianmin Zheng committed
80
81
82
83
84
85
86
87

state = multi_turn_question.run(
    question_1="What is the capital of the United States?",
    question_2="List two local attractions.",
)

for m in state.messages():
    print(m["role"], ":", m["content"])
88
89

print(state["answer_1"])
Lianmin Zheng's avatar
Lianmin Zheng committed
90
91
```

92
93
### Using OpenAI Models
Set the OpenAI API Key
Lianmin Zheng's avatar
Lianmin Zheng committed
94
```
95
export OPENAI_API_KEY=sk-******
Lianmin Zheng's avatar
Lianmin Zheng committed
96
97
```

98
Then, answer a multi-turn question.
Lianmin Zheng's avatar
Lianmin Zheng committed
99
```python
100
from sglang import function, system, user, assistant, gen, set_default_backend, OpenAI
Lianmin Zheng's avatar
Lianmin Zheng committed
101
102
103
104
105
106
107
108
109

@function
def multi_turn_question(s, question_1, question_2):
    s += system("You are a helpful assistant.")
    s += user(question_1)
    s += assistant(gen("answer_1", max_tokens=256))
    s += user(question_2)
    s += assistant(gen("answer_2", max_tokens=256))

110
set_default_backend(OpenAI("gpt-3.5-turbo"))
Lianmin Zheng's avatar
Lianmin Zheng committed
111
112
113
114
115
116
117
118

state = multi_turn_question.run(
    question_1="What is the capital of the United States?",
    question_2="List two local attractions.",
)

for m in state.messages():
    print(m["role"], ":", m["content"])
119
120

print(state["answer_1"])
Lianmin Zheng's avatar
Lianmin Zheng committed
121
122
123
124
```

### More Examples

125
Anthropic and VertexAI (Gemini) models are also supported.
Lianmin Zheng's avatar
Lianmin Zheng committed
126
127
You can find more examples at [examples/quick_start](examples/quick_start).

128
## Frontend: Structured Generation Language (SGLang)
Lianmin Zheng's avatar
Lianmin Zheng committed
129

Lianmin Zheng's avatar
Lianmin Zheng committed
130
131
132
133
134
To begin with, import sglang.
```python
import sglang as sgl
```

Lianmin Zheng's avatar
Lianmin Zheng committed
135
`sglang` provides some simple primitives such as `gen`, `select`, `fork`, `image`.
Lianmin Zheng's avatar
Lianmin Zheng committed
136
137
You can implement your prompt flow in a function decorated by `sgl.function`.
You can then invoke the function with `run` or `run_batch`.
138
The system will manage the state, chat template, parallelism and batching for you.
Lianmin Zheng's avatar
Lianmin Zheng committed
139

140
141
The complete code for the examples below can be found at [readme_examples.py](examples/usage/readme_examples.py)

Lianmin Zheng's avatar
Lianmin Zheng committed
142
### Control Flow
Lianmin Zheng's avatar
Lianmin Zheng committed
143
144
You can use any Python code within the function body, including control flow, nested function calls, and external libraries.

Lianmin Zheng's avatar
Lianmin Zheng committed
145
146
```python
@sgl.function
147
148
149
def tool_use(s, question):
    s += "To answer this question: " + question + ". "
    s += "I need to use a " + sgl.gen("tool", choices=["calculator", "search engine"]) + ". "
Lianmin Zheng's avatar
Lianmin Zheng committed
150
151
152

    if s["tool"] == "calculator":
        s += "The math expression is" + sgl.gen("expression")
153
154
    elif s["tool"] == "search engine":
        s += "The key word to search is" + sgl.gen("word")
Lianmin Zheng's avatar
Lianmin Zheng committed
155
```
Lianmin Zheng's avatar
Lianmin Zheng committed
156
157

### Parallelism
Lianmin Zheng's avatar
Lianmin Zheng committed
158
159
160
Use `fork` to launch parallel prompts.
Because `sgl.gen` is non-blocking, the for loop below issues two generation calls in parallel.

Lianmin Zheng's avatar
Lianmin Zheng committed
161
162
163
164
165
166
167
168
```python
@sgl.function
def tip_suggestion(s):
    s += (
        "Here are two tips for staying healthy: "
        "1. Balanced Diet. 2. Regular Exercise.\n\n"
    )

Lianmin Zheng's avatar
Lianmin Zheng committed
169
    forks = s.fork(2)
Lianmin Zheng's avatar
Lianmin Zheng committed
170
171
172
173
174
175
176
177
    for i, f in enumerate(forks):
        f += f"Now, expand tip {i+1} into a paragraph:\n"
        f += sgl.gen(f"detailed_tip", max_tokens=256, stop="\n\n")

    s += "Tip 1:" + forks[0]["detailed_tip"] + "\n"
    s += "Tip 2:" + forks[1]["detailed_tip"] + "\n"
    s += "In summary" + sgl.gen("summary")
```
Lianmin Zheng's avatar
Lianmin Zheng committed
178
179

### Multi Modality
Lianmin Zheng's avatar
Lianmin Zheng committed
180
181
Use `sgl.image` to pass an image as input.

Lianmin Zheng's avatar
Lianmin Zheng committed
182
183
```python
@sgl.function
Lianmin Zheng's avatar
Lianmin Zheng committed
184
def image_qa(s, image_file, question):
Lianmin Zheng's avatar
Lianmin Zheng committed
185
    s += sgl.user(sgl.image(image_file) + question)
Lianmin Zheng's avatar
Lianmin Zheng committed
186
    s += sgl.assistant(sgl.gen("answer", max_tokens=256)
Lianmin Zheng's avatar
Lianmin Zheng committed
187
188
```

189
190
See also [srt_example_llava.py](examples/quick_start/srt_example_llava.py).

Lianmin Zheng's avatar
Lianmin Zheng committed
191
### Constrained Decoding
192
193
Use `regex` to specify a regular expression as a decoding constraint.
This is only supported for local models.
Lianmin Zheng's avatar
Lianmin Zheng committed
194

Lianmin Zheng's avatar
Lianmin Zheng committed
195
```python
Lianmin Zheng's avatar
Lianmin Zheng committed
196
@sgl.function
Lianmin Zheng's avatar
Lianmin Zheng committed
197
198
def regular_expression_gen(s):
    s += "Q: What is the IP address of the Google DNS servers?\n"
Lianmin Zheng's avatar
Lianmin Zheng committed
199
    s += "A: " + sgl.gen(
Lianmin Zheng's avatar
Lianmin Zheng committed
200
201
202
203
204
        "answer",
        temperature=0,
        regex=r"((25[0-5]|2[0-4]\d|[01]?\d\d?).){3}(25[0-5]|2[0-4]\d|[01]?\d\d?)",
    )
```
Lianmin Zheng's avatar
Lianmin Zheng committed
205

206
### JSON Decoding
Lianmin Zheng's avatar
Lianmin Zheng committed
207
Use `regex` to specify a JSON schema with a regular expression.
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228

```python
character_regex = (
    r"""\{\n"""
    + r"""    "name": "[\w\d\s]{1,16}",\n"""
    + r"""    "house": "(Gryffindor|Slytherin|Ravenclaw|Hufflepuff)",\n"""
    + r"""    "blood status": "(Pure-blood|Half-blood|Muggle-born)",\n"""
    + r"""    "occupation": "(student|teacher|auror|ministry of magic|death eater|order of the phoenix)",\n"""
    + r"""    "wand": \{\n"""
    + r"""        "wood": "[\w\d\s]{1,16}",\n"""
    + r"""        "core": "[\w\d\s]{1,16}",\n"""
    + r"""        "length": [0-9]{1,2}\.[0-9]{0,2}\n"""
    + r"""    \},\n"""
    + r"""    "alive": "(Alive|Deceased)",\n"""
    + r"""    "patronus": "[\w\d\s]{1,16}",\n"""
    + r"""    "bogart": "[\w\d\s]{1,16}"\n"""
    + r"""\}"""
)

@sgl.function
def character_gen(s, name):
Lianmin Zheng's avatar
Lianmin Zheng committed
229
    s += name + " is a character in Harry Potter. Please fill in the following information about this character.\n"
230
231
232
    s += sgl.gen("json_output", max_tokens=256, regex=character_regex)
```

Lianmin Zheng's avatar
Lianmin Zheng committed
233
See also [json_decode.py](examples/usage/json_decode.py) for an additional example on specifying formats with Pydantic models.
234
235


Lianmin Zheng's avatar
Lianmin Zheng committed
236
### Batching
Lianmin Zheng's avatar
Lianmin Zheng committed
237
238
Use `run_batch` to run a batch of requests with continuous batching.

Lianmin Zheng's avatar
Lianmin Zheng committed
239
240
241
242
243
244
245
246
247
248
249
250
```python
@sgl.function
def text_qa(s, question):
    s += "Q: " + question + "\n"
    s += "A:" + sgl.gen("answer", stop="\n")

states = text_qa.run_batch(
    [
        {"question": "What is the capital of the United Kingdom?"},
        {"question": "What is the capital of France?"},
        {"question": "What is the capital of Japan?"},
    ],
Lianmin Zheng's avatar
Lianmin Zheng committed
251
    progress_bar=True
Lianmin Zheng's avatar
Lianmin Zheng committed
252
253
)
```
Lianmin Zheng's avatar
Lianmin Zheng committed
254
255

### Streaming
Lianmin Zheng's avatar
Lianmin Zheng committed
256
257
Add `stream=True` to enable streaming.

Lianmin Zheng's avatar
Lianmin Zheng committed
258
259
260
261
262
263
```python
@sgl.function
def text_qa(s, question):
    s += "Q: " + question + "\n"
    s += "A:" + sgl.gen("answer", stop="\n")

264
state = text_qa.run(
Lianmin Zheng's avatar
Lianmin Zheng committed
265
    question="What is the capital of France?",
Lianmin Zheng's avatar
Lianmin Zheng committed
266
267
268
    temperature=0.1,
    stream=True
)
Lianmin Zheng's avatar
Lianmin Zheng committed
269

Lianmin Zheng's avatar
Lianmin Zheng committed
270
271
272
for out in state.text_iter():
    print(out, end="", flush=True)
```
Lianmin Zheng's avatar
Lianmin Zheng committed
273

Lianmin Zheng's avatar
Lianmin Zheng committed
274
275
276
277
### Tips and Implementation Details
- The `choices` argument in `sgl.gen` is implemented by computing the normalized log probabilities of all choices and selecting the one with the highest probability.
- The `regex` argument in `sgl.gen` is implemented through autoregressive decoding with logit bias masking, according to the constraints set by the regex.

Lianmin Zheng's avatar
Lianmin Zheng committed
278
279
280
## Backend: SGLang Runtime (SRT)
The SGLang Runtime (SRT) is designed to work best with the SGLang frontend.
However, it can also be used as a standalone API server.
Ying Sheng's avatar
Ying Sheng committed
281
In this case, the [RadixAttention](https://arxiv.org/abs/2312.07104) can still greatly accelerate many use cases with automatic KV cache reuse.
Lianmin Zheng's avatar
Lianmin Zheng committed
282
283
284
285
286
287
288
289
290

### Usage
Launch a server
```
python -m sglang.launch_server --model-path meta-llama/Llama-2-7b-chat-hf --port 30000
```

Send a request
```
291
curl http://localhost:30000/generate \
Lianmin Zheng's avatar
Lianmin Zheng committed
292
293
  -H "Content-Type: application/json" \
  -d '{
294
    "text": "Once upon a time,",
295
    "sampling_params": {
296
297
298
      "max_new_tokens": 16,
      "temperature": 0
    }
Lianmin Zheng's avatar
Lianmin Zheng committed
299
300
  }'
```
301
302
Learn more about the argument format [here](docs/sampling_params.md).

303
304
305
306
307
308
309
### OpenAI Compatible API
In addition, the server supports an experimental OpenAI-compatible API.

```python
import openai
client = openai.Client(
    base_url="http://127.0.0.1:30000/v1", api_key="EMPTY")
Cody Yu's avatar
Cody Yu committed
310
311

# Text completion
312
313
314
315
316
317
318
response = client.completions.create(
	model="default",
	prompt="The capital of France is",
	temperature=0,
	max_tokens=32,
)
print(response)
Cody Yu's avatar
Cody Yu committed
319
320
321
322
323
324
325
326
327
328
329
330
331
332

# Chat completion
response = client.chat.completions.create(
    model="default",
    messages=[
        {"role": "system", "content": "You are a helpful AI assistant"},
        {"role": "user", "content": "List 3 countries and their capitals."},
    ],
    temperature=0,
    max_tokens=64,
)
print(response)
```

Lianmin Zheng's avatar
Lianmin Zheng committed
333
334
335
336

By default, the server uses the chat template specified in the model tokenizer from Hugging Face. It should just work for most official models such as Llama-2/Llama-3.

If needed, you can also override the chat template when launching the server:
Cody Yu's avatar
Cody Yu committed
337
338

```
339
python -m sglang.launch_server --model-path meta-llama/Llama-2-7b-chat-hf --port 30000 --chat-template llama-2
Cody Yu's avatar
Cody Yu committed
340
341
342
```

If the chat template you are looking for is missing, you are welcome to contribute it.
Lianmin Zheng's avatar
Lianmin Zheng committed
343
Meanwhile, you can also temporarily register your chat template as follows:
Cody Yu's avatar
Cody Yu committed
344
345
346
347
348
349
350
351
352
353
354
355
356
357

```json
{
  "name": "my_model",
  "system": "<|im_start|>system",
  "user": "<|im_start|>user",
  "assistant": "<|im_start|>assistant",
  "sep_style": "CHATML",
  "sep": "<|im_end|>",
  "stop_str": ["<|im_end|>", "<|im_start|>"]
}
```

```
358
python -m sglang.launch_server --model-path meta-llama/Llama-2-7b-chat-hf --port 30000 --chat-template ./my_model_template.json
359
360
```

Lianmin Zheng's avatar
Lianmin Zheng committed
361
362
363
364
365
### Additional Arguments
- Add `--tp 2` to enable tensor parallelism.
```
python -m sglang.launch_server --model-path meta-llama/Llama-2-7b-chat-hf --port 30000 --tp 2
```
Lianmin Zheng's avatar
Lianmin Zheng committed
366
367
368
369
- Add `--dp 2` to enable data parallelism. It can also be used together with tp. Data parallelism is better for throughput if there is enough memory.
```
python -m sglang.launch_server --model-path meta-llama/Llama-2-7b-chat-hf --port 30000 --dp 2 --tp 2
```
Ying Sheng's avatar
Ying Sheng committed
370
371
372
373
- If you see out-of-memory errors during serving, please try to reduce the memory usage of the KV cache pool by setting a smaller value of `--mem-fraction-static`. The default value is `0.9`
```
python -m sglang.launch_server --model-path meta-llama/Llama-2-7b-chat-hf --port 30000 --mem-fraction-static 0.7
```
Lianmin Zheng's avatar
Lianmin Zheng committed
374
- See [hyperparameter_tuning.md](docs/hyperparameter_tuning.md) on tuning hyperparameters for better performance.
Lianmin Zheng's avatar
Lianmin Zheng committed
375
376
377
378
379

### Supported Models
- Llama
- Mistral
- Mixtral
Lianmin Zheng's avatar
Lianmin Zheng committed
380
- Qwen / Qwen 2
381
382
383
- Gemma
  - Please add a new flag `--attention-reduce-in-fp32` to avoid some precision errors.
  - `python -m sglang.launch_server --model-path google/gemma-7b-it --port 30000 --attention-reduce-in-fp32`
Lianmin Zheng's avatar
Lianmin Zheng committed
384
- LLaVA
385
  - `python3 -m sglang.launch_server --model-path liuhaotian/llava-v1.5-7b --tokenizer-path llava-hf/llava-1.5-7b-hf --chat-template vicuna_v1.1 --port 30000`
386
387
  - `python3 -m sglang.launch_server --model-path liuhaotian/llava-v1.6-vicuna-7b --tokenizer-path llava-hf/llava-1.5-7b-hf --chat-template vicuna_v1.1 --port 30000`
  - `python3 -m sglang.launch_server --model-path liuhaotian/llava-v1.6-34b --tokenizer-path liuhaotian/llava-v1.6-34b-tokenizer --port 3000`
388
389
- LLaVA-NeXT-Video
  - see [srt_example_llava_v.sh](examples/usage/llava_video/srt_example_llava_v.sh)
Lianmin Zheng's avatar
Lianmin Zheng committed
390
391
- Yi-VL
  - see [srt_example_yi_vl.py](examples/quick_start/srt_example_yi_vl.py).
392
393
394
395
396
397
- StableLM
- Command-R
- DBRX
- AWQ/GPTQ/Marlin quantization

Instructions for supporting a new model are [here](https://github.com/sgl-project/sglang/blob/main/docs/model_support.md).
Lianmin Zheng's avatar
Lianmin Zheng committed
398
399

## Benchmark And Performance
Lianmin Zheng's avatar
Lianmin Zheng committed
400
401
402
403
404
405
- Llama-7B on NVIDIA A10G, FP16, Tensor Parallelism=1
![llama_7b](assets/llama_7b.jpg)

- Mixtral-8x7B on NVIDIA A10G, FP16, Tensor Parallelism=8
![mixtral_8x7b](assets/mixtral_8x7b.jpg)

Lianmin Zheng's avatar
Lianmin Zheng committed
406
407
- Learn more about the above [results](docs/benchmark_results.md).
- Synthetic latency and throughput benchmark [scripts](https://github.com/sgl-project/sglang/tree/main/benchmark/latency_throughput).
Lianmin Zheng's avatar
Lianmin Zheng committed
408

Lianmin Zheng's avatar
Lianmin Zheng committed
409
## Roadmap
Ying Sheng's avatar
Ying Sheng committed
410
https://github.com/sgl-project/sglang/issues/157
Lianmin Zheng's avatar
Lianmin Zheng committed
411
412
413

## Citation And Acknowledgment
```
Lianmin Zheng's avatar
Lianmin Zheng committed
414
415
416
417
@misc{zheng2024sglang,
      title={SGLang: Efficient Execution of Structured Language Model Programs},
      author={Lianmin Zheng and Liangsheng Yin and Zhiqiang Xie and Chuyue Sun and Jeff Huang and Cody Hao Yu and Shiyi Cao and Christos Kozyrakis and Ion Stoica and Joseph E. Gonzalez and Clark Barrett and Ying Sheng},
      year={2024},
Lianmin Zheng's avatar
Lianmin Zheng committed
418
419
420
421
422
423
      eprint={2312.07104},
      archivePrefix={arXiv},
      primaryClass={cs.AI}
}
```

424
We learned from the design and reused some code of the following projects: [Guidance](https://github.com/guidance-ai/guidance), [vLLM](https://github.com/vllm-project/vllm), [LightLLM](https://github.com/ModelTC/lightllm), [FlashInfer](https://github.com/flashinfer-ai/flashinfer), [Outlines](https://github.com/outlines-dev/outlines), [LMQL](https://github.com/eth-sri/lmql).