index.rst 2.76 KB
Newer Older
1
SGLang Documentation
2
====================
3

4
5
6
SGLang is a fast serving framework for large language models and vision language models.
It makes your interaction with models faster and more controllable by co-designing the backend runtime and frontend language.
The core features include:
7

8
- **Fast Backend Runtime**: Provides efficient serving with RadixAttention for prefix caching, zero-overhead CPU scheduler, continuous batching, token attention (paged attention), speculative decoding, tensor parallelism, chunked prefill, structured outputs, quantization (FP8/INT4/AWQ/GPTQ), and multi-lora batching.
9
- **Flexible Frontend Language**: Offers an intuitive interface for programming LLM applications, including chained generation calls, advanced prompting, control flow, multi-modal inputs, parallelism, and external interactions.
江家瑋's avatar
江家瑋 committed
10
- **Extensive Model Support**: Supports a wide range of generative models (Llama, Gemma, Mistral, Qwen, DeepSeek, LLaVA, etc.), embedding models (e5-mistral, gte, mcdse) and reward models (Skywork), with easy extensibility for integrating new models.
11
- **Active Community**: SGLang is open-source and backed by an active community with industry adoption.
12

13
14
.. toctree::
   :maxdepth: 1
15
   :caption: Installation
16

Lianmin Zheng's avatar
Lianmin Zheng committed
17
   start/install.md
Chayenne's avatar
Chayenne committed
18
19
20
21

.. toctree::
   :maxdepth: 1
   :caption: Backend Tutorial
Lianmin Zheng's avatar
Lianmin Zheng committed
22

Yineng Zhang's avatar
Yineng Zhang committed
23
   references/deepseek
24
   references/llama4
25
   backend/send_request.ipynb
26
27
   backend/openai_api_completions.ipynb
   backend/openai_api_vision.ipynb
Lianmin Zheng's avatar
Lianmin Zheng committed
28
   backend/openai_api_embeddings.ipynb
Chayenne's avatar
Chayenne committed
29
   backend/native_api.ipynb
Chayenne's avatar
Chayenne committed
30
   backend/offline_engine_api.ipynb
31
32
33
34
35

.. toctree::
   :maxdepth: 1
   :caption: Advanced Backend Configurations

36
37
38
   backend/server_arguments.md
   backend/sampling_params.md
   backend/hyperparameter_tuning.md
39
   backend/attention_backend.md
40

41
42
43
44
45
.. toctree::
   :maxdepth: 1
   :caption: Supported Models

   supported_models/generative_models.md
46
   supported_models/multimodal_language_models.md
47
48
49
50
   supported_models/embedding_models.md
   supported_models/reward_models.md
   supported_models/support_new_models.md

51
52
53
54
.. toctree::
   :maxdepth: 1
   :caption: Advanced Features

55
   backend/speculative_decoding.ipynb
56
   backend/structured_outputs.ipynb
57
   backend/function_calling.ipynb
Xihuai Wang's avatar
Xihuai Wang committed
58
   backend/separate_reasoning.ipynb
59
   backend/structured_outputs_for_reasoning_models.ipynb
60
61
   backend/custom_chat_template.md
   backend/quantization.md
62
   backend/lora.ipynb
63
   backend/pd_disaggregation.md
Chayenne's avatar
Chayenne committed
64
65
66
67

.. toctree::
   :maxdepth: 1
   :caption: Frontend Tutorial
Lianmin Zheng's avatar
Lianmin Zheng committed
68

69
   frontend/frontend.ipynb
Lianmin Zheng's avatar
Lianmin Zheng committed
70
   frontend/choices_methods.md
71

72
73
74
75
76
77
.. toctree::
   :maxdepth: 1
   :caption: SGLang Router

   router/router.md

78
.. toctree::
79
80
      :maxdepth: 1
      :caption: References
81

82
83
84
      references/general
      references/hardware
      references/advanced_deploy
85
      references/performance_analysis_and_optimization