offline_engine_api.ipynb 7.27 KB
Newer Older
Chayenne's avatar
Chayenne committed
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Offline Engine API\n",
    "\n",
    "SGLang provides a direct inference engine without the need for an HTTP server, especially for use cases where additional HTTP server adds unnecessary complexity or overhead. Here are two general use cases:\n",
    "\n",
    "- Offline Batch Inference\n",
    "- Custom Server on Top of the Engine\n",
    "\n",
    "This document focuses on the offline batch inference, demonstrating four different inference modes:\n",
    "\n",
    "- Non-streaming synchronous generation\n",
    "- Streaming synchronous generation\n",
    "- Non-streaming asynchronous generation\n",
    "- Streaming asynchronous generation\n",
    "\n",
21
    "Additionally, you can easily build a custom server on top of the SGLang offline engine. A detailed example working in a python script can be found in [custom_server](https://github.com/sgl-project/sglang/blob/main/examples/runtime/engine/custom_server.py).\n",
22
    "\n",
23
    "## SPECIAL WARNING!!!!\n",
24
    "\n",
25
26
27
    "**To launch the offline engine in your python scripts,** `__main__` **condition is necessary, since we use** `spawn` **mode to create subprocesses. Please refer to this simple example**:\n",
    "\n",
    "https://github.com/sgl-project/sglang/blob/main/examples/runtime/engine/launch_engine.py"
Chayenne's avatar
Chayenne committed
28
29
   ]
  },
30
31
32
33
34
35
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Advanced Usage\n",
    "\n",
36
    "The engine supports [vlm inference](https://github.com/sgl-project/sglang/blob/main/examples/runtime/engine/offline_batch_inference_vlm.py) as well as [extracting hidden states](https://github.com/sgl-project/sglang/blob/main/examples/runtime/hidden_states). \n",
37
38
39
40
    "\n",
    "Please see [the examples](https://github.com/sgl-project/sglang/tree/main/examples/runtime/engine) for further use cases."
   ]
  },
Chayenne's avatar
Chayenne committed
41
42
43
44
45
46
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Offline Batch Inference\n",
    "\n",
47
    "SGLang offline engine supports batch inference with efficient scheduling."
Chayenne's avatar
Chayenne committed
48
49
50
51
52
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
53
   "metadata": {},
Chayenne's avatar
Chayenne committed
54
55
56
57
   "outputs": [],
   "source": [
    "# launch the offline engine\n",
    "import asyncio\n",
58
59
60
61
62
63
64
65
    "import io\n",
    "import os\n",
    "\n",
    "from PIL import Image\n",
    "import requests\n",
    "import sglang as sgl\n",
    "\n",
    "from sglang.srt.conversation import chat_templates\n",
66
    "from sglang.test.test_utils import is_in_ci\n",
67
    "from sglang.utils import async_stream_and_merge, stream_and_merge\n",
68
69
70
71
    "\n",
    "if is_in_ci():\n",
    "    import patch\n",
    "\n",
72
    "\n",
Chayenne's avatar
Chayenne committed
73
74
75
76
77
78
79
80
81
82
83
84
85
    "llm = sgl.Engine(model_path=\"meta-llama/Meta-Llama-3.1-8B-Instruct\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Non-streaming Synchronous Generation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
86
   "metadata": {},
Chayenne's avatar
Chayenne committed
87
88
89
90
91
92
93
94
95
96
97
98
99
   "outputs": [],
   "source": [
    "prompts = [\n",
    "    \"Hello, my name is\",\n",
    "    \"The president of the United States is\",\n",
    "    \"The capital of France is\",\n",
    "    \"The future of AI is\",\n",
    "]\n",
    "\n",
    "sampling_params = {\"temperature\": 0.8, \"top_p\": 0.95}\n",
    "\n",
    "outputs = llm.generate(prompts, sampling_params)\n",
    "for prompt, output in zip(prompts, outputs):\n",
Lianmin Zheng's avatar
Lianmin Zheng committed
100
101
    "    print(\"===============================\")\n",
    "    print(f\"Prompt: {prompt}\\nGenerated text: {output['text']}\")"
Chayenne's avatar
Chayenne committed
102
103
104
105
106
107
108
109
110
111
112
113
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Streaming Synchronous Generation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
114
   "metadata": {},
Chayenne's avatar
Chayenne committed
115
116
117
   "outputs": [],
   "source": [
    "prompts = [\n",
118
119
120
    "    \"Write a short, neutral self-introduction for a fictional character. Hello, my name is\",\n",
    "    \"Provide a concise factual statement about France’s capital city. The capital of France is\",\n",
    "    \"Explain possible future trends in artificial intelligence. The future of AI is\",\n",
Chayenne's avatar
Chayenne committed
121
122
    "]\n",
    "\n",
123
124
125
126
    "sampling_params = {\n",
    "    \"temperature\": 0.2,\n",
    "    \"top_p\": 0.9,\n",
    "}\n",
Chayenne's avatar
Chayenne committed
127
    "\n",
128
    "print(\"\\n=== Testing synchronous streaming generation with overlap removal ===\\n\")\n",
Chayenne's avatar
Chayenne committed
129
    "\n",
130
131
132
133
    "for prompt in prompts:\n",
    "    print(f\"Prompt: {prompt}\")\n",
    "    merged_output = stream_and_merge(llm, prompt, sampling_params)\n",
    "    print(\"Generated text:\", merged_output)\n",
Chayenne's avatar
Chayenne committed
134
135
136
137
138
139
140
141
142
143
144
145
146
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Non-streaming Asynchronous Generation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
147
   "metadata": {},
Chayenne's avatar
Chayenne committed
148
149
150
   "outputs": [],
   "source": [
    "prompts = [\n",
151
152
153
    "    \"Write a short, neutral self-introduction for a fictional character. Hello, my name is\",\n",
    "    \"Provide a concise factual statement about France’s capital city. The capital of France is\",\n",
    "    \"Explain possible future trends in artificial intelligence. The future of AI is\",\n",
Chayenne's avatar
Chayenne committed
154
155
156
157
    "]\n",
    "\n",
    "sampling_params = {\"temperature\": 0.8, \"top_p\": 0.95}\n",
    "\n",
Lianmin Zheng's avatar
Lianmin Zheng committed
158
    "print(\"\\n=== Testing asynchronous batch generation ===\")\n",
Chayenne's avatar
Chayenne committed
159
160
161
162
163
164
    "\n",
    "\n",
    "async def main():\n",
    "    outputs = await llm.async_generate(prompts, sampling_params)\n",
    "\n",
    "    for prompt, output in zip(prompts, outputs):\n",
Lianmin Zheng's avatar
Lianmin Zheng committed
165
166
    "        print(f\"\\nPrompt: {prompt}\")\n",
    "        print(f\"Generated text: {output['text']}\")\n",
Chayenne's avatar
Chayenne committed
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
    "\n",
    "\n",
    "asyncio.run(main())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Streaming Asynchronous Generation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
182
   "metadata": {},
Chayenne's avatar
Chayenne committed
183
184
185
   "outputs": [],
   "source": [
    "prompts = [\n",
186
187
188
    "    \"Write a short, neutral self-introduction for a fictional character. Hello, my name is\",\n",
    "    \"Provide a concise factual statement about France’s capital city. The capital of France is\",\n",
    "    \"Explain possible future trends in artificial intelligence. The future of AI is\",\n",
Chayenne's avatar
Chayenne committed
189
    "]\n",
190
    "\n",
Chayenne's avatar
Chayenne committed
191
192
    "sampling_params = {\"temperature\": 0.8, \"top_p\": 0.95}\n",
    "\n",
193
    "print(\"\\n=== Testing asynchronous streaming generation (no repeats) ===\")\n",
Chayenne's avatar
Chayenne committed
194
195
196
197
    "\n",
    "\n",
    "async def main():\n",
    "    for prompt in prompts:\n",
Lianmin Zheng's avatar
Lianmin Zheng committed
198
    "        print(f\"\\nPrompt: {prompt}\")\n",
Chayenne's avatar
Chayenne committed
199
200
    "        print(\"Generated text: \", end=\"\", flush=True)\n",
    "\n",
201
202
203
204
205
    "        # Replace direct calls to async_generate with our custom overlap-aware version\n",
    "        async for cleaned_chunk in async_stream_and_merge(llm, prompt, sampling_params):\n",
    "            print(cleaned_chunk, end=\"\", flush=True)\n",
    "\n",
    "        print()  # New line after each prompt\n",
Chayenne's avatar
Chayenne committed
206
207
208
209
210
211
212
    "\n",
    "\n",
    "asyncio.run(main())"
   ]
  },
  {
   "cell_type": "code",
213
214
   "execution_count": null,
   "metadata": {},
Chayenne's avatar
Chayenne committed
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
   "outputs": [],
   "source": [
    "llm.shutdown()"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
231
   "pygments_lexer": "ipython3"
Chayenne's avatar
Chayenne committed
232
233
234
235
236
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}