native_api.ipynb 14.9 KB
Newer Older
Chayenne's avatar
Chayenne committed
1
2
3
4
5
6
{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
7
    "# SGLang Native APIs\n",
Chayenne's avatar
Chayenne committed
8
    "\n",
Chayenne's avatar
Chayenne committed
9
    "Apart from the OpenAI compatible APIs, the SGLang Runtime also provides its native server APIs. We introduce these following APIs:\n",
Chayenne's avatar
Chayenne committed
10
    "\n",
Chayenne's avatar
Chayenne committed
11
    "- `/generate` (text generation model)\n",
Chayenne's avatar
Chayenne committed
12
    "- `/get_model_info`\n",
13
    "- `/get_server_info`\n",
Chayenne's avatar
Chayenne committed
14
15
16
    "- `/health`\n",
    "- `/health_generate`\n",
    "- `/flush_cache`\n",
Chayenne's avatar
Chayenne committed
17
    "- `/update_weights`\n",
Chayenne's avatar
Chayenne committed
18
    "- `/encode`(embedding model)\n",
woodx's avatar
woodx committed
19
    "- `/v1/rerank`(cross encoder rerank model)\n",
20
    "- `/classify`(reward model)\n",
21
22
23
    "- `/start_expert_distribution_record`\n",
    "- `/stop_expert_distribution_record`\n",
    "- `/dump_expert_distribution_record`\n",
Chayenne's avatar
Chayenne committed
24
25
26
27
28
29
30
31
32
33
34
35
36
37
    "\n",
    "We mainly use `requests` to test these APIs in the following examples. You can also use `curl`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Launch A Server"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
38
   "metadata": {},
Chayenne's avatar
Chayenne committed
39
40
   "outputs": [],
   "source": [
Chayenne's avatar
Chayenne committed
41
    "import requests\n",
42
    "from sglang.test.test_utils import is_in_ci\n",
Chayenne's avatar
Chayenne committed
43
    "\n",
44
45
46
47
48
49
50
51
52
    "if is_in_ci():\n",
    "    from patch import launch_server_cmd\n",
    "else:\n",
    "    from sglang.utils import launch_server_cmd\n",
    "\n",
    "from sglang.utils import wait_for_server, print_highlight, terminate_process\n",
    "\n",
    "\n",
    "server_process, port = launch_server_cmd(\n",
53
    "    \"python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct --host 0.0.0.0\"\n",
Chayenne's avatar
Chayenne committed
54
    ")\n",
55
56
57
58
    "## To run qwen2.5-0.5b-instruct model on the Ascend-Npu, you can execute the following command:\n",
    "# server_process, port = launch_server_cmd(\n",
    "#     \"python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct --host 0.0.0.0 --device npu --tp 2 --attention-backend torch_native\"\n",
    "# )\n",
Chayenne's avatar
Chayenne committed
59
    "\n",
60
    "wait_for_server(f\"http://localhost:{port}\")"
Chayenne's avatar
Chayenne committed
61
62
63
64
65
66
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
Chayenne's avatar
Chayenne committed
67
    "## Generate (text generation model)\n",
68
    "Generate completions. This is similar to the `/v1/completions` in OpenAI API. Detailed parameters can be found in the [sampling parameters](./sampling_params.md)."
Chayenne's avatar
Chayenne committed
69
70
71
72
73
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
74
   "metadata": {},
Chayenne's avatar
Chayenne committed
75
76
   "outputs": [],
   "source": [
77
    "url = f\"http://localhost:{port}/generate\"\n",
Chayenne's avatar
Chayenne committed
78
    "data = {\"text\": \"What is the capital of France?\"}\n",
Chayenne's avatar
Chayenne committed
79
80
    "\n",
    "response = requests.post(url, json=data)\n",
Lianmin Zheng's avatar
Lianmin Zheng committed
81
    "print_highlight(response.json())"
Chayenne's avatar
Chayenne committed
82
83
   ]
  },
84
85
86
87
88
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
Chayenne's avatar
Chayenne committed
89
90
91
92
93
94
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get Model Info\n",
    "\n",
Lianmin Zheng's avatar
Lianmin Zheng committed
95
    "Get the information of the model.\n",
Chayenne's avatar
Chayenne committed
96
97
    "\n",
    "- `model_path`: The path/name of the model.\n",
Chayenne's avatar
Chayenne committed
98
99
    "- `is_generation`: Whether the model is used as generation model or embedding model.\n",
    "- `tokenizer_path`: The path/name of the tokenizer."
Chayenne's avatar
Chayenne committed
100
101
102
103
104
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
105
   "metadata": {},
Chayenne's avatar
Chayenne committed
106
107
   "outputs": [],
   "source": [
108
    "url = f\"http://localhost:{port}/get_model_info\"\n",
Chayenne's avatar
Chayenne committed
109
110
111
112
    "\n",
    "response = requests.get(url)\n",
    "response_json = response.json()\n",
    "print_highlight(response_json)\n",
113
    "assert response_json[\"model_path\"] == \"qwen/qwen2.5-0.5b-instruct\"\n",
Lianmin Zheng's avatar
Lianmin Zheng committed
114
    "assert response_json[\"is_generation\"] is True\n",
115
    "assert response_json[\"tokenizer_path\"] == \"qwen/qwen2.5-0.5b-instruct\"\n",
Chayenne's avatar
Chayenne committed
116
    "assert response_json.keys() == {\"model_path\", \"is_generation\", \"tokenizer_path\"}"
Chayenne's avatar
Chayenne committed
117
118
119
120
121
122
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
123
124
125
126
127
128
    "## Get Server Info\n",
    "Gets the server information including CLI arguments, token limits, and memory pool sizes.\n",
    "- Note: `get_server_info` merges the following deprecated endpoints:\n",
    "  - `get_server_args`\n",
    "  - `get_memory_pool_size` \n",
    "  - `get_max_total_num_tokens`"
Chayenne's avatar
Chayenne committed
129
130
131
132
133
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
134
   "metadata": {},
Chayenne's avatar
Chayenne committed
135
136
   "outputs": [],
   "source": [
137
    "# get_server_info\n",
Chayenne's avatar
Chayenne committed
138
    "\n",
139
    "url = f\"http://localhost:{port}/get_server_info\"\n",
Chayenne's avatar
Chayenne committed
140
141
142
143
144
145
146
147
148
    "\n",
    "response = requests.get(url)\n",
    "print_highlight(response.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
149
150
151
    "## Health Check\n",
    "- `/health`: Check the health of the server.\n",
    "- `/health_generate`: Check the health of the server by generating one token."
Chayenne's avatar
Chayenne committed
152
153
154
155
156
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
157
   "metadata": {},
Chayenne's avatar
Chayenne committed
158
159
   "outputs": [],
   "source": [
160
    "url = f\"http://localhost:{port}/health_generate\"\n",
Chayenne's avatar
Chayenne committed
161
    "\n",
162
    "response = requests.get(url)\n",
Chayenne's avatar
Chayenne committed
163
164
165
166
167
168
    "print_highlight(response.text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
169
   "metadata": {},
Chayenne's avatar
Chayenne committed
170
171
   "outputs": [],
   "source": [
172
    "url = f\"http://localhost:{port}/health\"\n",
Chayenne's avatar
Chayenne committed
173
174
175
176
177
    "\n",
    "response = requests.get(url)\n",
    "print_highlight(response.text)"
   ]
  },
178
179
180
181
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
182
    "## Flush Cache\n",
183
    "\n",
184
    "Flush the radix cache. It will be automatically triggered when the model weights are updated by the `/update_weights` API."
185
186
187
188
189
190
191
192
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
193
    "# flush cache\n",
194
    "\n",
195
    "url = f\"http://localhost:{port}/flush_cache\"\n",
196
    "\n",
197
    "response = requests.post(url)\n",
198
199
200
    "print_highlight(response.text)"
   ]
  },
Chayenne's avatar
Chayenne committed
201
202
203
204
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
Chayenne's avatar
Chayenne committed
205
    "## Update Weights From Disk\n",
Chayenne's avatar
Chayenne committed
206
    "\n",
Chayenne's avatar
Chayenne committed
207
208
209
    "Update model weights from disk without restarting the server. Only applicable for models with the same architecture and parameter size.\n",
    "\n",
    "SGLang support `update_weights_from_disk` API for continuous evaluation during training (save checkpoint to disk and update weights from disk).\n"
Chayenne's avatar
Chayenne committed
210
211
212
213
214
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
215
   "metadata": {},
Chayenne's avatar
Chayenne committed
216
217
218
219
   "outputs": [],
   "source": [
    "# successful update with same architecture and size\n",
    "\n",
220
    "url = f\"http://localhost:{port}/update_weights_from_disk\"\n",
221
    "data = {\"model_path\": \"qwen/qwen2.5-0.5b-instruct\"}\n",
Chayenne's avatar
Chayenne committed
222
223
224
    "\n",
    "response = requests.post(url, json=data)\n",
    "print_highlight(response.text)\n",
225
    "assert response.json()[\"success\"] is True\n",
226
    "assert response.json()[\"message\"] == \"Succeeded to update model weights.\""
Chayenne's avatar
Chayenne committed
227
228
229
230
231
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
232
   "metadata": {},
Chayenne's avatar
Chayenne committed
233
234
   "outputs": [],
   "source": [
235
    "# failed update with different parameter size or wrong name\n",
Chayenne's avatar
Chayenne committed
236
    "\n",
237
    "url = f\"http://localhost:{port}/update_weights_from_disk\"\n",
238
    "data = {\"model_path\": \"qwen/qwen2.5-0.5b-instruct-wrong\"}\n",
Chayenne's avatar
Chayenne committed
239
240
241
242
    "\n",
    "response = requests.post(url, json=data)\n",
    "response_json = response.json()\n",
    "print_highlight(response_json)\n",
243
    "assert response_json[\"success\"] is False\n",
Chayenne's avatar
Chayenne committed
244
    "assert response_json[\"message\"] == (\n",
245
    "    \"Failed to get weights iterator: \"\n",
246
    "    \"qwen/qwen2.5-0.5b-instruct-wrong\"\n",
247
    "    \" (repository not found).\"\n",
Chayenne's avatar
Chayenne committed
248
249
250
    ")"
   ]
  },
251
252
253
254
255
256
257
258
259
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "terminate_process(server_process)"
   ]
  },
Chayenne's avatar
Chayenne committed
260
261
262
263
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
Chayenne's avatar
Chayenne committed
264
    "## Encode (embedding model)\n",
Chayenne's avatar
Chayenne committed
265
    "\n",
Chayenne's avatar
Chayenne committed
266
267
    "Encode text into embeddings. Note that this API is only available for [embedding models](openai_api_embeddings.html#openai-apis-embedding) and will raise an error for generation models.\n",
    "Therefore, we launch a new server to server an embedding model."
Chayenne's avatar
Chayenne committed
268
269
270
271
272
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
273
   "metadata": {},
Chayenne's avatar
Chayenne committed
274
275
   "outputs": [],
   "source": [
276
    "embedding_process, port = launch_server_cmd(\n",
Chayenne's avatar
Chayenne committed
277
    "    \"\"\"\n",
278
    "python3 -m sglang.launch_server --model-path Alibaba-NLP/gte-Qwen2-1.5B-instruct \\\n",
279
    "    --host 0.0.0.0 --is-embedding\n",
Chayenne's avatar
Chayenne committed
280
281
282
    "\"\"\"\n",
    ")\n",
    "\n",
283
    "wait_for_server(f\"http://localhost:{port}\")"
Chayenne's avatar
Chayenne committed
284
285
286
287
288
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
289
   "metadata": {},
Chayenne's avatar
Chayenne committed
290
291
292
293
   "outputs": [],
   "source": [
    "# successful encode for embedding model\n",
    "\n",
294
    "url = f\"http://localhost:{port}/encode\"\n",
295
    "data = {\"model\": \"Alibaba-NLP/gte-Qwen2-1.5B-instruct\", \"text\": \"Once upon a time\"}\n",
Chayenne's avatar
Chayenne committed
296
297
298
299
300
301
    "\n",
    "response = requests.post(url, json=data)\n",
    "response_json = response.json()\n",
    "print_highlight(f\"Text embedding (first 10): {response_json['embedding'][:10]}\")"
   ]
  },
302
303
304
305
306
307
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
308
    "terminate_process(embedding_process)"
309
310
   ]
  },
woodx's avatar
woodx committed
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## v1/rerank (cross encoder rerank model)\n",
    "Rerank a list of documents given a query using a cross-encoder model. Note that this API is only available for cross encoder model like [BAAI/bge-reranker-v2-m3](https://huggingface.co/BAAI/bge-reranker-v2-m3) with `attention-backend` `triton` and `torch_native`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "reranker_process, port = launch_server_cmd(\n",
    "    \"\"\"\n",
    "python3 -m sglang.launch_server --model-path BAAI/bge-reranker-v2-m3 \\\n",
    "    --host 0.0.0.0 --disable-radix-cache --chunked-prefill-size -1 --attention-backend triton --is-embedding\n",
    "\"\"\"\n",
    ")\n",
    "\n",
    "wait_for_server(f\"http://localhost:{port}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# compute rerank scores for query and documents\n",
    "\n",
    "url = f\"http://localhost:{port}/v1/rerank\"\n",
    "data = {\n",
    "    \"model\": \"BAAI/bge-reranker-v2-m3\",\n",
    "    \"query\": \"what is panda?\",\n",
    "    \"documents\": [\n",
    "        \"hi\",\n",
    "        \"The giant panda (Ailuropoda melanoleuca), sometimes called a panda bear or simply panda, is a bear species endemic to China.\",\n",
    "    ],\n",
    "}\n",
    "\n",
    "response = requests.post(url, json=data)\n",
    "response_json = response.json()\n",
    "for item in response_json:\n",
    "    print_highlight(f\"Score: {item['score']:.2f} - Document: '{item['document']}'\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "terminate_process(reranker_process)"
   ]
  },
Chayenne's avatar
Chayenne committed
368
369
370
371
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
372
    "## Classify (reward model)\n",
Chayenne's avatar
Chayenne committed
373
    "\n",
374
    "SGLang Runtime also supports reward models. Here we use a reward model to classify the quality of pairwise generations."
Chayenne's avatar
Chayenne committed
375
376
377
378
379
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
380
   "metadata": {},
Chayenne's avatar
Chayenne committed
381
382
383
384
385
   "outputs": [],
   "source": [
    "# Note that SGLang now treats embedding models and reward models as the same type of models.\n",
    "# This will be updated in the future.\n",
    "\n",
386
    "reward_process, port = launch_server_cmd(\n",
Chayenne's avatar
Chayenne committed
387
    "    \"\"\"\n",
388
    "python3 -m sglang.launch_server --model-path Skywork/Skywork-Reward-Llama-3.1-8B-v0.2 --host 0.0.0.0 --is-embedding\n",
Chayenne's avatar
Chayenne committed
389
390
391
    "\"\"\"\n",
    ")\n",
    "\n",
392
    "wait_for_server(f\"http://localhost:{port}\")"
Chayenne's avatar
Chayenne committed
393
394
395
396
397
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
398
   "metadata": {},
Chayenne's avatar
Chayenne committed
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
   "outputs": [],
   "source": [
    "from transformers import AutoTokenizer\n",
    "\n",
    "PROMPT = (\n",
    "    \"What is the range of the numeric output of a sigmoid node in a neural network?\"\n",
    ")\n",
    "\n",
    "RESPONSE1 = \"The output of a sigmoid node is bounded between -1 and 1.\"\n",
    "RESPONSE2 = \"The output of a sigmoid node is bounded between 0 and 1.\"\n",
    "\n",
    "CONVS = [\n",
    "    [{\"role\": \"user\", \"content\": PROMPT}, {\"role\": \"assistant\", \"content\": RESPONSE1}],\n",
    "    [{\"role\": \"user\", \"content\": PROMPT}, {\"role\": \"assistant\", \"content\": RESPONSE2}],\n",
    "]\n",
    "\n",
    "tokenizer = AutoTokenizer.from_pretrained(\"Skywork/Skywork-Reward-Llama-3.1-8B-v0.2\")\n",
    "prompts = tokenizer.apply_chat_template(CONVS, tokenize=False)\n",
    "\n",
418
    "url = f\"http://localhost:{port}/classify\"\n",
Chayenne's avatar
Chayenne committed
419
    "data = {\"model\": \"Skywork/Skywork-Reward-Llama-3.1-8B-v0.2\", \"text\": prompts}\n",
Chayenne's avatar
Chayenne committed
420
421
422
423
424
425
    "\n",
    "responses = requests.post(url, json=data).json()\n",
    "for response in responses:\n",
    "    print_highlight(f\"reward: {response['embedding'][0]}\")"
   ]
  },
Chayenne's avatar
Chayenne committed
426
427
  {
   "cell_type": "code",
428
429
   "execution_count": null,
   "metadata": {},
Chayenne's avatar
Chayenne committed
430
431
   "outputs": [],
   "source": [
432
    "terminate_process(reward_process)"
Chayenne's avatar
Chayenne committed
433
   ]
434
  },
435
436
437
438
439
440
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Capture expert selection distribution in MoE models\n",
    "\n",
441
442
443
    "SGLang Runtime supports recording the number of times an expert is selected in a MoE model run for each expert in the model. This is useful when analyzing the throughput of the model and plan for optimization.\n",
    "\n",
    "*Note: We only print out the first 10 lines of the csv below for better readability. Please adjust accordingly if you want to analyze the results more deeply.*"
444
445
446
447
448
449
450
451
452
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "expert_record_server_process, port = launch_server_cmd(\n",
453
    "    \"python3 -m sglang.launch_server --model-path Qwen/Qwen1.5-MoE-A2.7B --host 0.0.0.0 --expert-distribution-recorder-mode stat\"\n",
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
    ")\n",
    "\n",
    "wait_for_server(f\"http://localhost:{port}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = requests.post(f\"http://localhost:{port}/start_expert_distribution_record\")\n",
    "print_highlight(response)\n",
    "\n",
    "url = f\"http://localhost:{port}/generate\"\n",
    "data = {\"text\": \"What is the capital of France?\"}\n",
    "\n",
    "response = requests.post(url, json=data)\n",
    "print_highlight(response.json())\n",
    "\n",
    "response = requests.post(f\"http://localhost:{port}/stop_expert_distribution_record\")\n",
    "print_highlight(response)\n",
    "\n",
    "response = requests.post(f\"http://localhost:{port}/dump_expert_distribution_record\")\n",
478
    "print_highlight(response)"
479
480
481
482
483
484
485
486
487
488
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "terminate_process(expert_record_server_process)"
   ]
Chayenne's avatar
Chayenne committed
489
490
491
492
493
494
495
496
497
498
499
500
  }
 ],
 "metadata": {
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
501
   "pygments_lexer": "ipython3"
Chayenne's avatar
Chayenne committed
502
503
504
505
506
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}