native_api.ipynb 15.3 KB
Newer Older
Chayenne's avatar
Chayenne committed
1
2
3
4
5
6
{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
7
    "# SGLang Native APIs\n",
Chayenne's avatar
Chayenne committed
8
    "\n",
Chayenne's avatar
Chayenne committed
9
    "Apart from the OpenAI compatible APIs, the SGLang Runtime also provides its native server APIs. We introduce these following APIs:\n",
Chayenne's avatar
Chayenne committed
10
    "\n",
Chayenne's avatar
Chayenne committed
11
    "- `/generate` (text generation model)\n",
Chayenne's avatar
Chayenne committed
12
    "- `/get_model_info`\n",
13
    "- `/get_server_info`\n",
Chayenne's avatar
Chayenne committed
14
15
16
    "- `/health`\n",
    "- `/health_generate`\n",
    "- `/flush_cache`\n",
Chayenne's avatar
Chayenne committed
17
    "- `/update_weights`\n",
Chayenne's avatar
Chayenne committed
18
    "- `/encode`(embedding model)\n",
woodx's avatar
woodx committed
19
    "- `/v1/rerank`(cross encoder rerank model)\n",
20
    "- `/classify`(reward model)\n",
21
22
23
    "- `/start_expert_distribution_record`\n",
    "- `/stop_expert_distribution_record`\n",
    "- `/dump_expert_distribution_record`\n",
Chayenne's avatar
Chayenne committed
24
25
26
27
28
29
30
31
32
33
34
35
36
37
    "\n",
    "We mainly use `requests` to test these APIs in the following examples. You can also use `curl`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Launch A Server"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
38
   "metadata": {},
Chayenne's avatar
Chayenne committed
39
40
   "outputs": [],
   "source": [
Chayenne's avatar
Chayenne committed
41
    "import requests\n",
42
    "from sglang.test.test_utils import is_in_ci\n",
Chayenne's avatar
Chayenne committed
43
    "\n",
44
45
46
47
48
49
50
51
52
    "if is_in_ci():\n",
    "    from patch import launch_server_cmd\n",
    "else:\n",
    "    from sglang.utils import launch_server_cmd\n",
    "\n",
    "from sglang.utils import wait_for_server, print_highlight, terminate_process\n",
    "\n",
    "\n",
    "server_process, port = launch_server_cmd(\n",
53
    "    \"python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct --host 0.0.0.0\"\n",
Chayenne's avatar
Chayenne committed
54
    ")\n",
55
56
57
58
    "## To run qwen2.5-0.5b-instruct model on the Ascend-Npu, you can execute the following command:\n",
    "# server_process, port = launch_server_cmd(\n",
    "#     \"python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct --host 0.0.0.0 --device npu --tp 2 --attention-backend torch_native\"\n",
    "# )\n",
Chayenne's avatar
Chayenne committed
59
    "\n",
60
    "wait_for_server(f\"http://localhost:{port}\")"
Chayenne's avatar
Chayenne committed
61
62
63
64
65
66
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
Chayenne's avatar
Chayenne committed
67
    "## Generate (text generation model)\n",
68
    "Generate completions. This is similar to the `/v1/completions` in OpenAI API. Detailed parameters can be found in the [sampling parameters](./sampling_params.md)."
Chayenne's avatar
Chayenne committed
69
70
71
72
73
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
74
   "metadata": {},
Chayenne's avatar
Chayenne committed
75
76
   "outputs": [],
   "source": [
77
    "url = f\"http://localhost:{port}/generate\"\n",
Chayenne's avatar
Chayenne committed
78
    "data = {\"text\": \"What is the capital of France?\"}\n",
Chayenne's avatar
Chayenne committed
79
80
    "\n",
    "response = requests.post(url, json=data)\n",
Lianmin Zheng's avatar
Lianmin Zheng committed
81
    "print_highlight(response.json())"
Chayenne's avatar
Chayenne committed
82
83
   ]
  },
84
85
86
87
88
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
Chayenne's avatar
Chayenne committed
89
90
91
92
93
94
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get Model Info\n",
    "\n",
Lianmin Zheng's avatar
Lianmin Zheng committed
95
    "Get the information of the model.\n",
Chayenne's avatar
Chayenne committed
96
97
    "\n",
    "- `model_path`: The path/name of the model.\n",
Chayenne's avatar
Chayenne committed
98
    "- `is_generation`: Whether the model is used as generation model or embedding model.\n",
99
100
    "- `tokenizer_path`: The path/name of the tokenizer.\n",
    "- `preferred_sampling_params`: The default sampling params specified via `--preferred-sampling-params`. `None` is returned in this example as we did not explicitly configure it in server args."
Chayenne's avatar
Chayenne committed
101
102
103
104
105
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
106
   "metadata": {},
Chayenne's avatar
Chayenne committed
107
108
   "outputs": [],
   "source": [
109
    "url = f\"http://localhost:{port}/get_model_info\"\n",
Chayenne's avatar
Chayenne committed
110
111
112
113
    "\n",
    "response = requests.get(url)\n",
    "response_json = response.json()\n",
    "print_highlight(response_json)\n",
114
    "assert response_json[\"model_path\"] == \"qwen/qwen2.5-0.5b-instruct\"\n",
Lianmin Zheng's avatar
Lianmin Zheng committed
115
    "assert response_json[\"is_generation\"] is True\n",
116
    "assert response_json[\"tokenizer_path\"] == \"qwen/qwen2.5-0.5b-instruct\"\n",
117
    "assert response_json[\"preferred_sampling_params\"] is None\n",
118
119
120
121
122
123
    "assert response_json.keys() == {\n",
    "    \"model_path\",\n",
    "    \"is_generation\",\n",
    "    \"tokenizer_path\",\n",
    "    \"preferred_sampling_params\",\n",
    "}"
Chayenne's avatar
Chayenne committed
124
125
126
127
128
129
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
130
131
132
133
134
135
    "## Get Server Info\n",
    "Gets the server information including CLI arguments, token limits, and memory pool sizes.\n",
    "- Note: `get_server_info` merges the following deprecated endpoints:\n",
    "  - `get_server_args`\n",
    "  - `get_memory_pool_size` \n",
    "  - `get_max_total_num_tokens`"
Chayenne's avatar
Chayenne committed
136
137
138
139
140
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
141
   "metadata": {},
Chayenne's avatar
Chayenne committed
142
143
   "outputs": [],
   "source": [
144
    "# get_server_info\n",
Chayenne's avatar
Chayenne committed
145
    "\n",
146
    "url = f\"http://localhost:{port}/get_server_info\"\n",
Chayenne's avatar
Chayenne committed
147
148
149
150
151
152
153
154
155
    "\n",
    "response = requests.get(url)\n",
    "print_highlight(response.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
156
157
158
    "## Health Check\n",
    "- `/health`: Check the health of the server.\n",
    "- `/health_generate`: Check the health of the server by generating one token."
Chayenne's avatar
Chayenne committed
159
160
161
162
163
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
164
   "metadata": {},
Chayenne's avatar
Chayenne committed
165
166
   "outputs": [],
   "source": [
167
    "url = f\"http://localhost:{port}/health_generate\"\n",
Chayenne's avatar
Chayenne committed
168
    "\n",
169
    "response = requests.get(url)\n",
Chayenne's avatar
Chayenne committed
170
171
172
173
174
175
    "print_highlight(response.text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
176
   "metadata": {},
Chayenne's avatar
Chayenne committed
177
178
   "outputs": [],
   "source": [
179
    "url = f\"http://localhost:{port}/health\"\n",
Chayenne's avatar
Chayenne committed
180
181
182
183
184
    "\n",
    "response = requests.get(url)\n",
    "print_highlight(response.text)"
   ]
  },
185
186
187
188
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
189
    "## Flush Cache\n",
190
    "\n",
191
    "Flush the radix cache. It will be automatically triggered when the model weights are updated by the `/update_weights` API."
192
193
194
195
196
197
198
199
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
200
    "# flush cache\n",
201
    "\n",
202
    "url = f\"http://localhost:{port}/flush_cache\"\n",
203
    "\n",
204
    "response = requests.post(url)\n",
205
206
207
    "print_highlight(response.text)"
   ]
  },
Chayenne's avatar
Chayenne committed
208
209
210
211
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
Chayenne's avatar
Chayenne committed
212
    "## Update Weights From Disk\n",
Chayenne's avatar
Chayenne committed
213
    "\n",
Chayenne's avatar
Chayenne committed
214
215
216
    "Update model weights from disk without restarting the server. Only applicable for models with the same architecture and parameter size.\n",
    "\n",
    "SGLang support `update_weights_from_disk` API for continuous evaluation during training (save checkpoint to disk and update weights from disk).\n"
Chayenne's avatar
Chayenne committed
217
218
219
220
221
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
222
   "metadata": {},
Chayenne's avatar
Chayenne committed
223
224
225
226
   "outputs": [],
   "source": [
    "# successful update with same architecture and size\n",
    "\n",
227
    "url = f\"http://localhost:{port}/update_weights_from_disk\"\n",
228
    "data = {\"model_path\": \"qwen/qwen2.5-0.5b-instruct\"}\n",
Chayenne's avatar
Chayenne committed
229
230
231
    "\n",
    "response = requests.post(url, json=data)\n",
    "print_highlight(response.text)\n",
232
    "assert response.json()[\"success\"] is True\n",
233
    "assert response.json()[\"message\"] == \"Succeeded to update model weights.\""
Chayenne's avatar
Chayenne committed
234
235
236
237
238
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
239
   "metadata": {},
Chayenne's avatar
Chayenne committed
240
241
   "outputs": [],
   "source": [
242
    "# failed update with different parameter size or wrong name\n",
Chayenne's avatar
Chayenne committed
243
    "\n",
244
    "url = f\"http://localhost:{port}/update_weights_from_disk\"\n",
245
    "data = {\"model_path\": \"qwen/qwen2.5-0.5b-instruct-wrong\"}\n",
Chayenne's avatar
Chayenne committed
246
247
248
249
    "\n",
    "response = requests.post(url, json=data)\n",
    "response_json = response.json()\n",
    "print_highlight(response_json)\n",
250
    "assert response_json[\"success\"] is False\n",
Chayenne's avatar
Chayenne committed
251
    "assert response_json[\"message\"] == (\n",
252
    "    \"Failed to get weights iterator: \"\n",
253
    "    \"qwen/qwen2.5-0.5b-instruct-wrong\"\n",
254
    "    \" (repository not found).\"\n",
Chayenne's avatar
Chayenne committed
255
256
257
    ")"
   ]
  },
258
259
260
261
262
263
264
265
266
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "terminate_process(server_process)"
   ]
  },
Chayenne's avatar
Chayenne committed
267
268
269
270
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
Chayenne's avatar
Chayenne committed
271
    "## Encode (embedding model)\n",
Chayenne's avatar
Chayenne committed
272
    "\n",
Chayenne's avatar
Chayenne committed
273
274
    "Encode text into embeddings. Note that this API is only available for [embedding models](openai_api_embeddings.html#openai-apis-embedding) and will raise an error for generation models.\n",
    "Therefore, we launch a new server to server an embedding model."
Chayenne's avatar
Chayenne committed
275
276
277
278
279
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
280
   "metadata": {},
Chayenne's avatar
Chayenne committed
281
282
   "outputs": [],
   "source": [
283
    "embedding_process, port = launch_server_cmd(\n",
Chayenne's avatar
Chayenne committed
284
    "    \"\"\"\n",
285
    "python3 -m sglang.launch_server --model-path Alibaba-NLP/gte-Qwen2-1.5B-instruct \\\n",
286
    "    --host 0.0.0.0 --is-embedding\n",
Chayenne's avatar
Chayenne committed
287
288
289
    "\"\"\"\n",
    ")\n",
    "\n",
290
    "wait_for_server(f\"http://localhost:{port}\")"
Chayenne's avatar
Chayenne committed
291
292
293
294
295
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
296
   "metadata": {},
Chayenne's avatar
Chayenne committed
297
298
299
300
   "outputs": [],
   "source": [
    "# successful encode for embedding model\n",
    "\n",
301
    "url = f\"http://localhost:{port}/encode\"\n",
302
    "data = {\"model\": \"Alibaba-NLP/gte-Qwen2-1.5B-instruct\", \"text\": \"Once upon a time\"}\n",
Chayenne's avatar
Chayenne committed
303
304
305
306
307
308
    "\n",
    "response = requests.post(url, json=data)\n",
    "response_json = response.json()\n",
    "print_highlight(f\"Text embedding (first 10): {response_json['embedding'][:10]}\")"
   ]
  },
309
310
311
312
313
314
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
315
    "terminate_process(embedding_process)"
316
317
   ]
  },
woodx's avatar
woodx committed
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## v1/rerank (cross encoder rerank model)\n",
    "Rerank a list of documents given a query using a cross-encoder model. Note that this API is only available for cross encoder model like [BAAI/bge-reranker-v2-m3](https://huggingface.co/BAAI/bge-reranker-v2-m3) with `attention-backend` `triton` and `torch_native`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "reranker_process, port = launch_server_cmd(\n",
    "    \"\"\"\n",
    "python3 -m sglang.launch_server --model-path BAAI/bge-reranker-v2-m3 \\\n",
    "    --host 0.0.0.0 --disable-radix-cache --chunked-prefill-size -1 --attention-backend triton --is-embedding\n",
    "\"\"\"\n",
    ")\n",
    "\n",
    "wait_for_server(f\"http://localhost:{port}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# compute rerank scores for query and documents\n",
    "\n",
    "url = f\"http://localhost:{port}/v1/rerank\"\n",
    "data = {\n",
    "    \"model\": \"BAAI/bge-reranker-v2-m3\",\n",
    "    \"query\": \"what is panda?\",\n",
    "    \"documents\": [\n",
    "        \"hi\",\n",
    "        \"The giant panda (Ailuropoda melanoleuca), sometimes called a panda bear or simply panda, is a bear species endemic to China.\",\n",
    "    ],\n",
    "}\n",
    "\n",
    "response = requests.post(url, json=data)\n",
    "response_json = response.json()\n",
    "for item in response_json:\n",
    "    print_highlight(f\"Score: {item['score']:.2f} - Document: '{item['document']}'\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "terminate_process(reranker_process)"
   ]
  },
Chayenne's avatar
Chayenne committed
375
376
377
378
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
379
    "## Classify (reward model)\n",
Chayenne's avatar
Chayenne committed
380
    "\n",
381
    "SGLang Runtime also supports reward models. Here we use a reward model to classify the quality of pairwise generations."
Chayenne's avatar
Chayenne committed
382
383
384
385
386
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
387
   "metadata": {},
Chayenne's avatar
Chayenne committed
388
389
390
391
392
   "outputs": [],
   "source": [
    "# Note that SGLang now treats embedding models and reward models as the same type of models.\n",
    "# This will be updated in the future.\n",
    "\n",
393
    "reward_process, port = launch_server_cmd(\n",
Chayenne's avatar
Chayenne committed
394
    "    \"\"\"\n",
395
    "python3 -m sglang.launch_server --model-path Skywork/Skywork-Reward-Llama-3.1-8B-v0.2 --host 0.0.0.0 --is-embedding\n",
Chayenne's avatar
Chayenne committed
396
397
398
    "\"\"\"\n",
    ")\n",
    "\n",
399
    "wait_for_server(f\"http://localhost:{port}\")"
Chayenne's avatar
Chayenne committed
400
401
402
403
404
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
405
   "metadata": {},
Chayenne's avatar
Chayenne committed
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
   "outputs": [],
   "source": [
    "from transformers import AutoTokenizer\n",
    "\n",
    "PROMPT = (\n",
    "    \"What is the range of the numeric output of a sigmoid node in a neural network?\"\n",
    ")\n",
    "\n",
    "RESPONSE1 = \"The output of a sigmoid node is bounded between -1 and 1.\"\n",
    "RESPONSE2 = \"The output of a sigmoid node is bounded between 0 and 1.\"\n",
    "\n",
    "CONVS = [\n",
    "    [{\"role\": \"user\", \"content\": PROMPT}, {\"role\": \"assistant\", \"content\": RESPONSE1}],\n",
    "    [{\"role\": \"user\", \"content\": PROMPT}, {\"role\": \"assistant\", \"content\": RESPONSE2}],\n",
    "]\n",
    "\n",
    "tokenizer = AutoTokenizer.from_pretrained(\"Skywork/Skywork-Reward-Llama-3.1-8B-v0.2\")\n",
    "prompts = tokenizer.apply_chat_template(CONVS, tokenize=False)\n",
    "\n",
425
    "url = f\"http://localhost:{port}/classify\"\n",
Chayenne's avatar
Chayenne committed
426
    "data = {\"model\": \"Skywork/Skywork-Reward-Llama-3.1-8B-v0.2\", \"text\": prompts}\n",
Chayenne's avatar
Chayenne committed
427
428
429
430
431
432
    "\n",
    "responses = requests.post(url, json=data).json()\n",
    "for response in responses:\n",
    "    print_highlight(f\"reward: {response['embedding'][0]}\")"
   ]
  },
Chayenne's avatar
Chayenne committed
433
434
  {
   "cell_type": "code",
435
436
   "execution_count": null,
   "metadata": {},
Chayenne's avatar
Chayenne committed
437
438
   "outputs": [],
   "source": [
439
    "terminate_process(reward_process)"
Chayenne's avatar
Chayenne committed
440
   ]
441
  },
442
443
444
445
446
447
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Capture expert selection distribution in MoE models\n",
    "\n",
448
449
450
    "SGLang Runtime supports recording the number of times an expert is selected in a MoE model run for each expert in the model. This is useful when analyzing the throughput of the model and plan for optimization.\n",
    "\n",
    "*Note: We only print out the first 10 lines of the csv below for better readability. Please adjust accordingly if you want to analyze the results more deeply.*"
451
452
453
454
455
456
457
458
459
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "expert_record_server_process, port = launch_server_cmd(\n",
460
    "    \"python3 -m sglang.launch_server --model-path Qwen/Qwen1.5-MoE-A2.7B --host 0.0.0.0 --expert-distribution-recorder-mode stat\"\n",
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
    ")\n",
    "\n",
    "wait_for_server(f\"http://localhost:{port}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = requests.post(f\"http://localhost:{port}/start_expert_distribution_record\")\n",
    "print_highlight(response)\n",
    "\n",
    "url = f\"http://localhost:{port}/generate\"\n",
    "data = {\"text\": \"What is the capital of France?\"}\n",
    "\n",
    "response = requests.post(url, json=data)\n",
    "print_highlight(response.json())\n",
    "\n",
    "response = requests.post(f\"http://localhost:{port}/stop_expert_distribution_record\")\n",
    "print_highlight(response)\n",
    "\n",
    "response = requests.post(f\"http://localhost:{port}/dump_expert_distribution_record\")\n",
485
    "print_highlight(response)"
486
487
488
489
490
491
492
493
494
495
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "terminate_process(expert_record_server_process)"
   ]
Chayenne's avatar
Chayenne committed
496
497
498
499
500
501
502
503
504
505
506
507
  }
 ],
 "metadata": {
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
508
   "pygments_lexer": "ipython3"
Chayenne's avatar
Chayenne committed
509
510
511
512
513
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}