openai_api_completions.ipynb 21.1 KB
Newer Older
Chayenne's avatar
Chayenne committed
1
2
3
4
5
6
{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
Lianmin Zheng's avatar
Lianmin Zheng committed
7
    "# OpenAI APIs - Completions\n",
Chayenne's avatar
Chayenne committed
8
    "\n",
9
10
    "SGLang provides OpenAI-compatible APIs to enable a smooth transition from OpenAI services to self-hosted local models.\n",
    "A complete reference for the API is available in the [OpenAI API Reference](https://platform.openai.com/docs/api-reference).\n",
11
    "\n",
12
    "This tutorial covers the following popular APIs:\n",
Chayenne's avatar
Chayenne committed
13
14
15
16
    "\n",
    "- `chat/completions`\n",
    "- `completions`\n",
    "- `batches`\n",
17
    "\n",
simveit's avatar
simveit committed
18
    "Check out other tutorials to learn about [vision APIs](https://docs.sglang.ai/backend/openai_api_vision.html) for vision-language models and [embedding APIs](https://docs.sglang.ai/backend/openai_api_embeddings.html) for embedding models."
Chayenne's avatar
Chayenne committed
19
20
21
22
23
24
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
25
    "## Launch A Server\n",
Chayenne's avatar
Chayenne committed
26
    "\n",
27
    "Launch the server in your terminal and wait for it to initialize."
Chayenne's avatar
Chayenne committed
28
29
30
31
   ]
  },
  {
   "cell_type": "code",
Chayenne's avatar
Chayenne committed
32
   "execution_count": null,
33
   "metadata": {},
Chayenne's avatar
Chayenne committed
34
   "outputs": [],
Chayenne's avatar
Chayenne committed
35
   "source": [
36
37
38
39
40
41
42
43
44
    "from sglang.test.test_utils import is_in_ci\n",
    "\n",
    "if is_in_ci():\n",
    "    from patch import launch_server_cmd\n",
    "else:\n",
    "    from sglang.utils import launch_server_cmd\n",
    "\n",
    "from sglang.utils import wait_for_server, print_highlight, terminate_process\n",
    "\n",
Chayenne's avatar
Chayenne committed
45
    "\n",
46
    "server_process, port = launch_server_cmd(\n",
47
    "    \"python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct --host 0.0.0.0 --mem-fraction-static 0.8\"\n",
Chayenne's avatar
Chayenne committed
48
49
    ")\n",
    "\n",
50
51
    "wait_for_server(f\"http://localhost:{port}\")\n",
    "print(f\"Server started on http://localhost:{port}\")"
Chayenne's avatar
Chayenne committed
52
53
   ]
  },
54
55
56
57
58
59
60
61
62
63
64
65
66
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Chat Completions\n",
    "\n",
    "### Usage\n",
    "\n",
    "The server fully implements the OpenAI API.\n",
    "It will automatically apply the chat template specified in the Hugging Face tokenizer, if one is available.\n",
    "You can also specify a custom chat template with `--chat-template` when launching the server."
   ]
  },
Chayenne's avatar
Chayenne committed
67
68
  {
   "cell_type": "code",
Chayenne's avatar
Chayenne committed
69
   "execution_count": null,
70
   "metadata": {},
Chayenne's avatar
Chayenne committed
71
   "outputs": [],
Chayenne's avatar
Chayenne committed
72
73
74
   "source": [
    "import openai\n",
    "\n",
75
    "client = openai.Client(base_url=f\"http://127.0.0.1:{port}/v1\", api_key=\"None\")\n",
Chayenne's avatar
Chayenne committed
76
77
    "\n",
    "response = client.chat.completions.create(\n",
78
    "    model=\"qwen/qwen2.5-0.5b-instruct\",\n",
Chayenne's avatar
Chayenne committed
79
80
81
82
83
84
    "    messages=[\n",
    "        {\"role\": \"user\", \"content\": \"List 3 countries and their capitals.\"},\n",
    "    ],\n",
    "    temperature=0,\n",
    "    max_tokens=64,\n",
    ")\n",
85
86
    "\n",
    "print_highlight(f\"Response: {response}\")"
Chayenne's avatar
Chayenne committed
87
88
89
90
91
92
93
94
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Parameters\n",
    "\n",
95
    "The chat completions API accepts OpenAI Chat Completions API's parameters. Refer to [OpenAI Chat Completions API](https://platform.openai.com/docs/api-reference/chat/create) for more details.\n",
Chayenne's avatar
Chayenne committed
96
    "\n",
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
    "SGLang extends the standard API with the `extra_body` parameter, allowing for additional customization. One key option within `extra_body` is `chat_template_kwargs`, which can be used to pass arguments to the chat template processor.\n",
    "\n",
    "#### Enabling Model Thinking/Reasoning\n",
    "\n",
    "You can use `chat_template_kwargs` to enable or disable the model's internal thinking or reasoning process output. Set `\"enable_thinking\": True` within `chat_template_kwargs` to include the reasoning steps in the response. This requires launching the server with a compatible reasoning parser (e.g., `--reasoning-parser qwen3` for Qwen3 models).\n",
    "\n",
    "Here's an example demonstrating how to enable thinking and retrieve the reasoning content separately (using `separate_reasoning: True`):\n",
    "\n",
    "```python\n",
    "# Ensure the server is launched with a compatible reasoning parser, e.g.:\n",
    "# python3 -m sglang.launch_server --model-path QwQ/Qwen3-32B-250415 --reasoning-parser qwen3 ...\n",
    "\n",
    "from openai import OpenAI\n",
    "\n",
    "# Modify OpenAI's API key and API base to use SGLang's API server.\n",
    "openai_api_key = \"EMPTY\"\n",
    "openai_api_base = f\"http://127.0.0.1:{port}/v1\" # Use the correct port\n",
    "\n",
    "client = OpenAI(\n",
    "    api_key=openai_api_key,\n",
    "    base_url=openai_api_base,\n",
    ")\n",
    "\n",
    "model = \"QwQ/Qwen3-32B-250415\" # Use the model loaded by the server\n",
    "messages = [{\"role\": \"user\", \"content\": \"9.11 and 9.8, which is greater?\"}]\n",
    "\n",
    "response = client.chat.completions.create(\n",
    "    model=model,\n",
    "    messages=messages,\n",
    "    extra_body={\n",
    "        \"chat_template_kwargs\": {\"enable_thinking\": True},\n",
    "        \"separate_reasoning\": True\n",
    "    }\n",
    ")\n",
    "\n",
    "print(\"response.choices[0].message.reasoning_content: \\n\", response.choices[0].message.reasoning_content)\n",
    "print(\"response.choices[0].message.content: \\n\", response.choices[0].message.content)\n",
    "```\n",
    "\n",
    "**Example Output:**\n",
    "\n",
    "```\n",
    "response.choices[0].message.reasoning_content: \n",
    " Okay, so I need to figure out which number is greater between 9.11 and 9.8. Hmm, let me think. Both numbers start with 9, right? So the whole number part is the same. That means I need to look at the decimal parts to determine which one is bigger.\n",
    "...\n",
    "Therefore, after checking multiple methods—aligning decimals, subtracting, converting to fractions, and using a real-world analogy—it's clear that 9.8 is greater than 9.11.\n",
    "\n",
    "response.choices[0].message.content: \n",
    " To determine which number is greater between **9.11** and **9.8**, follow these steps:\n",
    "...\n",
    "**Answer**:  \n",
    "9.8 is greater than 9.11.\n",
    "```\n",
    "\n",
    "Setting `\"enable_thinking\": False` (or omitting it) will result in `reasoning_content` being `None`.\n",
    "\n",
    "Here is an example of a detailed chat completion request using standard OpenAI parameters:"
Chayenne's avatar
Chayenne committed
154
155
156
157
   ]
  },
  {
   "cell_type": "code",
Chayenne's avatar
Chayenne committed
158
   "execution_count": null,
159
   "metadata": {},
Chayenne's avatar
Chayenne committed
160
   "outputs": [],
Chayenne's avatar
Chayenne committed
161
162
   "source": [
    "response = client.chat.completions.create(\n",
163
    "    model=\"qwen/qwen2.5-0.5b-instruct\",\n",
Chayenne's avatar
Chayenne committed
164
165
166
167
168
169
170
171
172
173
174
175
176
    "    messages=[\n",
    "        {\n",
    "            \"role\": \"system\",\n",
    "            \"content\": \"You are a knowledgeable historian who provides concise responses.\",\n",
    "        },\n",
    "        {\"role\": \"user\", \"content\": \"Tell me about ancient Rome\"},\n",
    "        {\n",
    "            \"role\": \"assistant\",\n",
    "            \"content\": \"Ancient Rome was a civilization centered in Italy.\",\n",
    "        },\n",
    "        {\"role\": \"user\", \"content\": \"What were their major achievements?\"},\n",
    "    ],\n",
    "    temperature=0.3,  # Lower temperature for more focused responses\n",
Lianmin Zheng's avatar
Lianmin Zheng committed
177
    "    max_tokens=128,  # Reasonable length for a concise response\n",
Chayenne's avatar
Chayenne committed
178
179
180
181
182
183
184
    "    top_p=0.95,  # Slightly higher for better fluency\n",
    "    presence_penalty=0.2,  # Mild penalty to avoid repetition\n",
    "    frequency_penalty=0.2,  # Mild penalty for more natural language\n",
    "    n=1,  # Single response is usually more stable\n",
    "    seed=42,  # Keep for reproducibility\n",
    ")\n",
    "\n",
Lianmin Zheng's avatar
Lianmin Zheng committed
185
186
187
188
189
190
191
    "print_highlight(response.choices[0].message.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
Chayenne's avatar
Chayenne committed
192
    "Streaming mode is also supported."
Lianmin Zheng's avatar
Lianmin Zheng committed
193
194
195
196
   ]
  },
  {
   "cell_type": "code",
Chayenne's avatar
Chayenne committed
197
   "execution_count": null,
198
   "metadata": {},
Chayenne's avatar
Chayenne committed
199
   "outputs": [],
Lianmin Zheng's avatar
Lianmin Zheng committed
200
201
   "source": [
    "stream = client.chat.completions.create(\n",
202
    "    model=\"qwen/qwen2.5-0.5b-instruct\",\n",
Lianmin Zheng's avatar
Lianmin Zheng committed
203
204
205
206
207
208
    "    messages=[{\"role\": \"user\", \"content\": \"Say this is a test\"}],\n",
    "    stream=True,\n",
    ")\n",
    "for chunk in stream:\n",
    "    if chunk.choices[0].delta.content is not None:\n",
    "        print(chunk.choices[0].delta.content, end=\"\")"
Chayenne's avatar
Chayenne committed
209
210
211
212
213
214
215
216
217
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Completions\n",
    "\n",
    "### Usage\n",
218
    "Completions API is similar to Chat Completions API, but without the `messages` parameter or chat templates."
Chayenne's avatar
Chayenne committed
219
220
221
222
   ]
  },
  {
   "cell_type": "code",
Chayenne's avatar
Chayenne committed
223
   "execution_count": null,
224
   "metadata": {},
Chayenne's avatar
Chayenne committed
225
   "outputs": [],
Chayenne's avatar
Chayenne committed
226
227
   "source": [
    "response = client.completions.create(\n",
228
    "    model=\"qwen/qwen2.5-0.5b-instruct\",\n",
Chayenne's avatar
Chayenne committed
229
230
231
232
233
234
    "    prompt=\"List 3 countries and their capitals.\",\n",
    "    temperature=0,\n",
    "    max_tokens=64,\n",
    "    n=1,\n",
    "    stop=None,\n",
    ")\n",
235
236
    "\n",
    "print_highlight(f\"Response: {response}\")"
Chayenne's avatar
Chayenne committed
237
238
239
240
241
242
243
244
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Parameters\n",
    "\n",
245
    "The completions API accepts OpenAI Completions API's parameters.  Refer to [OpenAI Completions API](https://platform.openai.com/docs/api-reference/completions/create) for more details.\n",
Chayenne's avatar
Chayenne committed
246
247
248
249
250
251
    "\n",
    "Here is an example of a detailed completions request:"
   ]
  },
  {
   "cell_type": "code",
Chayenne's avatar
Chayenne committed
252
   "execution_count": null,
253
   "metadata": {},
Chayenne's avatar
Chayenne committed
254
   "outputs": [],
Chayenne's avatar
Chayenne committed
255
256
   "source": [
    "response = client.completions.create(\n",
257
    "    model=\"qwen/qwen2.5-0.5b-instruct\",\n",
Chayenne's avatar
Chayenne committed
258
259
260
261
262
263
264
265
266
267
268
    "    prompt=\"Write a short story about a space explorer.\",\n",
    "    temperature=0.7,  # Moderate temperature for creative writing\n",
    "    max_tokens=150,  # Longer response for a story\n",
    "    top_p=0.9,  # Balanced diversity in word choice\n",
    "    stop=[\"\\n\\n\", \"THE END\"],  # Multiple stop sequences\n",
    "    presence_penalty=0.3,  # Encourage novel elements\n",
    "    frequency_penalty=0.3,  # Reduce repetitive phrases\n",
    "    n=1,  # Generate one completion\n",
    "    seed=123,  # For reproducible results\n",
    ")\n",
    "\n",
269
    "print_highlight(f\"Response: {response}\")"
Chayenne's avatar
Chayenne committed
270
271
   ]
  },
Lianmin Zheng's avatar
Lianmin Zheng committed
272
273
274
275
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
276
    "## Structured Outputs (JSON, Regex, EBNF)\n",
277
    "\n",
278
    "For OpenAI compatible structed outputs API, refer to [Structured Outputs](https://docs.sglang.ai/backend/structured_outputs.html#OpenAI-Compatible-API) for more details.\n"
279
280
   ]
  },
Chayenne's avatar
Chayenne committed
281
282
283
284
285
286
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Batches\n",
    "\n",
287
    "Batches API for chat completions and completions are also supported. You can upload your requests in `jsonl` files, create a batch job, and retrieve the results when the batch job is completed (which takes longer but costs less).\n",
Chayenne's avatar
Chayenne committed
288
289
290
291
292
293
294
295
296
297
298
299
    "\n",
    "The batches APIs are:\n",
    "\n",
    "- `batches`\n",
    "- `batches/{batch_id}/cancel`\n",
    "- `batches/{batch_id}`\n",
    "\n",
    "Here is an example of a batch job for chat completions, completions are similar.\n"
   ]
  },
  {
   "cell_type": "code",
Chayenne's avatar
Chayenne committed
300
   "execution_count": null,
301
   "metadata": {},
Chayenne's avatar
Chayenne committed
302
   "outputs": [],
Chayenne's avatar
Chayenne committed
303
304
305
306
307
   "source": [
    "import json\n",
    "import time\n",
    "from openai import OpenAI\n",
    "\n",
308
    "client = OpenAI(base_url=f\"http://127.0.0.1:{port}/v1\", api_key=\"None\")\n",
Chayenne's avatar
Chayenne committed
309
310
311
312
313
314
315
    "\n",
    "requests = [\n",
    "    {\n",
    "        \"custom_id\": \"request-1\",\n",
    "        \"method\": \"POST\",\n",
    "        \"url\": \"/chat/completions\",\n",
    "        \"body\": {\n",
316
    "            \"model\": \"qwen/qwen2.5-0.5b-instruct\",\n",
Chayenne's avatar
Chayenne committed
317
318
319
320
321
322
323
324
325
326
327
    "            \"messages\": [\n",
    "                {\"role\": \"user\", \"content\": \"Tell me a joke about programming\"}\n",
    "            ],\n",
    "            \"max_tokens\": 50,\n",
    "        },\n",
    "    },\n",
    "    {\n",
    "        \"custom_id\": \"request-2\",\n",
    "        \"method\": \"POST\",\n",
    "        \"url\": \"/chat/completions\",\n",
    "        \"body\": {\n",
328
    "            \"model\": \"qwen/qwen2.5-0.5b-instruct\",\n",
Chayenne's avatar
Chayenne committed
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
    "            \"messages\": [{\"role\": \"user\", \"content\": \"What is Python?\"}],\n",
    "            \"max_tokens\": 50,\n",
    "        },\n",
    "    },\n",
    "]\n",
    "\n",
    "input_file_path = \"batch_requests.jsonl\"\n",
    "\n",
    "with open(input_file_path, \"w\") as f:\n",
    "    for req in requests:\n",
    "        f.write(json.dumps(req) + \"\\n\")\n",
    "\n",
    "with open(input_file_path, \"rb\") as f:\n",
    "    file_response = client.files.create(file=f, purpose=\"batch\")\n",
    "\n",
    "batch_response = client.batches.create(\n",
    "    input_file_id=file_response.id,\n",
    "    endpoint=\"/v1/chat/completions\",\n",
    "    completion_window=\"24h\",\n",
    ")\n",
    "\n",
350
    "print_highlight(f\"Batch job created with ID: {batch_response.id}\")"
Chayenne's avatar
Chayenne committed
351
352
353
354
   ]
  },
  {
   "cell_type": "code",
Chayenne's avatar
Chayenne committed
355
   "execution_count": null,
356
   "metadata": {},
Chayenne's avatar
Chayenne committed
357
   "outputs": [],
Chayenne's avatar
Chayenne committed
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
   "source": [
    "while batch_response.status not in [\"completed\", \"failed\", \"cancelled\"]:\n",
    "    time.sleep(3)\n",
    "    print(f\"Batch job status: {batch_response.status}...trying again in 3 seconds...\")\n",
    "    batch_response = client.batches.retrieve(batch_response.id)\n",
    "\n",
    "if batch_response.status == \"completed\":\n",
    "    print(\"Batch job completed successfully!\")\n",
    "    print(f\"Request counts: {batch_response.request_counts}\")\n",
    "\n",
    "    result_file_id = batch_response.output_file_id\n",
    "    file_response = client.files.content(result_file_id)\n",
    "    result_content = file_response.read().decode(\"utf-8\")\n",
    "\n",
    "    results = [\n",
    "        json.loads(line) for line in result_content.split(\"\\n\") if line.strip() != \"\"\n",
    "    ]\n",
    "\n",
    "    for result in results:\n",
377
378
    "        print_highlight(f\"Request {result['custom_id']}:\")\n",
    "        print_highlight(f\"Response: {result['response']}\")\n",
Chayenne's avatar
Chayenne committed
379
    "\n",
380
    "    print_highlight(\"Cleaning up files...\")\n",
Chayenne's avatar
Chayenne committed
381
382
383
    "    # Only delete the result file ID since file_response is just content\n",
    "    client.files.delete(result_file_id)\n",
    "else:\n",
384
    "    print_highlight(f\"Batch job failed with status: {batch_response.status}\")\n",
Chayenne's avatar
Chayenne committed
385
    "    if hasattr(batch_response, \"errors\"):\n",
386
    "        print_highlight(f\"Errors: {batch_response.errors}\")"
Chayenne's avatar
Chayenne committed
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It takes a while to complete the batch job. You can use these two APIs to retrieve the batch job status or cancel the batch job.\n",
    "\n",
    "1. `batches/{batch_id}`: Retrieve the batch job status.\n",
    "2. `batches/{batch_id}/cancel`: Cancel the batch job.\n",
    "\n",
    "Here is an example to check the batch job status."
   ]
  },
  {
   "cell_type": "code",
Chayenne's avatar
Chayenne committed
403
   "execution_count": null,
404
   "metadata": {},
Chayenne's avatar
Chayenne committed
405
   "outputs": [],
Chayenne's avatar
Chayenne committed
406
407
408
409
410
   "source": [
    "import json\n",
    "import time\n",
    "from openai import OpenAI\n",
    "\n",
411
    "client = OpenAI(base_url=f\"http://127.0.0.1:{port}/v1\", api_key=\"None\")\n",
Chayenne's avatar
Chayenne committed
412
413
    "\n",
    "requests = []\n",
414
    "for i in range(20):\n",
Chayenne's avatar
Chayenne committed
415
416
417
418
419
420
    "    requests.append(\n",
    "        {\n",
    "            \"custom_id\": f\"request-{i}\",\n",
    "            \"method\": \"POST\",\n",
    "            \"url\": \"/chat/completions\",\n",
    "            \"body\": {\n",
421
    "                \"model\": \"qwen/qwen2.5-0.5b-instruct\",\n",
Chayenne's avatar
Chayenne committed
422
423
424
425
426
427
428
429
430
431
    "                \"messages\": [\n",
    "                    {\n",
    "                        \"role\": \"system\",\n",
    "                        \"content\": f\"{i}: You are a helpful AI assistant\",\n",
    "                    },\n",
    "                    {\n",
    "                        \"role\": \"user\",\n",
    "                        \"content\": \"Write a detailed story about topic. Make it very long.\",\n",
    "                    },\n",
    "                ],\n",
432
    "                \"max_tokens\": 64,\n",
Chayenne's avatar
Chayenne committed
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
    "            },\n",
    "        }\n",
    "    )\n",
    "\n",
    "input_file_path = \"batch_requests.jsonl\"\n",
    "with open(input_file_path, \"w\") as f:\n",
    "    for req in requests:\n",
    "        f.write(json.dumps(req) + \"\\n\")\n",
    "\n",
    "with open(input_file_path, \"rb\") as f:\n",
    "    uploaded_file = client.files.create(file=f, purpose=\"batch\")\n",
    "\n",
    "batch_job = client.batches.create(\n",
    "    input_file_id=uploaded_file.id,\n",
    "    endpoint=\"/v1/chat/completions\",\n",
    "    completion_window=\"24h\",\n",
    ")\n",
    "\n",
451
452
    "print_highlight(f\"Created batch job with ID: {batch_job.id}\")\n",
    "print_highlight(f\"Initial status: {batch_job.status}\")\n",
Chayenne's avatar
Chayenne committed
453
454
455
456
457
458
    "\n",
    "time.sleep(10)\n",
    "\n",
    "max_checks = 5\n",
    "for i in range(max_checks):\n",
    "    batch_details = client.batches.retrieve(batch_id=batch_job.id)\n",
459
460
461
462
463
464
465
    "\n",
    "    print_highlight(\n",
    "        f\"Batch job details (check {i+1} / {max_checks}) // ID: {batch_details.id} // Status: {batch_details.status} // Created at: {batch_details.created_at} // Input file ID: {batch_details.input_file_id} // Output file ID: {batch_details.output_file_id}\"\n",
    "    )\n",
    "    print_highlight(\n",
    "        f\"<strong>Request counts: Total: {batch_details.request_counts.total} // Completed: {batch_details.request_counts.completed} // Failed: {batch_details.request_counts.failed}</strong>\"\n",
    "    )\n",
Chayenne's avatar
Chayenne committed
466
467
468
469
470
471
472
473
474
475
476
477
478
    "\n",
    "    time.sleep(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is an example to cancel a batch job."
   ]
  },
  {
   "cell_type": "code",
Chayenne's avatar
Chayenne committed
479
   "execution_count": null,
480
   "metadata": {},
Chayenne's avatar
Chayenne committed
481
   "outputs": [],
Chayenne's avatar
Chayenne committed
482
483
484
485
   "source": [
    "import json\n",
    "import time\n",
    "from openai import OpenAI\n",
Chayenne's avatar
Chayenne committed
486
    "import os\n",
Chayenne's avatar
Chayenne committed
487
    "\n",
488
    "client = OpenAI(base_url=f\"http://127.0.0.1:{port}/v1\", api_key=\"None\")\n",
Chayenne's avatar
Chayenne committed
489
490
    "\n",
    "requests = []\n",
491
    "for i in range(5000):\n",
Chayenne's avatar
Chayenne committed
492
493
494
495
496
497
    "    requests.append(\n",
    "        {\n",
    "            \"custom_id\": f\"request-{i}\",\n",
    "            \"method\": \"POST\",\n",
    "            \"url\": \"/chat/completions\",\n",
    "            \"body\": {\n",
498
    "                \"model\": \"qwen/qwen2.5-0.5b-instruct\",\n",
Chayenne's avatar
Chayenne committed
499
500
501
502
503
504
505
506
507
508
    "                \"messages\": [\n",
    "                    {\n",
    "                        \"role\": \"system\",\n",
    "                        \"content\": f\"{i}: You are a helpful AI assistant\",\n",
    "                    },\n",
    "                    {\n",
    "                        \"role\": \"user\",\n",
    "                        \"content\": \"Write a detailed story about topic. Make it very long.\",\n",
    "                    },\n",
    "                ],\n",
509
    "                \"max_tokens\": 128,\n",
Chayenne's avatar
Chayenne committed
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
    "            },\n",
    "        }\n",
    "    )\n",
    "\n",
    "input_file_path = \"batch_requests.jsonl\"\n",
    "with open(input_file_path, \"w\") as f:\n",
    "    for req in requests:\n",
    "        f.write(json.dumps(req) + \"\\n\")\n",
    "\n",
    "with open(input_file_path, \"rb\") as f:\n",
    "    uploaded_file = client.files.create(file=f, purpose=\"batch\")\n",
    "\n",
    "batch_job = client.batches.create(\n",
    "    input_file_id=uploaded_file.id,\n",
    "    endpoint=\"/v1/chat/completions\",\n",
    "    completion_window=\"24h\",\n",
    ")\n",
    "\n",
528
529
    "print_highlight(f\"Created batch job with ID: {batch_job.id}\")\n",
    "print_highlight(f\"Initial status: {batch_job.status}\")\n",
Chayenne's avatar
Chayenne committed
530
531
532
533
534
    "\n",
    "time.sleep(10)\n",
    "\n",
    "try:\n",
    "    cancelled_job = client.batches.cancel(batch_id=batch_job.id)\n",
535
    "    print_highlight(f\"Cancellation initiated. Status: {cancelled_job.status}\")\n",
Chayenne's avatar
Chayenne committed
536
537
538
539
540
541
    "    assert cancelled_job.status == \"cancelling\"\n",
    "\n",
    "    # Monitor the cancellation process\n",
    "    while cancelled_job.status not in [\"failed\", \"cancelled\"]:\n",
    "        time.sleep(3)\n",
    "        cancelled_job = client.batches.retrieve(batch_job.id)\n",
542
    "        print_highlight(f\"Current status: {cancelled_job.status}\")\n",
Chayenne's avatar
Chayenne committed
543
544
545
    "\n",
    "    # Verify final status\n",
    "    assert cancelled_job.status == \"cancelled\"\n",
546
    "    print_highlight(\"Batch job successfully cancelled\")\n",
Chayenne's avatar
Chayenne committed
547
548
    "\n",
    "except Exception as e:\n",
549
    "    print_highlight(f\"Error during cancellation: {e}\")\n",
Chayenne's avatar
Chayenne committed
550
551
552
553
554
555
    "    raise e\n",
    "\n",
    "finally:\n",
    "    try:\n",
    "        del_response = client.files.delete(uploaded_file.id)\n",
    "        if del_response.deleted:\n",
556
    "            print_highlight(\"Successfully cleaned up input file\")\n",
Chayenne's avatar
Chayenne committed
557
558
559
    "        if os.path.exists(input_file_path):\n",
    "            os.remove(input_file_path)\n",
    "            print_highlight(\"Successfully deleted local batch_requests.jsonl file\")\n",
Chayenne's avatar
Chayenne committed
560
    "    except Exception as e:\n",
561
    "        print_highlight(f\"Error cleaning up: {e}\")\n",
Chayenne's avatar
Chayenne committed
562
563
564
565
566
    "        raise e"
   ]
  },
  {
   "cell_type": "code",
567
568
   "execution_count": null,
   "metadata": {},
Lianmin Zheng's avatar
Lianmin Zheng committed
569
   "outputs": [],
Chayenne's avatar
Chayenne committed
570
   "source": [
571
    "terminate_process(server_process)"
Chayenne's avatar
Chayenne committed
572
573
574
575
   ]
  }
 ],
 "metadata": {
Chayenne's avatar
Chayenne committed
576
577
578
579
580
581
582
583
584
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
585
   "pygments_lexer": "ipython3"
Chayenne's avatar
Chayenne committed
586
587
588
589
590
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}