function_calling.ipynb 23.5 KB
Newer Older
Tanjiro's avatar
Tanjiro committed
1
2
3
4
5
6
{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
YAMY's avatar
YAMY committed
7
    "# Tool and Function Calling\n",
Tanjiro's avatar
Tanjiro committed
8
    "\n",
simveit's avatar
simveit committed
9
    "This guide demonstrates how to use SGLang’s [Funcion calling](https://platform.openai.com/docs/guides/function-calling) functionality."
Tanjiro's avatar
Tanjiro committed
10
11
12
13
14
15
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
YAMY's avatar
YAMY committed
16
17
18
19
20
21
22
23
    "## OpenAI Compatible API"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Launching the Server"
Tanjiro's avatar
Tanjiro committed
24
25
26
27
28
29
30
31
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
YAMY's avatar
YAMY committed
32
33
    "from openai import OpenAI\n",
    "import json\n",
34
35
36
37
38
39
40
    "from sglang.utils import wait_for_server, print_highlight, terminate_process\n",
    "from sglang.test.test_utils import is_in_ci\n",
    "\n",
    "if is_in_ci():\n",
    "    from patch import launch_server_cmd\n",
    "else:\n",
    "    from sglang.utils import launch_server_cmd\n",
41
    "    import nest_asyncio\n",
42
    "\n",
43
    "    nest_asyncio.apply()\n",
Tanjiro's avatar
Tanjiro committed
44
    "\n",
45
    "server_process, port = launch_server_cmd(\n",
46
    "    \"python3 -m sglang.launch_server --model-path Qwen/Qwen2.5-7B-Instruct --tool-call-parser qwen25 --host 0.0.0.0\"  # qwen25\n",
Tanjiro's avatar
Tanjiro committed
47
    ")\n",
48
    "wait_for_server(f\"http://localhost:{port}\")"
YAMY's avatar
YAMY committed
49
50
51
52
53
54
55
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that `--tool-call-parser` defines the parser used to interpret responses. Currently supported parsers include:\n",
Tanjiro's avatar
Tanjiro committed
56
    "\n",
YAMY's avatar
YAMY committed
57
58
59
    "- llama3: Llama 3.1 / 3.2 (e.g. meta-llama/Llama-3.1-8B-Instruct, meta-llama/Llama-3.2-1B-Instruct).\n",
    "- mistral: Mistral (e.g. mistralai/Mistral-7B-Instruct-v0.3, mistralai/Mistral-Nemo-Instruct-2407, mistralai/\n",
    "Mistral-Nemo-Instruct-2407, mistralai/Mistral-7B-v0.3).\n",
60
    "- qwen25: Qwen 2.5 (e.g. Qwen/Qwen2.5-1.5B-Instruct, Qwen/Qwen2.5-7B-Instruct) and QwQ (i.e. Qwen/QwQ-32B). Especially, for QwQ, we can enable the reasoning parser together with tool call parser, details about reasoning parser can be found in [reasoning parser](https://docs.sglang.ai/backend/separate_reasoning.html)."
Tanjiro's avatar
Tanjiro committed
61
62
63
64
65
66
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
YAMY's avatar
YAMY committed
67
68
    "### Define Tools for Function Call\n",
    "Below is a Python snippet that shows how to define a tool as a dictionary. The dictionary includes a tool name, a description, and property defined Parameters."
Tanjiro's avatar
Tanjiro committed
69
70
71
72
73
74
75
76
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
YAMY's avatar
YAMY committed
77
    "# Define tools\n",
Tanjiro's avatar
Tanjiro committed
78
79
80
81
82
83
84
85
86
    "tools = [\n",
    "    {\n",
    "        \"type\": \"function\",\n",
    "        \"function\": {\n",
    "            \"name\": \"get_current_weather\",\n",
    "            \"description\": \"Get the current weather in a given location\",\n",
    "            \"parameters\": {\n",
    "                \"type\": \"object\",\n",
    "                \"properties\": {\n",
YAMY's avatar
YAMY committed
87
88
89
90
91
    "                    \"city\": {\n",
    "                        \"type\": \"string\",\n",
    "                        \"description\": \"The city to find the weather for, e.g. 'San Francisco'\",\n",
    "                    },\n",
    "                    \"state\": {\n",
Tanjiro's avatar
Tanjiro committed
92
    "                        \"type\": \"string\",\n",
YAMY's avatar
YAMY committed
93
94
95
96
97
98
99
    "                        \"description\": \"the two-letter abbreviation for the state that the city is\"\n",
    "                        \" in, e.g. 'CA' which would mean 'California'\",\n",
    "                    },\n",
    "                    \"unit\": {\n",
    "                        \"type\": \"string\",\n",
    "                        \"description\": \"The unit to fetch the temperature in\",\n",
    "                        \"enum\": [\"celsius\", \"fahrenheit\"],\n",
Tanjiro's avatar
Tanjiro committed
100
101
    "                    },\n",
    "                },\n",
YAMY's avatar
YAMY committed
102
    "                \"required\": [\"city\", \"state\", \"unit\"],\n",
Tanjiro's avatar
Tanjiro committed
103
104
105
    "            },\n",
    "        },\n",
    "    }\n",
YAMY's avatar
YAMY committed
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define Messages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_messages():\n",
    "    return [\n",
    "        {\n",
    "            \"role\": \"user\",\n",
126
    "            \"content\": \"What's the weather like in Boston today? Output a reasoning before act, then use the tools to help you.\",\n",
YAMY's avatar
YAMY committed
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
    "        }\n",
    "    ]\n",
    "\n",
    "\n",
    "messages = get_messages()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Initialize the Client"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize OpenAI-like client\n",
148
    "client = OpenAI(api_key=\"None\", base_url=f\"http://0.0.0.0:{port}/v1\")\n",
YAMY's avatar
YAMY committed
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
    "model_name = client.models.list().data[0].id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  Non-Streaming Request"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Non-streaming mode test\n",
    "response_non_stream = client.chat.completions.create(\n",
    "    model=model_name,\n",
    "    messages=messages,\n",
169
    "    temperature=0,\n",
170
171
    "    top_p=0.95,\n",
    "    max_tokens=1024,\n",
YAMY's avatar
YAMY committed
172
173
174
175
    "    stream=False,  # Non-streaming\n",
    "    tools=tools,\n",
    ")\n",
    "print_highlight(\"Non-stream response:\")\n",
176
177
178
179
180
    "print(response_non_stream)\n",
    "print_highlight(\"==== content ====\")\n",
    "print(response_non_stream.choices[0].message.content)\n",
    "print_highlight(\"==== tool_calls ====\")\n",
    "print(response_non_stream.choices[0].message.tool_calls)"
YAMY's avatar
YAMY committed
181
182
183
184
185
186
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
187
188
    "#### Handle Tools\n",
    "When the engine determines it should call a particular tool, it will return arguments or partial arguments through the response. You can parse these arguments and later invoke the tool accordingly."
YAMY's avatar
YAMY committed
189
190
191
192
193
194
195
196
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
197
198
199
    "name_non_stream = response_non_stream.choices[0].message.tool_calls[0].function.name\n",
    "arguments_non_stream = (\n",
    "    response_non_stream.choices[0].message.tool_calls[0].function.arguments\n",
YAMY's avatar
YAMY committed
200
201
    ")\n",
    "\n",
202
203
    "print_highlight(f\"Final streamed function call name: {name_non_stream}\")\n",
    "print_highlight(f\"Final streamed function call arguments: {arguments_non_stream}\")"
YAMY's avatar
YAMY committed
204
205
206
207
208
209
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
210
    "### Streaming Request"
YAMY's avatar
YAMY committed
211
212
213
214
215
216
217
218
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
219
220
221
222
223
    "# Streaming mode test\n",
    "print_highlight(\"Streaming response:\")\n",
    "response_stream = client.chat.completions.create(\n",
    "    model=model_name,\n",
    "    messages=messages,\n",
224
    "    temperature=0,\n",
225
226
227
228
    "    top_p=0.95,\n",
    "    max_tokens=1024,\n",
    "    stream=True,  # Enable streaming\n",
    "    tools=tools,\n",
YAMY's avatar
YAMY committed
229
230
    ")\n",
    "\n",
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
    "texts = \"\"\n",
    "tool_calls = []\n",
    "name = \"\"\n",
    "arguments = \"\"\n",
    "for chunk in response_stream:\n",
    "    if chunk.choices[0].delta.content:\n",
    "        texts += chunk.choices[0].delta.content\n",
    "    if chunk.choices[0].delta.tool_calls:\n",
    "        tool_calls.append(chunk.choices[0].delta.tool_calls[0])\n",
    "print_highlight(\"==== Text ====\")\n",
    "print(texts)\n",
    "\n",
    "print_highlight(\"==== Tool Call ====\")\n",
    "for tool_call in tool_calls:\n",
    "    print(tool_call)"
YAMY's avatar
YAMY committed
246
247
248
249
250
251
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
252
253
    "#### Handle Tools\n",
    "When the engine determines it should call a particular tool, it will return arguments or partial arguments through the response. You can parse these arguments and later invoke the tool accordingly."
YAMY's avatar
YAMY committed
254
255
256
257
258
259
260
261
262
263
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Parse and combine function call arguments\n",
    "arguments = []\n",
264
265
266
267
268
269
    "for tool_call in tool_calls:\n",
    "    if tool_call.function.name:\n",
    "        print_highlight(f\"Streamed function call name: {tool_call.function.name}\")\n",
    "\n",
    "    if tool_call.function.arguments:\n",
    "        arguments.append(tool_call.function.arguments)\n",
YAMY's avatar
YAMY committed
270
271
272
    "\n",
    "# Combine all fragments into a single JSON string\n",
    "full_arguments = \"\".join(arguments)\n",
273
    "print_highlight(f\"streamed function call arguments: {full_arguments}\")"
YAMY's avatar
YAMY committed
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define a Tool Function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This is a demonstration, define real function according to your usage.\n",
    "def get_current_weather(city: str, state: str, unit: \"str\"):\n",
    "    return (\n",
    "        f\"The weather in {city}, {state} is 85 degrees {unit}. It is \"\n",
    "        \"partly cloudly, with highs in the 90's.\"\n",
    "    )\n",
    "\n",
    "\n",
    "available_tools = {\"get_current_weather\": get_current_weather}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Execute the Tool"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
314
    "messages.append(response_non_stream.choices[0].message)\n",
YAMY's avatar
YAMY committed
315
    "\n",
316
317
318
319
320
321
322
    "# Call the corresponding tool function\n",
    "tool_call = messages[-1].tool_calls[0]\n",
    "tool_name = tool_call.function.name\n",
    "tool_to_call = available_tools[tool_name]\n",
    "result = tool_to_call(**(json.loads(tool_call.function.arguments)))\n",
    "print_highlight(f\"Function call result: {result}\")\n",
    "# messages.append({\"role\": \"tool\", \"content\": result, \"name\": tool_name})\n",
YAMY's avatar
YAMY committed
323
324
    "messages.append(\n",
    "    {\n",
325
326
327
328
    "        \"role\": \"tool\",\n",
    "        \"tool_call_id\": tool_call.id,\n",
    "        \"content\": str(result),\n",
    "        \"name\": tool_name,\n",
YAMY's avatar
YAMY committed
329
330
    "    }\n",
    ")\n",
Tanjiro's avatar
Tanjiro committed
331
    "\n",
YAMY's avatar
YAMY committed
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
    "print_highlight(f\"Updated message history: {messages}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Send Results Back to Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "final_response = client.chat.completions.create(\n",
Tanjiro's avatar
Tanjiro committed
349
350
    "    model=model_name,\n",
    "    messages=messages,\n",
351
    "    temperature=0,\n",
352
    "    top_p=0.95,\n",
Tanjiro's avatar
Tanjiro committed
353
354
355
    "    stream=False,\n",
    "    tools=tools,\n",
    ")\n",
YAMY's avatar
YAMY committed
356
    "print_highlight(\"Non-stream response:\")\n",
357
358
359
360
    "print(final_response)\n",
    "\n",
    "print_highlight(\"==== Text ====\")\n",
    "print(final_response.choices[0].message.content)"
YAMY's avatar
YAMY committed
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Native API and SGLang Runtime (SRT)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import AutoTokenizer\n",
    "import requests\n",
    "\n",
    "# generate an answer\n",
380
    "tokenizer = AutoTokenizer.from_pretrained(\"Qwen/Qwen2.5-7B-Instruct\")\n",
YAMY's avatar
YAMY committed
381
382
383
384
385
386
387
388
389
    "\n",
    "messages = get_messages()\n",
    "\n",
    "input = tokenizer.apply_chat_template(\n",
    "    messages,\n",
    "    tokenize=False,\n",
    "    add_generation_prompt=True,\n",
    "    tools=tools,\n",
    ")\n",
Tanjiro's avatar
Tanjiro committed
390
    "\n",
391
    "gen_url = f\"http://localhost:{port}/generate\"\n",
392
393
394
395
396
    "gen_data = {\n",
    "    \"text\": input,\n",
    "    \"sampling_params\": {\n",
    "        \"skip_special_tokens\": False,\n",
    "        \"max_new_tokens\": 1024,\n",
397
    "        \"temperature\": 0,\n",
398
399
400
    "        \"top_p\": 0.95,\n",
    "    },\n",
    "}\n",
YAMY's avatar
YAMY committed
401
    "gen_response = requests.post(gen_url, json=gen_data).json()[\"text\"]\n",
402
    "print_highlight(\"==== Reponse ====\")\n",
YAMY's avatar
YAMY committed
403
    "print(gen_response)\n",
Tanjiro's avatar
Tanjiro committed
404
    "\n",
YAMY's avatar
YAMY committed
405
    "# parse the response\n",
406
    "parse_url = f\"http://localhost:{port}/parse_function_call\"\n",
Tanjiro's avatar
Tanjiro committed
407
    "\n",
YAMY's avatar
YAMY committed
408
409
    "function_call_input = {\n",
    "    \"text\": gen_response,\n",
410
    "    \"tool_call_parser\": \"qwen25\",\n",
YAMY's avatar
YAMY committed
411
412
    "    \"tools\": tools,\n",
    "}\n",
Tanjiro's avatar
Tanjiro committed
413
    "\n",
YAMY's avatar
YAMY committed
414
415
    "function_call_response = requests.post(parse_url, json=function_call_input)\n",
    "function_call_response_json = function_call_response.json()\n",
416
417
418
419
    "\n",
    "print_highlight(\"==== Text ====\")\n",
    "print(function_call_response_json[\"normal_text\"])\n",
    "print_highlight(\"==== Calls ====\")\n",
YAMY's avatar
YAMY committed
420
421
    "print(\"function name: \", function_call_response_json[\"calls\"][0][\"name\"])\n",
    "print(\"function arguments: \", function_call_response_json[\"calls\"][0][\"parameters\"])"
Tanjiro's avatar
Tanjiro committed
422
423
424
425
426
427
428
429
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
430
    "terminate_process(server_process)"
Tanjiro's avatar
Tanjiro committed
431
432
433
434
435
436
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
YAMY's avatar
YAMY committed
437
438
439
440
441
442
443
444
445
446
447
448
449
    "## Offline Engine API"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sglang as sgl\n",
    "from sglang.srt.function_call_parser import FunctionCallParser\n",
    "from sglang.srt.managers.io_struct import Tool, Function\n",
    "\n",
450
    "llm = sgl.Engine(model_path=\"Qwen/Qwen2.5-7B-Instruct\")\n",
451
    "tokenizer = llm.tokenizer_manager.tokenizer\n",
YAMY's avatar
YAMY committed
452
453
454
455
456
    "input_ids = tokenizer.apply_chat_template(\n",
    "    messages, tokenize=True, add_generation_prompt=True, tools=tools\n",
    ")\n",
    "\n",
    "sampling_params = {\n",
457
    "    \"max_new_tokens\": 1024,\n",
458
    "    \"temperature\": 0,\n",
YAMY's avatar
YAMY committed
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
    "    \"top_p\": 0.95,\n",
    "    \"skip_special_tokens\": False,\n",
    "}\n",
    "\n",
    "# 1) Offline generation\n",
    "result = llm.generate(input_ids=input_ids, sampling_params=sampling_params)\n",
    "generated_text = result[\"text\"]  # Assume there is only one prompt\n",
    "\n",
    "print(\"=== Offline Engine Output Text ===\")\n",
    "print(generated_text)\n",
    "\n",
    "\n",
    "# 2) Parse using FunctionCallParser\n",
    "def convert_dict_to_tool(tool_dict: dict) -> Tool:\n",
    "    function_dict = tool_dict.get(\"function\", {})\n",
    "    return Tool(\n",
    "        type=tool_dict.get(\"type\", \"function\"),\n",
    "        function=Function(\n",
    "            name=function_dict.get(\"name\"),\n",
    "            description=function_dict.get(\"description\"),\n",
    "            parameters=function_dict.get(\"parameters\"),\n",
    "        ),\n",
    "    )\n",
    "\n",
    "\n",
    "tools = [convert_dict_to_tool(raw_tool) for raw_tool in tools]\n",
    "\n",
486
    "parser = FunctionCallParser(tools=tools, tool_call_parser=\"qwen25\")\n",
YAMY's avatar
YAMY committed
487
488
    "normal_text, calls = parser.parse_non_stream(generated_text)\n",
    "\n",
489
    "print(\"=== Parsing Result ===\")\n",
YAMY's avatar
YAMY committed
490
491
492
493
494
495
    "print(\"Normal text portion:\", normal_text)\n",
    "print(\"Function call portion:\")\n",
    "for call in calls:\n",
    "    # call: ToolCallItem\n",
    "    print(f\"  - tool name: {call.name}\")\n",
    "    print(f\"    parameters: {call.parameters}\")\n",
Tanjiro's avatar
Tanjiro committed
496
    "\n",
YAMY's avatar
YAMY committed
497
498
499
500
501
502
503
504
505
506
507
508
    "# 3) If needed, perform additional logic on the parsed functions, such as automatically calling the corresponding function to obtain a return value, etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "llm.shutdown()"
   ]
  },
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pythonic Tool Call Format (Llama-3.2 / Llama-3.3 / Llama-4)\n",
    "\n",
    "Some Llama models (such as Llama-3.2-1B, Llama-3.2-3B, Llama-3.3-70B, and Llama-4) support a \"pythonic\" tool call format, where the model outputs function calls as Python code, e.g.:\n",
    "\n",
    "```python\n",
    "[get_current_weather(city=\"San Francisco\", state=\"CA\", unit=\"celsius\")]\n",
    "```\n",
    "\n",
    "- The output is a Python list of function calls, with arguments as Python literals (not JSON).\n",
    "- Multiple tool calls can be returned in the same list:\n",
    "```python\n",
    "[get_current_weather(city=\"San Francisco\", state=\"CA\", unit=\"celsius\"),\n",
    " get_current_weather(city=\"New York\", state=\"NY\", unit=\"fahrenheit\")]\n",
    "```\n",
    "\n",
    "For more information, refer to Meta’s documentation on  [Zero shot function calling](https://github.com/meta-llama/llama-models/blob/main/models/llama4/prompt_format.md#zero-shot-function-calling---system-message).\n",
    "\n",
    "### How to enable\n",
    "- Launch the server with `--tool-call-parser pythonic`\n",
    "- You may also specify --chat-template with the improved template for the model (e.g., `--chat-template=examples/chat_template/tool_chat_template_llama4_pythonic.jinja`).\n",
    "This is recommended because the model expects a special prompt format to reliably produce valid pythonic tool call outputs. The template ensures that the prompt structure (e.g., special tokens, message boundaries like `<|eom|>`, and function call delimiters) matches what the model was trained or fine-tuned on. If you do not use the correct chat template, tool calling may fail or produce inconsistent results.\n",
    "\n",
    "#### Forcing Pythonic Tool Call Output Without a Chat Template\n",
    "If you don't want to specify a chat template, you must give the model extremely explicit instructions in your messages to enforce pythonic output. For example, for `Llama-3.2-1B-Instruct`, you need:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import openai\n",
    "\n",
    "server_process, port = launch_server_cmd(\n",
    "    \" python3 -m sglang.launch_server --model-path meta-llama/Llama-3.2-1B-Instruct --tool-call-parser pythonic --tp 1\"  # llama-3.2-1b-instruct\n",
    ")\n",
    "wait_for_server(f\"http://localhost:{port}\")\n",
    "\n",
    "tools = [\n",
    "    {\n",
    "        \"type\": \"function\",\n",
    "        \"function\": {\n",
    "            \"name\": \"get_weather\",\n",
    "            \"description\": \"Get the current weather for a given location.\",\n",
    "            \"parameters\": {\n",
    "                \"type\": \"object\",\n",
    "                \"properties\": {\n",
    "                    \"location\": {\n",
    "                        \"type\": \"string\",\n",
    "                        \"description\": \"The name of the city or location.\",\n",
    "                    }\n",
    "                },\n",
    "                \"required\": [\"location\"],\n",
    "            },\n",
    "        },\n",
    "    },\n",
    "    {\n",
    "        \"type\": \"function\",\n",
    "        \"function\": {\n",
    "            \"name\": \"get_tourist_attractions\",\n",
    "            \"description\": \"Get a list of top tourist attractions for a given city.\",\n",
    "            \"parameters\": {\n",
    "                \"type\": \"object\",\n",
    "                \"properties\": {\n",
    "                    \"city\": {\n",
    "                        \"type\": \"string\",\n",
    "                        \"description\": \"The name of the city to find attractions for.\",\n",
    "                    }\n",
    "                },\n",
    "                \"required\": [\"city\"],\n",
    "            },\n",
    "        },\n",
    "    },\n",
    "]\n",
    "\n",
    "\n",
    "def get_messages():\n",
    "    return [\n",
    "        {\n",
    "            \"role\": \"system\",\n",
    "            \"content\": (\n",
    "                \"You are a travel assistant. \"\n",
    "                \"When asked to call functions, ALWAYS respond ONLY with a python list of function calls, \"\n",
    "                \"using this format: [func_name1(param1=value1, param2=value2), func_name2(param=value)]. \"\n",
    "                \"Do NOT use JSON, do NOT use variables, do NOT use any other format. \"\n",
    "                \"Here is an example:\\n\"\n",
    "                '[get_weather(location=\"Paris\"), get_tourist_attractions(city=\"Paris\")]'\n",
    "            ),\n",
    "        },\n",
    "        {\n",
    "            \"role\": \"user\",\n",
    "            \"content\": (\n",
    "                \"I'm planning a trip to Tokyo next week. What's the weather like and what are some top tourist attractions? \"\n",
    "                \"Propose parallel tool calls at once, using the python list of function calls format as shown above.\"\n",
    "            ),\n",
    "        },\n",
    "    ]\n",
    "\n",
    "\n",
    "messages = get_messages()\n",
    "\n",
    "client = openai.Client(base_url=f\"http://localhost:{port}/v1\", api_key=\"xxxxxx\")\n",
    "model_name = client.models.list().data[0].id\n",
    "\n",
    "\n",
    "response_non_stream = client.chat.completions.create(\n",
    "    model=model_name,\n",
    "    messages=messages,\n",
622
623
    "    temperature=0,\n",
    "    top_p=0.9,\n",
624
625
626
627
628
629
630
631
632
    "    stream=False,  # Non-streaming\n",
    "    tools=tools,\n",
    ")\n",
    "print_highlight(\"Non-stream response:\")\n",
    "print(response_non_stream)\n",
    "\n",
    "response_stream = client.chat.completions.create(\n",
    "    model=model_name,\n",
    "    messages=messages,\n",
633
634
    "    temperature=0,\n",
    "    top_p=0.9,\n",
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
    "    stream=True,\n",
    "    tools=tools,\n",
    ")\n",
    "texts = \"\"\n",
    "tool_calls = []\n",
    "name = \"\"\n",
    "arguments = \"\"\n",
    "\n",
    "for chunk in response_stream:\n",
    "    if chunk.choices[0].delta.content:\n",
    "        texts += chunk.choices[0].delta.content\n",
    "    if chunk.choices[0].delta.tool_calls:\n",
    "        tool_calls.append(chunk.choices[0].delta.tool_calls[0])\n",
    "\n",
    "print_highlight(\"Streaming Response:\")\n",
    "print_highlight(\"==== Text ====\")\n",
    "print(texts)\n",
    "\n",
    "print_highlight(\"==== Tool Call ====\")\n",
    "for tool_call in tool_calls:\n",
    "    print(tool_call)\n",
    "\n",
    "terminate_process(server_process)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **Note:**  \n",
    "> The model may still default to JSON if it was heavily finetuned on that format. Prompt engineering (including examples) is the only way to increase the chance of pythonic output if you are not using a chat template."
   ]
  },
YAMY's avatar
YAMY committed
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How to support a new model?\n",
    "1. Update the TOOLS_TAG_LIST in sglang/srt/function_call_parser.py with the model’s tool tags. Currently supported tags include:\n",
    "```\n",
    "\tTOOLS_TAG_LIST = [\n",
    "\t    “<|plugin|>“,\n",
    "\t    “<function=“,\n",
    "\t    “<tool_call>“,\n",
    "\t    “<|python_tag|>“,\n",
    "\t    “[TOOL_CALLS]”\n",
    "\t]\n",
    "```\n",
    "2. Create a new detector class in sglang/srt/function_call_parser.py that inherits from BaseFormatDetector. The detector should handle the model’s specific function call format. For example:\n",
    "```\n",
    "    class NewModelDetector(BaseFormatDetector):\n",
    "```\n",
    "3. Add the new detector to the MultiFormatParser class that manages all the format detectors."
Tanjiro's avatar
Tanjiro committed
688
689
690
691
692
   ]
  }
 ],
 "metadata": {
  "language_info": {
693
694
695
696
697
698
699
700
701
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3"
Tanjiro's avatar
Tanjiro committed
702
703
704
  }
 },
 "nbformat": 4,
705
 "nbformat_minor": 4
Tanjiro's avatar
Tanjiro committed
706
}