added chatglm3

requirements.txt

+protobuf
+pydantic==1.10.9
+transformers==4.33.0
+sentencepiece==0.1.99
+accelerate==0.21.0
+astunparse
+sse-starlette
+

resources/WECHAT.md

+<div align="center">
+<img src=wechat.jpg width="60%"/>
+
+<p> 扫码关注公众号，加入「ChatGLM交流群」 </p>
+<p> Scan the QR code to follow the official account and join the "ChatGLM Discussion Group" </p>
+</div>
+

tool_using/README.md

+# 工具调用
+本文档将介绍如何使用 ChatGLM3-6B 进行工具调用。目前只有 ChatGLM3-6B 模型支持工具调用，而 ChatGLM3-6B-Base 和 ChatGLM3-6B-32K 模型不支持。
+
+## 构建 System Prompt
+这里以两个工具调用为例，首先准备好要构建的数据的描述信息。
+
+```python
+tools = [
+    {
+        "name": "track",
+        "description": "追踪指定股票的实时价格",
+        "parameters": {
+            "type": "object",
+            "properties": {
+                "symbol": {
+                    "description": "需要追踪的股票代码"
+                }
+            },
+            "required": ['symbol']
+        }
+    },
+    {
+        "name": "text-to-speech",
+        "description": "将文本转换为语音",
+        "parameters": {
+            "type": "object",
+            "properties": {
+                "text": {
+                    "description": "需要转换成语音的文本"
+                },
+                "voice": {
+                    "description": "要使用的语音类型（男声、女声等）"
+                },
+                "speed": {
+                    "description": "语音的速度（快、中等、慢等）"
+                }
+            },
+            "required": ['text']
+        }
+    }
+]
+system_info = {"role": "system", "content": "Answer the following questions as best as you can. You have access to the following tools:", "tools": tools}
+```
+请确保工具的定义格式与例子中一致以获得最优的性能
+
+## 提出问题
+注意：目前 ChatGLM3-6B 的工具调用只支持通过 `chat` 方法，不支持  `stream_chat` 方法。
+```python
+history = [system_info]
+query = "帮我查询股票10111的价格"
+response, history = model.chat(tokenizer, query, history=history)
+print(response)
+```
+这里期望得到的输出为
+```json
+{"name": "track", "parameters": {"symbol": "10111"}}
+```
+这表示模型需要调用工具 `track`，并且需要传入参数 `symbol`。
+
+## 调用工具，生成回复
+这里需要自行实现调用工具的逻辑。假设已经得到了返回结果，将结果以 json 格式返回给模型并得到回复。
+```python
+result = json.dumps({"price": 12412}, ensure_ascii=False)
+response, history = model.chat(tokenizer, result, history=history, role="observation")
+print(response)
+```
+这里 `role="observation"` 表示输入的是工具调用的返回值而不是用户输入，不能省略。
+
+期望得到的输出为
+```
+根据您的查询，经过API的调用，股票10111的价格是12412。
+```
+
+这表示本次工具调用已经结束，模型根据返回结果生成回复。对于比较复杂的问题，模型可能需要进行多次工具调用。这时，可以根据返回的 `response` 是 `str` 还是 `dict` 来判断返回的是生成的回复还是工具调用请求。
\ No newline at end of file

tool_using/README_en.md

+# Tool Invocation
+This document will introduce how to use the ChatGLM3-6B for tool invocation. Currently, only the ChatGLM3-6B model supports tool invocation, while the ChatGLM3-6B-Base and ChatGLM3-6B-32K models do not support it.
+
+## Building System Prompt
+Here are two examples of tool invocation. First, prepare the description information of the data to be built.
+
+```python
+tools = [
+    {
+        "name": "track",
+        "description": "Track the real-time price of a specified stock",
+        "parameters": {
+            "type": "object",
+            "properties": {
+                "symbol": {
+                    "description": "The stock code that needs to be tracked"
+                }
+            },
+            "required": ['symbol']
+        }
+    },
+    {
+        "name": "text-to-speech",
+        "description": "Convert text to speech",
+        "parameters": {
+            "type": "object",
+            "properties": {
+                "text": {
+                    "description": "The text that needs to be converted into speech"
+                },
+                "voice": {
+                    "description": "The type of voice to use (male, female, etc.)"
+                },
+                "speed": {
+                    "description": "The speed of the speech (fast, medium, slow, etc.)"
+                }
+            },
+            "required": ['text']
+        }
+    }
+]
+system_info = {"role": "system", "content": "Answer the following questions as best as you can. You have access to the following tools:", "tools": tools}
+```
+
+Please ensure that the definition format of the tool is consistent with the example to obtain optimal performance.
+
+## Asking Questions
+Note: Currently, the tool invocation of ChatGLM3-6B only supports the `chat` method and does not support the `stream_chat` method.
+```python
+history = [system_info]
+query = "Help me inquire the price of stock 10111"
+response, history = model.chat(tokenizer, query, history=history)
+print(response)
+```
+The expected output here is
+```json
+{"name": "track", "parameters": {"symbol": "10111"}}
+```
+This indicates that the model needs to call the tool `track`, and the parameter `symbol` needs to be passed in.
+
+## Invoke Tool, Generate Response
+Here, you need to implement the logic of calling the tool yourself. Assuming that the return result has been obtained, return the result to the model in json format and get a response.
+```python
+result = json.dumps({"price": 12412}, ensure_ascii=False)
+response, history = model.chat(tokenizer, result, history=history, role="observation")
+print(response)
+```
+Here `role="observation"` indicates that the input is the return value of the tool invocation rather than user input, and it cannot be omitted.
+
+The expected output is
+```
+Based on your query, after the API call, the price of stock 10111 is 12412.
+```
+
+This indicates that this tool invocation has ended, and the model generates a response based on the return result. For more complex questions, the model may need to make multiple tool invocations. At this time, you can judge whether the returned `response` is `str` or `dict` to determine whether the return is a generated response or a tool invocation request.
\ No newline at end of file

tool_using/cli_demo_tool.py

+import os
+import platform
+from transformers import AutoTokenizer, AutoModel
+import torch
+
+MODEL_PATH = os.environ.get('MODEL_PATH', 'THUDM/chatglm3-6b')
+TOKENIZER_PATH = os.environ.get("TOKENIZER_PATH", MODEL_PATH)
+DEVICE = 'cuda' if torch.cuda.is_available() else 'cpu'
+
+tokenizer = AutoTokenizer.from_pretrained(TOKENIZER_PATH, trust_remote_code=True)
+if 'cuda' in DEVICE: # AMD, NVIDIA GPU can use Half Precision
+    model = AutoModel.from_pretrained(MODEL_PATH, trust_remote_code=True).to(DEVICE).eval()
+else: # CPU, Intel GPU and other GPU can use Float16 Precision Only
+    model = AutoModel.from_pretrained(MODEL_PATH, trust_remote_code=True).float().to(DEVICE).eval()
+
+os_name = platform.system()
+clear_command = 'cls' if os_name == 'Windows' else 'clear'
+stop_stream = False
+
+
+def build_prompt(history):
+    prompt = "欢迎使用 ChatGLM3-6B 模型，输入内容即可进行对话，clear 清空对话历史，stop 终止程序"
+    for query, response in history:
+        prompt += f"\n\n用户：{query}"
+        prompt += f"\n\nChatGLM3-6B：{response}"
+    return prompt
+
+
+tools = [{'name': 'track', 'description': '追踪指定股票的实时价格', 'parameters': {'type': 'object', 'properties': {'symbol': {'description': '需要追踪的股票代码'}}, 'required': []}}, {'name': '/text-to-speech', 'description': '将文本转换为语音', 'parameters': {'type': 'object', 'properties': {'text': {'description': '需要转换成语音的文本'}, 'voice': {'description': '要使用的语音类型（男声、女声等）'}, 'speed': {'description': '语音的速度（快、中等、慢等）'}}, 'required': []}}, {'name': '/image_resizer', 'description': '调整图片的大小和尺寸', 'parameters': {'type': 'object', 'properties': {'image_file': {'description': '需要调整大小的图片文件'}, 'width': {'description': '需要调整的宽度值'}, 'height': {'description': '需要调整的高度值'}}, 'required': []}}, {'name': '/foodimg', 'description': '通过给定的食品名称生成该食品的图片', 'parameters': {'type': 'object', 'properties': {'food_name': {'description': '需要生成图片的食品名称'}}, 'required': []}}]
+system_item = {"role": "system",
+               "content": "Answer the following questions as best as you can. You have access to the following tools:",
+               "tools": tools}
+
+def main():
+    past_key_values, history = None, [system_item]
+    role = "user"
+    global stop_stream
+    print("欢迎使用 ChatGLM3-6B 模型，输入内容即可进行对话，clear 清空对话历史，stop 终止程序")
+    while True:
+        query = input("\n用户：") if role == "user" else input("\n结果：")
+        if query.strip() == "stop":
+            break
+        if query.strip() == "clear":
+            past_key_values, history = None,  [system_item]
+            role = "user"
+            os.system(clear_command)
+            print("欢迎使用 ChatGLM3-6B 模型，输入内容即可进行对话，clear 清空对话历史，stop 终止程序")
+            continue
+        print("\nChatGLM：", end="")
+        response, history = model.chat(tokenizer, query, history=history, role=role)
+        print(response, end="", flush=True)
+        print("")
+        if isinstance(response, dict):
+            role = "observation"
+
+
+if __name__ == "__main__":
+    main()
\ No newline at end of file

tool_using/openai_api_demo.py

+import json
+
+from openai import OpenAI
+from colorama import init, Fore
+from loguru import logger
+
+from tool_register import get_tools, dispatch_tool
+
+init(autoreset=True)
+client = OpenAI(
+  base_url="http://127.0.0.1:8000/v1",
+  api_key = "xxx"
+)
+
+functions = get_tools()
+
+
+def run_conversation(query: str, stream=False, functions=None, max_retry=5):
+    params = dict(model="chatglm3", messages=[{"role": "user", "content": query}], stream=stream)
+    if functions:
+        params["functions"] = functions
+    response = client.chat.completions.create(**params)
+
+    for _ in range(max_retry):
+        if not stream:
+            if response.choices[0].message.function_call:
+                function_call = response.choices[0].message.function_call
+                logger.info(f"Function Call Response: {function_call.model_dump()}")
+
+                function_args = json.loads(function_call.arguments)
+                tool_response = dispatch_tool(function_call.name, function_args)
+                logger.info(f"Tool Call Response: {tool_response}")
+
+                params["messages"].append(response.choices[0].message)
+                params["messages"].append(
+                    {
+                        "role": "function",
+                        "name": function_call.name,
+                        "content": tool_response,  # 调用函数返回结果
+                    }
+                )
+            else:
+                reply = response.choices[0].message.content
+                logger.info(f"Final Reply: \n{reply}")
+                return
+
+        else:
+            output = ""
+            for chunk in response:
+                content = chunk.choices[0].delta.content or ""
+                print(Fore.BLUE + content, end="", flush=True)
+                output += content
+
+                if chunk.choices[0].finish_reason == "stop":
+                    return
+
+                elif chunk.choices[0].finish_reason == "function_call":
+                    print("\n")
+
+                    function_call = chunk.choices[0].delta.function_call
+                    logger.info(f"Function Call Response: {function_call.model_dump()}")
+
+                    function_args = json.loads(function_call.arguments)
+                    tool_response = dispatch_tool(function_call.name, function_args)
+                    logger.info(f"Tool Call Response: {tool_response}")
+
+                    params["messages"].append(
+                        {
+                            "role": "assistant",
+                            "content": output
+                        }
+                    )
+                    params["messages"].append(
+                        {
+                            "role": "function",
+                            "name": function_call.name,
+                            "content": tool_response,  # 调用函数返回结果
+                        }
+                    )
+
+                    break
+
+        response = client.chat.completions.create(**params)
+
+
+if __name__ == "__main__":
+    query = "你是谁"
+    run_conversation(query, stream=True)
+
+    logger.info("\n=========== next conversation ===========")
+
+    query = "帮我查询北京的天气怎么样"
+    run_conversation(query, functions=functions, stream=True)

tool_using/requirements.txt

+openai>=1.2.0

tool_using/test.py

+tools = [
+    {
+        "name": "track",
+        "description": "追踪指定股票的实时价格",
+        "parameters": {
+            "type": "object",
+            "properties": {
+                "symbol": {
+                    "description": "需要追踪的股票代码"
+                }
+            },
+            "required": ['symbol']
+        }
+    },
+    {
+        "name": "text-to-speech",
+        "description": "将文本转换为语音",
+        "parameters": {
+            "type": "object",
+            "properties": {
+                "text": {
+                    "description": "需要转换成语音的文本"
+                },
+                "voice": {
+                    "description": "要使用的语音类型（男声、女声等）"
+                },
+                "speed": {
+                    "description": "语音的速度（快、中等、慢等）"
+                }
+            },
+            "required": ['text']
+        }
+    }
+]
+system_info = {"role": "system", "content": "Answer the following questions as best as you can. You have access to the following tools:", "tools": tools}
+
+import os
+import platform
+from transformers import AutoTokenizer, AutoModel
+import torch
+
+MODEL_PATH = os.environ.get('MODEL_PATH', '../../chatglm3-6b')
+TOKENIZER_PATH = os.environ.get("TOKENIZER_PATH", MODEL_PATH)
+DEVICE = 'cuda' if torch.cuda.is_available() else 'cpu'
+tokenizer = AutoTokenizer.from_pretrained(TOKENIZER_PATH, trust_remote_code=True)
+if 'cuda' in DEVICE: # AMD, NVIDIA GPU can use Half Precision
+    model = AutoModel.from_pretrained(MODEL_PATH, trust_remote_code=True).to(DEVICE).eval()
+else: # CPU, Intel GPU and other GPU can use Float16 Precision Only
+    model = AutoModel.from_pretrained(MODEL_PATH, trust_remote_code=True).float().to(DEVICE).eval()
+
+history = [system_info]
+query = "帮我查询股票10111的价格"
+response, history = model.chat(tokenizer, query, history=history)
+print(response)
\ No newline at end of file

tool_using/tool_register.py

+import inspect
+import traceback
+from copy import deepcopy
+from pprint import pformat
+from types import GenericAlias
+from typing import get_origin, Annotated
+
+_TOOL_HOOKS = {}
+_TOOL_DESCRIPTIONS = {}
+
+
+def register_tool(func: callable):
+    tool_name = func.__name__
+    tool_description = inspect.getdoc(func).strip()
+    python_params = inspect.signature(func).parameters
+    tool_params = []
+    for name, param in python_params.items():
+        annotation = param.annotation
+        if annotation is inspect.Parameter.empty:
+            raise TypeError(f"Parameter `{name}` missing type annotation")
+        if get_origin(annotation) != Annotated:
+            raise TypeError(f"Annotation type for `{name}` must be typing.Annotated")
+
+        typ, (description, required) = annotation.__origin__, annotation.__metadata__
+        typ: str = str(typ) if isinstance(typ, GenericAlias) else typ.__name__
+        if not isinstance(description, str):
+            raise TypeError(f"Description for `{name}` must be a string")
+        if not isinstance(required, bool):
+            raise TypeError(f"Required for `{name}` must be a bool")
+
+        tool_params.append({
+            "name": name,
+            "description": description,
+            "type": typ,
+            "required": required
+        })
+    tool_def = {
+        "name": tool_name,
+        "description": tool_description,
+        "params": tool_params
+    }
+
+    print("[registered tool] " + pformat(tool_def))
+    _TOOL_HOOKS[tool_name] = func
+    _TOOL_DESCRIPTIONS[tool_name] = tool_def
+
+    return func
+
+
+def dispatch_tool(tool_name: str, tool_params: dict) -> str:
+    if tool_name not in _TOOL_HOOKS:
+        return f"Tool `{tool_name}` not found. Please use a provided tool."
+    tool_call = _TOOL_HOOKS[tool_name]
+    try:
+        ret = tool_call(**tool_params)
+    except:
+        ret = traceback.format_exc()
+    return str(ret)
+
+
+def get_tools() -> dict:
+    return deepcopy(_TOOL_DESCRIPTIONS)
+
+
+# Tool Definitions
+
+@register_tool
+def random_number_generator(
+        seed: Annotated[int, 'The random seed used by the generator', True],
+        range: Annotated[tuple[int, int], 'The range of the generated numbers', True],
+) -> int:
+    """
+    Generates a random number x, s.t. range[0] <= x < range[1]
+    """
+    if not isinstance(seed, int):
+        raise TypeError("Seed must be an integer")
+    if not isinstance(range, tuple):
+        raise TypeError("Range must be a tuple")
+    if not isinstance(range[0], int) or not isinstance(range[1], int):
+        raise TypeError("Range must be a tuple of integers")
+
+    import random
+    return random.Random(seed).randint(*range)
+
+
+@register_tool
+def get_weather(
+        city_name: Annotated[str, 'The name of the city to be queried', True],
+) -> str:
+    """
+    Get the current weather for `city_name`
+    """
+
+    if not isinstance(city_name, str):
+        raise TypeError("City name must be a string")
+
+    key_selection = {
+        "current_condition": ["temp_C", "FeelsLikeC", "humidity", "weatherDesc", "observation_time"],
+    }
+    import requests
+    try:
+        resp = requests.get(f"https://wttr.in/{city_name}?format=j1")
+        resp.raise_for_status()
+        resp = resp.json()
+        ret = {k: {_v: resp[k][0][_v] for _v in v} for k, v in key_selection.items()}
+    except:
+        import traceback
+        ret = "Error encountered while fetching weather data!\n" + traceback.format_exc()
+
+    return str(ret)
+
+
+if __name__ == "__main__":
+    print(dispatch_tool("get_weather", {"city_name": "beijing"}))
+    print(get_tools())