Skip to content

GitLab

Switch branch/tag
Find file Normal viewHistoryPermalink
openllm.rst 4.14 KB
Newer Older
chenzk's avatar
v1.0  
chenzk committed
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125 
OpenLLM
=======

.. attention:: 
    To be updated for Qwen3.

OpenLLM allows developers to run Qwen2.5 models of different sizes as OpenAI-compatible APIs with a single command. It features a built-in chat UI, state-of-the-art inference backends, and a simplified workflow for creating enterprise-grade cloud deployment with Qwen2.5. Visit `the OpenLLM repository <https://github.com/bentoml/OpenLLM/>`_ to learn more.

Installation
------------

Install OpenLLM using ``pip``.

.. code:: bash

   pip install openllm

Verify the installation and display the help information:

.. code:: bash

   openllm --help

Quickstart
----------

Before you run any Qwen2.5 model, ensure your model repository is up to date by syncing it with OpenLLM's latest official repository.

.. code:: bash

   openllm repo update

List the supported Qwen2.5 models:

.. code:: bash

   openllm model list --tag qwen2.5

The results also display the required GPU resources and supported platforms:

.. code:: bash

   model    version                repo     required GPU RAM    platforms
   -------  ---------------------  -------  ------------------  -----------
   qwen2.5  qwen2.5:0.5b           default  12G                 linux
            qwen2.5:1.5b           default  12G                 linux
            qwen2.5:3b             default  12G                 linux
            qwen2.5:7b             default  24G                 linux
            qwen2.5:14b            default  80G                 linux
            qwen2.5:14b-ggml-q4    default                      macos
            qwen2.5:14b-ggml-q8    default                      macos
            qwen2.5:32b            default  80G                 linux
            qwen2.5:32b-ggml-fp16  default                      macos
            qwen2.5:72b            default  80Gx2               linux
            qwen2.5:72b-ggml-q4    default                      macos

To start a server with one of the models, use ``openllm serve`` like this:

.. code:: bash

   openllm serve qwen2.5:7b

By default, the server starts at ``http://localhost:3000/``.

Interact with the model server
------------------------------

With the model server up and running, you can call its APIs in the following ways:

.. tab-set::

    .. tab-item:: CURL

       Send an HTTP request to its ``/generate`` endpoint via CURL:

       .. code-block:: bash

            curl -X 'POST' \
               'http://localhost:3000/api/generate' \
               -H 'accept: text/event-stream' \
               -H 'Content-Type: application/json' \
               -d '{
               "prompt": "Tell me something about large language models.",
               "model": "Qwen/Qwen2.5-7B-Instruct",
               "max_tokens": 2048,
               "stop": null
            }'

    .. tab-item:: Python client

       Call the OpenAI-compatible endpoints with frameworks and tools that support the OpenAI API protocol. Here is an example:

       .. code-block:: python

            from openai import OpenAI

            client = OpenAI(base_url='http://localhost:3000/v1', api_key='na')

            # Use the following func to get the available models
            # model_list = client.models.list()
            # print(model_list)

            chat_completion = client.chat.completions.create(
               model="Qwen/Qwen2.5-7B-Instruct",
               messages=[
                  {
                        "role": "user",
                        "content": "Tell me something about large language models."
                  }
               ],
               stream=True,
            )
            for chunk in chat_completion:
               print(chunk.choices[0].delta.content or "", end="")

    .. tab-item:: Chat UI

       OpenLLM provides a chat UI at the ``/chat`` endpoint for the LLM server at http://localhost:3000/chat.

       .. image:: ../../source/assets/qwen-openllm-ui-demo.png

Model repository
----------------

A model repository in OpenLLM represents a catalog of available LLMs. You can add your own repository to OpenLLM with custom Qwen2.5 variants for your specific needs. See our `documentation to learn details <https://github.com/bentoml/OpenLLM?tab=readme-ov-file#model-repository>`_.