No commit message

No commit message

README.md

@@ -152,13 +152,94 @@ curl http://localhost:8000/v1/chat/completions \
 
     output:               The image features a close-up view of a stop sign on a city street
 
-### result
+### **gradio和vllm结合使用**
+
+1.安装gradio
 
-使用的加速卡:1张 DCU-K100_AI-64G
+```
+pip install gradio
+```
 
+1.1 基于gradio的llava对话
+
+```python
+#gradio_openai_vlm_webserver.py
+import argparse
+import gradio as gr
+from openai import OpenAI
+
+parser = argparse.ArgumentParser(description='LLaVA Chatbot Interface')
+parser.add_argument('--model-url', type=str, default='http://localhost:8000/v1', help='Model URL')
+parser.add_argument('-m', '--model', type=str, required=True, help='Model path')
+parser.add_argument("--host", type=str, default=None)
+parser.add_argument("--port", type=int, default=8001)
+
+args = parser.parse_args()
+
+openai_api_key = "EMPTY"
+openai_api_base = args.model_url
+
+client = OpenAI(api_key=openai_api_key, base_url=openai_api_base)
+
+def predict(message, history, image):
+    history_openai_format = []
+    user_message = message
+    if image:
+        user_message += f" [local file]({image})"
+    history_openai_format.append({"role": "user", "content": user_message})
+    try:
+        chat_response = client.chat.completions.create(
+            model=args.model,
+            messages=history_openai_format
+        )
+        return chat_response.choices[0].message.content
+    except Exception as e:
+        return f"An error occurred: {str(e)}"
+gr.ChatInterface(
+    predict,
+    additional_inputs=[
+        gr.Image(label="Upload Image", type="filepath",scale=1, width="50%")
+    ],
+    title="LLaVA Chatbot",
+    description="Chat with the LLaVA model and upload images for visual understanding."
+).queue().launch(server_name=args.host, server_port=args.port, share=True)
 ```
-Prompt: 'What is deep learning?', Generated text: ' Deep learning is a subset of machine learning that involves the use of neural networks to model and solve complex problems. Neural networks are a network of interconnected nodes or " neurons" that are designed to recognize patterns in data, learn from examples, and make predictions or decisions.\nThe term "deep" in deep learning refers to the use of multiple layers or hidden layers in these neural networks. Each layer processes the input data in a different way, extracting increasingly abstract features as the data passes through.'
+
+2.安装必要文件
+
+    2.1 启动gradio服务，根据提示操作
+
 ```
+python  gradio_openai_vlm_webserver.py --model "/mnt/data/llm-models/llava/llava-1.5-7b-hf" --model-url http://localhost:8000/v1
+```
+
+    2.2 更改文件权限
+
+打开提示下载文件目录，输入以下命令给予权限
+
+```
+chmod +x frpc_linux_amd64_v0.*
+```
+
+3.启动OpenAI兼容服务
+
+`cd examples`
+
+```
+python -m vllm.entrypoints.openai.api_server     --model /mnt/data/llm-models/llava/llava-1.5-7b-hf     --image-input-type pixel_values     --image-token-id 32000     --image-input-shape 1,3,336,336     --image-feature-size 576     --chat-template template_llava.jinja
+```
+
+5.使用对话服务
+
+在浏览器中输入本地 URL，可以使用 Gradio 提供的对话服务。
+
+### result
+
+使用的加速卡:单卡K100_AI     模型：[llava-v1.5-7b](http://113.200.138.88:18080/aimodels/llava-v1.5-7b)
+
+<div align=center>
+    <img src="./doc/llava_gradio.png" width="500" height="300"/>
+</div>
 
 ### 精度