Commit a72e268a authored by weishb's avatar weishb
Browse files

support vllm_0.15.1

parent bd12e34d
...@@ -19,9 +19,9 @@ Qwen3-VL-Embedding 和 Qwen3-VL-Reranker 模型系列是通义千问(Qwen) ...@@ -19,9 +19,9 @@ Qwen3-VL-Embedding 和 Qwen3-VL-Reranker 模型系列是通义千问(Qwen)
| DTK | 26.04 | | DTK | 26.04 |
| python | 3.10.12 | | python | 3.10.12 |
| transformers | 4.57.6 | | transformers | 4.57.6 |
| vllm | 0.11.0+das.opt1.rc2.dtk2604 | | vllm | 0.15.1+das.opt1.alpha.dtk2604 |
推荐使用镜像:harbor.sourcefind.cn:5443/dcu/admin/base/vllm:0.11.0-ubuntu22.04-dtk26.04-0130-py3.10-20260202 推荐使用镜像:harbor.sourcefind.cn:5443/dcu/admin/base/custom:vllm0.15.1-ubuntu22.04-dtk26.04-0130-py3.10-20260220
```bash ```bash
docker run -it \ docker run -it \
...@@ -38,7 +38,7 @@ docker run -it \ ...@@ -38,7 +38,7 @@ docker run -it \
-u root \ -u root \
-v /opt/hyhal/:/opt/hyhal/:ro \ -v /opt/hyhal/:/opt/hyhal/:ro \
-v /path/your_code_data/:/path/your_code_data/ \ -v /path/your_code_data/:/path/your_code_data/ \
harbor.sourcefind.cn:5443/dcu/admin/base/vllm:0.11.0-ubuntu22.04-dtk26.04-0130-py3.10-20260202 bash harbor.sourcefind.cn:5443/dcu/admin/base/custom:vllm0.15.1-ubuntu22.04-dtk26.04-0130-py3.10-20260220 bash
``` ```
更多镜像可前往[光源](https://sourcefind.cn/#/service-list)下载使用。 更多镜像可前往[光源](https://sourcefind.cn/#/service-list)下载使用。
...@@ -46,9 +46,7 @@ docker run -it \ ...@@ -46,9 +46,7 @@ docker run -it \
镜像内其他环境配置 镜像内其他环境配置
``` ```
pip install pycountry
1.解压vllm.zip到/usr/local/lib/python3.10/dist-packages直接覆盖需要修改的文件
unzip -o vllm.zip -d /usr/local/lib/python3.10/dist-packages
``` ```
...@@ -62,18 +60,17 @@ docker run -it \ ...@@ -62,18 +60,17 @@ docker run -it \
### vllm ### vllm
#### 单机推理 #### 单机推理
启动服务
```bash
```bash
## serve启动
vllm serve Qwen/Qwen3-VL-Reranker-8B \ vllm serve Qwen/Qwen3-VL-Reranker-8B \
--runner pooling \ --runner pooling \
--hf-overrides '{"architectures": ["Qwen3VLForSequenceClassification"],"classifier_from_token":["no","yes"],"is_original_qwen3_reranker":true}' \ --hf-overrides '{"architectures": ["Qwen3VLForSequenceClassification"],"classifier_from_token":["no","yes"],"is_original_qwen3_reranker":true}' \
--chat-template qwen3_vl_reranker.jinja \ --chat-template qwen3_vl_reranker.jinja \
--max-model-len 4096 \ --max-model-len 4096 \
--served-model-name qwen3-vl-reranker --served-model-name qwen3-vl-reranker
```
调用服务: ## client访问
```
curl -s http://127.0.0.1:8000/rerank \ curl -s http://127.0.0.1:8000/rerank \
-H "Content-Type: application/json" \ -H "Content-Type: application/json" \
-d '{ -d '{
......
File suppressed by a .gitattributes entry or the file's encoding is unsupported.
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment