Commit 94cad9f3 authored by dengjb's avatar dengjb
Browse files

update

parent abd00af2
......@@ -60,6 +60,32 @@ pip install pycountry
## 推理
### vLLM
#### 单机推理
1.启动vllm server
```bash
## serve启动
vllm serve moonshotai/Kimi-K2.6 \
-tp 16 \
--distributed-executor-backend ray \
--gpu-memory-utilization 0.95 \
--mm-encoder-tp-mode data \
--trust-remote-code \
--tool-call-parser kimi_k2 \
--reasoning-parser kimi_k2
## client访问
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "moonshotai/Kimi-K2.6",
"messages": [
{"role": "user", "content": "牛顿提出了哪三大运动定律?请简要说明。"}
],
"temperature": 0.6
}'
```
#### 多机推理
1. 加入环境变量
> 请注意:
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment