Commit fd1ada03 authored by chenych's avatar chenych
Browse files

Update README.

parent 8bb665c7
......@@ -45,6 +45,8 @@ docker run -it \
`暂无`
## 推理
> 如果出现`ImportError: librocm_smi64.so.2: cannot open shaned object file: No such file or directory`报错,系机器hyhal版本较低所致,请进行升级。
### vllm
#### 单机推理
1. 加入环境变量
......@@ -171,7 +173,7 @@ ray start --address='x.x.x.x:6379' --num-gpus=8 --num-cpus=32
vllm serve ZhipuAI/GLM-5 \
--port 8001 \
--trust-remote-code \
--tensor-parallel-size 32 \
--tensor-parallel-size 32 \ # BW1000是32, BW1100是16
--gpu-memory-utilization 0.85 \
--distributed-executor-backend ray \
--dtype bfloat16 \
......@@ -211,6 +213,7 @@ curl http://localhost:8001/v1/chat/completions \
| 模型名称 | 权重大小 | DCU型号 | 最低卡数需求 |下载地址|
|:-----:|:----------:|:----------:|:---------------------:|:----------:|
| GLM-5 | 744B | BW1000 | 32 | [ModelScope](https://modelscope.cn/models/ZhipuAI/GLM-5) |
| GLM-5 | 744B | BW1100 | 16 | [ModelScope](https://modelscope.cn/models/ZhipuAI/GLM-5) |
| GLM-5-FP8 | 744B | BW1100 | 8 | [ModelScope](https://modelscope.cn/models/ZhipuAI/GLM-5-FP8) |
## 源码仓库及问题反馈
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment