Commit c5ba7b50 authored by chenych's avatar chenych
Browse files

Update README.

parent 41e7937b
......@@ -51,6 +51,8 @@ docker run -it \
`暂无`
## 推理
> 如果出现`ImportError: librocm_smi64.so.2: cannot open shaned object file: No such file or directory`报错,系机器hyhal版本较低所致,请进行升级。
### vllm
#### 单机推理
1. 加入环境变量
......@@ -58,7 +60,6 @@ docker run -it \
# 环境变量
rm -rf ~/.cache
rm -rf ~/.triton
export VLLM_TORCH_PROFILER_DIR=/home/GLM-5/
export HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
export ALLREDUCE_STREAM_WITH_COMPUTE=1
export NCCL_MIN_NCHANNELS=16
......@@ -174,7 +175,7 @@ ray start --address='x.x.x.x:6379' --num-gpus=8 --num-cpus=32
vllm serve ZhipuAI/GLM-5.1 \
--port 8001 \
--trust-remote-code \
--tensor-parallel-size 32 \
--tensor-parallel-size 32 \ # BW1000是32, BW1100是16
--gpu-memory-utilization 0.85 \
--distributed-executor-backend ray \
--dtype bfloat16 \
......@@ -211,7 +212,8 @@ curl http://localhost:8001/v1/chat/completions \
## 预训练权重
| 模型名称 | 权重大小 | DCU型号 | 最低卡数需求 |下载地址|
|:-----:|:----------:|:----------:|:---------------------:|:----------:|
| GLM-5.1 | 754B | BW1000,BW1100 | 32 | [ModelScope](https://modelscope.cn/models/ZhipuAI/GLM-5.1) |
| GLM-5.1 | 754B | BW1000 | 32 | [ModelScope](https://modelscope.cn/models/ZhipuAI/GLM-5.1) |
| GLM-5.1 | 754B | BW1100 | 16 | [ModelScope](https://modelscope.cn/models/ZhipuAI/GLM-5.1) |
| GLM-5.1-FP8 | 754B | BW1100 | 8 | [ModelScope](https://modelscope.cn/models/ZhipuAI/GLM-5.1-FP8) |
## 源码仓库及问题反馈
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment