Skip to content

GitLab

You need to sign in or sign up before continuing.
Commit c5ba7b50 authored by chenych's avatar chenych
Browse files

Update README.

parent 41e7937b
Hide whitespace changes
Inline Side-by-side
Showing with 5 additions and 3 deletions
README.md
View file @ c5ba7b50
...@@ -51,6 +51,8 @@ docker run -it \ ...@@ -51,6 +51,8 @@ docker run -it \
`暂无` `暂无`
## 推理 ## 推理
> 如果出现`ImportError: librocm_smi64.so.2: cannot open shaned object file: No such file or directory`报错，系机器hyhal版本较低所致，请进行升级。
### vllm ### vllm
#### 单机推理 #### 单机推理
1. 加入环境变量 1. 加入环境变量
...@@ -58,7 +60,6 @@ docker run -it \ ...@@ -58,7 +60,6 @@ docker run -it \
# 环境变量 # 环境变量
rm -rf ~/.cache rm -rf ~/.cache
rm -rf ~/.triton rm -rf ~/.triton
export VLLM_TORCH_PROFILER_DIR=/home/GLM-5/
export HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 export HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
export ALLREDUCE_STREAM_WITH_COMPUTE=1 export ALLREDUCE_STREAM_WITH_COMPUTE=1
export NCCL_MIN_NCHANNELS=16 export NCCL_MIN_NCHANNELS=16
...@@ -174,7 +175,7 @@ ray start --address='x.x.x.x:6379' --num-gpus=8 --num-cpus=32 ...@@ -174,7 +175,7 @@ ray start --address='x.x.x.x:6379' --num-gpus=8 --num-cpus=32
vllm serve ZhipuAI/GLM-5.1 \ vllm serve ZhipuAI/GLM-5.1 \
--port 8001 \ --port 8001 \
--trust-remote-code \ --trust-remote-code \
--tensor-parallel-size 32 \ --tensor-parallel-size 32 \ # BW1000是32, BW1100是16
--gpu-memory-utilization 0.85 \ --gpu-memory-utilization 0.85 \
--distributed-executor-backend ray \ --distributed-executor-backend ray \
--dtype bfloat16 \ --dtype bfloat16 \
...@@ -211,7 +212,8 @@ curl http://localhost:8001/v1/chat/completions \ ...@@ -211,7 +212,8 @@ curl http://localhost:8001/v1/chat/completions \
## 预训练权重 ## 预训练权重
| 模型名称 | 权重大小 | DCU型号 | 最低卡数需求 |下载地址| | 模型名称 | 权重大小 | DCU型号 | 最低卡数需求 |下载地址|
|:-----:|:----------:|:----------:|:---------------------:|:----------:| |:-----:|:----------:|:----------:|:---------------------:|:----------:|
| GLM-5.1 | 754B | BW1000,BW1100 | 32 | [ModelScope](https://modelscope.cn/models/ZhipuAI/GLM-5.1) | | GLM-5.1 | 754B | BW1000 | 32 | [ModelScope](https://modelscope.cn/models/ZhipuAI/GLM-5.1) |
| GLM-5.1 | 754B | BW1100 | 16 | [ModelScope](https://modelscope.cn/models/ZhipuAI/GLM-5.1) |
| GLM-5.1-FP8 | 754B | BW1100 | 8 | [ModelScope](https://modelscope.cn/models/ZhipuAI/GLM-5.1-FP8) | | GLM-5.1-FP8 | 754B | BW1100 | 8 | [ModelScope](https://modelscope.cn/models/ZhipuAI/GLM-5.1-FP8) |
## 源码仓库及问题反馈 ## 源码仓库及问题反馈
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment