Commit fcb745b5 authored by zhouxiang's avatar zhouxiang
Browse files

完善readme

parent 56215723
...@@ -24,7 +24,25 @@ fastllm是纯c++实现的高性能大模型推理库 ...@@ -24,7 +24,25 @@ fastllm是纯c++实现的高性能大模型推理库
在光源可拉取推理的docker镜像,拉取方式如下: 在光源可拉取推理的docker镜像,拉取方式如下:
``` ```
docker pull image.sourcefind.cn:5000/dcu/admin/base/custom:glm-ft-v1.0 docker pull image.sourcefind.cn:5000/dcu/admin/base/pytorch:2.1.0-centos7.6-dtk23.10.1-py38
```
### 容器启动
模型推理容器启动命令参考如下,用户根据需要修改:
```
# <container_name> 自定义容器名
# <project_path> 当前工程所在路径
docker run -it --name=<container_name> -v <project_path>:/work -w /work --privileged -v /opt/hyhal:/opt/hyhal --device=/dev/kfd --device=/dev/dri --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --network host --shm-size=16G --group-add video image.sourcefind.cn:5000/dcu/admin/base/pytorch:2.1.0-centos7.6-dtk23.10.1-py38 /bin/bash
```
### 加载环境
进入容器后执行如下命令,加载运行环境变量
```
source /opt/dtk/cuda/env.sh
``` ```
### 编译 ### 编译
......
...@@ -3,7 +3,7 @@ from transformers import AutoTokenizer, AutoModel ...@@ -3,7 +3,7 @@ from transformers import AutoTokenizer, AutoModel
from fastllm_pytools import torch2flm from fastllm_pytools import torch2flm
if __name__ == "__main__": if __name__ == "__main__":
model_path = "/home/ZhipuAI/chatglm3-6b" model_path = "THUDM/chatglm2-6b"
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True) tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
model = AutoModel.from_pretrained(model_path, trust_remote_code=True) model = AutoModel.from_pretrained(model_path, trust_remote_code=True)
model = model.eval() model = model.eval()
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment