Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
OpenDAS
fastllm
Commits
fcb745b5
Commit
fcb745b5
authored
Jan 31, 2024
by
zhouxiang
Browse files
完善readme
parent
56215723
Changes
2
Hide whitespace changes
Inline
Side-by-side
Showing
2 changed files
with
20 additions
and
2 deletions
+20
-2
README.md
README.md
+19
-1
tools/scripts/chatglm_export.py
tools/scripts/chatglm_export.py
+1
-1
No files found.
README.md
View file @
fcb745b5
...
@@ -24,7 +24,25 @@ fastllm是纯c++实现的高性能大模型推理库
...
@@ -24,7 +24,25 @@ fastllm是纯c++实现的高性能大模型推理库
在光源可拉取推理的docker镜像,拉取方式如下:
在光源可拉取推理的docker镜像,拉取方式如下:
```
```
docker pull image.sourcefind.cn:5000/dcu/admin/base/custom:glm-ft-v1.0
docker pull image.sourcefind.cn:5000/dcu/admin/base/pytorch:2.1.0-centos7.6-dtk23.10.1-py38
```
### 容器启动
模型推理容器启动命令参考如下,用户根据需要修改:
```
# <container_name> 自定义容器名
# <project_path> 当前工程所在路径
docker run -it --name=<container_name> -v <project_path>:/work -w /work --privileged -v /opt/hyhal:/opt/hyhal --device=/dev/kfd --device=/dev/dri --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --network host --shm-size=16G --group-add video image.sourcefind.cn:5000/dcu/admin/base/pytorch:2.1.0-centos7.6-dtk23.10.1-py38 /bin/bash
```
### 加载环境
进入容器后执行如下命令,加载运行环境变量
```
source /opt/dtk/cuda/env.sh
```
```
### 编译
### 编译
...
...
tools/scripts/chatglm_export.py
View file @
fcb745b5
...
@@ -3,7 +3,7 @@ from transformers import AutoTokenizer, AutoModel
...
@@ -3,7 +3,7 @@ from transformers import AutoTokenizer, AutoModel
from
fastllm_pytools
import
torch2flm
from
fastllm_pytools
import
torch2flm
if
__name__
==
"__main__"
:
if
__name__
==
"__main__"
:
model_path
=
"
/home/ZhipuAI
/chatglm
3
-6b"
model_path
=
"
THUDM
/chatglm
2
-6b"
tokenizer
=
AutoTokenizer
.
from_pretrained
(
model_path
,
trust_remote_code
=
True
)
tokenizer
=
AutoTokenizer
.
from_pretrained
(
model_path
,
trust_remote_code
=
True
)
model
=
AutoModel
.
from_pretrained
(
model_path
,
trust_remote_code
=
True
)
model
=
AutoModel
.
from_pretrained
(
model_path
,
trust_remote_code
=
True
)
model
=
model
.
eval
()
model
=
model
.
eval
()
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment