Skip to content

GitLab

Unverified Commit 78603072 authored by Ning Xie's avatar Ning Xie Committed by GitHub
Browse files

[Docs] add runai_streamer_sharded to LoadConfig (#27937) 


Signed-off-by: default avatarAndy Xie <andy.xning@gmail.com>
parent 145c00a4
Hide whitespace changes
Inline Side-by-side
Showing with 2 additions and 0 deletions
vllm/config/load.py
View file @ 78603072
......@@ -40,6 +40,8 @@ class LoadConfig:
more information.\n
- "runai_streamer" will load the Safetensors weights using Run:ai Model
Streamer.\n
- "runai_streamer_sharded" will load weights from pre-sharded checkpoint
files using Run:ai Model Streamer.\n
- "bitsandbytes" will load the weights using bitsandbytes quantization.\n
- "sharded_state" will load weights from pre-sharded checkpoint files,
supporting efficient loading of tensor-parallel models.\n
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment