"vscode:/vscode.git/clone" did not exist on "d07efb31c5efdb16eb386493b326cf3e90047978"
Unverified Commit 82c25151 authored by Joe Runde's avatar Joe Runde Committed by GitHub
Browse files

[Doc] update gpu-memory-utilization flag docs (#9507)


Signed-off-by: default avatarJoe Runde <Joseph.Runde@ibm.com>
parent 1325872e
......@@ -428,7 +428,11 @@ class EngineArgs:
help='The fraction of GPU memory to be used for the model '
'executor, which can range from 0 to 1. For example, a value of '
'0.5 would imply 50%% GPU memory utilization. If unspecified, '
'will use the default value of 0.9.')
'will use the default value of 0.9. This is a global gpu memory '
'utilization limit, for example if 50%% of the gpu memory is '
'already used before vLLM starts and --gpu-memory-utilization is '
'set to 0.9, then only 40%% of the gpu memory will be allocated '
'to the model executor.')
parser.add_argument(
'--num-gpu-blocks-override',
type=int,
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment