Skip to content

GitLab

"...git@developer.sourcefind.cn:modelzoo/solov2-pytorch.git" did not exist on "8d010d7de9c2643e715aaf6033ff7fd5c60ebdc2"
Commit fa7a696d authored by Lianmin Zheng's avatar Lianmin Zheng
Browse files

Fix max_new_tokens for limited memory

parent bef0b359
Hide whitespace changes
Inline Side-by-side
Showing with 1 addition and 0 deletions
python/sglang/srt/managers/router/model_rpc.py
View file @ fa7a696d
......@@ -229,6 +229,7 @@ class ModelRpcServer(rpyc.Service):
req.sampling_params.max_new_tokens = min(
req.sampling_params.max_new_tokens,
self.model_config.context_len - 1 - len(req.input_ids),
self.max_total_num_token - 128 - len(req.input_ids),
)
self.forward_queue.append(req)
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment