Skip to content

GitLab

Unverified Commit e41a0fa3 authored by Nick Hill's avatar Nick Hill Committed by GitHub
Browse files

[Perf] Freeze core engine proc heap after init (#24008) 


Signed-off-by: default avatarNick Hill <nhill@redhat.com>
parent 37241077
Hide whitespace changes
Inline Side-by-side
Showing with 6 additions and 0 deletions
vllm/v1/engine/core.py
View file @ e41a0fa3
# SPDX-License-Identifier: Apache-2.0 # SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
import gc
import os import os
import queue import queue
import signal import signal
...@@ -536,6 +537,11 @@ class EngineCoreProc(EngineCore): ...@@ -536,6 +537,11 @@ class EngineCoreProc(EngineCore):
self.step_fn = (self.step if self.batch_queue is None else self.step_fn = (self.step if self.batch_queue is None else
self.step_with_batch_queue) self.step_with_batch_queue)
# Mark the startup heap as static so that it's ignored by GC.
# Reduces pause times of oldest generation collections.
gc.collect()
gc.freeze()
@contextmanager @contextmanager
def _perform_handshakes( def _perform_handshakes(
self, self,
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment