Unverified Commit 0c730268 authored by Woosuk Kwon's avatar Woosuk Kwon Committed by GitHub
Browse files

[V1][PP] Fix memory profiling in PP (#13315)


Signed-off-by: default avatarWoosuk Kwon <woosuk.kwon@berkeley.edu>
parent 6a854c7a
...@@ -1158,11 +1158,12 @@ class GPUModelRunner(LoRAModelRunnerMixin): ...@@ -1158,11 +1158,12 @@ class GPUModelRunner(LoRAModelRunnerMixin):
# Trigger compilation for general shape. # Trigger compilation for general shape.
hidden_states = self._dummy_run(self.max_num_tokens, hidden_states = self._dummy_run(self.max_num_tokens,
dummy_kv_caches) dummy_kv_caches)
if not get_pp_group().is_last_rank: if get_pp_group().is_last_rank:
return hidden_states hidden_states = hidden_states[logit_indices]
hidden_states = hidden_states[logit_indices] logits = self.model.compute_logits(hidden_states, None)
logits = self.model.compute_logits(hidden_states, None) # TODO(woosuk): Consider the memory usage of the sampler.
# TODO(woosuk): Consider the memory usage of the sampler. else:
logits = None
torch.cuda.synchronize() torch.cuda.synchronize()
del hidden_states, logits del hidden_states, logits
self.encoder_cache.clear() self.encoder_cache.clear()
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment