[fix]修复开启并行解码后，在极端测试情况下，由于设置了speculative-disable-by-batch-size导致不跑并行解码导致previo...

[fix]修复开启并行解码后，在极端测试情况下，由于设置了speculative-disable-by-batch-size导致不跑并行解码导致previous_hidden_states不断增加，最终导致显存用尽服务无响应问题

vllm/spec_decode/spec_decode_worker.py

@@ -690,14 +690,16 @@ class SpecDecodeWorker(LoraNotSupportedWorkerBase):
                 hidden_states = hidden_states[
                     torch.where(sampler_output.sampled_token_ids -
                                 VLLM_INVALID_TOKEN_ID)[0]]
-            if self.previous_hidden_states is None and len(
-                    seq_group_meta_with_hidden):
-                self.previous_hidden_states = HiddenStates(
-                    hidden_states, seq_group_meta_with_hidden)
-            elif self.previous_hidden_states and len(
-                    seq_group_meta_with_hidden):
-                self.previous_hidden_states.update(hidden_states,
-                                                   seq_group_meta_with_hidden)
+            
+            if not skip_proposer:
+                if self.previous_hidden_states is None and len(
+                        seq_group_meta_with_hidden):
+                    self.previous_hidden_states = HiddenStates(
+                        hidden_states, seq_group_meta_with_hidden)
+                elif self.previous_hidden_states and len(
+                        seq_group_meta_with_hidden):
+                    self.previous_hidden_states.update(hidden_states,
+                                                    seq_group_meta_with_hidden)
                 
             # Store logits from target model execution.
             if self.tree_decoding: