Merge pull request #1064 from kvcache-ai/fix-temperature

fix temperature=0, flashinfer sample error

Merge pull request #1064 from kvcache-ai/fix-temperature
fix temperature=0, flashinfer sample error
6463070b · wang jiahao · GitHub · aac0c91d · ec03bcbd · 6463070b
Unverified Commit 6463070b authored Apr 07, 2025 by wang jiahao Committed by GitHub Apr 07, 2025
Show whitespace changes
Inline Side-by-side

Showing with 2 additions and 0 deletions

ktransformers/server/backend/interfaces/balance_serve.py ktransformers/server/backend/interfaces/balance_serve.py +2 -0

No files found.
--- a/ktransformers/server/backend/interfaces/balance_serve.py
+++ b/ktransformers/server/backend/interfaces/balance_serve.py
@@ -367,6 +367,8 @@ class BalanceServeInterface(BackendInterfaceBase):
        #@TODO add server
        stop_criteria =  [self.tokenizer.encode(self.tokenizer.eos_token, add_special_tokens=False),self.tokenizer.encode("<|im_end|>")]
        query_add.stop_criteria = stop_criteria
+        if temperature == 0:
+            temperature = 0.0001
        query_add.sample_options.temperature = temperature
        if top_p == 0:
            top_p = 0.0001