"test/git@developer.sourcefind.cn:change/sglang.git" did not exist on "f842853a40e3a663e3374f4384377b44c13fa9d8"
Unverified Commit 7b5e943c authored by Joao Gante's avatar Joao Gante Committed by GitHub
Browse files

Generate: TF contrastive search must pop `use_cache` from `model_kwargs` (#21149)

parent 7f3dab39
...@@ -2437,6 +2437,8 @@ class TFGenerationMixin: ...@@ -2437,6 +2437,8 @@ class TFGenerationMixin:
else self.generation_config.return_dict_in_generate else self.generation_config.return_dict_in_generate
) )
use_cache = True # In contrastive search, we always use cache use_cache = True # In contrastive search, we always use cache
model_kwargs.pop("use_cache", None)
use_xla = not tf.executing_eagerly() use_xla = not tf.executing_eagerly()
# TODO (Joao): fix cache format or find programatic way to detect cache index # TODO (Joao): fix cache format or find programatic way to detect cache index
# GPT2 and other models has a slightly different cache structure, with a different batch axis # GPT2 and other models has a slightly different cache structure, with a different batch axis
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment