Unverified Commit 91a07ff6 authored by Wei Zhao's avatar Wei Zhao Committed by GitHub
Browse files

[Bugfix] Fix DeepSeek v3.2 tokenizer outputting None issue (#33832)


Signed-off-by: default avatarwzhao18 <wzhao18.sz@gmail.com>
parent d5c48001
......@@ -157,6 +157,10 @@ def detokenize_incrementally(
)
if isinstance(new_tokens, str):
new_tokens = [new_tokens]
else:
# This is required to guard against out-of-vocab prompt token ids
# (for example when using dummy weights)
_replace_none_with_empty(new_tokens) # type: ignore[arg-type]
else:
new_tokens = [""]
output_tokens = prev_tokens + new_tokens
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment