Skip to content

GitLab

"...git@developer.sourcefind.cn:chenpangpang/transformers.git" did not exist on "5f25855b3efb3c92f26428b5b28a4cd668d2feff"
Unverified Commit 22dab246 authored by Sigbjørn Skjæret's avatar Sigbjørn Skjæret Committed by GitHub
Browse files

Render chat template tojson filter as unicode (#31041) 

* Render chat template tojson filter as unicode

* ruff--
parent 4f98b144
Hide whitespace changes
Inline Side-by-side
Showing with 1 addition and 0 deletions
src/transformers/tokenization_utils_base.py
View file @ 22dab246
...@@ -1853,6 +1853,7 @@ class PreTrainedTokenizerBase(SpecialTokensMixin, PushToHubMixin): ...@@ -1853,6 +1853,7 @@ class PreTrainedTokenizerBase(SpecialTokensMixin, PushToHubMixin):
raise TemplateError(message) raise TemplateError(message)
jinja_env = ImmutableSandboxedEnvironment(trim_blocks=True, lstrip_blocks=True) jinja_env = ImmutableSandboxedEnvironment(trim_blocks=True, lstrip_blocks=True)
jinja_env.policies["json.dumps_kwargs"]["ensure_ascii"] = False
jinja_env.globals["raise_exception"] = raise_exception jinja_env.globals["raise_exception"] = raise_exception
return jinja_env.from_string(chat_template) return jinja_env.from_string(chat_template)
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment