Render chat template tojson filter as unicode (#31041)

* Render chat template tojson filter as unicode * ruff--

Render chat template tojson filter as unicode (#31041)
* Render chat template tojson filter as unicode * ruff--
22dab246 · Sigbjørn Skjæret · GitHub · 4f98b144 · 22dab246
Unverified Commit 22dab246 authored May 28, 2024 by Sigbjørn Skjæret Committed by GitHub May 28, 2024
Hide whitespace changes
Inline Side-by-side

Showing with 1 addition and 0 deletions

src/transformers/tokenization_utils_base.py src/transformers/tokenization_utils_base.py +1 -0

No files found.
--- a/src/transformers/tokenization_utils_base.py
+++ b/src/transformers/tokenization_utils_base.py
@@ -1853,6 +1853,7 @@ class PreTrainedTokenizerBase(SpecialTokensMixin, PushToHubMixin):
            raise TemplateError(message)

        jinja_env = ImmutableSandboxedEnvironment(trim_blocks=True, lstrip_blocks=True)
+        jinja_env.policies["json.dumps_kwargs"]["ensure_ascii"] = False
        jinja_env.globals["raise_exception"] = raise_exception
        return jinja_env.from_string(chat_template)