Unverified Commit c3b51891 authored by Guillaume Calmettes's avatar Guillaume Calmettes Committed by GitHub
Browse files

[Bugfix] catch AssertionError in MistralTokenizer as ValueError (#16344)


Signed-off-by: default avatarGuillaume Calmettes <gcalmettes@scaleway.com>
parent a25866ac
...@@ -1193,8 +1193,15 @@ def apply_mistral_chat_template( ...@@ -1193,8 +1193,15 @@ def apply_mistral_chat_template(
**kwargs, **kwargs,
) )
return tokenizer.apply_chat_template( try:
messages=messages, return tokenizer.apply_chat_template(
tools=tools, messages=messages,
**kwargs, tools=tools,
) **kwargs,
)
# mistral-common uses assert statements to stop processing of input
# if input does not comply with the expected format.
# We convert those assertion errors to ValueErrors so they can be
# are properly caught in the preprocessing_input step
except AssertionError as e:
raise ValueError from e
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment