fix : escape key of start_token from special characters before search...

fix : escape key of start_token from special characters before search end_token in token2json function of DonutProcessor (#25472) fix : escape key of start_token from special characters before searching for end_token

fix : escape key of start_token from special characters before search...
fix : escape key of start_token from special characters before search end_token in token2json function of DonutProcessor (#25472) fix : escape key of start_token from special characters before searching for end_token
e97deca9 · nour-elkamel · GitHub · 0ebe7ae1 · e97deca9
Unverified Commit e97deca9 authored Aug 14, 2023 by nour-elkamel Committed by GitHub Aug 14, 2023
Show whitespace changes
Inline Side-by-side

Showing with 3 additions and 1 deletion

src/transformers/models/donut/processing_donut.py src/transformers/models/donut/processing_donut.py +3 -1

No files found.
--- a/src/transformers/models/donut/processing_donut.py
+++ b/src/transformers/models/donut/processing_donut.py
@@ -138,7 +138,9 @@ class DonutProcessor(ProcessorMixin):
            if start_token is None:
                break
            key = start_token.group(1)
-            end_token = re.search(rf"</s_{key}>", tokens, re.IGNORECASE)
+            key_escaped = re.escape(key)
+
+            end_token = re.search(rf"</s_{key_escaped}>", tokens, re.IGNORECASE)
            start_token = start_token.group()
            if end_token is None:
                tokens = tokens.replace(start_token, "")