Skip to content

GitLab

"git@developer.sourcefind.cn:modelzoo/llama3_pytorch.git" did not exist on "cc7fcbbba8aaf1a96210200bc3e16913878f4e20"
Unverified Commit 9a23af4a authored by Lysandre Debut's avatar Lysandre Debut Committed by GitHub
Browse files

Add XLMProphetNetTokenizer to tokenization auto (#8245)

parent 5b178f3c
Hide whitespace changes
Inline Side-by-side
Showing with 1 addition and 0 deletions
src/transformers/tokenization_auto.py
View file @ 9a23af4a
...@@ -113,6 +113,7 @@ else: ...@@ -113,6 +113,7 @@ else:
T5Tokenizer = None T5Tokenizer = None
XLMRobertaTokenizer = None XLMRobertaTokenizer = None
XLNetTokenizer = None XLNetTokenizer = None
XLMProphetNetTokenizer = None
if is_tokenizers_available(): if is_tokenizers_available():
from .tokenization_albert_fast import AlbertTokenizerFast from .tokenization_albert_fast import AlbertTokenizerFast
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment