Skip to content

GitLab

"git@developer.sourcefind.cn:chenpangpang/transformers.git" did not exist on "9fbb4cdc8074b4f74cae20ad68f4ba4f93fe0005"
Unverified Commit 2772d3e7 authored by Lysandre Debut's avatar Lysandre Debut Committed by GitHub
Browse files

Add RemBert to AutoTokenizer (#13224)

parent f1bb6f08
Hide whitespace changes
Inline Side-by-side
Showing with 7 additions and 0 deletions
src/transformers/models/auto/tokenization_auto.py
View file @ 2772d3e7
...@@ -199,6 +199,13 @@ else: ...@@ -199,6 +199,13 @@ else:
"MBart50TokenizerFast" if is_tokenizers_available() else None, "MBart50TokenizerFast" if is_tokenizers_available() else None,
), ),
), ),
(
"rembert",
(
"RemBertTokenizer" if is_sentencepiece_available() else None,
"RemBertTokenizerFast" if is_tokenizers_available() else None,
),
),
] ]
) )
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment