[WIP]remove XLMTokenizer inheritance from FlaubertTokenizer (#19330)

* remove XLMTokenizer inheritance from FlaubertTokenizer * remove XLMTokenizer inheritance from FlaubertTokenizer * remove XLMTokenizer inheritance from FlaubertTokenizer * remove XLMTokenizer inheritance from FlaubertTokenizer: fixed styling * removed repo-consistensy issue

[WIP]remove XLMTokenizer inheritance from FlaubertTokenizer (#19330)
* remove XLMTokenizer inheritance from FlaubertTokenizer * remove XLMTokenizer inheritance from FlaubertTokenizer * remove XLMTokenizer inheritance from FlaubertTokenizer * remove XLMTokenizer inheritance from FlaubertTokenizer: fixed styling * removed repo-consistensy issue
c54bb1ad · Druhin Abrol · GitHub · e12bbe3b · c54bb1ad
Unverified Commit c54bb1ad authored Oct 05, 2022 by Druhin Abrol Committed by GitHub Oct 05, 2022
Expand all Hide whitespace changes
Inline Side-by-side

Showing with 454 additions and 8 deletions

src/transformers/models/flaubert/tokenization_flaubert.py src/transformers/models/flaubert/tokenization_flaubert.py +454 -8

No files found.
--- a/src/transformers/models/flaubert/tokenization_flaubert.py
+++ b/src/transformers/models/flaubert/tokenization_flaubert.py