Skip to content

GitLab

"...git@developer.sourcefind.cn:chenpangpang/open-webui.git" did not exist on "a0935dec60eb635e7f8b3bf796aab9047cd6ee88"
Unverified Commit 7418a48e authored by Blip blop's avatar Blip blop Committed by GitHub
Browse files

Removed `Bert` interdependency in `tokenization_electra.py` (#19356) 



* Copied from BertTokenizer() in tokenization_bert

* Added BasicTokenizer and WordPieceTokenizer Class

* Update src/transformers/models/electra/tokenization_electra.py
Co-authored-by: default avatarSylvain Gugger <35901082+sgugger@users.noreply.github.com>

* Added copied from comments for basicTokenizer and WordPieceTokenizer

* Updated the comments for the tokenizerClasses

* Update src/transformers/models/electra/tokenization_electra.py
Co-authored-by: default avatarSylvain Gugger <35901082+sgugger@users.noreply.github.com>

* Update src/transformers/models/electra/tokenization_electra.py
Co-authored-by: default avatarSylvain Gugger <35901082+sgugger@users.noreply.github.com>

* Formatted tokenization_electra with `make style`

* Fix repo inconsistencies

* Update src/transformers/models/electra/tokenization_electra.py
Co-authored-by: default avatarSylvain Gugger <35901082+sgugger@users.noreply.github.com>

* Set the logger
Co-authored-by: default avatarSylvain Gugger <35901082+sgugger@users.noreply.github.com>
parent 6ef16f2b
Expand all Show whitespace changes
Inline Side-by-side
Showing with 464 additions and 7 deletions
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment