Assertions to exceptions (#13692)

* Raise exceptions instead of using assertions for control flow #12789

* # coding=utf-8

* Raise exceptions instead of using assertions for control flow

* Raise exceptions instead of using assertions for control flow

* Update src/transformers/tokenization_utils.py

Raise exceptions instead of using assertions for control flow
Co-authored-by: Suraj Patil <surajp815@gmail.com>

* Update src/transformers/tokenization_utils.py

Raise exceptions instead of using assertions for control flow
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* Raise exceptions instead of using assertions for control flow

* test

* Raise exceptions instead of using assertions for control flow
Co-authored-by: MocktaiLEngineer <kavinarasu22@gmail.com>
Co-authored-by: Suraj Patil <surajp815@gmail.com>
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

src/transformers/tokenization_utils.py

@@ -362,7 +362,8 @@ class PreTrainedTokenizer(PreTrainedTokenizerBase):
 
         tokens_to_add = []
         for token in new_tokens:
-            assert isinstance(token, str)
+            if not isinstance(token, str):
+                raise TypeError(f"Token {token} is not a string but a {type(token)}.")
             if not special_tokens and hasattr(self, "do_lower_case") and self.do_lower_case:
                 token = token.lower()
             if (