Unverified Commit a0be960d authored by Uri Alon's avatar Uri Alon Committed by GitHub
Browse files

Fixing tokenizer when `transformers` is installed without `tokenizers` (#26236)

* Fixing tokenizer when tokenizers is not installed

* Adding __repr__ function and repr=True in dataclass

* Revert "Adding __repr__ function and repr=True in dataclass"

This reverts commit 18839505d1cada3170ed623744d3e75008a18bdc.
parent 777f2243
...@@ -101,6 +101,9 @@ else: ...@@ -101,6 +101,9 @@ else:
def __getstate__(self): def __getstate__(self):
return self.__dict__ return self.__dict__
def __str__(self):
return self.content
@dataclass @dataclass
class EncodingFast: class EncodingFast:
"""This is dummy class because without the `tokenizers` library we don't have these objects anyway""" """This is dummy class because without the `tokenizers` library we don't have these objects anyway"""
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment