Unverified Commit 532e60be authored by Arthur's avatar Arthur Committed by GitHub
Browse files

Fix blender bot missleading doc (#20301)

* fix the doc to specify that add_prefix_space = False

* add correct expected output
parent df56c843
......@@ -99,8 +99,9 @@ class BlenderbotTokenizer(PreTrainedTokenizer):
```
>>> from transformers import BlenderbotTokenizer
>>> tokenizer = BlenderbotTokenizer.from_pretrained("facebook/blenderbot-3B")
>>> tokenizer.add_prefix_space = False
>>> tokenizer("Hello world")['input_ids']
[6950, 1085, 2]
[47, 921, 86, 1085, 2]
>>> tokenizer(" Hello world")['input_ids']
[6950, 1085, 2]
```
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment