Unverified Commit 6e3b0144 authored by TilmannR's avatar TilmannR Committed by GitHub
Browse files

Fix docstring of CLIPTokenizer(Fast) (#20233)

parent cf7b98b8
......@@ -275,7 +275,7 @@ class CLIPTokenizer(PreTrainedTokenizer):
unk_token (`str`, *optional*, defaults to `<|endoftext|>`):
The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
token instead.
bos_token (`str`, *optional*, defaults to `<|endoftext|>`):
bos_token (`str`, *optional*, defaults to `<|startoftext|>`):
The beginning of sequence token.
eos_token (`str`, *optional*, defaults to `<|endoftext|>`):
The end of sequence token.
......
......@@ -63,7 +63,7 @@ class CLIPTokenizerFast(PreTrainedTokenizerFast):
unk_token (`str`, *optional*, defaults to `<|endoftext|>`):
The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
token instead.
bos_token (`str`, *optional*, defaults to `<|endoftext|>`):
bos_token (`str`, *optional*, defaults to `<|startoftext|>`):
The beginning of sequence token.
eos_token (`str`, *optional*, defaults to `<|endoftext|>`):
The end of sequence token.
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment