• Jesse Gross's avatar
    model: Don't unconditionally add special tokens · b70fc4d5
    Jesse Gross authored
    We sometimes tokenize partial strings. For example, with
    multimodal inputs, we split the input string around the images
    and then tokenize each piece. In these cases, we should only add
    the special tokens on the first piece.
    b70fc4d5
process_text.go 7 KB