• Jesse Gross's avatar
    vocab: Use int32 for special tokens · 7916f550
    Jesse Gross authored
    Special tokens are currently read as uint32 from the model metadata.
    However, all other parts of the system (including the tokenizer) use
    int32 to represent tokens so it is impossible to represent the high
    portion of the unsigned range. For consistency and to avoid casts,
    we should just use int32 everywhere.
    7916f550
process_text.go 6.39 KB