-
d.transposed authored
[Bugfix] Handle the edge case in detokenizer where processed tokens contain both `stop` str and `eos` token (#23938) Signed-off-by:dtransposed <damian.bogunowicz@gmail.com>
922d3b40
[Bugfix] Handle the edge case in detokenizer where processed tokens contain both `stop` str and `eos` token (#23938)
Signed-off-by: dtransposed <damian.bogunowicz@gmail.com>