Skip to content

GitLab

Commit 20e35938 authored by jmorganca's avatar jmorganca Committed by Michael Yang
Browse files

model: validate left and right pairs before merging them

parent 63a39406
Hide whitespace changes
Inline Side-by-side
Showing with 4 additions and 0 deletions
model/process_text_spm.go
View file @ 20e35938
...@@ -169,6 +169,10 @@ func (spm SentencePieceModel) Encode(s string, addSpecial bool) ([]int32, error) ...@@ -169,6 +169,10 @@ func (spm SentencePieceModel) Encode(s string, addSpecial bool) ([]int32, error)
continue continue
} }
if id := spm.vocab.Encode(string(left.runes) + string(right.runes)); id < 0 {
continue
}
merges[pair.a].runes = append(left.runes, right.runes...) merges[pair.a].runes = append(left.runes, right.runes...)
merges[pair.b].runes = nil merges[pair.b].runes = nil
merges[pair.a].n = right.n merges[pair.a].n = right.n
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment