Skip to content

GitLab

Switch branch/tag
Find file BlameHistoryPermalink
  • Oren Amsalem's avatar
    Actually the extra_id are from 0-99 and not from 1-100 (#5967) · d24ea708
    Oren Amsalem authored 
    a = tokenizer.encode("we got a <extra_id_99>", return_tensors='pt',add_special_tokens=True)
print(a)
>tensor([[   62,   530,     3,     9, 32000]])
a = tokenizer.encode("we got a <extra_id_100>", return_tensors='pt',add_special_tokens=True)
print(a)
>tensor([[   62,   530,     3,     9,     3,     2, 25666,   834,    23,    26,
           834,  2915,  3155]])
    d24ea708
t5.rst 5.48 KB