GitLab

Signed-off-by: Aaron Pham <contact@aarnphm.xyz>

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

Signed-off-by: Travis Johnson <tsjohnso@us.ibm.com>
Signed-off-by: Prashant Gupta <prashantgupta@us.ibm.com>
Co-authored-by: Prashant Gupta <prashantgupta@us.ibm.com>
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

Co-authored-by: Zifei Tong <zifeitong@gmail.com>

Co-authored-by: Roger Wang <ywang@roblox.com>

Some simplifications made for clarity.

Also moves detokenization-related functions from tokenizer.py to detokenizer.py.

Co-authored-by: MeloYang <meloyang05@gmail.com>

Signed-off-by: Antoni Baum <antoni.baum@protonmail.com>