GitLab

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

Co-authored-by: Aurick Qiao <aurick.qiao@snowflake.com>

Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>

Co-authored-by: simon-mo <xmo@berkeley.edu>
Co-authored-by: Chang Su <chang.s.su@oracle.com>
Co-authored-by: Simon Mo <simon.mo@hey.com>
Co-authored-by: Roger Wang <136131678+ywang96@users.noreply.github.com>
Co-authored-by: Roger Wang <ywang@roblox.com>

Co-authored-by: Roger Wang <ywang@roblox.com>

Co-authored-by: Michael Goin <michael@neuralmagic.com>

Co-authored-by: Antoni Baum <antoni.baum@protonmail.com>

Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>

Signed-off-by: Prashant Gupta <prashantgupta@us.ibm.com>
Co-authored-by: Travis Johnson <tjohnson31415@gmail.com>

Co-authored-by: Danny Guinther <dguinther@neuralmagic.com>

Some simplifications made for clarity.

Also moves detokenization-related functions from tokenizer.py to detokenizer.py.

Co-authored-by: MeloYang <meloyang05@gmail.com>

Co-authored-by: Chen Shen <scv119@gmail.com>
Co-authored-by: Shreyas Krishnaswamy <shrekris@anyscale.com>
Co-authored-by: Avnish Narayan <avnish@anyscale.com>

Co-authored-by: Zhuohan Li <zhuohan123@gmail.com>

Signed-off-by: Antoni Baum <antoni.baum@protonmail.com>