GitLab

Co-authored-by: Roger Wang <ywang@roblox.com>

Co-authored-by: Michael Goin <michael@neuralmagic.com>

Co-authored-by: Antoni Baum <antoni.baum@protonmail.com>

Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>

Signed-off-by: Prashant Gupta <prashantgupta@us.ibm.com>
Co-authored-by: Travis Johnson <tjohnson31415@gmail.com>

Co-authored-by: Danny Guinther <dguinther@neuralmagic.com>

Some simplifications made for clarity.

Also moves detokenization-related functions from tokenizer.py to detokenizer.py.

Co-authored-by: MeloYang <meloyang05@gmail.com>

Co-authored-by: Chen Shen <scv119@gmail.com>
Co-authored-by: Shreyas Krishnaswamy <shrekris@anyscale.com>
Co-authored-by: Avnish Narayan <avnish@anyscale.com>

Co-authored-by: Zhuohan Li <zhuohan123@gmail.com>

Signed-off-by: Antoni Baum <antoni.baum@protonmail.com>

conjuction -> conjunction