GitLab

Co-authored-by: Dash Desai <1723932+iamontheinet@users.noreply.github.com>
Co-authored-by: Aurick Qiao <qiao@aurick.net>
Co-authored-by: Aurick Qiao <aurick.qiao@snowflake.com>
Co-authored-by: Aurick Qiao <aurickq@users.noreply.github.com>
Co-authored-by: Cody Yu <hao.yu.cody@gmail.com>

Signed-off-by: Prashant Gupta <prashantgupta@us.ibm.com>
Co-authored-by: Travis Johnson <tjohnson31415@gmail.com>

Co-authored-by: Danny Guinther <dguinther@neuralmagic.com>

Co-authored-by: Roger Wang <136131678+ywang96@users.noreply.github.com>

Signed-off-by: Tao He <sighingnow@gmail.com>

Some simplifications made for clarity.

Also moves detokenization-related functions from tokenizer.py to detokenizer.py.

Co-authored-by: MeloYang <meloyang05@gmail.com>

Co-authored-by: Roy <jasonailu87@gmail.com>
Co-authored-by: Roger Meier <r.meier@siemens.com>

Co-authored-by: Roy <jasonailu87@gmail.com>

This reverts commit 5c976a7e.

Co-authored-by: Chen Shen <scv119@gmail.com>
Co-authored-by: Shreyas Krishnaswamy <shrekris@anyscale.com>
Co-authored-by: Avnish Narayan <avnish@anyscale.com>