GitLab

Signed-off-by: raushan <raushan@huggingface.co>
Signed-off-by: Isotr0py <2037008807@qq.com>
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Co-authored-by: Isotr0py <2037008807@qq.com>
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>

Signed-off-by: Seiji Eicher <seiji@anyscale.com>

Signed-off-by: Chengji Yao <chengjiyao@google.com>

Signed-off-by: zRzRzRzRzRzRzR <2448370773@qq.com>
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Signed-off-by: Lu Fang <fanglu@fb.com>
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Co-authored-by: Lu Fang <fanglu@fb.com>

Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>

Signed-off-by: Thomas Parnell <tpa@zurich.ibm.com>

Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>

Signed-off-by: Rabi Mishra <ramishra@redhat.com>

Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>

Signed-off-by: Shixian Cui <shixian@amazon.com>
Co-authored-by: Shixian Cui <shixian@amazon.com>

Signed-off-by: Isotr0py <2037008807@qq.com>

Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>

Signed-off-by: Huy Do <huydhn@gmail.com>

Signed-off-by: Deepfocused <rlawhdrhs27@gmail.com>
Signed-off-by: woongsik <rlawhdrhs27@gmail.com>

Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>

Signed-off-by: Lucia Fang <fanglu@fb.com>
Signed-off-by: Lu Fang <fanglu@meta.com>
Signed-off-by: Lu Fang <fanglu@fb.com>
Co-authored-by: Lu Fang <fanglu@meta.com>

Signed-off-by: Jialin Ouyang <jialino@meta.com>

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

[Bugfix] The special_tokens in tokenizer should also be controlled by do_lower_case in encoder_config. (#20750)
Signed-off-by: wang.yuqi <noooop@126.com>

Signed-off-by: wang.yuqi <noooop@126.com>

Signed-off-by: Shixian Cui <shixian@amazon.com>
Co-authored-by: Shixian Cui <shixian@amazon.com>

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

Signed-off-by: ElizaWszola <ewszola@redhat.com>

Signed-off-by: Kyle Huang <kylhuang@nvidia.com>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>

Signed-off-by: Asher Zhang <asherszhang@tencent.com>

Signed-off-by: Varun Sundar Rabindranath <vsundarr@redhat.com>
Co-authored-by: Varun Sundar Rabindranath <vsundarr@redhat.com>

Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>

Signed-off-by: David Ben-David <davidb@pliops.com>
Co-authored-by: David Ben-David <davidb@pliops.com>

Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>

Signed-off-by: mgoin <mgoin64@gmail.com>

feat - add a new endpoint `get_tokenizer_info` to provide tokenizer/chat-template information (#20575)
Signed-off-by: m-misiura <mmisiura@redhat.com>

Signed-off-by: Seiji Eicher <seiji@anyscale.com>

Signed-off-by: qizixi <qizixi@meta.com>
Co-authored-by: qizixi <qizixi@meta.com>

Signed-off-by: Peter Pan <Peter.Pan@daocloud.io>

Signed-off-by: Max de Bayser <mbayser@br.ibm.com>

Signed-off-by: Patrick von Platen <patrick.v.platen@gmail.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>

Signed-off-by: Patrick von Platen <patrick.v.platen@gmail.com>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>

Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>