GitLab

* move tokenizer

* remove Tokenizer in init

* update deploy.py

* free runner disk

* limit cpu

* docker.yml

* keep swap

* keep swap

* [doc] Update benchmark command in w4a16.md

* Update w4a16.md

* Update w4a16.md

add pip install nvidia-ml-py

* [doc] Update w4a16.md

* fix lint error
Signed-off-by: del-zhenwu <dele.zhenwu@gmail.com>

* [doc] update model_path & prompt_tokens
Signed-off-by: del-zhenwu <dele.zhenwu@gmail.com>

---------
Signed-off-by: del-zhenwu <dele.zhenwu@gmail.com>

* add tp hint for deploy

* fix lint

* assert tp in turbomind

* fix lint

* support deploy qwen-14b-chat

* update README

* load safetensors first

* make IPv6 compatible, safe run for coroutine interrupting

* instance_id -> session_id and fix api_client.py

* update doc

* remove useless faq

* safe ip mapping

* update app.py

* remove print

* update doc

* change shared_instances_ from weakptr to sharedptr

* update

* Support CORS for openai api server

* Remove unnecessary var

* Add CORS support follow the same style with vllm

* Fix memory leak

* modern c++

* fix benchmark serving cannot use Qwen tokenizer

* update benchmark readme

* expose stop words

* support string

* fix

* remove eoa from chatbot

* remove eoa of turbomind

* fix ut

* suffix wheel and fix InternLM no system bug

* cuda allocator fix

* graceful termination

* lint and compilation fix

Fix side effect brought by supporting codellama: `sequence_start` is always true when calling `model.get_prompt` (#466)

quantilized -> quantized

* better profiler

* wait for releasing mem

* remove fire

* remove support for multiple model benchmark

* comments

* support actual seqlen

* change chat template

* update

* fix ut

* int->size_t

* output more details

* correct tp

* rollback

* update

* update readme

* add 'internlm-chat' as the default tag for internlm chat models

* rollback tokenizer

---------
Co-authored-by: AllentDan <AllentDan@yeah.net>
Co-authored-by: grimoire <yaoqian@pjlab.org.cn>

* better profiler

* wait for releasing mem

* remove fire

* remove support for multiple model benchmark

* comments

* output more details

* correct tp

* support actual seqlen

* fix lint

* update variable types

* lint

* update type

* fix lint

---------

* fix token count bug

* fix error response

* reduce gil switching

* ffi lock func

* remove unused

* remove unused

* remove unused

* fix the build step

* Fix the build step

This reverts commit 7368b88692ecca3f5b39f92a8cc41cf21e3fd71e.

* tmp

* add demo for codellama inference

* update

* update

* update

* update codellama.md

* export rope_theta

* update

* update doc

* fix client.py

* define SamplingParam

* rollback 'end'

* rotary_emb_base to rotary_embedding_base

* change to baichuan2-7b

* support baichuan2-chat

* update args from generation config

* update deploy.py

* update readme

* tested with tp

* step-1 when last id is eos

* add news

---------
Co-authored-by: chenxin <chenxin@pjlab.org.cn>