GitLab

* [Doc] add persistent batch inference

* update

* update

* Update README.md

* Update README_zh-CN.md

* feat(template): remote diff

* feat(cmake): use c++17

* change target tritonfastertransformerbackend to tritonturbomindbackend

* install targets to backends/turbomind

* changge model_dir

* build turbomind

* change namespace fastertransformer to turbomind

* change logger name

* temp

* fix lint

* csrc->src

* remove clang-format

* skip .rst

* skip doc

* clang-format

version

version

* mat_B

* rename lmdeploy/serve/fastertransformer to lmdeploy/serve/turbomind

* update

* update

* change llmdeploy to lmdeploy

* update logo

* update readme

* add hf tokenizer

* format

* fix for comment

* don't skip speical tokens

* add webui

* update readme

* resolve comments

* readme

* feat(src): add int8 and compile passed

* feat(kernels): fix

* feat(llama): update kernel

* feat(src): add debug

* fix(kernel): k_cache use int8_t pointer

* style(llama): clean code

* feat(deploy.py): revert to enable fmha

* style(LlamaV2): clean code

* feat(deploy.py): add default quant policy

* remove constraints on model name

* remove duplicate model converter

* add profile

* get eos and bos from server

* update stop_words

* update sequence_length when the last generated token is eos_id

* fix

* fix

* check-in models

* valicate model_name

* make stop_words as property

* debug profiling

* better stats

* fix assistant reponse

* update profile serving

* update

* update

* support attention bias

* fix conflict

* remove constraints on model name

* remove duplicate model converter

* update arch

* clang-format

* remove comment

---------
Co-authored-by: yaoqian <yaoqian@localhost.localdomain>

* support fmha

* update sm by cudaarch

* update ldscript path

* clang-format

* clang-format

---------

* check-in dockerfile

* check-in dockerfile

* add ft code

* gitignore

* fix lint

* revert fmha

* add logo

* update readme

* add scripts for deploying llama family models via fastertransformer

* fix

* fix

* set symlinks True when copying triton models templates

* pack model repository for triton inference server

* add exception

* fix

* update config.pbtxt and launching scripts