GitLab

issue/16: 多请求多并发测试脚本

support iluvatar device type

fix: assign scale_* as class members instead of local variables

fix: fix 9G4B model by add coefficent to weight tensor when load

issue/14: 兼容openai协议

issue/12: 沐曦多卡推理卡死问题修复

Added memory alignment in the memory pool

Fixed a double free bug

Fixed a bug where allocating 0 workspace causes overlapping memory in memory pool which then results in a memory double free error.

支持多请求连续batch以及kvcache池

* Tensor memory pool - initial commit

* Adjusted workspace management to use the memory pool.

* Re-differentiated create and createAsync in Storage

* Set initial memory pool size to 0

* Separated creatAsync and createFromPool

* Removed the redundant WorkspaceHandle

* Adjusted naming

* Set default mempool size to 0 in declaration

* Adjusted format

Fixed Build Cemplexity

Modified xmake script to avoid a potential complexity of having to manually delete $INFINI_ROOT/include/infinicore_infer.h