examples/offline_inference_chat.py · ad385667103b7a98fee5a4fc46d2a7745407333e · OpenDAS / vllm_cscc

[Frontend] Batch inference for llm.chat() API (#8648) · 2529d09b

Andy authored Sep 24, 2024


Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
Co-authored-by: Roger Wang <ywang@roblox.com>
Co-authored-by: Roger Wang <136131678+ywang96@users.noreply.github.com>

2529d09b

offline_inference_chat.py 1.99 KB

Replace offline_inference_chat.py