router/src/queue.rs · ebc74d5666bf2a304181a5fe8cc7dfc6ddbe1d95 · OpenDAS / text-generation-inference · GitLab

"cuda/cuda_stream_manager.cpp" did not exist on "ef83c8932adc3b5e9c11b0e8bb24d104df856914"

Find file Blame History Permalink

feat(router): use number of tokens in batch as input for dynamic batching (#226) · ebc74d56
OlivierDehaene authored Apr 24, 2023
```
Co-authored-by: Nick Hill <nickhill@us.ibm.com>
```
ebc74d56

queue.rs 14.9 KB