GitLab

Co-authored-by: One <imone@tuta.io>

Co-authored-by: Zhuohan Li <zhuohan123@gmail.com>

* [FIX] Fix Alibi implementation in PagedAttention kernel

* Fix test_attention

* Fix

---------
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
Co-authored-by: Oliver-ss <yuansongwx@outlook.com>

Signed-off-by: Antoni Baum <antoni.baum@protonmail.com>
Co-authored-by: Avnish Narayan <38871737+avnishn@users.noreply.github.com>

Signed-off-by: Tao Peng <jiankeng.pt@alibaba-inc.com>

Co-authored-by: oliveryuan <oliveryuan@basemind.com>

Co-authored-by: woWoosuk Kwon <woosuk.kwon@berkeley.edu>

* optimize

* add benchmark

* add assert

* add test

* Add query stride to multi_query_cached_kv_attention

* Add kernel benchmark script