GitLab

Co-authored-by: Robert Irvine <robert@seamlessml.com>
Co-authored-by: root <rirv938@gmail.com>
Co-authored-by: Casper <casperbh.96@gmail.com>
Co-authored-by: julian-q <julianhquevedo@gmail.com>

* [FIX] Fix Alibi implementation in PagedAttention kernel

* Fix test_attention

* Fix

---------
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
Co-authored-by: Oliver-ss <yuansongwx@outlook.com>

Co-authored-by: woWoosuk Kwon <woosuk.kwon@berkeley.edu>

* optimize

* add benchmark

* add assert

* add test

* Add query stride to multi_query_cached_kv_attention

* Add kernel benchmark script