GitLab

Signed-off-by: lijian <lijian6@sugon.com>

Signed-off-by: lijian <lijian6@sugon.com>

2. Fix rocshmem internode hang on 508.
Signed-off-by: lijian <lijian6@sugon.com>

2. Add internode ll mode.
3. Add test internode ll mode.
Signed-off-by: lijian <lijian6@sugon.com>

Signed-off-by: lijian <lijian6@sugon.com>

     backup pass way.
Signed-off-by: lishen <lishen@sugon.com>

Signed-off-by: lijian <lijian6@sugon.com>

Signed-off-by: lijian <lijian6@sugon.com>

Signed-off-by: lijian <lijian6@sugon.com>

Co-authored-by: Yifei Zhang <219273404+yifeizhang-c@users.noreply.github.com>

* Fix hidden_size % 128 != 0

* Add `align_down()` function

* Use the full warp to wait TMA store

* Support arbitrary hidden sizes in fp8 cast

* lint

* Remove redundant TMA flushes

* Less barrier initialization overhead

* Simplify `elect_one_sync`

* Use `elect_one_sync` instead of lanes

* Minor fix

* Polish testing prints

* Refactor for internode kernels

* Better performance

* Fix mbarrier

* Remove redundant store

Each thread is responsible for one target rank

* nits

* hack unrolled warp copy

* Revert "nits"

This reverts commit 3e1b28d9b17f2c1cc46403d432ca576dbf15bd45.

* Fix: avoid floating point exception.
Signed-off-by: wangfakang <fakangwang@gmail.com>

* simplify the code.
Signed-off-by: wangfakang <fakangwang@gmail.com>

---------
Signed-off-by: wangfakang <fakangwang@gmail.com>

* fix combine timeout due to forwarder min head update

* Update head before and after combine_token; add assertion for nvl_buffer_size_per_rdma_rank

---------
Co-authored-by: zhiyi Hu <zhiyihu@U-NYQQMGK0-2250.local>

Signed-off-by: wangfakang <fakangwang@gmail.com>