GitLab

[Speculative Decoding 1/2 ] Add typical acceptance sampling as one of the sampling techniques in the verifier (#5131)

Signed-off-by: Lei Wen <wenlei03@qiyi.com>
Co-authored-by: Lei Wen <wenlei03@qiyi.com>

Co-authored-by: Michael Goin <michael@neuralmagic.com>
Co-authored-by: youkaichao <youkaichao@gmail.com>
Co-authored-by: zifeitong <zifei.tong@parasail.io>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-neuralmagic@users.noreply.github.com>

Co-authored-by: Robert Shaw <114415538+robertgshaw2-neuralmagic@users.noreply.github.com>

[Core][Distributed] add coordinator to reduce code duplication in tp and pp (#5293)

Signed-off-by: Travis Johnson <tsjohnso@us.ibm.com>
Co-authored-by: Sanger Steel <sangersteel@gmail.com>
Co-authored-by: Roger Wang <ywang@roblox.com>

Inspired by #5146, this PR improves FP8 quantize kernel by vectorizing data transfer to better utilize memory bandwidth. Microbenchmark shows that this improved kernel can achieve 1.0x-1.5x speedup (especially when hidden size is large).

In details, we applied 3 optimizations:

- Use inverted scale so that most divisions are changed to multiplications.
- Unroll the loop by 4 times to improve ILP.
- Use vectorized 4 to transfer data between HBM and SRAM.

Revert "[CI/Build] Add `is_quant_method_supported` to control quantization test configurations" (#5463)

Co-authored-by: Antoni Baum <antoni.baum@protonmail.com>

Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>

Co-authored-by: Roger Wang <ywang@roblox.com>

Co-authored-by: Michael Goin <michael@neuralmagic.com>

[mis][ci/test] fix flaky test in tests/test_sharded_state_loader.py (#5361)

[CI/Test] improve robustness of test by replacing del with context manager (vllm_runner) (#5357)

[CI/Test] improve robustness of test by replacing del with context manager (hf_runner) (#5347)

Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>

Co-authored-by: Varun Sundar Rabindranath <varunsundar08@gmail.com>
Co-authored-by: Varun Sundar Rabindranath <varun@neuralmagic.com>

Co-authored-by: mulin.lyh <mulin.lyh@taobao.com>