GitLab

Signed-off-by: kevin <kevin@anyscale.com>

Signed-off-by: kevin <kevin@anyscale.com>

Co-authored-by: Roger Wang <ywang@roblox.com>

Co-authored-by: Michael Goin <michael@neuralmagic.com>

[mis][ci/test] fix flaky test in tests/test_sharded_state_loader.py (#5361)

[Core][CUDA Graph] add output buffer for cudagraph to reduce memory footprint (#5074)

[CI/Test] improve robustness of test by replacing del with context manager (vllm_runner) (#5357)

[CI/Test] improve robustness of test by replacing del with context manager (hf_runner) (#5347)

Bug description:
With torch 2.4.0.dev20240603+cu121,
cutlass_fp8_supported outputs False, and the (capability, version) before the comparison is (90, 11111111112)

This PR fixes the support check for FP8 CUTLASS ( cutlass_fp8_supported) which was introduced in https://github.com/vllm-project/vllm/pull/5183.

Co-authored-by: team <calvinn.ng@ahrefs.com>

Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>

Co-authored-by: Varun Sundar Rabindranath <varunsundar08@gmail.com>
Co-authored-by: Varun Sundar Rabindranath <varun@neuralmagic.com>

Switching from torch._scaled_mm to vLLM's cutlass fp8 kernels when supported as we are seeing 5-15% improvement in e2e performance on neuralmagic/Meta-Llama-3-8B-Instruct-FP8

see https://docs.google.com/spreadsheets/d/1GiAnmzyGHgZ6zL_LDSTm35Bdrt4A8AaFEurDlISYYA4/ for some quick e2e benchmarks and #5144 for comparisons across different GEMM sizes.

Co-authored-by: mulin.lyh <mulin.lyh@taobao.com>

Co-authored-by: Simon Mo <simon.mo@hey.com>
Co-authored-by: Breno Faria <breno.faria@intrafind.com>