GitLab

[Misc] Refactor linear layer weight loading; introduce `BasevLLMParameter` and `weight_loader_v2` (#5874)

Co-authored-by: Michael Goin <michael@neuralmagic.com>

Co-authored-by: Michael Goin <michael@neuralmagic.com>

Co-authored-by: Robert Shaw <rshaw@neuralmagic.com>

Co-authored-by: Robert Shaw <rshaw@neuralmagic>

Co-authored-by: Robert Shaw <rshaw@neuralmagic.com>
Co-authored-by: ZX <zx@lbx.dev>

Co-authored-by: Robert Shaw <rshaw@neuralmagic>

[Misc] Add per channel support for static activation quantization; update w8a8 schemes to share base classes (#5650)

[Misc] Add channel-wise quantization support for w8a8 dynamic per token activation quantization (#5542)

Co-authored-by: Robert Shaw <114415538+robertgshaw2-neuralmagic@users.noreply.github.com>

Inspired by #5146, this PR improves FP8 quantize kernel by vectorizing data transfer to better utilize memory bandwidth. Microbenchmark shows that this improved kernel can achieve 1.0x-1.5x speedup (especially when hidden size is large).

In details, we applied 3 optimizations:

- Use inverted scale so that most divisions are changed to multiplications.
- Unroll the loop by 4 times to improve ILP.
- Use vectorized 4 to transfer data between HBM and SRAM.

Revert "[CI/Build] Add `is_quant_method_supported` to control quantization test configurations" (#5463)

Co-authored-by: Michael Goin <michael@neuralmagic.com>

[CI/Test] improve robustness of test by replacing del with context manager (vllm_runner) (#5357)

Co-authored-by: Varun Sundar Rabindranath <varunsundar08@gmail.com>
Co-authored-by: Varun Sundar Rabindranath <varun@neuralmagic.com>

Co-authored-by: Varun Sundar Rabindranath <varunsundar08@gmail.com>
Co-authored-by: Varun Sundar Rabindranath <varun@neuralmagic.com>

Since #4335 was merged, I've noticed that the definition of ServerRunner in the tests is the same as in the test for OpenAI API. I have moved the class to the test utilities to avoid code duplication. (Although it only has been repeated twice so far, I will add another similar test suite in #4200 which would duplicate the code a third time)

Also, I have moved the test utilities file (test_utils.py) to under the test directory (tests/utils.py), since none of its code is actually used in the main package. Note that I have added __init__.py to each test subpackage and updated the ray.init() call in the test utilities file in order to relative import tests/utils.py.

Co-authored-by: alexm <alexm@neuralmagic.com>
Co-authored-by: mgoin <michael@neuralmagic.com>