• Daniël de Kok's avatar
    Add support for GPTQ Marlin (#2052) · 093a27c5
    Daniël de Kok authored
    Add support for GPTQ Marlin kernels
    
    GPTQ Marlin extends the Marlin kernels to support common GPTQ
    configurations:
    
    - bits: 4 or 8
    - groupsize: -1, 32, 64, or 128
    - desc_act: true/false
    
    Using the GPTQ Marlin kernels requires repacking the parameters in the
    Marlin quantizer format.
    
    The kernels were contributed by Neural Magic to VLLM. We vendor them
    here for convenience.
    093a27c5
flash_starcoder2.py 2.6 KB