Skip to content

GitLab

Commit 504a12b8 authored by zhuwenwen's avatar zhuwenwen
Browse files

add kvcache fp8

parent d074a953
Expand all Hide whitespace changes
Inline Side-by-side
Showing with 1088 additions and 359 deletions
csrc/quantization/fp8/amd/quant_utils.cuh
View file @ 504a12b8
This diff is collapsed.
csrc/quantization/fp8/amd/quant_utils_src.cuh 0 → 100644
View file @ 504a12b8
This diff is collapsed.
vllm/utils.py
View file @ 504a12b8
...@@ -172,7 +172,7 @@ STR_DTYPE_TO_TORCH_DTYPE = { ...@@ -172,7 +172,7 @@ STR_DTYPE_TO_TORCH_DTYPE = {
"half": torch.half, "half": torch.half,
"bfloat16": torch.bfloat16, "bfloat16": torch.bfloat16,
"float": torch.float, "float": torch.float,
# "fp8": torch.uint8, "fp8": torch.uint8,
# "fp8_e4m3": torch.uint8, # "fp8_e4m3": torch.uint8,
# "fp8_e5m2": torch.uint8, # "fp8_e5m2": torch.uint8,
"int8": torch.int8, "int8": torch.int8,
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment