Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in
Toggle navigation
Menu
Open sidebar
OpenDAS
vllm_cscc
Commits
c53dc466
Unverified
Commit
c53dc466
authored
Feb 05, 2025
by
Michael Goin
Committed by
GitHub
Feb 04, 2025
Browse files
[Doc] Remove performance warning for auto_awq.md (#12743)
parent
3d09e592
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
0 additions
and
6 deletions
+0
-6
docs/source/features/quantization/auto_awq.md
docs/source/features/quantization/auto_awq.md
+0
-6
No files found.
docs/source/features/quantization/auto_awq.md
View file @
c53dc466
...
...
@@ -2,12 +2,6 @@
# AutoAWQ
:::{warning}
Please note that AWQ support in vLLM is under-optimized at the moment. We would recommend using the unquantized version of the model for better
accuracy and higher throughput. Currently, you can use AWQ as a way to reduce memory footprint. As of now, it is more suitable for low latency
inference with small number of concurrent requests. vLLM's AWQ implementation have lower throughput than unquantized version.
:::
To create a new 4-bit quantized model, you can leverage
[
AutoAWQ
](
https://github.com/casper-hansen/AutoAWQ
)
.
Quantizing reduces the model's precision from FP16 to INT4 which effectively reduces the file size by ~70%.
The main benefits are lower latency and memory usage.
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment