"cacheflow/vscode:/vscode.git/clone" did not exist on "825d8892b54af80516ce98a89b595018f756a6d3"
Unverified Commit 3f9b7ab9 authored by Kyle Sayers's avatar Kyle Sayers Committed by GitHub
Browse files

[Doc] Update examples to remove SparseAutoModelForCausalLM (#12062)


Signed-off-by: default avatarKyle Sayers <kylesayrs@gmail.com>
parent ad34c0df
...@@ -54,16 +54,15 @@ The quantization process involves three main steps: ...@@ -54,16 +54,15 @@ The quantization process involves three main steps:
### 1. Loading the Model ### 1. Loading the Model
Use `SparseAutoModelForCausalLM`, which wraps `AutoModelForCausalLM`, for saving and loading quantized models: Load your model and tokenizer using the standard `transformers` AutoModel classes:
```python ```python
from llmcompressor.transformers import SparseAutoModelForCausalLM from transformers import AutoTokenizer, AutoModelForCausalLM
from transformers import AutoTokenizer
MODEL_ID = "meta-llama/Meta-Llama-3-8B-Instruct" MODEL_ID = "meta-llama/Meta-Llama-3-8B-Instruct"
model = AutoModelForCausalLM.from_pretrained(
model = SparseAutoModelForCausalLM.from_pretrained( MODEL_ID, device_map="auto", torch_dtype="auto",
MODEL_ID, device_map="auto", torch_dtype="auto") )
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID) tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
``` ```
......
...@@ -30,14 +30,13 @@ The quantization process involves four main steps: ...@@ -30,14 +30,13 @@ The quantization process involves four main steps:
### 1. Loading the Model ### 1. Loading the Model
Use `SparseAutoModelForCausalLM`, which wraps `AutoModelForCausalLM`, for saving and loading quantized models: Load your model and tokenizer using the standard `transformers` AutoModel classes:
```python ```python
from llmcompressor.transformers import SparseAutoModelForCausalLM from transformers import AutoTokenizer, AutoModelForCausalLM
from transformers import AutoTokenizer
MODEL_ID = "meta-llama/Meta-Llama-3-8B-Instruct" MODEL_ID = "meta-llama/Meta-Llama-3-8B-Instruct"
model = SparseAutoModelForCausalLM.from_pretrained( model = AutoModelForCausalLM.from_pretrained(
MODEL_ID, device_map="auto", torch_dtype="auto", MODEL_ID, device_map="auto", torch_dtype="auto",
) )
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID) tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment