Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
chenpangpang
transformers
Commits
dc0c1029
Unverified
Commit
dc0c1029
authored
Aug 29, 2023
by
Younes Belkada
Committed by
GitHub
Aug 29, 2023
Browse files
[`Docs`] More clarifications on BT + FA (#25823)
parent
c9bae84e
Changes
1
Show whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
4 additions
and
2 deletions
+4
-2
docs/source/en/perf_infer_gpu_one.md
docs/source/en/perf_infer_gpu_one.md
+4
-2
No files found.
docs/source/en/perf_infer_gpu_one.md
View file @
dc0c1029
...
...
@@ -74,7 +74,7 @@ import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("facebook/opt-350m")
model = AutoModelForCausalLM.from_pretrained("facebook/opt-350m").to("cuda")
model = AutoModelForCausalLM.from_pretrained("facebook/opt-350m"
, torch_dtype=torch.float16
).to("cuda")
#
convert the model to BetterTransformer
model.to_bettertransformer()
...
...
@@ -99,6 +99,8 @@ try using the PyTorch nightly version, which may have a broader coverage for Fla
pip3
install
-U
--pre
torch torchvision torchaudio
--index-url
https://download.pytorch.org/whl/nightly/cu118
```
Or make sure your model is correctly casted in float16 or bfloat16
Have a look at
[
this detailed blogpost
](
https://pytorch.org/blog/out-of-the-box-acceleration/
)
to read more about what is possible to do with
`BetterTransformer`
+ SDPA API.
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment