"...git@developer.sourcefind.cn:OpenDAS/bitsandbytes.git" did not exist on "5a4263f4dc05fe8f78f4111beab9f68a81deeab1"
Unverified Commit 9a0e10c4 authored by Tim Dettmers's avatar Tim Dettmers Committed by GitHub
Browse files

Merge pull request #729 from merveenoyan/small_fix

Small fix to getting started code in README
parents 1a5d43eb 9f293ffc
...@@ -119,7 +119,7 @@ torch.nn.Embedding(...) -> bnb.nn.StableEmbedding(...) # recommended for NLP mo ...@@ -119,7 +119,7 @@ torch.nn.Embedding(...) -> bnb.nn.StableEmbedding(...) # recommended for NLP mo
``` ```
Note that by default all parameter tensors with less than 4096 elements are kept at 32-bit even if you initialize those parameters with 8-bit optimizers. This is done since such small tensors do not save much memory and often contain highly variable parameters (biases) or parameters that require high precision (batch norm, layer norm). You can change this behavior like so: Note that by default all parameter tensors with less than 4096 elements are kept at 32-bit even if you initialize those parameters with 8-bit optimizers. This is done since such small tensors do not save much memory and often contain highly variable parameters (biases) or parameters that require high precision (batch norm, layer norm). You can change this behavior like so:
``` ```python
# parameter tensors with less than 16384 values are optimized in 32-bit # parameter tensors with less than 16384 values are optimized in 32-bit
# it is recommended to use multiplies of 4096 # it is recommended to use multiplies of 4096
adam = bnb.optim.Adam8bit(model.parameters(), min_8bit_size=16384) adam = bnb.optim.Adam8bit(model.parameters(), min_8bit_size=16384)
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment