Unverified Commit 00934026 authored by Sylvain Gugger's avatar Sylvain Gugger Committed by GitHub
Browse files

LLaMA house-keeping (#22216)

* LLaMA house-keeping

* Doc links
parent 42f8f764
...@@ -33,8 +33,10 @@ python src/transformers/models/llama/convert_llama_weights_to_hf.py \ ...@@ -33,8 +33,10 @@ python src/transformers/models/llama/convert_llama_weights_to_hf.py \
- After conversion, the model and tokenizer can be loaded via: - After conversion, the model and tokenizer can be loaded via:
```python ```python
tokenizer = transformers.LlamaTokenizer.from_pretrained("/output/path/tokenizer/") from transformers import LlamaForCausalLM, LlamaTokenizer
model = transformers.LlamaForCausalLM.from_pretrained("/output/path/llama-7b/")
tokenizer = LlamaTokenizer.from_pretrained("/output/path/tokenizer/")
model = LlamaForCausalLM.from_pretrained("/output/path/llama-7b/")
``` ```
- The LLaMA tokenizer is based on [sentencepiece](https://github.com/google/sentencepiece). One quirk of sentencepiece is that when decoding a sequence, if the first token is the start of the word (e.g. "Banana"), the tokenizer does not prepend the prefix space to the string. To have the tokenizer output the prefix space, set `decode_with_prefix_space=True` in the `LlamaTokenizer` object or in the tokenizer configuration. - The LLaMA tokenizer is based on [sentencepiece](https://github.com/google/sentencepiece). One quirk of sentencepiece is that when decoding a sequence, if the first token is the start of the word (e.g. "Banana"), the tokenizer does not prepend the prefix space to the string. To have the tokenizer output the prefix space, set `decode_with_prefix_space=True` in the `LlamaTokenizer` object or in the tokenizer configuration.
......
...@@ -4486,9 +4486,9 @@ if TYPE_CHECKING: ...@@ -4486,9 +4486,9 @@ if TYPE_CHECKING:
TypicalLogitsWarper, TypicalLogitsWarper,
top_k_top_p_filtering, top_k_top_p_filtering,
) )
from .modeling_utils import PreTrainedModel
# PyTorch model imports # PyTorch model imports
from .modeling_utils import PreTrainedModel
from .models.albert import ( from .models.albert import (
ALBERT_PRETRAINED_MODEL_ARCHIVE_LIST, ALBERT_PRETRAINED_MODEL_ARCHIVE_LIST,
AlbertForMaskedLM, AlbertForMaskedLM,
......
...@@ -30,7 +30,7 @@ LLAMA_PRETRAINED_CONFIG_ARCHIVE_MAP = {} ...@@ -30,7 +30,7 @@ LLAMA_PRETRAINED_CONFIG_ARCHIVE_MAP = {}
class LlamaConfig(PretrainedConfig): class LlamaConfig(PretrainedConfig):
r""" r"""
This is the configuration class to store the configuration of a [`~LlamaModel`]. It is used to instantiate an LLaMA This is the configuration class to store the configuration of a [`LlamaModel`]. It is used to instantiate an LLaMA
model according to the specified arguments, defining the model architecture. Instantiating a configuration with the model according to the specified arguments, defining the model architecture. Instantiating a configuration with the
defaults will yield a similar configuration to that of the LLaMA-7B. defaults will yield a similar configuration to that of the LLaMA-7B.
...@@ -41,7 +41,7 @@ class LlamaConfig(PretrainedConfig): ...@@ -41,7 +41,7 @@ class LlamaConfig(PretrainedConfig):
Args: Args:
vocab_size (`int`, *optional*, defaults to 32000): vocab_size (`int`, *optional*, defaults to 32000):
Vocabulary size of the LLaMA model. Defines the number of different tokens that can be represented by the Vocabulary size of the LLaMA model. Defines the number of different tokens that can be represented by the
`inputs_ids` passed when calling [`~LlamaModel`] `inputs_ids` passed when calling [`LlamaModel`]
hidden_size (`int`, *optional*, defaults to 4096): hidden_size (`int`, *optional*, defaults to 4096):
Dimension of the hidden representations. Dimension of the hidden representations.
intermediate_size (`int`, *optional*, defaults to 11008): intermediate_size (`int`, *optional*, defaults to 11008):
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment