Skip to content

GitLab

Unverified Commit 955fd4fe authored by Yaser Abdelaziz's avatar Yaser Abdelaziz Committed by GitHub
Browse files

[docs/gpt-j] fix typo (#13851)

parent de948350
Show whitespace changes
Inline Side-by-side
Showing with 0 additions and 5 deletions
docs/source/model_doc/gptj.rst
View file @ 955fd4fe
...@@ -53,11 +53,6 @@ Tips: ...@@ -53,11 +53,6 @@ Tips:
size, the tokenizer for `GPT-J <https://huggingface.co/EleutherAI/gpt-j-6B>`__ contains 143 extra tokens size, the tokenizer for `GPT-J <https://huggingface.co/EleutherAI/gpt-j-6B>`__ contains 143 extra tokens
``<|extratoken_1|>... <|extratoken_143|>``, so the ``vocab_size`` of tokenizer also becomes 50400. ``<|extratoken_1|>... <|extratoken_143|>``, so the ``vocab_size`` of tokenizer also becomes 50400.
- Although the embedding matrix has a size of 50400, only 50257 entries are used by the GPT-2 tokenizer. These extra
tokens are added for the sake of efficiency on TPUs. To avoid the mis-match between embedding matrix size and vocab
size, the tokenizer for [GPT-J](https://huggingface.co/EleutherAI/gpt-j-6B) contains 143 extra tokens
``<|extratoken_1|>... <|extratoken_143|>``, so the ``vocab_size`` of tokenizer also becomes 50400.
Generation Generation
_______________________________________________________________________________________________________________________ _______________________________________________________________________________________________________________________
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment