Unverified Commit 364a5ae1 authored by Lysandre Debut's avatar Lysandre Debut Committed by GitHub
Browse files

Refactor Code samples; Test code samples (#5036)



* Refactor code samples

* Test docstrings

* Style

* Tokenization examples

* Run rust of tests

* First step to testing source docs

* Style and BART comment

* Test the remainder of the code samples

* Style

* let to const

* Formatting fixes

* Ready for merge

* Fix fixture + Style

* Fix last tests

* Update docs/source/quicktour.rst
Co-authored-by: default avatarSylvain Gugger <35901082+sgugger@users.noreply.github.com>

* Addressing @sgugger's comments + Fix MobileBERT in TF
Co-authored-by: default avatarSylvain Gugger <35901082+sgugger@users.noreply.github.com>
parent 315f464b
This diff is collapsed.
This diff is collapsed.
...@@ -66,13 +66,15 @@ class MBartTokenizer(XLMRobertaTokenizer): ...@@ -66,13 +66,15 @@ class MBartTokenizer(XLMRobertaTokenizer):
The tokenization method is <tokens> <eos> <language code>. There is no BOS token. The tokenization method is <tokens> <eos> <language code>. There is no BOS token.
Examples:: Examples::
from transformers import MBartTokenizer
tokenizer = MBartTokenizer.from_pretrained('mbart-large-en-ro') >>> from transformers import MBartTokenizer
example_english_phrase = " UN Chief Says There Is No Military Solution in Syria" >>> tokenizer = MBartTokenizer.from_pretrained('facebook/mbart-large-en-ro')
expected_translation_romanian = "Şeful ONU declară că nu există o soluţie militară în Siria" >>> example_english_phrase = " UN Chief Says There Is No Military Solution in Syria"
batch: dict = tokenizer.prepare_translation_batch( >>> expected_translation_romanian = "Şeful ONU declară că nu există o soluţie militară în Siria"
example_english_phrase, src_lang="en_XX", tgt_lang="ro_RO", tgt_texts=expected_translation_romanian >>> batch: dict = tokenizer.prepare_translation_batch(
) ... example_english_phrase, src_lang="en_XX", tgt_lang="ro_RO", tgt_texts=expected_translation_romanian
... )
""" """
vocab_files_names = {"vocab_file": "sentencepiece.bpe.model"} vocab_files_names = {"vocab_file": "sentencepiece.bpe.model"}
......
This diff is collapsed.
...@@ -81,6 +81,7 @@ class ReformerTokenizer(PreTrainedTokenizer): ...@@ -81,6 +81,7 @@ class ReformerTokenizer(PreTrainedTokenizer):
vocab_files_names = VOCAB_FILES_NAMES vocab_files_names = VOCAB_FILES_NAMES
pretrained_vocab_files_map = PRETRAINED_VOCAB_FILES_MAP pretrained_vocab_files_map = PRETRAINED_VOCAB_FILES_MAP
max_model_input_sizes = PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES max_model_input_sizes = PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES
model_input_names = ["attention_mask"]
def __init__( def __init__(
self, self,
......
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment