- 23 Feb, 2020 1 commit
-
-
Lysandre Debut authored
Don't know of a use case where that would be useful, but this is more consistent
-
- 22 Feb, 2020 1 commit
-
-
Funtowicz Morgan authored
* enable_padding should pad up to max_length if set. Signed-off-by:
Morgan Funtowicz <morgan@huggingface.co> * Added more testing on padding. Signed-off-by:
-
- 20 Feb, 2020 2 commits
-
-
Joe Davison authored
-
Funtowicz Morgan authored
* Remove warning when pad_to_max_length is not set. Signed-off-by:
-
- 19 Feb, 2020 3 commits
-
-
Funtowicz Morgan authored
* Correctly return the tuple of generated file(s) when calling save_pretrained Signed-off-by:
-
Funtowicz Morgan authored
* Override build_inputs_with_special_tokens for fast impl + unittest. Signed-off-by:
-
Funtowicz Morgan authored
* Implemented fast version of tokenizers Signed-off-by:
馃殌 Signed-off-by:MOI Anthony <xn1t0x@gmail.com>
-
- 13 Feb, 2020 1 commit
-
-
Joe Davison authored
* Preserve spaces in GPT-2 tokenizers Preserves spaces after special tokens in GPT-2 and inhereted (RoBERTa) tokenizers, enabling correct BPE encoding. Automatically inserts a space in front of first token in encode function when adding special tokens. * Add tokenization preprocessing method * Add framework argument to pipeline factory Also fixes pipeline test issue. Each test input now treated as a distinct sequence.
-
- 29 Jan, 2020 3 commits
-
-
Bram Vanroy authored
Requesting pad_token_id would cause an error message when it is None. Use private _pad_token instead.
-
BramVanroy authored
In batch_encode_plus we have to ensure that the tokenizer has a pad_token_id so that, when padding, no None values are added as padding. That would happen with gpt2, openai, transfoxl. closes https://github.com/huggingface/transformers/issues/2640
-
Lysandre authored
-
- 16 Jan, 2020 3 commits
-
-
Julien Chaumond authored
-
Julien Chaumond authored
-
Julien Chaumond authored
-
- 15 Jan, 2020 2 commits
-
-
Julien Chaumond authored
-
Aditya Bhargava authored
Fix an issue where `prepare_for_model()` gives a `KeyError` when `return_token_type_ids` is set to `False` and `return_tensors` is enabled.
-
- 14 Jan, 2020 2 commits
- 09 Jan, 2020 1 commit
-
-
Yohei Tamura authored
-
- 08 Jan, 2020 1 commit
-
-
Julien Chaumond authored
cc @n1t0
-
- 07 Jan, 2020 1 commit
-
-
Genta Indra Winata authored
-
- 06 Jan, 2020 2 commits
-
-
alberduris authored
-
alberduris authored
-
- 05 Jan, 2020 1 commit
-
-
Aymeric Augustin authored
This should stabilize formatting.
-
- 26 Dec, 2019 4 commits
-
-
Anthony MOI authored
-
Anthony MOI authored
As we discussed, this is handled here directly cc @thomwolf
-
Anthony MOI authored
-
Anthony MOI authored
-
- 25 Dec, 2019 2 commits
-
-
vitaliyradchenko authored
-
vitaliyradchenko authored
-
- 24 Dec, 2019 2 commits
-
-
Anthony MOI authored
-
Anthony MOI authored
-
- 23 Dec, 2019 1 commit
-
-
Alan deLevie authored
avoir -> avoid
-
- 22 Dec, 2019 7 commits
-
-
Aymeric Augustin authored
On Python 3, `open is io.open`.
-
Aymeric Augustin authored
-
Aymeric Augustin authored
-
Aymeric Augustin authored
This prevents transformers from being importable simply because the CWD is the root of the git repository, while not being importable from other directories. That led to inconsistent behavior, especially in examples. Once you fetch this commit, in your dev environment, you must run: $ pip uninstall transformers $ pip install -e . -
Aymeric Augustin authored
Ignore warnings related to Python 2, because it's going away soon.
-
Aymeric Augustin authored
Fixes flake8 warning W291 (x224).
-
Aymeric Augustin authored
This is the result of: $ isort --recursive examples templates transformers utils hubconf.py setup.py
-
