Commits · 1cbd566c63535878bf6b9ba90655420eece636e7 · chenpangpang / transformers

24 Sep, 2019 5 commits
- various updates · a6981076
  thomwolf authored Sep 24, 2019
  
  a6981076
- create_mask_from_sequences -> create_token_type_ids_from_sequences · d340e232
  LysandreJik authored Sep 24, 2019
  
  d340e232
- `output_token_type` -> `token_type_ids` · c832f43a
  LysandreJik authored Sep 24, 2019
  
  c832f43a
- Updated tests · 0ea82b24
  LysandreJik authored Sep 24, 2019
  
  0ea82b24
- Updated DistilBERT · 9d44236f
  LysandreJik authored Sep 24, 2019
  
  9d44236f
19 Sep, 2019 13 commits
- Python 2 compatibility · ab984a8b
  LysandreJik authored Sep 19, 2019
  
  ab984a8b
- Tokenizer accepts token list as well as string · 3df208c9
  LysandreJik authored Sep 19, 2019
  
  3df208c9
- prepare_for_model and prepare_pair_for_model methods. Added an option to... · 66ea76b8
  LysandreJik authored Sep 19, 2019
```
prepare_for_model and prepare_pair_for_model methods. Added an option to select which sequence will be truncated.
```
  66ea76b8
- Stride + tests + small fixes · baa74326
  LysandreJik authored Sep 19, 2019
  
  baa74326
- Mask computing in standalone method. Tests. · c10c7d59
  LysandreJik authored Sep 19, 2019
  
  c10c7d59
- Sentence -> Sequence. Removed output_mask from the special token addition methods. · bf503158
  LysandreJik authored Sep 19, 2019
  
  bf503158
- Doc + remove artefacts · 8cba0572
  LysandreJik authored Sep 19, 2019
  
  8cba0572
- Modified encode to return only lists. Added a more complete encode_plus method · dcc9bb32
  LysandreJik authored Sep 19, 2019
  
  dcc9bb32
- Max encoding length + corresponding tests · af23b626
  LysandreJik authored Sep 11, 2019
  
  af23b626
- Number of added tokens calculator · d572d702
  LysandreJik authored Sep 11, 2019
  
  d572d702
- Changed warning to be more explicit · 2d8ec5a6
  LysandreJik authored Sep 06, 2019
```
Co-authored by: julien_c <chaumond@gmail.com>
```
  2d8ec5a6
- Added binary masking tests · c3df2136
  LysandreJik authored Sep 02, 2019
  
  c3df2136
- Tokenizers' encode function can output binary masks · e391d473
  LysandreJik authored Sep 02, 2019
  
  e391d473
05 Sep, 2019 2 commits
- adding test for common properties and cleaning up a bit base class · 5c6cac10
  thomwolf authored Sep 05, 2019
  
  5c6cac10
- Fix typo · d7379477
  maru0kun authored Sep 05, 2019
  
  d7379477
02 Sep, 2019 2 commits
- Appends space before special token · 31d3373b
  LysandreJik authored Sep 01, 2019
  
  31d3373b
- fixing #1133 · fede4ef4
  thomwolf authored Sep 02, 2019
  
  fede4ef4
30 Aug, 2019 5 commits
- added test and debug tokenizer configuration serialization · 69da972a
  thomwolf authored Aug 30, 2019
  
  69da972a
- saving and reloading tokenizer configurations · 88111de0
  thomwolf authored Aug 30, 2019
  
  88111de0
- Added option to setup pretrained tokenizer arguments · 82462c5c
  thomwolf authored Aug 30, 2019
  
  82462c5c
- more doc on special tokens · 8faf2e08
  thomwolf authored Aug 30, 2019
  
  8faf2e08
- adding shortcut to the ids of all the special tokens · d51f72d5
  thomwolf authored Aug 30, 2019
  
  d51f72d5
29 Aug, 2019 1 commit
- fix adding special tokens · 07e21307
  epwalsh authored Aug 29, 2019
  
  07e21307
27 Aug, 2019 1 commit
- add kwargs to base encode function · a175a9dc
  thomwolf authored Aug 27, 2019
  
  a175a9dc
26 Aug, 2019 1 commit
- Generic encoding implementation. · 529a16de
  LysandreJik authored Aug 26, 2019
  
  529a16de
23 Aug, 2019 2 commits
- max_len_single_sentence & max_len_sentences_pair as attributes so they can be modified · 3bcbebd4
  thomwolf authored Aug 23, 2019
  
  3bcbebd4
- adding max_lengths for single sentences and sentences pairs · 47d68534
  thomwolf authored Aug 23, 2019
  
  47d68534
22 Aug, 2019 2 commits
- reraise EnvironmentError in from_pretrained functions of Model and Tokenizer · c603d099
  Abhishek Rao authored Aug 22, 2019
  
  c603d099
- Decode now calls private property instead of public method · 2ba1a14f
  LysandreJik authored Aug 22, 2019
  
  2ba1a14f
20 Aug, 2019 2 commits
- adding proxies options for the from_pretrained methods · 43489756
  thomwolf authored Aug 20, 2019
  
  43489756
- add force_download option to from_pretrained methods · fecaed0e
  thomwolf authored Aug 20, 2019
  
  fecaed0e
14 Aug, 2019 2 commits
- Doc · 572dcfd1
  LysandreJik authored Aug 14, 2019
  
  572dcfd1
- Re-implemented tokenize() iteratively in PreTrainedTokenizer. · 9ce36e3e
  samvelyan authored Aug 14, 2019
  
  9ce36e3e
13 Aug, 2019 1 commit
- Fixed error with encoding · 3d87991f
  LysandreJik authored Aug 13, 2019
  
  3d87991f
12 Aug, 2019 1 commit
- Added documentation and changed parameters for special_tokens_sentences_pair. · 22ac004a
  LysandreJik authored Aug 12, 2019
  
  22ac004a