Commits · dcc9bb3252fdad12897851f22f57c3130ba38a7a · chenpangpang / transformers

19 Sep, 2019 6 commits
- Modified encode to return only lists. Added a more complete encode_plus method · dcc9bb32
  LysandreJik authored Sep 19, 2019
  
  dcc9bb32
- Max encoding length + corresponding tests · af23b626
  LysandreJik authored Sep 11, 2019
  
  af23b626
- Number of added tokens calculator · d572d702
  LysandreJik authored Sep 11, 2019
  
  d572d702
- Changed warning to be more explicit · 2d8ec5a6
  LysandreJik authored Sep 06, 2019
```
Co-authored by: julien_c <chaumond@gmail.com>
```
  2d8ec5a6
- Added binary masking tests · c3df2136
  LysandreJik authored Sep 02, 2019
  
  c3df2136
- Tokenizers' encode function can output binary masks · e391d473
  LysandreJik authored Sep 02, 2019
  
  e391d473
05 Sep, 2019 2 commits
- adding test for common properties and cleaning up a bit base class · 5c6cac10
  thomwolf authored Sep 05, 2019
  
  5c6cac10
- Fix typo · d7379477
  maru0kun authored Sep 05, 2019
  
  d7379477
02 Sep, 2019 2 commits
- Appends space before special token · 31d3373b
  LysandreJik authored Sep 01, 2019
  
  31d3373b
- fixing #1133 · fede4ef4
  thomwolf authored Sep 02, 2019
  
  fede4ef4
30 Aug, 2019 5 commits
- added test and debug tokenizer configuration serialization · 69da972a
  thomwolf authored Aug 30, 2019
  
  69da972a
- saving and reloading tokenizer configurations · 88111de0
  thomwolf authored Aug 30, 2019
  
  88111de0
- Added option to setup pretrained tokenizer arguments · 82462c5c
  thomwolf authored Aug 30, 2019
  
  82462c5c
- more doc on special tokens · 8faf2e08
  thomwolf authored Aug 30, 2019
  
  8faf2e08
- adding shortcut to the ids of all the special tokens · d51f72d5
  thomwolf authored Aug 30, 2019
  
  d51f72d5
29 Aug, 2019 1 commit
- fix adding special tokens · 07e21307
  epwalsh authored Aug 29, 2019
  
  07e21307
27 Aug, 2019 1 commit
- add kwargs to base encode function · a175a9dc
  thomwolf authored Aug 27, 2019
  
  a175a9dc
26 Aug, 2019 1 commit
- Generic encoding implementation. · 529a16de
  LysandreJik authored Aug 26, 2019
  
  529a16de
23 Aug, 2019 2 commits
- max_len_single_sentence & max_len_sentences_pair as attributes so they can be modified · 3bcbebd4
  thomwolf authored Aug 23, 2019
  
  3bcbebd4
- adding max_lengths for single sentences and sentences pairs · 47d68534
  thomwolf authored Aug 23, 2019
  
  47d68534
22 Aug, 2019 2 commits
- reraise EnvironmentError in from_pretrained functions of Model and Tokenizer · c603d099
  Abhishek Rao authored Aug 22, 2019
  
  c603d099
- Decode now calls private property instead of public method · 2ba1a14f
  LysandreJik authored Aug 22, 2019
  
  2ba1a14f
20 Aug, 2019 2 commits
- adding proxies options for the from_pretrained methods · 43489756
  thomwolf authored Aug 20, 2019
  
  43489756
- add force_download option to from_pretrained methods · fecaed0e
  thomwolf authored Aug 20, 2019
  
  fecaed0e
14 Aug, 2019 2 commits
- Doc · 572dcfd1
  LysandreJik authored Aug 14, 2019
  
  572dcfd1
- Re-implemented tokenize() iteratively in PreTrainedTokenizer. · 9ce36e3e
  samvelyan authored Aug 14, 2019
  
  9ce36e3e
13 Aug, 2019 1 commit
- Fixed error with encoding · 3d87991f
  LysandreJik authored Aug 13, 2019
  
  3d87991f
12 Aug, 2019 1 commit
- Added documentation and changed parameters for special_tokens_sentences_pair. · 22ac004a
  LysandreJik authored Aug 12, 2019
  
  22ac004a
09 Aug, 2019 1 commit
- Tokenization encode/decode class-based sequence handling · 14e970c2
  LysandreJik authored Aug 09, 2019
  
  14e970c2
08 Aug, 2019 1 commit
- Encode and Decode are back in the superclass. They now handle sentence pairs special tokens. · 6c41a8f5
  LysandreJik authored Aug 08, 2019
  
  6c41a8f5
05 Aug, 2019 1 commit
- cleaning up tokenizer tests structure (at last) - last remaining ppb refs · 328afb70
  thomwolf authored Aug 05, 2019
  
  328afb70
04 Aug, 2019 2 commits
- updating docs - adding few tests to tokenizers · 00132b7a
  thomwolf authored Aug 04, 2019
  
  00132b7a
- big doc update [WIP] · 009273db
  thomwolf authored Aug 04, 2019
  
  009273db
26 Jul, 2019 3 commits
- dictionnary => dictionary · c717d385
  thomwolf authored Jul 26, 2019
  
  c717d385
- fix #901 · 7b6e474c
  thomwolf authored Jul 26, 2019
  
  7b6e474c
- clean up pretrained · 27b0f86d
  thomwolf authored Jul 26, 2019
  
  27b0f86d
24 Jul, 2019 1 commit

make save_pretrained work with added tokens · ae152cec

Joel Grus authored Jul 24, 2019

right now it's dumping the *decoder* when it should be dumping the *encoder*. this fixes that.

ae152cec

16 Jul, 2019 2 commits
- update readme and pretrained model weight files · 1849aa7d
  thomwolf authored Jul 16, 2019
  
  1849aa7d
- update conversion scripts and __main__ · 1b35d05d
  thomwolf authored Jul 16, 2019
  
  1b35d05d
15 Jul, 2019 1 commit
- update tokenizer - update squad example for xlnet · 15d8b126
  thomwolf authored Jul 15, 2019
  
  15d8b126