Commits · 50e62a4cb4d503e3559b88838b8cf9f745fef516 · chenpangpang / transformers

09 Jul, 2019 1 commit
- unified tokenizer api and serialization + tests · b1978698
  thomwolf authored Jul 09, 2019
  
  b1978698
05 Jul, 2019 3 commits
- tokenization abstract class - tests for examples · 36bca545
  thomwolf authored Jul 05, 2019
  
  36bca545
- [BIG] name change · 0bab55d5
  thomwolf authored Jul 05, 2019
  
  0bab55d5
- standardizing tokenizers API and adding tests · e75c3f70
  thomwolf authored Jul 05, 2019
  
  e75c3f70
17 Jun, 2019 1 commit
- better error messages · 8415a38b
  thomwolf authored Jun 17, 2019
  
  8415a38b
01 Jun, 2019 1 commit
- add GPT2 torchhub compatibility · a92b6dc3
  VictorSanh authored Jun 01, 2019
  
  a92b6dc3
08 May, 2019 1 commit
- clean up in tokenization · 366a3b02
  thomwolf authored May 08, 2019
  
  366a3b02
07 May, 2019 1 commit
- GPT-2 option to avoid predicting special tokens · d1b6979a
  thomwolf authored May 07, 2019
  
  d1b6979a
01 May, 2019 2 commits
- gpt-2 tokenizer · db98a4a4
  thomwolf authored May 01, 2019
  
  db98a4a4
- Fix #537 · 74f7906d
  Ben Mann authored Apr 30, 2019
  
  74f7906d
17 Apr, 2019 4 commits
- fix #497 · bcde2c61
  thomwolf authored Apr 17, 2019
  
  bcde2c61
- fix #497 · 929579f3
  thomwolf authored Apr 17, 2019
  
  929579f3
- fix GPT-2 tokenization to work also on python 3... · 5afa497c
  thomwolf authored Apr 17, 2019
  
  5afa497c
- fixed GPT-2 tokenization on python 2 · bc70779b
  thomwolf authored Apr 17, 2019
  
  bc70779b
16 Apr, 2019 1 commit
- improving GPT2 tokenization and adding tests · 18a8a15f
  thomwolf authored Apr 16, 2019
  
  18a8a15f
15 Apr, 2019 3 commits
- fixing tests · e8568a3b
  thomwolf authored Apr 15, 2019
  
  e8568a3b
- added tokenizers serialization tests · 870b734b
  thomwolf authored Apr 15, 2019
  
  870b734b
- add serialization semantics to tokenizers - fix transfo-xl tokenizer · 3e65f255
  thomwolf authored Apr 15, 2019
  
  3e65f255
06 Mar, 2019 1 commit
- fix typo - logger info · 5c85fc39
  thomwolf authored Mar 06, 2019
  
  5c85fc39
05 Mar, 2019 1 commit
- Warn instead of raising in BERT and GPT-2 tokenizers as well, to allow for pre-caching of tokens · 4a49c225
  Catalin Voss authored Mar 05, 2019
  
  4a49c225
18 Feb, 2019 3 commits
- simple · ab7f5d29
  thomwolf authored Feb 18, 2019
  
  ab7f5d29
- clean up tokenization - fix python 2 tests · b450a7fa
  thomwolf authored Feb 18, 2019
  
  b450a7fa
- update readme · d44db114
  thomwolf authored Feb 18, 2019
  
  d44db114
17 Feb, 2019 1 commit
- adding gpt2 · ffd62382
  thomwolf authored Feb 17, 2019
  
  ffd62382