Skip to content

GitLab

"tests/fixtures/test_sentencepiece_no_bos.model" did not exist on "81d6841b4be25a164235975e5ebdcf99d7a26633"
Switch branch/tag
  1. 29 Jan, 2021 2 commits
  3. 28 Jan, 2021 4 commits
  5. 27 Jan, 2021 1 commit
  7. 24 Jan, 2021 1 commit
  9. 23 Jan, 2021 2 commits
  11. 22 Jan, 2021 3 commits
  13. 21 Jan, 2021 1 commit
  15. 16 Jan, 2021 4 commits
  17. 13 Jan, 2021 1 commit
  19. 12 Jan, 2021 2 commits
  21. 10 Jan, 2021 2 commits
  23. 09 Jan, 2021 2 commits
  25. 06 Jan, 2021 3 commits
  27. 05 Jan, 2021 2 commits
  29. 03 Jan, 2021 1 commit
  31. 02 Jan, 2021 1 commit
  33. 28 Dec, 2020 2 commits
  35. 30 Nov, 2020 6 commits
    • Leo Gao's avatar
      Update docstring · 75db3899
      Leo Gao authored
      75db3899
    • Leo Gao's avatar
      Remove num_tokens · e3031e84
      Leo Gao authored
      e3031e84
    • Leo Gao's avatar
      Add missing import · cf69ba9c
      Leo Gao authored
      cf69ba9c
    • Leo Gao's avatar
      Refactor to remove generate and fix some bad tokenization · 90e50b4c
      Leo Gao authored 
      In particular, the following assumptions are FALSE in general:
tokenize(context + continuation) = tokenize(context) + tokenize(continuation)
len(tokenize(context + continuation)) = len(tokenize(context)) + len(tokenize(continuation))
tokenize(context + continuation)[:len(tokenize(context))] = tokenize(context)

So we need to tip-toe around the problem by being careful with how we do it.

In particular, using Fast is not just for performance; while behavour of GPT2Tokenizer differs across Transformers 2 and 3, GPT2TokenizerFast doesn't.
      90e50b4c
    • Leo Gao's avatar
      Make fewshot_examples fast · 6de520af
      Leo Gao authored
      6de520af
    • Leo Gao's avatar
      Undo MNLI changes · ff3adfe2
      Leo Gao authored
      ff3adfe2