1. 09 Jan, 2023 1 commit
  2. 08 Jan, 2023 5 commits
  3. 07 Jan, 2023 1 commit
  4. 06 Jan, 2023 4 commits
  5. 05 Jan, 2023 8 commits
  6. 04 Jan, 2023 20 commits
  7. 03 Jan, 2023 1 commit
    • Motoki Wu's avatar
      Add custom stop token ids for generation (#20727) · 45da7cec
      Motoki Wu authored
      * Add StopIdStoppingCriteria
      
      * add a working test for stop id criteria
      
      * add to global scope
      
      * add stop_ids to generate
      
      * add pipeline test
      
      * use tokenizer encode in test
      
      * add test to generation utils
      
      * reformat
      
      * fixup
      
      * make-fix-copies
      
      * rename to stop_token_id
      
      * use stop_tokens instead
      
      * add to text to text generation
      
      * make fixup
      
      * make repo-consistency
      
      * Add support for list of ints for eos_token_id inside generation/utils.py
      
      * Instead of having if elses, cast the eos_token_id into a List[int]
      
      * Add List[int] support for logits_process.py
      
      * add List[int] for beam_search.py
      
      * add List[int] for forced_eos_token_id
      
      * revert stop token id stopping criteria changes
      
      * make fixup
      
      * fix tests
      
      * add eos_token_id to generation/utils.py and added tests test_utils.py
      
      * add eos_token_id type hints and fix for pad tokens
      
      * add comments
      
      * remove some prints and remove forced false test
      
      * fix
      
      * put back test_stop_sequence_stopping_criteria
      
      * remove unused import and make fixup
      
      * add a none check
      
      * update docstring
      
      * add more docstring for list ints
      
      * make fixup
      45da7cec