1. 08 Jan, 2023 5 commits
  2. 07 Jan, 2023 1 commit
  3. 06 Jan, 2023 4 commits
  4. 05 Jan, 2023 8 commits
  5. 04 Jan, 2023 20 commits
  6. 03 Jan, 2023 2 commits
    • Motoki Wu's avatar
      Add custom stop token ids for generation (#20727) · 45da7cec
      Motoki Wu authored
      * Add StopIdStoppingCriteria
      
      * add a working test for stop id criteria
      
      * add to global scope
      
      * add stop_ids to generate
      
      * add pipeline test
      
      * use tokenizer encode in test
      
      * add test to generation utils
      
      * reformat
      
      * fixup
      
      * make-fix-copies
      
      * rename to stop_token_id
      
      * use stop_tokens instead
      
      * add to text to text generation
      
      * make fixup
      
      * make repo-consistency
      
      * Add support for list of ints for eos_token_id inside generation/utils.py
      
      * Instead of having if elses, cast the eos_token_id into a List[int]
      
      * Add List[int] support for logits_process.py
      
      * add List[int] for beam_search.py
      
      * add List[int] for forced_eos_token_id
      
      * revert stop token id stopping criteria changes
      
      * make fixup
      
      * fix tests
      
      * add eos_token_id to generation/utils.py and added tests test_utils.py
      
      * add eos_token_id type hints and fix for pad tokens
      
      * add comments
      
      * remove some prints and remove forced false test
      
      * fix
      
      * put back test_stop_sequence_stopping_criteria
      
      * remove unused import and make fixup
      
      * add a none check
      
      * update docstring
      
      * add more docstring for list ints
      
      * make fixup
      45da7cec
    • radcheb's avatar
      Fix race condition on cleaning checkpoints when save_total_limit set to 1 (#20989) · cd918492
      radcheb authored
      
      
      * Update trainer.py
      
      * fix style
      Co-authored-by: default avatarRadhwane Chebaane <rchebaane.external@epo.org>
      cd918492