• Motoki Wu's avatar
    Add custom stop token ids for generation (#20727) · 45da7cec
    Motoki Wu authored
    * Add StopIdStoppingCriteria
    
    * add a working test for stop id criteria
    
    * add to global scope
    
    * add stop_ids to generate
    
    * add pipeline test
    
    * use tokenizer encode in test
    
    * add test to generation utils
    
    * reformat
    
    * fixup
    
    * make-fix-copies
    
    * rename to stop_token_id
    
    * use stop_tokens instead
    
    * add to text to text generation
    
    * make fixup
    
    * make repo-consistency
    
    * Add support for list of ints for eos_token_id inside generation/utils.py
    
    * Instead of having if elses, cast the eos_token_id into a List[int]
    
    * Add List[int] support for logits_process.py
    
    * add List[int] for beam_search.py
    
    * add List[int] for forced_eos_token_id
    
    * revert stop token id stopping criteria changes
    
    * make fixup
    
    * fix tests
    
    * add eos_token_id to generation/utils.py and added tests test_utils.py
    
    * add eos_token_id type hints and fix for pad tokens
    
    * add comments
    
    * remove some prints and remove forced false test
    
    * fix
    
    * put back test_stop_sequence_stopping_criteria
    
    * remove unused import and make fixup
    
    * add a none check
    
    * update docstring
    
    * add more docstring for list ints
    
    * make fixup
    45da7cec
test_utils.py 131 KB