• Nicolas Patry's avatar
    Adding new argument `max_new_tokens` for generate. (#11476) · 80d712fa
    Nicolas Patry authored
    * Adding new argument `max_new_tokens` for generate.
    
    This is a proposal to add a new argument `max_new_tokens` to `generate`.
    This include a `MaxNewTokensCriteria` that enables callers that don't
    know about the token length ahead (like pipelines callers) to manage
    more easily the length of their generated output.
    
    * Adding a test for the user warning when both`max_length` and
    `max_new_tokens` are used together.
    
    * Removed redundant `no_grad`.
    80d712fa
test_generation_utils.py 68.6 KB