1. 03 Jun, 2024 1 commit
    • Ahmed Moubtahij's avatar
      Token healing (#30081) · 39b2ff69
      Ahmed Moubtahij authored
      
      
      * token healing impl + trie with extensions
      
      * make fixup
      
      * prefix-robust space tokenization
      
      * examples readme and requirements
      
      * make fixup
      
      * allow input prompt and model
      
      * redundant defaults
      
      * Specialized Trie
      
      * make fixup
      
      * updated tests with new inherited Tree
      
      * input ids to auto device_map
      
      * rm unused import
      
      * Update src/transformers/generation/utils.py
      Co-authored-by: default avatarArthur <48595927+ArthurZucker@users.noreply.github.com>
      
      * naming convention
      
      * Revert "naming convention"
      
      This reverts commit dd39d9c5b7a969e2d8a8d2a8e54f121b82dc44f0.
      
      * naming convention
      
      * last -hopefully- changes
      
      ---------
      Co-authored-by: default avatarArthur <48595927+ArthurZucker@users.noreply.github.com>
      39b2ff69