"vscode:/vscode.git/clone" did not exist on "4a035c05987a76344dc09279cb122d26cc213e54"
  • Ahmed Moubtahij's avatar
    Token healing (#30081) · 39b2ff69
    Ahmed Moubtahij authored
    
    
    * token healing impl + trie with extensions
    
    * make fixup
    
    * prefix-robust space tokenization
    
    * examples readme and requirements
    
    * make fixup
    
    * allow input prompt and model
    
    * redundant defaults
    
    * Specialized Trie
    
    * make fixup
    
    * updated tests with new inherited Tree
    
    * input ids to auto device_map
    
    * rm unused import
    
    * Update src/transformers/generation/utils.py
    Co-authored-by: default avatarArthur <48595927+ArthurZucker@users.noreply.github.com>
    
    * naming convention
    
    * Revert "naming convention"
    
    This reverts commit dd39d9c5b7a969e2d8a8d2a8e54f121b82dc44f0.
    
    * naming convention
    
    * last -hopefully- changes
    
    ---------
    Co-authored-by: default avatarArthur <48595927+ArthurZucker@users.noreply.github.com>
    39b2ff69
tokenization_utils.py 45.8 KB