"vscode:/vscode.git/clone" did not exist on "31dfde74290bbfbbb05a5919eb9424962aeabd99"
  • Nicolas Patry's avatar
    [WIP] Hard error when ignoring tensors. (#27484) · 2da28c4b
    Nicolas Patry authored
    
    
    * [WIP] Hard error when ignoring tensors.
    
    * Better selection/error when saving a checkpoint.
    
    - Find all names we should normally drop (those are in the transformers
      config)
    - Find all disjoint tensors (for those we can safely trigger a copy to
      get rid of the sharing before saving)
    - Clone those disjoint tensors getting rid of the issue
    - Find all identical names (those should be declared in the config
      but we try to find them all anyway.)
    - For all identical names:
      - If they are in the config, just ignore them everything is fine
      - If they are not, warn about them.
    - For all remainder tensors which are shared yet neither identical NOR
      disjoint. raise a hard error.
    
    * Adding a failing test on `main` that passes here.
    
    * We don't need to keep the subfolder logic in this test.
    
    * Apply suggestions from code review
    Co-authored-by: default avatarArthur <48595927+ArthurZucker@users.noreply.github.com>
    
    ---------
    Co-authored-by: default avatarArthur <48595927+ArthurZucker@users.noreply.github.com>
    2da28c4b
test_modeling_utils.py 91.9 KB