• Paul McCann's avatar
    Don't silence errors when loading tasks (#1148) · 34b563b1
    Paul McCann authored
    
    
    * Add example failing task
    
    This task includes an invalid import. This will cause an exception and
    the task will not be loaded. But this just results in a DEBUG level log
    message, so in normal usage you'll see no error, and will be told the
    task doesn't exist.
    
    Here's an example command line to run the task:
    
        python -m lm_eval --model hf --model_args pretrained=rinna/japanese-gpt-1b --tasks fail
    
    This task is based on a Japanese Winograd task, but that's not
    important, and was just used due to familiarity.
    
    * Do not ignore errors when loading tasks
    
    * Change how task errors are logged
    
    This makes the proposed changes from PR discussion.
    
    1. Exceptions not related to missing modules/imports are logged as
       warnings.
    
    2. module/import related exceptions are still logged at debug level, but
       if any of them happen there is a warning about it with instructions
       on how to show logs.
    
    * Remove intentionally failing task
    
    ---------
    Co-authored-by: default avatarPaul O'Leary McCann <polm@dampfkraft.com>
    34b563b1
__init__.py 9.43 KB