• LSinev's avatar
    Provide ability for custom sampler for ConfigurableTask (#1616) · ae72cebc
    LSinev authored
    * Added fewshot sampling seeds to evaluator.simple_evaluate signature
    
    Way to control seed of fewshot sampling
    may help with #1591
    
    * Added ability for custom sampler for ConfigurableTask
    
    May be set in config like
    ```
    fewshot_config:
      sampler: !function utils.MyFewshotSampler
    ```
    
    * explicitly set fewshot random generator seed for HFLM generate_until_task test
    
    * add backward compatibility for three args seed setup
    
    * save seeds info to logs/reports
    ae72cebc
task.py 57.7 KB