"tools/travis/install.sh" did not exist on "3f122ae1a1267771d987bc0810237f6456cef867"
  • LSinev's avatar
    Provide ability for custom sampler for ConfigurableTask (#1616) · ae72cebc
    LSinev authored
    * Added fewshot sampling seeds to evaluator.simple_evaluate signature
    
    Way to control seed of fewshot sampling
    may help with #1591
    
    * Added ability for custom sampler for ConfigurableTask
    
    May be set in config like
    ```
    fewshot_config:
      sampler: !function utils.MyFewshotSampler
    ```
    
    * explicitly set fewshot random generator seed for HFLM generate_until_task test
    
    * add backward compatibility for three args seed setup
    
    * save seeds info to logs/reports
    ae72cebc
__main__.py 14.7 KB