1. 22 Feb, 2024 1 commit
    • Ayush Thakur's avatar
      feat: Add Weights and Biases support (#1339) · 2683fbbb
      Ayush Thakur authored
      
      
      * add wandb as extra dependency
      
      * wandb metrics logging
      
      * refactor
      
      * log samples as tables
      
      * fix linter
      
      * refactor: put in a class
      
      * change dir
      
      * add panels
      
      * log eval as table
      
      * improve tables logging
      
      * improve reports logging
      
      * precommit run
      
      * ruff check
      
      * handle importing reports api gracefully
      
      * ruff
      
      * compare results
      
      * minor pre-commit fixes
      
      * build comparison report
      
      * ruff check
      
      * log results as artifacts
      
      * remove comparison script
      
      * update dependency
      
      * type annotate and docstring
      
      * add example
      
      * update readme
      
      * fix typo
      
      * teardown
      
      * handle outside wandb run
      
      * gracefully fail reports creation
      
      * precommit checks
      
      * add report url to summary
      
      * use wandb  printer for better url stdout
      
      * fix ruff
      
      * handle N/A and groups
      
      * fix eval table
      
      * remove unused var
      
      * update wandb version req + disable reports stdout
      
      * remove reports feature to TODO
      
      * add label to multi-choice question data
      
      * log model predictions
      
      * lints
      
      * loglikelihood_rolling
      
      * log eval result for groups
      
      * log tables by group for better handling
      
      * precommit
      
      * choices column for multi-choice
      
      * graciously fail wandb
      
      * remove reports feature
      
      * track system metrics + total eval time + stdout
      
      ---------
      Co-authored-by: default avatarLintang Sutawika <lintang@eleuther.ai>
      2683fbbb
  2. 21 Dec, 2023 1 commit
  3. 20 Dec, 2023 1 commit
    • Alex Bäuerle's avatar
      feat: add option to upload results to Zeno (#990) · 21d4ae98
      Alex Bäuerle authored
      
      
      * feat: add option to upload results to Zeno
      
      * config-based upload supporting different task types and metrics
      
      * upload tasks as individual projects
      
      * wording
      
      * readme
      
      * add example notebook
      
      * Update documentation for Zeno integration
      
      * Make zeno deps an extra
      
      * Update README.md
      
      * Document extra deps installation
      
      * Update zeno_visualize.py
      
      * fix: balance parens
      
      * fix typo
      
      * fix merge commit I botched
      
      * Update zeno_visualize.py
      
      * Update logger warning stmt
      
      * fix whitespace
      
      ---------
      Co-authored-by: default avatarHailey Schoelkopf <65563625+haileyschoelkopf@users.noreply.github.com>
      21d4ae98
  4. 30 Nov, 2023 1 commit
  5. 29 Nov, 2023 1 commit
  6. 27 Nov, 2023 2 commits
  7. 22 Nov, 2023 1 commit
  8. 15 Jun, 2023 1 commit
  9. 12 Jun, 2023 1 commit
  10. 19 May, 2023 1 commit
  11. 27 Apr, 2023 1 commit
  12. 24 Apr, 2023 1 commit
  13. 19 Apr, 2023 1 commit