Commits · d6ceced5ee81a0afae7ffcd8158abe273f044814 · gaoqiong / lm-evaluation-harness

28 Apr, 2023 1 commit

Add non-programmatic BIG-bench-hard tasks (#406) · 602abceb

yurodiviy authored Apr 28, 2023



* Support bigbench-hard json tasks using multiple_choice_grade

* Add support for greedy decoding in bigbench tasks

* move bigbench_resources to datasets

* rectify changes to rf.greedy_until w upstream

* make path to resource import reflect new location

---------
Co-authored-by: haileyschoelkopf <hailey.schoelkopf@yale.edu>

602abceb

24 Jun, 2022 1 commit
- fix key access in squad evaluation metrics · 1a153185
  Konstantin Schulz authored Jun 24, 2022
  
  1a153185
03 May, 2022 2 commits
- Revert `tests/testdata` changes and address flake8 issues · 8c997e53
  jon-tow authored May 03, 2022
  
  8c997e53
- add pre-commit · 121b7096
  Fabrizio Milo authored May 02, 2022
  
  121b7096
18 Mar, 2022 1 commit
- Refactor `Task` download · 7c9da714
  Jonathan Tow authored Mar 11, 2022
  
  7c9da714
28 Feb, 2022 2 commits
- Make citations module-level constants · 3f13d15f
  Jonathan Tow authored Feb 28, 2022
  
  3f13d15f
- Add citations and descriptions to all tasks · a1aceacd
  Jonathan Tow authored Feb 28, 2022
  
  a1aceacd
05 Feb, 2022 1 commit
- Added decontamination to remaining evals · dae7b868
  Quentin Gregory Anthony authored Feb 05, 2022
  
  dae7b868
30 Oct, 2021 1 commit
- Replace the `fewshot_description` API with a `description_dict` based interface · 8ac99269
  Jonathan Tow authored Oct 30, 2021
  
  8ac99269
09 Aug, 2021 1 commit
- fix squad breakage due to HF change · 198ca732
  Leo Gao authored Aug 09, 2021
  
  198ca732
05 Jun, 2021 1 commit
- Add task versioning · 105fa974
  Leo Gao authored Jun 04, 2021
  
  105fa974
28 Mar, 2021 2 commits
- Rename task · 8de85534
  Leo Gao authored Mar 28, 2021
  
  8de85534
- squad: fix aggregation · cbc5c9c8
  Leo Gao authored Mar 28, 2021
  
  cbc5c9c8
27 Mar, 2021 1 commit
- Remove unused imports and format imports · d5d19219
  Jonathan Tow authored Mar 26, 2021
  
  d5d19219
24 Mar, 2021 2 commits
- Fixed calling of loglikelihood within SQuAD task · 14dd29c4
  Charles Foster authored Mar 23, 2021
  
  14dd29c4
- SQuAD fixed to use loglikelihood API to calculate the probability of an unanswerable question. · 5be42b4d
  Charles Foster authored Mar 23, 2021
  
  5be42b4d
13 Feb, 2021 1 commit
- Fixes SQuAD v2 metric computation. · 232c9ab6
  Charles Foster authored Feb 13, 2021
  
  232c9ab6
11 Feb, 2021 2 commits
- Fixes to make greedy_until work · 432bd44c
  Leo Gao authored Feb 10, 2021
```
# Conflicts:
#	lm_eval/models/gpt2.py
#	lm_eval/tasks/squad.py
```
  432bd44c
- Fixes to make greedy_until work · 7b649ded
  Leo Gao authored Feb 10, 2021
  
  7b649ded
10 Feb, 2021 3 commits
- Removed unnecessary import. · eb4c8407
  Charles Foster authored Feb 09, 2021
  
  eb4c8407
- Passes tests, except for NotImplementedError for request type greedy_until. · bba6e0e9
  Charles Foster authored Feb 09, 2021
  
  bba6e0e9
- Skeleton of SQuADv2. Not yet tested. · f48b119d
  Charles Foster authored Feb 09, 2021
  
  f48b119d
01 Feb, 2021 1 commit
- Fix linting problems · fe4a1efd
  Leo Gao authored Feb 01, 2021
  
  fe4a1efd
22 Jan, 2021 2 commits
- Remove rewrite reminder comment from everything except SuperGLUE · f4120e59
  Leo Gao authored Jan 21, 2021
  
  f4120e59
- Clean up code, remove some footguns · e31b4b31
  Leo Gao authored Jan 21, 2021
  
  e31b4b31
13 Jan, 2021 1 commit
- Move `doc_to_text` target code into `doc_to_target` · d77241eb
  Jonathan Tow authored Jan 13, 2021
  
  d77241eb
06 Jan, 2021 1 commit
- Add comments to remind that evaluation needs to be written for the new framework · 2d2dbf96
  Leo Gao authored Jan 05, 2021
  
  2d2dbf96
05 Jan, 2021 1 commit
- Add reminder to rewrite · 08dc67ea
  Leo Gao authored Jan 05, 2021
  
  08dc67ea
22 Oct, 2020 2 commits
- Clarified that SQuaD does not include a test set. · 76b08133
  Charles Foster authored Oct 21, 2020
  
  76b08133
- Added SQuAD v2 dataset · 5c58b267
  Charles Foster authored Oct 21, 2020
  
  5c58b267