Commits · 42caa660ef5897a25e2cdeba97dea3fd8ea75926 · gaoqiong / lm-evaluation-harness

16 Jun, 2023 2 commits
- Turn bigbench_resources into module such that it's included in the build · 42caa660
  svenhendrikx authored Jun 16, 2023
  
  42caa660
- Allow HFLM model to be initialized with transformers.PreTrainedModel instance · a28c0193
  svenhendrikx authored Jun 16, 2023
  
  a28c0193
08 Jun, 2023 4 commits
- Merge pull request #568 from EleutherAI/revert-537-mmlu · b21c8f3d
  Stella Biderman authored Jun 08, 2023
```
Revert "Add MultipleChoiceExactTask"
```
  b21c8f3d
- Revert "Add MultipleChoiceExactTask" · 21970120
  Stella Biderman authored Jun 08, 2023
  
  21970120
- Merge pull request #537 from gakada/mmlu · 38c74278
  Stella Biderman authored Jun 08, 2023
```
Add MultipleChoiceExactTask
```
  38c74278
- Add Anthropic support (#562) · cc4eab6a
  Jason Phang authored Jun 08, 2023
```
* add anthropic support

* move requirement
```
  cc4eab6a
07 Jun, 2023 1 commit
- Update task_guide.md (#564) · 9c89d1fc
  Hailey Schoelkopf authored Jun 07, 2023
  
  9c89d1fc
06 Jun, 2023 4 commits
- Add MultipleChoiceExactTask · c2fcf688
  gk authored May 31, 2023
  
  c2fcf688
- Add load_in_4bit and fix peft loading (#556) · 4b701e22
  gakada authored Jun 06, 2023
  
  4b701e22
- Merge pull request #557 from kwikiel/master · 01e0626f
  Lintang Sutawika authored Jun 06, 2023
  
  01e0626f
- Fix typo · 6690afaf
  Kacper Wikieł authored Jun 06, 2023
  
  6690afaf
05 Jun, 2023 3 commits
- Use max_length in AutoSeq2SeqLM (#551) · 2976f69b
  gakada authored Jun 05, 2023
  
  2976f69b
- Move spaces from context to continuation (#546) · 92929bd2
  gakada authored Jun 05, 2023
  
  92929bd2
- Update CODEOWNERS · 76fed776
  Stella Biderman authored Jun 05, 2023
  
  76fed776
01 Jun, 2023 1 commit

Fix LLaMA tokenization issue (#531) · 23f30926

gakada authored Jun 02, 2023



* Fix tokenization issue in BaseLM.loglikelihood

* Add a regression script

* Use entire non-continuation length as context

---------
Co-authored-by: Hailey Schoelkopf <65563625+haileyschoelkopf@users.noreply.github.com>

23f30926

30 May, 2023 1 commit
- changed quantized type signature and default val (#532) · 441e6ac1
  Sam Passaglia authored May 30, 2023
  
  441e6ac1
27 May, 2023 1 commit
- Merge pull request #519 from gakada/gptq · 095d8406
  Stella Biderman authored May 26, 2023
```
Add support for loading GPTQ models via AutoGPTQ
```
  095d8406
26 May, 2023 1 commit
- GPTQ: add auto-gptq extra, add gptq_use_triton parameter · c11ad4f2
  gk authored May 26, 2023
  
  c11ad4f2
25 May, 2023 2 commits
- Extend `dtype` command line flag to `HFLM` (#523) · 8cff2bea
  Hailey Schoelkopf authored May 25, 2023
```
* allow for hf-causal to take dtype arg

* document this change
```
  8cff2bea
- GPTQ: fix README and support default names · b465cd01
  gk authored May 25, 2023
  
  b465cd01
23 May, 2023 4 commits
- Merge pull request #522 from EleutherAI/fix-mgpt-fewshot · 4e94af6f
  Lintang Sutawika authored May 24, 2023
  
  4e94af6f
- fixed fewshot prompt by filling [mask] · 25699d3e
  lintangsutawika authored May 23, 2023
  
  25699d3e
- re implemented fewshot_context method in the class to allow custom prompt for fewshot · 5a8ac198
  lintangsutawika authored May 23, 2023
  
  5a8ac198
- Add support for loading GPTQ models via AutoGPTQ · b296c4f6
  gk authored May 23, 2023
  
  b296c4f6
21 May, 2023 4 commits
- Merge pull request #481 from janEbert/json-task · 84ef60ee
  Stella Biderman authored May 21, 2023
```
Add perplexity task on arbitrary JSON data
```
  84ef60ee
- Merge branch 'master' into json-task · 4de8a74e
  Stella Biderman authored May 21, 2023
  
  4de8a74e
- Merge pull request #492 from juletx/eval-info · bda68845
  Stella Biderman authored May 21, 2023
```
Add option to dump prompts and completions to a JSON file
```
  bda68845
- Merge pull request #480 from kenhktsui/float-limit · 96a83d45
  Stella Biderman authored May 21, 2023
```
Evaluation Against Portion of Benchmark Data
```
  96a83d45
19 May, 2023 2 commits
- Merge pull request #477 from juletx/results · e53eb332
  Stella Biderman authored May 19, 2023
```
Add results of various models in json and md format
```
  e53eb332
- Merge pull request #483 from janEbert/out-dir · d1327193
  Stella Biderman authored May 19, 2023
```
Create output path directory if necessary
```
  d1327193
18 May, 2023 1 commit
- Update README.md · e7a212ff
  Stella Biderman authored May 18, 2023
  
  e7a212ff
12 May, 2023 1 commit
- Merge remote-tracking branch 'upstream/master' into results · 92a50856
  Julen Etxaniz authored May 12, 2023
  
  92a50856
11 May, 2023 3 commits

Add multilingual datasets (XCOPA, XStoryCloze, XWinograd, PAWS-X, XNLI, MGSM) (#426) · d1451679

Julen Etxaniz authored May 11, 2023

* add xcopa dataset

* add xstory_cloze dataset and run pre-commit

* fix xcopa validation and test sets

* add xwinograd dataset

* add pawsx task

* add xnli task

* update task table with recently added tasks

* remove unused metrics from paws-x

* add mgsm task and fix gsm8k

* fix gsm8k until

* update task table

d1451679

update write out variable name · af913422
Julen Etxaniz authored May 11, 2023

af913422
update parameter names and add docs · 99b0a42d
Julen Etxaniz authored May 11, 2023

99b0a42d

10 May, 2023 4 commits
- Merge pull request #490 from jquesnelle/auto-batch-size-fix · 05550ef3
  Stella Biderman authored May 10, 2023
```
fix adaptive batch crash when there are no new requests
```
  05550ef3
- Update README.md · f71bffb0
  Stella Biderman authored May 10, 2023
  
  f71bffb0
- add --write_detailed_eval_info to dump JSON with prompts and completions · 2e046ce3
  Julen Etxaniz authored May 10, 2023
  
  2e046ce3
- fix adaptive batch crash when there are no new requests (e.g. when pulling from cache) · d424f26b
  Jeffrey Quesnelle authored May 10, 2023
  
  d424f26b
09 May, 2023 1 commit
- Update README.md · 8fc04fe5
  Stella Biderman authored May 09, 2023
  
  8fc04fe5