Commits · 25dfd3f6697edd7fbefc10dcb858337ed742910d · gaoqiong / lm-evaluation-harness

03 Jul, 2023 1 commit
- Update babi.py · caadcbc9
  Hailey Schoelkopf authored Jul 03, 2023
  
  caadcbc9
02 Jul, 2023 1 commit
- Fix trust_remote_code, bnb_4bit_*, max_batch_size for gpt2 · a946c6cd
  gk authored Jul 02, 2023
  
  a946c6cd
30 Jun, 2023 1 commit
- add trust_remote_code to tokenizer.from_pretrained · e9f1af36
  windows_on_wsl authored Jun 30, 2023
  
  e9f1af36
29 Jun, 2023 1 commit
- add err handling for multi-tok stopseq · c9c141d2
  haileyschoelkopf authored Jun 29, 2023
  
  c9c141d2
27 Jun, 2023 7 commits
- update docstring · 1e98c74e
  haileyschoelkopf authored Jun 27, 2023
  
  1e98c74e
- switch to using get_model() · 4221d30f
  haileyschoelkopf authored Jun 27, 2023
  
  4221d30f
- pass batch size · d362dfee
  haileyschoelkopf authored Jun 27, 2023
  
  d362dfee
- remove tokenizer assert · 5f1d18d4
  haileyschoelkopf authored Jun 27, 2023
  
  5f1d18d4
- fix issue with dumping model_name to results · 2a92cde0
  haileyschoelkopf authored Jun 27, 2023
  
  2a92cde0
- change self.gpt2 -> self.model · e0498dd7
  haileyschoelkopf authored Jun 27, 2023
  
  e0498dd7
- Revert "[triviaqa] The ground truth must be a *substring* of the generated an…" · ac0cb7e4
  Hailey Schoelkopf authored Jun 27, 2023
  
  ac0cb7e4
25 Jun, 2023 1 commit
- Change collation order for greedy_until · a6952a0b
  Hailey Schoelkopf authored Jun 25, 2023
  
  a6952a0b
21 Jun, 2023 1 commit
- [triviaqa] The ground truth must be a *substring* of the generated answer · 0570991a
  Guillaume "Vermeille" Sanchez authored Jun 21, 2023
  
  0570991a
16 Jun, 2023 3 commits
- Add logic to simple_evaluate to instantiate HFLM from transformers.PreTrainedModel instance · ddc634f2
  svenhendrikx authored Jun 16, 2023
  
  ddc634f2
- Turn bigbench_resources into module such that it's included in the build · 42caa660
  svenhendrikx authored Jun 16, 2023
  
  42caa660
- Allow HFLM model to be initialized with transformers.PreTrainedModel instance · a28c0193
  svenhendrikx authored Jun 16, 2023
  
  a28c0193
14 Jun, 2023 3 commits
- Update triviaqa.py · 50cf0984
  Hailey Schoelkopf authored Jun 14, 2023
  
  50cf0984
- rm triviaqa to use HF filtered dev set · 859e5920
  Hailey Schoelkopf authored Jun 14, 2023
  
  859e5920
- Change triviaqa dataset path · a1160392
  Hailey Schoelkopf authored Jun 14, 2023
  
  a1160392
13 Jun, 2023 4 commits
- add error handling for moving model to device · 2d27f9e1
  haileyschoelkopf authored Jun 13, 2023
  
  2d27f9e1
- Fix non-callable attributes in CachingLM · d4889e1a
  gk authored Jun 14, 2023
  
  d4889e1a
- fix seqlen issues for bloom, OPT tokenizer check · fa43ab2e
  haileyschoelkopf authored Jun 13, 2023
  
  fa43ab2e
- Add - 4bit-related args · 990bc542
  ynot authored Jun 13, 2023
  
  990bc542
12 Jun, 2023 3 commits
- Update evaluator.py - extra .model added · 811252b8
  Poedator authored Jun 13, 2023
```
fixed the reference to the inner model
```
  811252b8
- fix for non-str output from evaluator · 69ed600d
  Poedator authored Jun 12, 2023
  
  69ed600d
- Update evaluator.py cache_db argument str · b9a6a3df
  Poedator authored Jun 12, 2023
```
in case `isinstance(model, lm_eval.base.LM)` line 80 gives error, because `model` is not `str`. A simple inline if statement fixes this.
```
  b9a6a3df
11 Jun, 2023 1 commit
- Add --max_batch_size and --batch_size auto:N · 8cec82b2
  gk authored Jun 11, 2023
  
  8cec82b2
08 Jun, 2023 2 commits
- Revert "Add MultipleChoiceExactTask" · 21970120
  Stella Biderman authored Jun 08, 2023
  
  21970120
- Add Anthropic support (#562) · cc4eab6a
  Jason Phang authored Jun 08, 2023
```
* add anthropic support

* move requirement
```
  cc4eab6a
07 Jun, 2023 1 commit
- fix unreachable return · c117e787
  Oleh Shliazhko authored Jun 08, 2023
  
  c117e787
06 Jun, 2023 3 commits
- Add MultipleChoiceExactTask · c2fcf688
  gk authored May 31, 2023
  
  c2fcf688
- address review comments · f034974a
  Oleh Shliazhko authored Jun 06, 2023
  
  f034974a
- Add load_in_4bit and fix peft loading (#556) · 4b701e22
  gakada authored Jun 06, 2023
  
  4b701e22
05 Jun, 2023 2 commits
- Use max_length in AutoSeq2SeqLM (#551) · 2976f69b
  gakada authored Jun 05, 2023
  
  2976f69b
- Move spaces from context to continuation (#546) · 92929bd2
  gakada authored Jun 05, 2023
  
  92929bd2
01 Jun, 2023 1 commit

Fix LLaMA tokenization issue (#531) · 23f30926

gakada authored Jun 02, 2023



* Fix tokenization issue in BaseLM.loglikelihood

* Add a regression script

* Use entire non-continuation length as context

---------
Co-authored-by: Hailey Schoelkopf <65563625+haileyschoelkopf@users.noreply.github.com>

23f30926

30 May, 2023 1 commit
- changed quantized type signature and default val (#532) · 441e6ac1
  Sam Passaglia authored May 30, 2023
  
  441e6ac1
28 May, 2023 1 commit
- bump eval version · 3a424af4
  Oleh Shliazhko authored May 28, 2023
  
  3a424af4
26 May, 2023 2 commits
- Fix triviaqa task · 8c419c83
  seopbo authored May 26, 2023
  
  8c419c83
- GPTQ: add auto-gptq extra, add gptq_use_triton parameter · c11ad4f2
  gk authored May 26, 2023
  
  c11ad4f2