Commits · 0991dd33a0161c3e713abca4007553c7378833b7 · OpenDAS / opencompass

24 Jan, 2024 5 commits

[Sync] Updata dataset cfg for internMath (#837) · 0991dd33
Fengzhe Zhou authored Jan 24, 2024
```
Co-authored-by: liuhongwei <liuhongwei@pjlab.org.cn>
```
0991dd33
add fail notify (#836) · f7d7837a
zhulinJulia24 authored Jan 24, 2024

f7d7837a
update doc (#830) · f3675516
Fengzhe Zhou authored Jan 24, 2024

f3675516
[Feature] Update API implementation (#834) · 793e32c9
Songyang Zhang authored Jan 24, 2024

793e32c9

bittersweet1999 authored Jan 24, 2024



* add mtbench

* add mtbench

* Update configs/datasets/subjective/multiround/mtbench_judgeby_gpt4.py
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>

* Update configs/datasets/subjective/multiround/mtbench_judgeby_gpt4.py
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>

* Update opencompass/datasets/subjective/__init__.py
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>

* Update opencompass/datasets/subjective/mtbench.py
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>

* fix mtbench

---------
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>

2ee8e8a1

23 Jan, 2024 3 commits

[Feature] Add IFEval (#813) · e059a5c2
Jingming authored Jan 23, 2024
```
* [Feature] Add IFEval

* [Doc] add introduction of IFEval
```
e059a5c2

[Fix] fix strings (#833) · 3d9bb4ae

bittersweet1999 authored Jan 23, 2024



* add compass arena

* add compass_arena

* add compass arena

* Update opencompass/summarizers/subjective/compass_arena.py
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>

* Update opencompass/summarizers/subjective/__init__.py
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>

* Update opencompass/datasets/subjective/compass_arena.py
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>

* Update opencompass/datasets/subjective/__init__.py
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>

* Update configs/eval_subjective_compassarena.py
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>

* Update configs/datasets/subjective/compassarena/compassarena_compare.py
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>

* Update configs/eval_subjective_compassarena.py
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>

* Update configs/datasets/subjective/compassarena/compassarena_compare.py
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>

* fix check position bias

* fix string

---------
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>

3d9bb4ae

[Feature] Add CompassArena (#828) · 2d4da8dd

bittersweet1999 authored Jan 23, 2024



* add compass arena

* add compass_arena

* add compass arena

* Update opencompass/summarizers/subjective/compass_arena.py
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>

* Update opencompass/summarizers/subjective/__init__.py
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>

* Update opencompass/datasets/subjective/compass_arena.py
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>

* Update opencompass/datasets/subjective/__init__.py
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>

* Update configs/eval_subjective_compassarena.py
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>

* Update configs/datasets/subjective/compassarena/compassarena_compare.py
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>

* Update configs/eval_subjective_compassarena.py
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>

* Update configs/datasets/subjective/compassarena/compassarena_compare.py
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>

* fix check position bias

---------
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>

2d4da8dd

19 Jan, 2024 2 commits

Update hf_internlm2_chat template (#823) · 40a2441d
RangiLyu authored Jan 19, 2024
```
* Update hf_internlm2_chat template

* Update 20B
```
40a2441d

Update cdme config and evaluator (#812) · e975a96f

Guo Qipeng authored Jan 19, 2024



* update cdme config and evaluator

* fix cdme prompt

* move CDME trim post-processor as a separate evaluator

---------
Co-authored-by: 郭琦鹏 <guoqipeng@pjlab.org.cn>

e975a96f

18 Jan, 2024 4 commits
- Add LightllmApi KeyError log & Update doc (#816) · f09a2ff4
  Yang Yong authored Jan 18, 2024
```
* Add LightllmApi KeyError log

* Update LightllmApi doc
```
  f09a2ff4
- Test runner update - split step, change schedule time and disable hf cache (#814) · 8b5c467c
  zhulinJulia24 authored Jan 18, 2024
```
* Update pr-run-test.yml

* Update pr-run-test.yml

* Update pr-run-test.yml

* split step and change order, change schedule time and disable hf cache
```
  8b5c467c
- [Fix] Update yi 200k config (#815) · dcc32ed8
  Mo Li authored Jan 18, 2024
  
  dcc32ed8
- [Fix] Fix turbomind and update docs (#808) · 61fe873c
  RunningLeon authored Jan 18, 2024
```
* update

* update docs

* add engine_config and gen_config in eval_config

* update

* fix

* fix

* fix

* fix docstr

* fix url
```
  61fe873c
17 Jan, 2024 4 commits

[Doc] Update News (#810) · 9e5746d3
Fengzhe Zhou authored Jan 17, 2024

9e5746d3
[Sync] Add InternLM2 Keyset Evaluation Demo (#807) · b4afe3e7
Fengzhe Zhou authored Jan 17, 2024
```
Co-authored-by: zhangyifan1 <zhangyifan1@pjlab.org.cn>
```
b4afe3e7

Added support for multi-needle testing in needle-in-a-haystack test (#802) · acae5609

Mo Li authored Jan 17, 2024



* Add NeedleInAHaystack Test

* Apply pre-commit formatting

* Update configs/eval_hf_internlm_chat_20b_cdme.py
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>

* add needle in haystack test

* update needle in haystack test

* update plot function in tools_needleinahaystack.py

* optimizing needleinahaystack dataset generation strategy

* modify minor formatting issues

* add English version support

* change NeedleInAHaystackDataset to dynamic loading

* change NeedleInAHaystackDataset to dynamic loading

* fix needleinahaystack test eval bug

* fix needleinahaystack config bug

* Added support for multi-needle testing in needle-in-a-haystack test

* Optimize the code for plotting in the needle-in-a-haystack test.

* Correct the typo in the dataset parameters.

* update needleinahaystack test docs

---------
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>

acae5609

[Feature] Update evaluate turbomind (#804) · 0836aec6
RunningLeon authored Jan 17, 2024
```
* update

* fix

* fix

* fix
```
0836aec6

16 Jan, 2024 2 commits
- reorganize subject files (#801) · 814b3f73
  bittersweet1999 authored Jan 16, 2024
  
  814b3f73
- Add test runner, one case, daily and pr trigger (#751) · 2cd09164
  zhulinJulia24 authored Jan 16, 2024
```
* init test yaml

* add simple pr

* update

* update

* change name

* Update pr-run-test.yml

* Update pr-run-test.yml

---------
Co-authored-by: zhulin1 <zhulin1@pjlab.org.cn>
```
  2cd09164
12 Jan, 2024 1 commit

[Feature] Add configs for creationbench (#791) · 83d6c483

bittersweet1999 authored Jan 12, 2024

* add creationv2_zh

* add creationv2_zh

* add eng config for creationbench

* add eng config for creationbench

* add eng config for creationbench

83d6c483

11 Jan, 2024 4 commits
- [Fix] hot fix for requirements (#789) · d0dc3534
  Hubert authored Jan 11, 2024
  
  d0dc3534
- Update gsm8k agent prompt (#788) · 467ad0ac
  Songyang Zhang authored Jan 11, 2024
  
  467ad0ac
- [Feature] Add support for Nanbeige API (#786) · d3a0ddc3
  notoschord authored Jan 11, 2024
```
Co-authored-by: notoschord <wangzekai@kanzhun.com>
```
  d3a0ddc3
- add temperature in alles (#787) · 5679edb4
  bittersweet1999 authored Jan 11, 2024
  
  5679edb4
09 Jan, 2024 1 commit

[Feature] Update MedBench (#779) · ad872a5d

Xiaoming Shi authored Jan 09, 2024



* update medbench

* medbench update

* format medbench

* format

* Update

* update

* update

* update suffix

---------
Co-authored-by: 施晓明 <PJLAB\shixiaoming@pjnl104220118l.pjlab.org>
Co-authored-by: Leymore <zfz-960727@163.com>

ad872a5d

08 Jan, 2024 7 commits
- [Sync] Bump version to 0.2.1 (#778) · a74e4c1a
  Fengzhe Zhou authored Jan 08, 2024
  
  a74e4c1a
- [Sync] Sync with internal codes 2023.01.08 (#777) · 32f40a8f
  Fengzhe Zhou authored Jan 08, 2024
  
  32f40a8f
- [Feature] *_batch_generate* function, add the MultiTokenEOSCriteria (#772) · 8194199d
  jiangjin1999 authored Jan 08, 2024
```
* jiangjin1999: in the _batch_generate function, add the MultiTokenEOSCriteria feature to speed up inference.

* jiangjin1999: in the _batch_generate function, add the MultiTokenEOSCriteria feature to speed up inference.

---------
Co-authored-by: jiangjin08 <jiangjin08@MBP-2F32S5MD6P-0029.local>
Co-authored-by: jiangjin08 <jiangjin08@a.sh.vip.dianping.com>
```
  8194199d
- [Docs] Update contamination docs (#775) · f78fcf6e
  Fengzhe Zhou authored Jan 08, 2024
  
  f78fcf6e
- [Feature] Contamination analysis for MMLU, Hellaswag, and ARC_c (#699) · 0b286303
  liyucheng09 authored Jan 08, 2024
```
* Contamination analysis for ARC_c, mmlu, and Hellaswag

* update `eval_contamination.py`

* update `contamination.py` summarizer

* fix `eval_contamination.py`

* add mmlu groups for contamination analysis
```
  0b286303
- typo(installation.md): fix unzip commands (#774) · ba1b684f
  tpoisonooo authored Jan 08, 2024
```
* Update installation.md

* Update installation.md
```
  ba1b684f
- [Fix] fix typos in drop prompt (#773) · 11f3b91e
  Yuchen Yan authored Jan 08, 2024
```
Co-authored-by: yanyuchen04 <yanyuchen04@meituan.com>
```
  11f3b91e
05 Jan, 2024 4 commits

Support Mbpp_plus dataset (#770) · 30a90d8d

Connor-Shen authored Jan 05, 2024



* support mbpp+

* support mbpp+

* minor fix

* [Feat] minor fix

---------
Co-authored-by: yingfhu <yingfhu@gmail.com>

30a90d8d

quick fix for postprocess pred extraction (#771) · 3c606cb7
bittersweet1999 authored Jan 05, 2024

3c606cb7

[Update] Update introduction of CompassBench-2024-Q1 (#769) · 0c75f0f9

Songyang Zhang authored Jan 05, 2024



* [Doc] Update Example of CompassBench

* [Doc] Update Example of CompassBench

* [Doc] Update Example of CompassBench

* update

* Update docs/zh_cn/advanced_guides/compassbench_intro.md
Co-authored-by: Fengzhe Zhou <zfz-960727@163.com>

---------
Co-authored-by: Fengzhe Zhou <zfz-960727@163.com>

0c75f0f9

[Feature] add subject ir dataset (#755) · 2163f939
bittersweet1999 authored Jan 05, 2024
```
* add subject ir

* Add ir dataset

* Add ir dataset
```
2163f939

04 Jan, 2024 1 commit
- [Feature] Add multi_round dataset evaluation (#766) · be369c3e
  bittersweet1999 authored Jan 04, 2024
```
* multi_round dataset

* add multi_round evaluation
```
  be369c3e
03 Jan, 2024 1 commit
- [Fix] Fix small bug in alignbench (#764) · 7cd65d49
  bittersweet1999 authored Jan 03, 2024
```
* fix small bugs

* fix small bugs
```
  7cd65d49
02 Jan, 2024 1 commit

[Feature] Support LLaMA2-Accessory (#732) · 3eb225a5

Chris Liu authored Jan 02, 2024

* Support LLaMA2-Accessory

* remove strip

* clear imports

* reformat

* fix lint

* fix lint

* update readme

* update readme

* update readme

* update readme

3eb225a5