Commits · b39f501563ab14bf81fc3c3bd76bd4de92680344 · OpenDAS / opencompass

09 Apr, 2024 1 commit
- [Sync] update taco (#1030) · b39f5015
  Fengzhe Zhou authored Apr 09, 2024
  
  b39f5015
07 Apr, 2024 3 commits

[Fix] Simplify needlebench summarizer (#1024) · 16f29b25
Mo Li authored Apr 07, 2024
```
* Conflicts:
	configs/summarizers/needlebench.py

* fix lint problems
```
16f29b25

[Feature] Add ATC Choice Version (#1019) · f2af4933

Mo Li authored Apr 07, 2024

* Squashed commit of the following:

commit c48ad194c3976dc63d1b60d8c8ab2d5ff9e1cbfe
Author: DseidLi <2568818204@qq.com>
Date:   Tue Apr 2 16:57:43 2024 +0800

    add atc_choice

commit 3ac6efea29619573e6fac8fa3cce464853dcead0
Merge: 2d4e5597 8e3a9c3
Author: DseidLi <2568818204@qq.com>
Date:   Tue Apr 2 16:41:38 2024 +0800

    Merge branch 'atc_choice' into atc_add_choice

commit 8e3a9c396a3e5546d3faf584183f6fd60b974d5e
Merge: 150a036 0a6a03fe
Author: DseidLi <2568818204@qq.com>
Date:   Tue Mar 26 04:47:07 2024 +0800

    Merge branch 'main' into atc_choice

    Conflicts:
    	configs/summarizers/needlebench.py
    	opencompass/datasets/needlebench/multi.py
    	opencompass/datasets/needlebench/origin.py
    	opencompass/datasets/needlebench/parallel.py

commit 150a036d6d990f26a57c974d1af83d88c31a0f9d
Merge: 8d6ac9a 940dd18
Author: DseidLi <2568818204@qq.com>
Date:   Wed Mar 20 03:49:08 2024 +0800

    Merge branch 'needlebench_fix' into atc_choice

commit 8d6ac9a1a43b1c9d0f0ea27e7d58968a203ea898
Author: DseidLi <2568818204@qq.com>
Date:   Wed Mar 20 03:41:49 2024 +0800

    optimize needlebench code

commit 940dd18a4270f24bc69edd2a780182c68918e1a9
Author: DseidLi <2568818204@qq.com>
Date:   Wed Mar 20 03:39:46 2024 +0800

    fix vllm

commit d8be6877bc41051f3edcc0421c462c834c0f1c9a
Merge: ecad78a 2527fda
Author: DseidLi <2568818204@qq.com>
Date:   Tue Mar 19 21:07:08 2024 +0800

    Merge remote-tracking branch 'origin/add_1M_dataset' into atc_choice

commit 2527fda8a546595bcaea1e5261367bc1097faec8
Author: DseidLi <2568818204@qq.com>
Date:   Tue Mar 19 16:03:40 2024 +0800

    add model configs

commit 75425acdf80d6d25ee24bb0aa60ac48539262e76
Author: DseidLi <2568818204@qq.com>
Date:   Tue Mar 19 16:02:15 2024 +0800

    add prompt postion args

commit 367ba1ba612a8cec5df1f80d5e5ae4e285baf38b
Author: DseidLi <2568818204@qq.com>
Date:   Wed Feb 28 21:40:00 2024 +0800

    add Needlebench-1000K configs

commit ecad78af14c4bb00fe325779114b384c57ab30bf
Author: DseidLi <2568818204@qq.com>
Date:   Thu Mar 14 22:08:32 2024 +0800

    fix atc

commit 08772c0787b18872abadc9ffec3223941a5ee0c2
Merge: 9f3f8cf caf1cf8a
Author: DseidLi <2568818204@qq.com>
Date:   Thu Mar 14 22:07:28 2024 +0800

    Merge branch 'main' into atc_choice

    Conflicts:
    	configs/datasets/needlebench/readme.md
    	configs/datasets/needlebench/readme_zh-CN.md
    	configs/summarizers/needlebench.py
    	opencompass/datasets/needlebench/atc.py
    	opencompass/summarizers/needlebench.py

commit 9f3f8cfb4452722734d334114ac1d14110e57406
Author: DseidLi <2568818204@qq.com>
Date:   Thu Mar 14 21:35:53 2024 +0800

    add atc-choice test

commit 52be7c1202376b4e09821188b826f1a805328129
Author: DseidLi <2568818204@qq.com>
Date:   Wed Mar 6 02:54:15 2024 +0800

    update needlebench randomseed and add vllm qwen14b

commit fc1effce596ae2e5ece4933e8cd34aef8e64a6f9
Merge: 4e747ed caf1cf8a
Author: DseidLi <2568818204@qq.com>
Date:   Wed Mar 6 02:51:14 2024 +0800

    Merge branch 'main' into add_model_configs

commit 31834f9b23af3354ac3581ec86d693d0f05cdd1c
Merge: 7dabc82 120bf8b3
Author: DseidLi <2568818204@qq.com>
Date:   Sun Mar 3 23:29:42 2024 +0800

    Merge branch 'main' of https://github.com/open-compass/opencompass into atc_choice

commit 4e747ed1988ddbcfcc7fff334601259ade72d363
Author: DseidLi <2568818204@qq.com>
Date:   Sun Mar 3 22:15:25 2024 +0800

    add internlm2-lmdeploy model and gemma configs

commit 7dabc828123d711c8cf834d6aab4137bb55e85ed
Author: DseidLi <2568818204@qq.com>
Date:   Sat Mar 2 17:26:15 2024 +0800

    add atc choice version -ZH

commit 996f8ae43d3f946a052f736717ead139d153e2dd
Author: DseidLi <2568818204@qq.com>
Date:   Wed Feb 28 16:58:56 2024 +0800

    update readme for needlebench

commit f7266e873cb34ccf18a7f20b2c5821af8416a14f
Author: DseidLi <2568818204@qq.com>
Date:   Wed Feb 28 16:44:53 2024 +0800

    move readme.md

commit 1c7375681dea13996802e45b878dc4929ea8fa65
Author: DseidLi <2568818204@qq.com>
Date:   Wed Feb 28 16:38:31 2024 +0800

    fix linting error

commit b6524f3ebfb8a3a12a5ad3e3fa7a8a0921fcb6c1
Author: DseidLi <2568818204@qq.com>
Date:   Wed Feb 28 16:33:51 2024 +0800

    lint summarizer

commit c0d1190e39d3b6724f677346df2572df9af59f25
Author: DseidLi <2568818204@qq.com>
Date:   Wed Feb 28 16:29:03 2024 +0800

    add needlebench intro, fix summarizer

commit 0965baf78588e29d813b61d73f0ebd868a0ce3d0
Author: DseidLi <2568818204@qq.com>
Date:   Mon Feb 26 13:31:26 2024 +0800

    fix bug in needlebench summarizer

commit 5d32b31eb85382026935f356190ad92b103afd98
Author: DseidLi <2568818204@qq.com>
Date:   Sat Feb 24 03:19:08 2024 +0800

    update act prompt

commit af82a7f085e394d83aa84043e2881dd50115942c
Merge: 32bf9fe 53fe788d
Author: DseidLi <2568818204@qq.com>
Date:   Fri Feb 23 17:50:32 2024 +0800

    Merge remote-tracking branch 'upstream/main' into needlebench

commit 32bf9fe802eaf8e8e5b33ff17b2a897058f8b66b
Author: DseidLi <2568818204@qq.com>
Date:   Fri Feb 23 17:31:32 2024 +0800

    simplify needlebench 32k, 128k, 200k for eval

commit a7cb025e05a48449de9839005fada02bd5bff15a
Author: DseidLi <2568818204@qq.com>
Date:   Fri Feb 23 14:48:58 2024 +0800

    add needlebench

* fix summarizer

* remove repeated code

* remove chinese comments

f2af4933

[Fix] Refactor Needlebench Configs for CLI Testing Support (#1020) · b50d1632

Mo Li authored Apr 07, 2024

* add needlebench datasets suffix

* fix import

* update run.py args for summarizer key and dataset suffix

* update utils/run.py

b50d1632

02 Apr, 2024 1 commit

[Feature] Add multi-model judge and fix some problems (#1016) · 2d4e5597

bittersweet1999 authored Apr 02, 2024

* support multi-model judge and moe judge

* test_moe

* test_moe

* test

* add moe judge

* support multi-judge-model

2d4e5597

28 Mar, 2024 1 commit
- [Feature] Support AlpacaEval_V2 (#1006) · 02e7eec9
  bittersweet1999 authored Mar 28, 2024
```
* support alpacaeval_v2

* support alpacaeval

* update docs

* update docs
```
  02e7eec9
25 Mar, 2024 1 commit

[Feature] update needlebench and configs (#986) · 0a6a03fe

Mo Li authored Mar 25, 2024

* add Needlebench-1000K configs

* add prompt postion args

* add model configs

* Update parallel.py

* fix lint

0a6a03fe

19 Mar, 2024 4 commits
- [Fix] Update APPS/TACO (#988) · 0221d308
  Connor-Shen authored Mar 19, 2024
```
* [Feature] update apps/taco

* [Feature] update apps/taco
```
  0221d308
- [Feature] Update APPS (#985) · 8a3c6e51
  Connor-Shen authored Mar 19, 2024
```
* update post process

* update post process
```
  8a3c6e51
- [Feat] Support TACO (#966) · d92595b6
  Connor-Shen authored Mar 19, 2024
```
* [Feat] Support TACO

* update README

* update README
```
  d92595b6
- add support for set prediction path (#984) · c78a4df9
  bittersweet1999 authored Mar 19, 2024
  
  c78a4df9
15 Mar, 2024 1 commit
- [Feature] Add the implement of QuALITY datasets (#976) · 89a8a891
  Jingming authored Mar 15, 2024
```
#976 
```
  89a8a891
13 Mar, 2024 1 commit

[Bench] Support APPS (#963) · 3098d788

Connor-Shen authored Mar 13, 2024

* [Feat] support apps

* [Feat] support apps

* [Feat] support apps

* update README

3098d788

11 Mar, 2024 2 commits
- [Sync] update 20240308 (#953) · bdd85358
  Fengzhe Zhou authored Mar 11, 2024
  
  bdd85358
- [fix] add different temp for different question in mtbench (#954) · 848e7c8a
  bittersweet1999 authored Mar 11, 2024
```
* add temp for mtbench

* add document for mtbench

* add document for mtbench
```
  848e7c8a
05 Mar, 2024 2 commits
- [Fix] FinanceIQ_datasets import error (#939) · 2e993989
  Xu Song authored Mar 05, 2024
```
* [Fix] Fix KeyError: 'FinanceIQ_datasets'

* [Fix] Fix KeyError: 'FinanceIQ_datasets'
```
  2e993989
- [Fix] fix a bug of humanevalplus config (#944) · d0550268
  Jingming authored Mar 05, 2024
  
  d0550268
04 Mar, 2024 3 commits

[Sync] Sync Internal (#941) · b03d5dc5
Fengzhe Zhou authored Mar 04, 2024

b03d5dc5

[Feature] add lveval benchmark (#914) · bbec7d87

yuantao2108 authored Mar 04, 2024



* add lveval benchmark

* add LVEval readme file

* update LVEval readme file

* Update configs/eval_bluelm_32k_lveval.py

* Update configs/eval_llama2_7b_lveval.py

---------
Co-authored-by: yuantao <yuantao@infini-ai.com>
Co-authored-by: Mo Li <82895469+DseidLi@users.noreply.github.com>

bbec7d87

[Feature] Upgrade the needle-in-a-haystack experiment to Needlebench (#913) · 8142f399

Mo Li authored Mar 04, 2024

* add needlebench

* simplify needlebench 32k, 128k, 200k for eval

* update act prompt

* fix bug in needlebench summarizer

* add needlebench intro, fix summarizer

* lint summarizer

* fix linting error

* move readme.md

* update readme for needlebench

* update docs of needlebench

* simplify needlebench summarizers

8142f399

29 Feb, 2024 1 commit
- [Feature] Support OpenFinData (#896) · 4c45a71b
  Skyfall-xzz authored Feb 29, 2024
```
* [Feature] Support OpenFinData

* add README for OpenFinData

* update README
```
  4c45a71b
28 Feb, 2024 1 commit
- [Feature] add support for gemini (#931) · 001e77fe
  bittersweet1999 authored Feb 28, 2024
```
* add gemini

* add gemini

* add gemini
```
  001e77fe
23 Feb, 2024 1 commit
- [Fix] fix ifeval (#909) · 53fe788d
  Jingming authored Feb 23, 2024
  
  53fe788d
22 Feb, 2024 1 commit
- [Fix] Fix IFEval (#906) · 45c606bc
  bittersweet1999 authored Feb 22, 2024
```
* fix ifeval

* fix ifeval

* fix ifeval

* fix ifeval
```
  45c606bc
06 Feb, 2024 2 commits

fix bug of gsm8k_postprocess (#863) · dd444685

hailsham authored Feb 06, 2024



* fix bug of gsm8k_postprocess

* update postprocess

---------
Co-authored-by: Lei Fei <SENSETIME\leifei1@cn3114002087l.domain.sensetime.com>
Co-authored-by: Leymore <zfz-960727@163.com>

dd444685

[feat] support multipl-e (#846) · 444d8d95

Connor-Shen authored Feb 06, 2024



* [feat] support humaneval_multipl-e

* format

---------
Co-authored-by: Leymore <zfz-960727@163.com>

444d8d95

05 Feb, 2024 2 commits
- [Sync] Merge branch 'dev' into zfz/update-keyset-demo (#876) · d34ba111
  Fengzhe Zhou authored Feb 05, 2024
  
  d34ba111
- Support NPHardEval (#835) · 7ad11680
  Skyfall-xzz authored Feb 05, 2024
```
* support NPHardEval

* add .md file and fix minor bugs

* refactor and minor fix

---------
Co-authored-by: Leymore <zfz-960727@163.com>
```
  7ad11680
04 Feb, 2024 1 commit

[Feature] support alpacaeval (#809) · 7806cd0f

bittersweet1999 authored Feb 04, 2024



* support alpacaeval_v1

* Update opencompass/summarizers/subjective/__init__.py
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>

* Update opencompass/summarizers/subjective/alpacaeval_v1.py
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>

* fix conflict

* support alpacaeval v2

* support alpacav2

---------
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>

7806cd0f

30 Jan, 2024 1 commit
- fix compass arena (#854) · 5c6dc908
  bittersweet1999 authored Jan 30, 2024
  
  5c6dc908
27 Jan, 2024 1 commit
- [Fix] Fix acc of IFEval (#849) · 28018833
  Jingming authored Jan 27, 2024
```
* [Feature] Add IFEval

* [Fix] Changing the Score Rule.
```
  28018833
26 Jan, 2024 1 commit
- [Fix] Update MedBench (#845) · 35aace77
  Xiaoming Shi authored Jan 26, 2024
  
  35aace77
24 Jan, 2024 3 commits

[Fix] fix corev2 (#838) · 77be07db
bittersweet1999 authored Jan 24, 2024
```
* fix corev2

* fix corev2
```
77be07db
[Sync] Updata dataset cfg for internMath (#837) · 0991dd33
Fengzhe Zhou authored Jan 24, 2024
```
Co-authored-by: liuhongwei <liuhongwei@pjlab.org.cn>
```
0991dd33

[Feature] add mtbench (#829) · 2ee8e8a1

bittersweet1999 authored Jan 24, 2024



* add mtbench

* add mtbench

* Update configs/datasets/subjective/multiround/mtbench_judgeby_gpt4.py
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>

* Update configs/datasets/subjective/multiround/mtbench_judgeby_gpt4.py
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>

* Update opencompass/datasets/subjective/__init__.py
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>

* Update opencompass/datasets/subjective/mtbench.py
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>

* fix mtbench

---------
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>

2ee8e8a1

23 Jan, 2024 2 commits

[Feature] Add IFEval (#813) · e059a5c2
Jingming authored Jan 23, 2024
```
* [Feature] Add IFEval

* [Doc] add introduction of IFEval
```
e059a5c2

[Feature] Add CompassArena (#828) · 2d4da8dd

bittersweet1999 authored Jan 23, 2024



* add compass arena

* add compass_arena

* add compass arena

* Update opencompass/summarizers/subjective/compass_arena.py
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>

* Update opencompass/summarizers/subjective/__init__.py
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>

* Update opencompass/datasets/subjective/compass_arena.py
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>

* Update opencompass/datasets/subjective/__init__.py
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>

* Update configs/eval_subjective_compassarena.py
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>

* Update configs/datasets/subjective/compassarena/compassarena_compare.py
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>

* Update configs/eval_subjective_compassarena.py
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>

* Update configs/datasets/subjective/compassarena/compassarena_compare.py
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>

* fix check position bias

---------
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>

2d4da8dd

19 Jan, 2024 1 commit

Update cdme config and evaluator (#812) · e975a96f

Guo Qipeng authored Jan 19, 2024



* update cdme config and evaluator

* fix cdme prompt

* move CDME trim post-processor as a separate evaluator

---------
Co-authored-by: 郭琦鹏 <guoqipeng@pjlab.org.cn>

e975a96f

17 Jan, 2024 2 commits

[Sync] Add InternLM2 Keyset Evaluation Demo (#807) · b4afe3e7
Fengzhe Zhou authored Jan 17, 2024
```
Co-authored-by: zhangyifan1 <zhangyifan1@pjlab.org.cn>
```
b4afe3e7

Added support for multi-needle testing in needle-in-a-haystack test (#802) · acae5609

Mo Li authored Jan 17, 2024



* Add NeedleInAHaystack Test

* Apply pre-commit formatting

* Update configs/eval_hf_internlm_chat_20b_cdme.py
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>

* add needle in haystack test

* update needle in haystack test

* update plot function in tools_needleinahaystack.py

* optimizing needleinahaystack dataset generation strategy

* modify minor formatting issues

* add English version support

* change NeedleInAHaystackDataset to dynamic loading

* change NeedleInAHaystackDataset to dynamic loading

* fix needleinahaystack test eval bug

* fix needleinahaystack config bug

* Added support for multi-needle testing in needle-in-a-haystack test

* Optimize the code for plotting in the needle-in-a-haystack test.

* Correct the typo in the dataset parameters.

* update needleinahaystack test docs

---------
Co-authored-by: Songyang Zhang <tonysy@users.noreply.github.com>

acae5609