GitLab

* add AceGPT-MMLUArabic benchmark

* update readme and fix lint issue

* remove unused package

* add MMLUArabic zero-shot settings

* rename filename and update readme

* s3eval_branch

* update s3eval

* [Fix] Fix AGIEval chinese sets

* Create agieval_gen_617738.py

* [Fix] Fix AGIEval chinese sets

* Restore agieval_gen_64afd3.py

* Update agieval_gen.py

* Create agieval_mixed_0fa998.py

* Update agieval_mixed.py

* add mgsm datasets

* fix lint

* fix lint

* update mgsm

* update mgsm

* ease code spell

* update

* update

* update

---------
Co-authored-by: Leymore <zfz-960727@163.com>

* [Feature] update drop dataset from openai simple eval

* update drop template presentation

* update

---------
Co-authored-by: Leymore <zfz-960727@163.com>

* add mmlu prompt from simple_evals, openai

* return empty str on failure

* fixed formatting based on pre-commit tests

* fixed typo in comments; reduced the number of models in the eval config

* fixed a bug in LLMCompressionDataset, where setting samples=None would result in passing test[:None] to load_dataset

* removed unnecessary variable in _format_table_pivot; changed lark_reporter message to English

requiresments -> requirements

* [Update] Update performance of common benchmarks

* [Update] Update performance of common benchmarks

* [Update] Update performance of common benchmarks

* Add Math Evaluation with Judge Model Evaluator

* Add Math Evaluation with Judge Model Evaluator

* Add Math Evaluation with Judge Model Evaluator

* Add Math Evaluation with Judge Model Evaluator

* Fix Llama-3 meta template

* Fix MATH with JudgeLM Evaluation

* Fix MATH with JudgeLM Evaluation

* Fix MATH with JudgeLM Evaluation

* Fix MATH with JudgeLM Evaluation

---------
Co-authored-by: liuhongwei <liuhongwei@pjlab.org.cn>

* adapt to lmdeploy v0.4.0

* compatible

* add flames datasets

* fix lint

* rm quota

* add judgemodel info and fix os path

* support flames dataset

* support flames dataset

---------
Co-authored-by: bittersweet1999 <1487910649@qq.com>

* update NeedleInAHaystack Test Docs

* update docs

* fix output typing, change mutable list to immutable tuple

* import missed type

* format

---------
Co-authored-by: Leymore <zfz-960727@163.com>

* fix relative path bug

* format

---------
Co-authored-by: hmp <505030475@qq.com>
Co-authored-by: Leymore <zfz-960727@163.com>

* add vllm get_ppl

* add vllm get_ppl

* format

---------
Co-authored-by: xingjin.wang <xingjin.wang@mihoyo.com>
Co-authored-by: Leymore <zfz-960727@163.com>

* Remove MultiModal

* update index.rst

* update README

* remove mmbench codes

* update news

---------
Co-authored-by: Leymore <zfz-960727@163.com>

* add gpqa_openai_simple_eval

* 触发CI构建

* reorg

---------
Co-authored-by: Leymore <zfz-960727@163.com>

* modify the requirements/runtime.txt: numpy==1.23.4 --> numpy>=1.23.4

* update cibench: dataset and evluation

* cibench summarizer bug

* update cibench

* move extract_code import

---------
Co-authored-by: zhangchuyu@pjlab.org.cn <zhangchuyu@pjlab.org.cn>
Co-authored-by: Leymore <zfz-960727@163.com>

* support arenahard

* support arenahard

* support arenahard

* support openai math evaluation

* support openai math evaluation

* support openai math evaluation

* support math llm judge

* support math llm judge

* [Feature] Add IFEval

* add humaneval prompt from simple_evals, openai

Co-authored-by: liuhongwei <liuhongwei@pjlab.org.cn>

* add lmdeploy tis python backend model

* fix pr check

* update

* add TheoremQA with 5-shot

* cherry pick from add-huggingface-above-v4.33, good TheoremQA results

* add LLaMA-3 Series configs

* update readme

* fix multiround

* fix

* [fix]Fixed the issue caused by the repeated loading of VLLM model during task segmentation.

* [fix] avoid TypeError: VLLM.__init__() got an unexpected keyword argument 'tokenizer_only'

* restore .pre-commit-config.yaml

* restore opencompass/tasks/openicl_infer.py

---------
Co-authored-by: IcyFeather <mengzhuo.happy@gmail.com>
Co-authored-by: Leymore <zfz-960727@163.com>

* [Update] Update readme

* [Update] Update readme

* [Update] Update readme

(cherry picked from commit 16ac6306c72fa202173289b55eaefe85e0fcb73c)
Co-authored-by: liuhongwei <liuhongwei@pjlab.org.cn>

* logger.error -> logger.info in OpenAI

* logger.info -> logger.debug in OpenAI

* add ChemBench

* update results

* molbench -> ChemBench

---------
Co-authored-by: Leymore <zfz-960727@163.com>