GitLab

* [Feature]: Add Openflamingo MMBench

* [Fix]: Fix import error

* [Fix]: Revert task config

* [Fix]: Fix path bug

* refactor instructblip

* add post processor

* add forward

* fix lint

* update

* update

* Add ToT method

* Update ToT

* Update ToT

* Update ToT

* Update ToT

* Update ToT

* Update ToT

* Update ToT

* Update chain_of_thought.md

* Update icl_tot_inferencer.py

---------
Co-authored-by: liuhongwei <liuhongwei@pjlab.org.cn>

* add llama2 native implements

* rename configs/eval_llama_7b.py

---------
Co-authored-by: zhoufengzhe <zhoufengzhe@pjlab.org.cn>

* [Feat] Support visualglm inference on MMBench.

* [Feat] Support llava inference on MMBench.

* [Fix] Fix pre-commit format.

* [Fix] Add docstring for llava

* [Fix] Fix multi-process inference error of LlaVA and add comments.
1. Set `low_cpu_mem_usage` to False to address device issue.
2. Add docstring and type hints.
3. Rename class and remove registry.

* [Fix] Pre-commit fix.

* [Fix] add forward entry, add dynamic import to seedbench

* [Fix] Fix pre-commit.

* [Fix] Fix missing context.

* [Fix] Fix docstring.

* [Feat] Support multi-modal evaluation on MME benchmark.

* [Fix] Remove debug code.

* [Fix] Remove redundant codes and add type hints.

* [Fix] Rename in config.

* [Fix] Rebase main.

* [Fix] Fix isort and yapf conflict.

Co-authored-by: wangchonghua <wangchonghua@pjlab.org.cn>

* support seedbench

* update docstrings

* update

* update

* update

* update according to review

* rebase

* fix lint

* update

* [Feature]: Add flickr30k

* [Feature]: Add GQA

* [Feature]: Add OCR VQA

* [Feature]: Add OK VQA

* [Feature]: Add text vqa

* [Feature]: Add other vqa

* add codegeex2

* add humanevalx dataset

* add evaluator

* update evaluator

* update configs

* update clean code

* update configs

* fix lint

* remove sleep

* fix lint

* update docs

* fix lint

* [Feat] support adv_glue dataset for adversarial robustness

* reorg files

* minor fix

* minor fix

* [Feature]: Refactor class name

* [Feature]: Add minigpt-4 coco caption

* [Feature]: Update minigpt-4 coco caption

* [Feature]: Add MiniGPT-4 ScienceQA

* [Feature]: Add minigpt-4 vqav2

* [Feature]: Add VSR

* [Feature]: Revert task to previous version

* [Fix] fix bug for postprocessor

* minor fix

Co-authored-by: kennymckormick <dhd@pku.edu.cn>

* [Feat] update postprocessor to get first option

* minor fix

* minor fix

* add llama-oriented dataset configs

* update

* revert cvalues & update llama_example

* [Feat] add safety to collections

* minor fix

[Enhancement] Add humaneval postprocessor for GPT models & eval config for GPT4, enhance the original humaneval postprocessor (#129)

* [Enhancement] Enhance humaneval postprocessor

* add human-eval testcase

* update

* update

---------
Co-authored-by: Leymore <zfz-960727@163.com>

* support turbomind

* update doc

* Update docs/en/advanced_guides/evaluation_turbomind.md
Co-authored-by: Tong Gao <gaotongxiao@gmail.com>

* Update docs/zh_cn/advanced_guides/evaluation_turbomind.md
Co-authored-by: Tong Gao <gaotongxiao@gmail.com>

* Update docs/zh_cn/advanced_guides/evaluation_turbomind.md
Co-authored-by: Tong Gao <gaotongxiao@gmail.com>

* Update docs/en/advanced_guides/evaluation_turbomind.md
Co-authored-by: Tong Gao <gaotongxiao@gmail.com>

* update

---------
Co-authored-by: Tong Gao <gaotongxiao@gmail.com>

* add Xiezhi SQuAD2.0 ANLI; update WSC

* update

* update

* update doc string

* [Feature]: Refactor input and output

* [Feature]: Update tasks

* update agieval data

* rename variables

* calulate max_out_len without hard code

* set default value

* update configs

* Update configs/eval_gpt3.5.py
Co-authored-by: Tong Gao <gaotongxiao@gmail.com>

---------
Co-authored-by: Tong Gao <gaotongxiao@gmail.com>

* [Feature]: Add minigpt-4

* [Feature]: Add mm local runner

* [Feature]: Add instructblip

* [Feature]: Delete redundant file

* [Feature]: Delete redundant file

* [Feature]: Add README to InstructBLIP

* [Feature]: Update MiniGPT-4

* [Fix]: Fix lint

* [Feature]add omnibenchmark readme (#49)

* add omnibenchmark readme

* fix

* Update OmniMMBench.md

* Update OmniMMBench.md

* Update OmniMMBench.md

* [Fix]: Refine name (#54)

* [Feature]: Unify out and err

* [Fix]: Fix lint

* [Feature]: Rename to mmbench and change weight path

* [Feature]: Delete Omni in instructblip

* [Feature]: Check the avaliablity of lavis

* [Fix]: Fix lint

* [Feature]: Refactor MM

* [Refactor]: Refactor path

* [Feature]: Delete redundant files

* [Refactor]: Delete redundant files

---------
Co-authored-by: Wangbo Zhao(黑色枷锁) <56866854+wangbo-zhao@users.noreply.github.com>

* [Feature] Support evaluating acc based on minimum edit distance, update SIQA

* update

* add self-consistency

* add CoT method Self-Consistency

* fix typo error and update openicl_eval

* add tydiQA-GoldP task

* fix sc

* rename gsm8k_sc

* fix sc

* add self-consistency doc

* refine sc

---------
Authored-by: liushz <qq1791167085@163.com>

* support internLM

* support internLM

* simplify intern model files

* update storage_manager

* support internLM

* Modify the file organization structure

* support internLM

* support internLM

* support internLM

* support internLM

* change some details

* [Refactor] Update crows-pairs evaluation

* [Refactor] Update crows-pairs evaluation

* minor

Co-authored-by: Leymore <zfz-960727@163.com>

* add CMMLU

* debug cmmlu

* add slurm args `qos`

* fix format: space before comment

* remove unused variable

* change the location of `answer is`

---------
Co-authored-by: 李浩楠 <lihaonan@lihaonandeMacBook-Air.local>
Co-authored-by: 李浩楠 <haonan.li>
Co-authored-by: Leymore <zfz-960727@163.com>

* add llama-2 models

* update docs

---------
Co-authored-by: gaotongxiao <gaotongxiao@gmail.com>

* [Feat] support CValues

* minor fix

Co-authored-by: liuhongwei <liuhongwei@pjlab.org.cn>

* [Feat] add falcon-40b

* minor fix

* [Feature] Add baichuan13b

* update num_gpus

* Add logger info and remove dataset bugs

* fix typo