To help users get started quickly, we recommend the following process:
- For users who want to use VLMEvalKit, we recommend reading the "Start Your First Step" section to set up the environment and start a mini-experiment to familiarize yourself with the process.
- If you want to customize more modules, such as adding datasets and models, we provide an "Advanced Tutorial."
We always welcome users' PRs (Pull Requests) and Issues to improve VLMEvalKit!
[](https://github.com/open-compass/VLMEvalKit/stargazers)
title={VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Models},
author={Haodong Duan and Junming Yang and Yuxuan Qiao and Xinyu Fang and Lin Chen and Yuan Liu and Xiaoyi Dong and Yuhang Zang and Pan Zhang and Jiaqi Wang and Dahua Lin and Kai Chen},
你可以通过支持 `chat_inner(message, dataset)` API 为你的模型新增多轮对话功能并兼容多轮对话评测。这个 API 输出一个字符串型回复,`message` 包含一个聊天记录的列表,格式如下:
```python
# Assume msg1, msg2, msg3, ... are multi-modal messages following the previously described format
# `chat_inner` take the following chat history list as input:
message=[
dict(role='user',content=msg1),
dict(role='assistant',content=msg2),
dict(role='user',content=msg3),
dict(role='assistant',content=msg4),
......
dict(role='user',content=msgn),
]
# `message` should contain an odd number of chat utterances, the role of utterances should be interleaved "user" and "assistant", with the role of the last utterance to be "user".
[](https://github.com/open-compass/VLMEvalKit/stargazers)
title={VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Models},
author={Haodong Duan and Junming Yang and Yuxuan Qiao and Xinyu Fang and Lin Chen and Yuan Liu and Xiaoyi Dong and Yuhang Zang and Pan Zhang and Jiaqi Wang and Dahua Lin and Kai Chen},
<pathstyle="stroke: none; stroke-width: 1; stroke-dasharray: none; stroke-linecap: butt; stroke-dashoffset: 0; stroke-linejoin: miter; stroke-miterlimit: 4; fill: rgb(88,120,180); fill-rule: nonzero; opacity: 1;"transform=" translate(-19.1, -28)"d="M 16.5 22.6 L 10.1 25.700000000000003 L 15.399999999999999 25.500000000000004 L 16.5 22.6 z M 12.3 33.6 L 13.4 30.700000000000003 L 8.100000000000001 30.900000000000002 L 12.3 33.6 z M 21.6 33.3 L 28 30.199999999999996 L 22.7 30.399999999999995 L 21.6 33.3 z M 25.8 22.4 L 24.7 25.299999999999997 L 30 25.099999999999998 L 25.8 22.4 z M 31.5 26.2 L 24.4 26.4 L 22.7 25.299999999999997 L 24.2 21.299999999999997 L 22.2 20 L 19 21.5 L 17.5 25.4 L 14.8 26.7 L 7.700000000000001 26.9 L 4.500000000000001 28.4 L 6.600000000000001 29.799999999999997 L 13.700000000000001 29.599999999999998 L 13.700000000000001 29.599999999999998 L 15.4 30.7 L 13.9 34.7 L 16 36 L 19.2 34.5 L 20.7 30.6 L 20.7 30.6 L 23.3 29.400000000000002 L 23.3 29.400000000000002 L 30.5 29.200000000000003 L 33.7 27.700000000000003 L 31.5 26.2 z M 20.2 28.7 C 19.2 29.2 17.9 29.2 17.2 28.8 C 16.599999999999998 28.400000000000002 16.8 27.6 17.8 27.2 C 18.8 26.7 20.1 26.7 20.8 27.099999999999998 C 21.5 27.5 21.2 28.2 20.2 28.7 z"stroke-linecap="round"/>
<pathstyle="stroke: none; stroke-width: 1; stroke-dasharray: none; stroke-linecap: butt; stroke-dashoffset: 0; stroke-linejoin: miter; stroke-miterlimit: 4; fill: rgb(54,86,155); fill-rule: nonzero; opacity: 1;"transform=" translate(-27.05, -12.8)"d="M 33.5 19.8 L 32.2 13.3 L 30.700000000000003 15.200000000000001 L 33.5 19.8 z M 27.5 5.1 L 23.3 2.3999999999999995 L 26 7 L 27.5 5.1 z M 20.7 5.7 L 22 12.2 L 23.5 10.299999999999999 L 20.7 5.7 z M 26.8 20.4 L 31 23.099999999999998 L 28.3 18.5 L 26.8 20.4 z M 34 22.3 L 30.4 16.1 L 30.4 16.1 L 29.9 13.400000000000002 L 31.9 10.800000000000002 L 31.299999999999997 7.600000000000002 L 29.199999999999996 6.200000000000003 L 27.199999999999996 8.800000000000002 L 25.499999999999996 7.700000000000003 L 21.799999999999997 1.400000000000003 L 19.6 0 L 20.200000000000003 3.2 L 23.900000000000002 9.5 L 23.900000000000002 9.5 L 24.400000000000002 12.1 L 24.400000000000002 12.1 L 22.400000000000002 14.7 L 23.000000000000004 17.9 L 25.100000000000005 19.299999999999997 L 27.000000000000004 16.799999999999997 L 28.700000000000003 17.9 L 32.400000000000006 24.2 L 34.50000000000001 25.599999999999998 L 34 22.3 z M 27.5 14.6 C 26.9 14.2 26.2 13 26 12 C 25.8 11 26.2 10.5 26.8 10.9 C 27.400000000000002 11.3 28.1 12.5 28.3 13.5 C 28.5 14.6 28.1 15.1 27.5 14.6 z"stroke-linecap="round"/>
<pathstyle="stroke: none; stroke-width: 1; stroke-dasharray: none; stroke-linecap: butt; stroke-dashoffset: 0; stroke-linejoin: miter; stroke-miterlimit: 4; fill: rgb(27,56,130); fill-rule: nonzero; opacity: 1;"transform=" translate(-10.55, -13.4)"d="M 12 2.8 L 5.6 5.9 L 9.399999999999999 7.6000000000000005 L 12 2.8 z M 1.1 14.4 L 2.4000000000000004 20.9 L 5 16.099999999999998 L 1.1 14.4 z M 9.1 24 L 15.5 20.9 L 11.7 19.2 L 9.1 24 z M 20 12.4 L 18.7 5.9 L 16.099999999999998 10.7 L 20 12.4 z M 20.4 14.9 L 15.299999999999999 12.600000000000001 L 15.299999999999999 12.600000000000001 L 14.799999999999999 9.900000000000002 L 18.299999999999997 3.400000000000002 L 17.699999999999996 0.20000000000000195 L 14.499999999999996 1.700000000000002 L 11 8.1 L 8.3 9.4 L 8.3 9.4 L 3.2 7.1 L 0 8.6 L 0.6 11.8 L 5.8 14.100000000000001 L 6.3 16.8 L 6.3 16.8 L 2.8 23.4 L 3.4 26.599999999999998 L 6.6 25.099999999999998 L 10.1 18.599999999999998 L 12.7 17.4 L 12.7 17.4 L 17.9 19.799999999999997 L 21.099999999999998 18.299999999999997 L 20.4 14.9 z M 10.9 15.2 C 9.9 15.7 9 15.2 8.8 14.2 C 8.600000000000001 13.2 9.200000000000001 12 10.200000000000001 11.5 C 11.200000000000001 11 12.100000000000001 11.5 12.3 12.5 C 12.5 13.5 11.9 14.7 10.9 15.2 z"stroke-linecap="round"/>