"vscode:/vscode.git/clone" did not exist on "29daf498cd05e2e94ba2e4189e977afdec675a2b"
README.md 3.88 KB
Newer Older
lvskiller's avatar
readme  
lvskiller committed
1
2
# Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models

Melos's avatar
Melos committed
3

Yuliang Liu's avatar
Yuliang Liu committed
4
5
6
7
8
<br>
<p align="center">
    <img src="images/logo_monkey.png" width="300"/>
<p>
<br>
lvskiller's avatar
readme  
lvskiller committed
9
10

<div align="center">
Yuliang Liu's avatar
Yuliang Liu committed
11
Zhang Li*, Biao Yang*, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu†, Xiang Bai†
lvskiller's avatar
readme  
lvskiller committed
12
13
14
15
</div>
<div align="center">
<strong>Huazhong University of Science and Technology, Kingsoft</strong>
</div>
Yuliang Liu's avatar
Yuliang Liu committed
16
17

<p align="center">
Yuliang Liu's avatar
Yuliang Liu committed
18
19
<a href="updating">Paper will be released soon</a>&nbsp&nbsp | &nbsp&nbsp<a href="http://221.232.49.195:7680/">Demo</a>&nbsp&nbsp 
<!--     | &nbsp&nbsp<a href="Monkey Model">Monkey Models</a>&nbsp | &nbsp <a href="updating">Tutorial</a> -->
Yuliang Liu's avatar
Yuliang Liu committed
20
</p>
Melos's avatar
Melos committed
21
22
-----

Yuliang Liu's avatar
Yuliang Liu committed
23
**Monkey** brings a training-efficient approach to effectively improve the input resolution capacity up to 896 x 1344 pixels without pretraining from the start. To bridge the gap between simple text labels and high input resolution, we propose a multi-level description generation method, which automatically provides rich information that can guide the model to learn the contextual association between scenes and objects. With the synergy of these two designs, our model achieved excellent results on multiple benchmarks. By comparing our model with various LMMs, including GPT4V, our model demonstrates promising performance in image captioning by paying attention to textual information and capturing fine details within the images; its improved input resolution also enables remarkable performance in document images with dense text. 
lvskiller's avatar
readme  
lvskiller committed
24
25
26

## Spotlights

Yuliang Liu's avatar
Yuliang Liu committed
27
- **Contextual associations.** Our method demonstrates a superior ability to infer the relationships between targets more effectively when answering questions, which results in delivering more comprehensive and insightful results.
Melos's avatar
Melos committed
28
29
- **Support resolution up to 1344 x 896.** Surpassing the standard 448 x 448 resolution typically employed for LMMs, this significant increase in resolution augments the ability to discern and understand unnoticeable or tightly clustered objects and dense text. 
- **Enhanced general performance.** We carried out testing across 16 diverse datasets, leading to impressive performance by our Monkey model in tasks such as Image Captioning, General Visual Question Answering, Text-centric Visual Question Answering, and Document-oriented Visual Question Answering.
lvskiller's avatar
readme  
lvskiller committed
30
31
32

## performance

Yuliang Liu's avatar
Yuliang Liu committed
33
<br>
lvskiller's avatar
lvskiller committed
34

Yuliang Liu's avatar
Yuliang Liu committed
35
<p align="center">
lvskiller's avatar
lvskiller committed
36
    <img src="images/radar.png" width="800"/>
Yuliang Liu's avatar
Yuliang Liu committed
37
38
<p>
<br>
lvskiller's avatar
lvskiller committed
39
40


lvskiller's avatar
readme  
lvskiller committed
41
42
## Demo

Yuliang Liu's avatar
Yuliang Liu committed
43
Have a try using the providing [Demo](http://221.232.49.195:7680/). All you need are to simpley upload or capture image from desktop or your phone, then click the generate. You may also generate multiple times to get more information. You can also generate Chinese answer by using “生成中文描述”: 
lvskiller's avatar
lvskiller committed
44

Yuliang Liu's avatar
Yuliang Liu committed
45
46
47
48
49
50
<br>
<p align="center">
    <img src="images/generation.png" width="900"/>
<p>
<br>
    
lvskiller's avatar
readme  
lvskiller committed
51
52
## Cases

Yuliang Liu's avatar
Yuliang Liu committed
53
Our model can accurately describe the details in the image.
lvskiller's avatar
readme  
lvskiller committed
54

Yuliang Liu's avatar
Yuliang Liu committed
55
56
57
58
59
<br>
<p align="center">
    <img src="images/caption_1.png" width="700"/>
<p>
<br>
lvskiller's avatar
lvskiller committed
60

Yuliang Liu's avatar
Yuliang Liu committed
61
Besides, our model has also demonstrated some capabilities in fine-grained question answering.
lvskiller's avatar
readme  
lvskiller committed
62

Yuliang Liu's avatar
Yuliang Liu committed
63
64
65
66
67
<br>
<p align="center">
    <img src="images/qa_1.png" width="700"/>
<p>
<br>
lvskiller's avatar
readme  
lvskiller committed
68

Yuliang Liu's avatar
Yuliang Liu committed
69
We have also achieved impressive performance on document-based tasks.
lvskiller's avatar
readme  
lvskiller committed
70

Yuliang Liu's avatar
Yuliang Liu committed
71
72
73
74
75
<br>
<p align="center">
    <img src="images/Doc_Chart.png" width="700"/>
<p>
<br>
lvskiller's avatar
readme  
lvskiller committed
76

Yuliang Liu's avatar
Yuliang Liu committed
77
We qualitatively compare with existing LMMs including GPT4V, Qwen-vl, etc, which shows inspiring results. One can have a try using the provided demo. 
lvskiller's avatar
readme  
lvskiller committed
78

Yuliang Liu's avatar
Yuliang Liu committed
79
80
81
82
83
<br>
<p align="center">
    <img src="images/compare.png" width="800"/>
<p>
<br>
lvskiller's avatar
lvskiller committed
84

lvskiller's avatar
readme  
lvskiller committed
85
86
## Acknowledgement

lvskiller's avatar
lvskiller committed
87

Melos's avatar
Melos committed
88
[Qwen-VL](https://github.com/QwenLM/Qwen-VL.git): the codebase we built upon. Thanks for the authors of Qwen for providing the framework.
lvskiller's avatar
readme  
lvskiller committed
89

Yuliang Liu's avatar
Yuliang Liu committed
90

lvskiller's avatar
lvskiller committed
91

Yuliang Liu's avatar
Yuliang Liu committed
92
93
## Copyright
We welcome suggestions to help us improve the little Monkey. For any query, please contact Dr. Yuliang Liu: ylliu@hust.edu.cn