**Monkey** brings a training-efficient approach to effectively improve the input resolution capacity up to 896 x 1344 pixels without pretraining from the start. To bridge the gap between simple text labels and high input resolution, we propose a multi-level description generation method, which automatically provides rich information that can guide the model to learn the contextual association between scenes and objects. With the synergy of these two designs, our model achieved excellent results on multiple benchmarks. By comparing our model with various LMMs, including GPT4V, our model demonstrates promising performance in image captioning by paying attention to textual information and capturing fine details within the images; its improved input resolution also enables remarkable performance in document images with dense text.
**Monkey** brings a training-efficient approach to effectively improve the input resolution capacity up to 896 x 1344 pixels without pretraining from the start. To bridge the gap between simple text labels and high input resolution, we propose a multi-level description generation method, which automatically provides rich information that can guide the model to learn the contextual association between scenes and objects. With the synergy of these two designs, our model achieved excellent results on multiple benchmarks. By comparing our model with various LMMs, including GPT4V, our model demonstrates promising performance in image captioning by paying attention to textual information and capturing fine details within the images; its improved input resolution also enables remarkable performance in document images with dense text.
Have a try using the providing [Demo](https://53965e0026f6da5097.gradio.live). All you need are to simpley upload or capture image from desktop or your phone, then click the generate. You will get like:
Have a try using the providing [Demo](https://53965e0026f6da5097.gradio.live). All you need are to simpley upload or capture image from desktop or your phone, then click the generate. You will get like:
...
@@ -46,7 +47,7 @@ Have a try using the providing [Demo](https://53965e0026f6da5097.gradio.live). A
...
@@ -46,7 +47,7 @@ Have a try using the providing [Demo](https://53965e0026f6da5097.gradio.live). A
<p>
<p>
<br>
<br>
## Cases
## Cases
Our model can accurately describe the details in the image.
Our model can accurately describe the details in the image.
...
@@ -56,7 +57,7 @@ Our model can accurately describe the details in the image.
...
@@ -56,7 +57,7 @@ Our model can accurately describe the details in the image.
<imgsrc="images/caption_1.png"width="700"/>
<imgsrc="images/caption_1.png"width="700"/>
<p>
<p>
<br>
<br>
Besides, our model has also demonstrated some capabilities in fine-grained question answering.
Besides, our model has also demonstrated some capabilities in fine-grained question answering.
<br>
<br>
...
@@ -80,13 +81,13 @@ We qualitatively compare with existing LMMs including GPT4V, Qwen-vl, etc, which
...
@@ -80,13 +81,13 @@ We qualitatively compare with existing LMMs including GPT4V, Qwen-vl, etc, which
<imgsrc="images/compare.png"width="800"/>
<imgsrc="images/compare.png"width="800"/>
<p>
<p>
<br>
<br>
## Acknowledgement
## Acknowledgement
[Qwen-VL](https://github.com/QwenLM/Qwen-VL.git): the codebase we built upon. Thanks for the authors of Qwen for providing the framework.
[Qwen-VL](https://github.com/QwenLM/Qwen-VL.git): the codebase we built upon. Thanks for the authors of Qwen for providing the framework.
## Copyright
## Copyright
We welcome suggestions to help us improve the little Monkey. For any query, please contact Dr. Yuliang Liu: ylliu@hust.edu.cn
We welcome suggestions to help us improve the little Monkey. For any query, please contact Dr. Yuliang Liu: ylliu@hust.edu.cn
