**Monkey** brings a training-efficient approach to effectively improve the input resolution capacity up to 896 x 1344 pixels without pretraining from the start. To bridge the gap between simple text labels and high input resolution, we propose a multi-level description generation method, which automatically provides rich information that can guide the model to learn the contextual association between scenes and objects. With the synergy of these two designs, our model achieved excellent results on multiple benchmarks. By comparing our model with various LMMs, including GPT4V, our model demonstrates promising performance in image captioning by paying attention to textual information and capturing fine details within the images; its improved input resolution also enables remarkable performance in document images with dense text.
**Monkey** brings a training-efficient approach to effectively improve the input resolution capacity up to 896 x 1344 pixels without pretraining from the start. To bridge the gap between simple text labels and high input resolution, we propose a multi-level description generation method, which automatically provides rich information that can guide the model to learn the contextual association between scenes and objects. With the synergy of these two designs, our model achieved excellent results on multiple benchmarks. By comparing our model with various LMMs, including GPT4V, our model demonstrates promising performance in image captioning by paying attention to textual information and capturing fine details within the images; its improved input resolution also enables remarkable performance in document images with dense text.
## News
## News
*```2024.1.3 ``` 🚀🚀🚀 Release the basic data generation pipeline. [Data Generation](./data_generation)
*```2024.2.27 ``` 🚀 Monkey is accepted by CVPR 2024. The [paper](https://arxiv.org/abs/2311.06607) has been carefully updated according to the valuable comments.
*```2023.12.21``` 🚀🚀🚀 The JSON file used for Monkey training is provided.
*```2024.1.3 ``` 🚀 Release the basic data generation pipeline. [Data Generation](./data_generation)
*```2023.12.16``` 🚀🚀🚀 Monkey can be trained using 8 NVIDIA 3090 GPUs. See subsection [train](#Train) for details.
*```2023.12.21``` 🚀 The JSON file used for Monkey training is provided.
*```2023.11.25``` 🚀🚀🚀 Monkey-chat demo is released.
*```2023.12.16``` 🚀 Monkey can be trained using 8 NVIDIA 3090 GPUs. See subsection [train](#Train) for details.
*```2023.11.06``` 🚀🚀🚀 Monkey [paper](https://arxiv.org/abs/2311.06607) is released.
*```2023.11.25``` 🚀 Monkey-chat demo is released.
*```2023.11.06``` 🚀 Monkey [paper](https://arxiv.org/abs/2311.06607) is released.
## Spotlights
## Spotlights
...
@@ -247,9 +248,6 @@ If you wish to refer to the baseline results published here, please use the foll
...
@@ -247,9 +248,6 @@ If you wish to refer to the baseline results published here, please use the foll
}
}
```
```
If you find the Monkey cute, please star. It would be a great encouragement for us.
## Acknowledgement
## Acknowledgement
[Qwen-VL](https://github.com/QwenLM/Qwen-VL.git): the codebase we built upon. Thanks for the authors of Qwen for providing the framework.
[Qwen-VL](https://github.com/QwenLM/Qwen-VL.git): the codebase we built upon. Thanks for the authors of Qwen for providing the framework.
