Initial commit

.dockerignore

+docs
+Dockerfile
\ No newline at end of file

.gitattributes

+# Basic .gitattributes for a python repo.
+
+# Source files
+# ============
+*.pxd    text diff=python
+*.py     text diff=python
+*.py3    text diff=python
+*.pyw    text diff=python
+*.pyx    text diff=python
+*.pyz    text diff=python
+*.pyi    text diff=python
+
+# Binary files
+# ============
+*.db     binary
+*.p      binary
+*.pkl    binary
+*.pickle binary
+*.pyc    binary export-ignore
+*.pyo    binary export-ignore
+*.pyd    binary
+
+# Jupyter notebook
+*.ipynb  text eol=lf
+
+# Others
+*           text=auto
+*.txt		text
+*.sh		text eol=lf
+
+# Note: .db, .p, and .pkl files are associated
+# with the python modules ``pickle``, ``dbm.*``,
+# ``shelve``, ``marshal``, ``anydbm``, & ``bsddb``
+# (among others).

.gitignore

+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+.hypothesis/
+.pytest_cache/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+doc/en/_build/
+doc/zh_cn/_build/
+
+# PyBuilder
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# pyenv
+.python-version
+
+# celery beat schedule file
+celerybeat-schedule
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+# data/
+# data
+.vscode
+.idea
+.DS_Store
+
+# custom
+*.pkl
+*.pkl.json
+*.log.json
+doc/modelzoo_statistics.md
+mmdet/.mim
+work_dirs
+
+# Pytorch
+*.pth
+*.py~
+*.sh~
+
+# venus
+venus_run.sh
+

.gitmodules

+[submodule "third_party/mmyolo"]
+	path = third_party/mmyolo
+	url = https://github.com/onuralpszr/mmyolo.git

Dockerfile

+FROM image.sourcefind.cn:5000/dcu/admin/base/pytorch:1.13.1-ubuntu20.04-dtk23.10-py38
\ No newline at end of file

README.md

+# YOLO-World
+## 论文
+`YOLO-World: Real-Time Open-Vocabulary Object Detection`
+- https://arxiv.org/abs/2401.17270
+## 模型结构
+YOLO-World是一种实时开放词汇目标检测系统，它通过视觉-语言建模和大规模数据集上的预训练，增强了YOLO（You Only Look Once）系列检测器的开放词汇检测能力。该模型没有使用在线词汇表，而是提供了一个提示-检测范式，其中用户根据需要生成一系列提示，提示将被编码到离线词汇表中。然后可以将其重新参数化为模型权重，用于部署和进一步加速。
+
+<div align=center>
+    <img src="./doc/YOLO-World.png"/>
+</div>
+
+## 算法原理
+YOLO-World包含一个YOLO检测器、一个文本编码器和一个可重参数化的视觉-语言路径聚合网络（RepVL-PAN）。给定输入文本，YOLO-World中的文本编码器将其编码为文本嵌入向量。YOLO检测器中的图像编码器从输入图像中提取多尺度特征，然后利用RepVL-PAN通过利用图像特征和文本嵌入向量之间的跨模态融合来增强文本和图像的表示。在推理过程中，可以移除文本编码器，并将文本嵌入重新参数化为RepVL-PAN的权重，以实现高效部署。
+
+<div align=center>
+    <img src="./doc/The_overview.png"/>
+</div>
+
+RepVL-PAN结构如下图，通过自顶向下和自底向上的方式，建立具有多尺度图像特征{C3, C4, C5}的特征金字塔{P3, P4, P5}。此外，提出了文本引导的CSPLayer（T-CSPLayer）和图像池化注意力（(I-Pooling Attention）来进一步增强图像特征和文本特征之间的交互作用，从而提高开放词汇表的视觉语义表示能力。在推理过程中，可以将脱机词汇表嵌入重新参数化为卷积层或线性层的权重，以进行部署。
+
+<div align=center>
+    <img src="./doc/RepVL-PAN.png"/>
+</div>
+
+
+## 环境配置
+### Docker（方法一）
+此处提供[光源](https://www.sourcefind.cn/#/service-details)拉取docker镜像的地址与使用步骤，以及[光合](https://developer.hpccube.com/tool/)开发者社区深度学习库下载地址
+```
+docker pull image.sourcefind.cn:5000/dcu/admin/base/pytorch:1.13.1-ubuntu20.04-dtk23.10-py38 
+docker run -it --shm-size=128G -v /path/your_code_data/:/path/your_code_data/ -v /opt/hyhal:/opt/hyhal:ro --privileged=true --device=/dev/kfd --device=/dev/dri/ --group-add video --name efficientsam_pytorch  <your IMAGE ID> bash # <your IMAGE ID>为以上拉取的docker的镜像ID替换，本镜像为：0a56ef1842a7
+cd /path/your_code_data/yolo_world_pytorch
+pip install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/  --trusted-host mirrors.aliyun.com
+#从光合社区下载mmcv-2.0.1
+pip install mmcv-2.0.1+gite2f0eed.abi0.dtk2310.torch1.13-cp38-cp38-manylinux2014_x86_64.whl
+pip uninstall deepspeed
+cd third_party
+git clone https://github.com/onuralpszr/mmyolo.git
+cd mmyolo
+pip install -v -e .
+```
+### Dockerfile（方法二）
+此处提供dockerfile的使用方法
+```
+docker build --no-cache -t yoloworld:latest .
+docker run -it --shm-size=128G -v /path/your_code_data/:/path/your_code_data/ -v /opt/hyhal:/opt/hyhal:ro --privileged=true --device=/dev/kfd --device=/dev/dri/ --group-add video --name yolo_world_pytorch  yoloworld  bash
+cd /path/your_code_data/yolo_world_pytorch
+pip install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/  --trusted-host mirrors.aliyun.com
+pip uninstall deepspeed
+cd third_party
+git clone https://github.com/onuralpszr/mmyolo.git
+cd mmyolo
+pip install -v -e .
+```
+### Anaconda（方法三）
+此处提供本地配置、编译的详细步骤，例如：
+
+关于本项目DCU显卡所需的特殊深度学习库可从[光合](https://developer.hpccube.com/tool/)开发者社区下载安装。
+```
+#DTK驱动：dtk23.10
+# python：python3.8
+# torch: 1.3.1
+# torchvision: 0.14.1
+# mmcv-2.0.1
+conda create -n yoloworld python=3.8
+conda activate yoloworld
+pip install torch-1.13.1+git7d2dd01.abi0.dtk2310-cp38-cp38-manylinux2014_x86_64.whl
+pip install torchvision-0.14.1+gitf78f29f.abi1.dtk2310.torch1.13-cp38-cp38-manylinux2014_x86_64.whl
+pip install mmcv-2.0.1+gite2f0eed.abi0.dtk2310.torch1.13-cp38-cp38-manylinux2014_x86_64.whl
+```
+`Tips：以上dtk驱动、python、torch等DCU相关工具版本需要严格一一对应`
+
+其它依赖环境安装如下：
+```
+cd /path/your_code_data/yolo_world_pytorch
+pip install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/  --trusted-host mirrors.aliyun.com
+cd third_party
+git clone https://github.com/onuralpszr/mmyolo.git
+cd mmyolo
+pip install -v -e .
+```
+## 数据集
+
+数据集和标注文件下载链接如下表所示：
+
+| images                                                                                          | Annotation File                                                                            |
+|-------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------|
+| [Objects365 train](https://opendatalab.com/OpenDataLab/Objects365_v1) | [objects365_train.json](https://opendatalab.com/OpenDataLab/Objects365_v1) |
+| [GQA](https://downloads.cs.stanford.edu/nlp/data/gqa/images.zip) | [final_mixed_train_no_coco.json](https://huggingface.co/GLIPModel/GLIP/tree/main/mdetr_annotations) |
+| [Flickr30k](https://opendatalab.com/OpenDataLab/Flickr30k) | [final_flickr_separateGT_train.json](https://huggingface.co/GLIPModel/GLIP/tree/main/mdetr_annotations) |
+| [COCO val2017](https://opendatalab.com/OpenDataLab/COCO_2017) | [lvis_v1_minival_inserted_image_name.json](https://huggingface.co/GLIPModel/GLIP/blob/main/lvis_v1_minival_inserted_image_name.json) |
+将所有数据下载并放置于data文件夹下，数据目录如下：
+
+注意：lvis_v1_minival_inserted_image_name.json放于coco/lvis/文件夹下，其余数据标注文件放于各数据文件的annotations文件夹下
+```
+├── coco
+│   ├── annotations
+│   ├── lvis
+│   ├── train2017
+│   ├── val2017
+├── flickr
+│   ├── annotations
+│   └── images
+├── mixed_grounding
+│   ├── annotations
+│   ├── images
+├── objects365v1
+│   ├── annotations
+│   ├── train
+│   ├── val
+
+```
+## 训练
+首先下载clip-vit模型文件，放于openai目录下:
+[clip-vit-large-patch14-336下载](https://huggingface.co/openai/clip-vit-large-patch14-336)
+[clip-vit-base-patch32下载](https://huggingface.co/openai/clip-vit-base-patch32)
+### 单机单卡
+```
+cd /path/your_code_data/yolo_world_pytorch
+chmod +x tools/dist_train.sh
+# sample command for pre-training, use AMP for mixed-precision training
+#./tools/dist_train.sh configs/pretrain/yolo_world_l_t2i_bn_2e-4_100e_4x8gpus_obj365v1_goldg_train_lvis_minival.py 1 --amp
+#样例如下
+./tools/dist_train.sh configs/pretrain/yolo_world_v2_s_vlpan_bn_2e-3_100e_4x8gpus_obj365v1_goldg_train_1280ft_lvis_minival.py 1 --amp
+```
+
+### 单机多卡
+```
+chmod +x tools/dist_train.sh
+# sample command for pre-training, use AMP for mixed-precision training
+#./tools/dist_train.sh configs/pretrain/yolo_world_l_t2i_bn_2e-4_100e_4x8gpus_obj365v1_goldg_train_lvis_minival.py 4 --amp
+#样例如下
+./tools/dist_train.sh configs/pretrain/yolo_world_v2_s_vlpan_bn_2e-3_100e_4x8gpus_obj365v1_goldg_train_1280ft_lvis_minival.py 4 --amp
+```
+## 推理
+根据以下链接下载想要的模型权重文件，放到weights文件夹下：
+[模型权重文件下载](https://hf-mirror.com/wondervictor/YOLO-World/tree/main)
+
+注意：模型配置文件与权重文件应一一对应
+
+### 单卡推理
+
+Evaluated on LVIS minival：
+
+```
+chmod +x tools/dist_test.sh
+#./tools/dist_test.sh path/to/config path/to/weights 1
+#样例如下
+./tools/dist_test.sh \
+configs/pretrain/yolo_world_v2_x_vlpan_bn_2e-3_100e_4x8gpus_obj365v1_goldg_train_1280ft_lvis_minival.py \
+weights/yolo_world_v2_x_obj365v1_goldg_cc3mlite_pretrain_1280ft-14996a36.pth 1
+```
+
+Simple_Demo：
+
+```
+#PYTHONPATH=/xxxx/YOLO-World python demo/simple_demo.py
+#样例如下：
+#注意需先下载模型权重文件yolo_world_v2_x_obj365v1_goldg_cc3mlite_pretrain_1280ft-14996a36.pth
+PYTHONPATH=/public/home/luopl/yolo_world_pytorch  python demo/simple_demo.py
+```
+
+Image_Demo：
+
+```
+#python demo/image_demo.py path/to/config path/to/weights image/path/directory 'person,dog,cat' --topk 100 --threshold 0.005 --output-dir demo_outputs
+#样例如下：
+PYTHONPATH=/public/home/luopl/yolo_world_pytorch \
+python demo/image_demo.py configs/pretrain/yolo_world_v2_x_vlpan_bn_2e-3_100e_4x8gpus_obj365v1_goldg_train_1280ft_lvis_minival.py \
+weights/yolo_world_v2_x_obj365v1_goldg_cc3mlite_pretrain_1280ft-14996a36.pth \
+demo/sample_images/ \
+'person,sky,bus' --topk 100 --threshold 0.005 --output-dir demo_outputs
+```
+
+### 多卡推理
+
+Evaluated on LVIS minival：
+
+```
+chmod +x tools/dist_test.sh
+#./tools/dist_test.sh path/to/config path/to/weights 4
+#样例如下
+./tools/dist_test.sh \
+configs/pretrain/yolo_world_v2_x_vlpan_bn_2e-3_100e_4x8gpus_obj365v1_goldg_train_1280ft_lvis_minival.py \
+weights/yolo_world_v2_x_obj365v1_goldg_cc3mlite_pretrain_1280ft-14996a36.pth 4
+```
+
+## result
+Simple_demo推理可视化结果如下：
+
+<div align=center>
+    <img src="./doc/detecting_results.png"/>
+</div>
+
+Image_demo推理可视化结果如下：
+
+<div align=center>
+    <img src="./doc/bus.jpg"/>
+</div>
+
+
+### 精度
+使用四张DCU-Z100SM
+
+|                 model                  |             Pre-train Data             | size | APmini | APr  | APc  | APf  |
+|:--------------------------------------:|:--------------------------------------:|------|--------|------|------|------|
+|             YOLO-Worldv2-S             |               O365+GoldG               | 1280 | 24.1   | 18.7 | 22.0 | 26.9 |
+|             YOLO-Worldv2-M             |               O365+GoldG               | 1280 | 31.6   | 24.5 | 29.0 | 35.1 |
+|             YOLO-Worldv2-L             |               O365+GoldG               | 1280 | 34.6   | 29.2 | 32.8 | 37.2 |
+|             YOLO-Worldv2-X             |          O365+GoldG+CC3M-Lite          | 1280 | 37.6   | 29.1 | 35.8 | 40.6 |
+## 应用场景
+### 算法类别
+`目标检测`
+### 热点应用行业
+`科研,制造,医疗,家居,教育`
+## 源码仓库及问题反馈
+- https://developer.hpccube.com/codes/modelzoo/yolo_world_pytorch
+## 参考资料
+- https://github.com/AILab-CVC/YOLO-World
+

configs/finetune_coco/README.md

+## Fine-tune YOLO-World on MS-COCO
+
+
+### Updates
+
+1. [2024-3-27]: Considering that fine-tuning YOLO-World on COCO **without `mask-refine`** obtains bad results, e.g., YOLO-World-L obtains 48.6 AP without `mask-refine` compared to 53.3 AP with `mask-refine`, we rethink the training process and explore new training schemes for fine-tuning without `mask-refine`.
+BTW, the COCO fine-tuning results are updated with higher performance (with `mask-refine`)!
+
+
+### COCO Results and Checkpoints
+
+**NOTE:**
+1. AP<sup>ZS</sup>: AP evaluated in the zero-shot setting (w/o fine-tuning on COCO dataset).
+2. `mask-refine`: refine the box annotations with masks, and add `CopyPaste` augmentation during training.
+
+| model | Schedule | `mask-refine` | efficient neck | AP<sup>ZS</sup>|  AP | AP<sub>50</sub> | AP<sub>75</sub> | weights | log |
+| :---- | :-------: | :----------: |:-------------: | :------------: | :-: | :--------------:| :-------------: |:------: | :-: |
+| [YOLO-World-v2-S](./yolo_world_v2_s_vlpan_bn_2e-4_80e_8gpus_mask-refine_finetune_coco.py) | AdamW, 2e-4, 80e | ✔️  | ✖️ | 37.5 | 46.1 | 62.0 | 49.9 | [HF Checkpoints](https://huggingface.co/wondervictor/YOLO-World/blob/main/yolo_world_v2_s_vlpan_bn_2e-4_80e_8gpus_mask-refine_finetune_coco_ep80-492dc329.pth) | [log](https://huggingface.co/wondervictor/YOLO-World/blob/main/yolo_world_v2_s_vlpan_bn_2e-4_80e_8gpus_mask-refine_finetune_coco_20240327_110411.log) |
+| [YOLO-World-v2-M](./yolo_world_v2_m_vlpan_bn_2e-4_80e_8gpus_mask-refine_finetune_coco.py) | AdamW, 2e-4, 80e | ✔️  | ✖️ | 42.8 | 51.0 | 67.5 | 55.2 | [HF Checkpoints](https://huggingface.co/wondervictor/YOLO-World/blob/main/yolo_world_v2_m_vlpan_bn_2e-4_80e_8gpus_mask-refine_finetune_coco_ep80-69c27ac7.pth) | [log](https://huggingface.co/wondervictor/YOLO-World/blob/main/yolo_world_v2_m_vlpan_bn_2e-4_80e_8gpus_mask-refine_finetune_coco_20240327_110411.log) |
+| [YOLO-World-v2-L](./yolo_world_v2_l_vlpan_bn_2e-4_80e_8gpus_mask-refine_finetune_coco.py) | AdamW, 2e-4, 80e | ✔️  | ✖️ | 45.1 | 53.9 | 70.9 | 58.8 | [HF Checkpoints](https://huggingface.co/wondervictor/YOLO-World/blob/main/yolo_world_v2_l_vlpan_bn_2e-4_80e_8gpus_mask-refine_finetune_coco_ep80-81c701ee.pth) | [log](https://huggingface.co/wondervictor/YOLO-World/blob/main/yolo_world_v2_l_vlpan_bn_2e-4_80e_8gpus_mask-refine_finetune_coco_20240326_160313.log) |
+| [YOLO-World-v2-X](./yolo_world_v2_x_vlpan_bn_2e-4_80e_8gpus_mask-refine_finetune_coco.py) | AdamW, 2e-4, 80e | ✔️  | ✖️ | 46.8 | 54.7 | 71.6 | 59.6 | [HF Checkpoints](https://huggingface.co/wondervictor/YOLO-World/blob/main/yolo_world_v2_x_vlpan_bn_2e-4_80e_8gpus_mask-refine_finetune_coco_ep80-76bc0cbd.pth) | [log](https://huggingface.co/wondervictor/YOLO-World/blob/main/yolo_world_v2_x_vlpan_bn_2e-4_80e_8gpus_mask-refine_finetune_coco_20240322_181232.log) |
+| [YOLO-World-v2-L](./yolo_world_v2_l_vlpan_bn_sgd_1e-3_40e_8gpus_finetune_coco.py) 🔥 | SGD, 1e-3, 40e | ✖️  | ✖️ | 45.1 | 52.8 | 69.5 | 57.8 | [HF Checkpoints](https://huggingface.co/wondervictor/YOLO-World/blob/main/yolo_world_v2_l_vlpan_bn_sgd_1e-3_40e_8gpus_finetune_coco_ep80-e1288152.pth) | [log](https://huggingface.co/wondervictor/YOLO-World/blob/main/yolo_world_v2_l_vlpan_bn_sgd_1e-3_40e_8gpus_finetuning_coco_20240327_014902.log) |
+
+
+### Reparameterized Training
+
+| model | Schedule | `mask-refine` | efficient neck | AP<sup>ZS</sup>|  AP | AP<sub>50</sub> | AP<sub>75</sub> | weights | log |
+| :---- | :-------: | :----------: |:-------------: | :------------: | :-: | :--------------:| :-------------: |:------: | :-: |
+| [YOLO-World-v2-S](./yolo_world_v2_s_rep_vlpan_bn_2e-4_80e_8gpus_mask-refine_finetune_coco.py) | AdamW, 2e-4, 80e | ✔️  | ✖️ | 37.5 | 46.3 | 62.8 | 50.4 | [HF Checkpoints]() | [log]() | 
\ No newline at end of file

configs/finetune_coco/yolo_world_l_dual_vlpan_2e-4_80e_8gpus_finetune_coco.py

+_base_ = (
+    '../../third_party/mmyolo/configs/yolov8/'
+    'yolov8_l_syncbn_fast_8xb16-500e_coco.py')
+custom_imports = dict(
+    imports=['yolo_world'],
+    allow_failed_imports=False)
+
+# hyper-parameters
+num_classes = 80
+num_training_classes = 80
+max_epochs = 80  # Maximum training epochs
+close_mosaic_epochs = 10
+save_epoch_intervals = 5
+text_channels = 512
+neck_embed_channels = [128, 256, _base_.last_stage_out_channels // 2]
+neck_num_heads = [4, 8, _base_.last_stage_out_channels // 2 // 32]
+base_lr = 2e-4
+weight_decay = 0.05
+train_batch_size_per_gpu = 16
+load_from='pretrained_models/yolo_world_l_clip_base_dual_vlpan_2e-3adamw_32xb16_100e_o365_goldg_train_pretrained-0e566235.pth'
+persistent_workers = False
+
+# model settings
+model = dict(
+    type='YOLOWorldDetector',
+    mm_neck=True,
+    num_train_classes=num_training_classes,
+    num_test_classes=num_classes,
+    data_preprocessor=dict(type='YOLOWDetDataPreprocessor'),
+    backbone=dict(
+        _delete_=True,
+        type='MultiModalYOLOBackbone',
+        image_model={{_base_.model.backbone}},
+        text_model=dict(
+            type='HuggingCLIPLanguageBackbone',
+            model_name='openai/clip-vit-base-patch32',
+            frozen_modules=['all'])),
+    neck=dict(type='YOLOWorldDualPAFPN',
+              guide_channels=text_channels,
+              embed_channels=neck_embed_channels,
+              num_heads=neck_num_heads,
+              block_cfg=dict(type='MaxSigmoidCSPLayerWithTwoConv'),
+              text_enhancder=dict(type='ImagePoolingAttentionModule',
+                                  embed_channels=256,
+                                  num_heads=8)),
+    bbox_head=dict(type='YOLOWorldHead',
+                   head_module=dict(type='YOLOWorldHeadModule',
+                                    embed_dims=text_channels,
+                                    num_classes=num_training_classes)),
+    train_cfg=dict(assigner=dict(num_classes=num_training_classes)))
+
+# dataset settings
+text_transform = [
+    dict(type='RandomLoadText',
+         num_neg_samples=(num_classes, num_classes),
+         max_num_samples=num_training_classes,
+         padding_to_max=True,
+         padding_value=''),
+    dict(type='mmdet.PackDetInputs',
+         meta_keys=('img_id', 'img_path', 'ori_shape', 'img_shape', 'flip',
+                    'flip_direction', 'texts'))
+]
+mosaic_affine_transform = [
+    dict(
+        type='MultiModalMosaic',
+        img_scale=_base_.img_scale,
+        pad_val=114.0,
+        pre_transform=_base_.pre_transform),
+    dict(
+        type='YOLOv5RandomAffine',
+        max_rotate_degree=0.0,
+        max_shear_degree=0.0,
+        max_aspect_ratio=100.,
+        scaling_ratio_range=(1 - _base_.affine_scale,
+                             1 + _base_.affine_scale),
+        # img_scale is (width, height)
+        border=(-_base_.img_scale[0] // 2, -_base_.img_scale[1] // 2),
+        border_val=(114, 114, 114))
+]
+train_pipeline = [
+    *_base_.pre_transform,
+    *mosaic_affine_transform,
+    dict(
+        type='YOLOv5MultiModalMixUp',
+        prob=_base_.mixup_prob,
+        pre_transform=[*_base_.pre_transform,
+                       *mosaic_affine_transform]),
+    *_base_.last_transform[:-1],
+    *text_transform
+]
+train_pipeline_stage2 = [
+    *_base_.train_pipeline_stage2[:-1],
+    *text_transform
+]
+coco_train_dataset = dict(
+    _delete_=True,
+    type='MultiModalDataset',
+    dataset=dict(
+        type='YOLOv5CocoDataset',
+        data_root='data/coco',
+        ann_file='annotations/instances_train2017.json',
+        data_prefix=dict(img='train2017/'),
+        filter_cfg=dict(filter_empty_gt=False, min_size=32)),
+    class_text_path='data/texts/coco_class_texts.json',
+    pipeline=train_pipeline)
+
+train_dataloader = dict(
+    persistent_workers=persistent_workers,
+    batch_size=train_batch_size_per_gpu,
+    collate_fn=dict(type='yolow_collate'),
+    dataset=coco_train_dataset)
+test_pipeline = [
+    *_base_.test_pipeline[:-1],
+    dict(type='LoadText'),
+    dict(
+        type='mmdet.PackDetInputs',
+        meta_keys=('img_id', 'img_path', 'ori_shape', 'img_shape',
+                   'scale_factor', 'pad_param', 'texts'))
+]
+coco_val_dataset = dict(
+    _delete_=True,
+    type='MultiModalDataset',
+    dataset=dict(
+        type='YOLOv5CocoDataset',
+        data_root='data/coco',
+        ann_file='annotations/instances_val2017.json',
+        data_prefix=dict(img='val2017/'),
+        filter_cfg=dict(filter_empty_gt=False, min_size=32)),
+    class_text_path='data/texts/coco_class_texts.json',
+    pipeline=test_pipeline)
+val_dataloader = dict(dataset=coco_val_dataset)
+test_dataloader = val_dataloader
+# training settings
+default_hooks = dict(
+    param_scheduler=dict(
+        scheduler_type='linear',
+        lr_factor=0.01,
+        max_epochs=max_epochs),
+    checkpoint=dict(
+        max_keep_ckpts=-1,
+        save_best=None,
+        interval=save_epoch_intervals))
+custom_hooks = [
+    dict(
+        type='EMAHook',
+        ema_type='ExpMomentumEMA',
+        momentum=0.0001,
+        update_buffers=True,
+        strict_load=False,
+        priority=49),
+    dict(
+        type='mmdet.PipelineSwitchHook',
+        switch_epoch=max_epochs - close_mosaic_epochs,
+        switch_pipeline=train_pipeline_stage2)
+]
+train_cfg = dict(
+    max_epochs=max_epochs,
+    val_interval=5,
+    dynamic_intervals=[((max_epochs - close_mosaic_epochs),
+                        _base_.val_interval_stage2)])
+optim_wrapper = dict(
+    optimizer=dict(
+        _delete_=True,
+        type='AdamW',
+        lr=base_lr,
+        weight_decay=weight_decay,
+        batch_size_per_gpu=train_batch_size_per_gpu),
+    paramwise_cfg=dict(
+        custom_keys={'backbone.text_model': dict(lr_mult=0.01),
+                     'logit_scale': dict(weight_decay=0.0)}),
+    constructor='YOLOWv5OptimizerConstructor')
+
+# evaluation settings
+val_evaluator = dict(
+    _delete_=True,
+    type='mmdet.CocoMetric',
+    proposal_nums=(100, 1, 10),
+    ann_file='data/coco/annotations/instances_val2017.json',
+    metric='bbox')

configs/finetune_coco/yolo_world_l_dual_vlpan_2e-4_80e_8gpus_mask-refine_finetune_coco.py

+_base_ = (
+    '../../third_party/mmyolo/configs/yolov8/'
+    'yolov8_l_mask-refine_syncbn_fast_8xb16-500e_coco.py')
+custom_imports = dict(
+    imports=['yolo_world'],
+    allow_failed_imports=False)
+
+# hyper-parameters
+num_classes = 80
+num_training_classes = 80
+max_epochs = 80  # Maximum training epochs
+close_mosaic_epochs = 10
+save_epoch_intervals = 5
+text_channels = 512
+neck_embed_channels = [128, 256, _base_.last_stage_out_channels // 2]
+neck_num_heads = [4, 8, _base_.last_stage_out_channels // 2 // 32]
+base_lr = 2e-4
+weight_decay = 0.05
+train_batch_size_per_gpu = 16
+load_from='pretrained_models/yolo_world_l_clip_base_dual_vlpan_2e-3adamw_32xb16_100e_o365_goldg_train_pretrained-0e566235.pth'
+persistent_workers = False
+
+# model settings
+model = dict(
+    type='YOLOWorldDetector',
+    mm_neck=True,
+    num_train_classes=num_training_classes,
+    num_test_classes=num_classes,
+    data_preprocessor=dict(type='YOLOWDetDataPreprocessor'),
+    backbone=dict(
+        _delete_=True,
+        type='MultiModalYOLOBackbone',
+        image_model={{_base_.model.backbone}},
+        text_model=dict(
+            type='HuggingCLIPLanguageBackbone',
+            model_name='openai/clip-vit-base-patch32',
+            frozen_modules=['all'])),
+    neck=dict(type='YOLOWorldDualPAFPN',
+              guide_channels=text_channels,
+              embed_channels=neck_embed_channels,
+              num_heads=neck_num_heads,
+              block_cfg=dict(type='MaxSigmoidCSPLayerWithTwoConv'),
+              text_enhancder=dict(type='ImagePoolingAttentionModule',
+                                  embed_channels=256,
+                                  num_heads=8)),
+    bbox_head=dict(type='YOLOWorldHead',
+                   head_module=dict(type='YOLOWorldHeadModule',
+                                    embed_dims=text_channels,
+                                    num_classes=num_training_classes)),
+    train_cfg=dict(assigner=dict(num_classes=num_training_classes)))
+
+# dataset settings
+text_transform = [
+    dict(type='RandomLoadText',
+         num_neg_samples=(num_classes, num_classes),
+         max_num_samples=num_training_classes,
+         padding_to_max=True,
+         padding_value=''),
+    dict(type='mmdet.PackDetInputs',
+         meta_keys=('img_id', 'img_path', 'ori_shape', 'img_shape', 'flip',
+                    'flip_direction', 'texts'))
+]
+mosaic_affine_transform = [
+    dict(
+        type='MultiModalMosaic',
+        img_scale=_base_.img_scale,
+        pad_val=114.0,
+        pre_transform=_base_.pre_transform),
+    dict(type='YOLOv5CopyPaste', prob=_base_.copypaste_prob),
+    dict(
+        type='YOLOv5RandomAffine',
+        max_rotate_degree=0.0,
+        max_shear_degree=0.0,
+        max_aspect_ratio=100.,
+        scaling_ratio_range=(1 - _base_.affine_scale,
+                             1 + _base_.affine_scale),
+        # img_scale is (width, height)
+        border=(-_base_.img_scale[0] // 2, -_base_.img_scale[1] // 2),
+        border_val=(114, 114, 114),
+        min_area_ratio=_base_.min_area_ratio,
+        use_mask_refine=_base_.use_mask2refine)
+]
+train_pipeline = [
+    *_base_.pre_transform,
+    *mosaic_affine_transform,
+    dict(
+        type='YOLOv5MultiModalMixUp',
+        prob=_base_.mixup_prob,
+        pre_transform=[*_base_.pre_transform,
+                       *mosaic_affine_transform]),
+    *_base_.last_transform[:-1],
+    *text_transform
+]
+train_pipeline_stage2 = [
+    *_base_.train_pipeline_stage2[:-1],
+    *text_transform
+]
+coco_train_dataset = dict(
+    _delete_=True,
+    type='MultiModalDataset',
+    dataset=dict(
+        type='YOLOv5CocoDataset',
+        data_root='data/coco',
+        ann_file='annotations/instances_train2017.json',
+        data_prefix=dict(img='train2017/'),
+        filter_cfg=dict(filter_empty_gt=False, min_size=32)),
+    class_text_path='data/texts/coco_class_texts.json',
+    pipeline=train_pipeline)
+
+train_dataloader = dict(
+    persistent_workers=persistent_workers,
+    batch_size=train_batch_size_per_gpu,
+    collate_fn=dict(type='yolow_collate'),
+    dataset=coco_train_dataset)
+test_pipeline = [
+    *_base_.test_pipeline[:-1],
+    dict(type='LoadText'),
+    dict(
+        type='mmdet.PackDetInputs',
+        meta_keys=('img_id', 'img_path', 'ori_shape', 'img_shape',
+                   'scale_factor', 'pad_param', 'texts'))
+]
+coco_val_dataset = dict(
+    _delete_=True,
+    type='MultiModalDataset',
+    dataset=dict(
+        type='YOLOv5CocoDataset',
+        data_root='data/coco',
+        ann_file='annotations/instances_val2017.json',
+        data_prefix=dict(img='val2017/'),
+        filter_cfg=dict(filter_empty_gt=False, min_size=32)),
+    class_text_path='data/texts/coco_class_texts.json',
+    pipeline=test_pipeline)
+val_dataloader = dict(dataset=coco_val_dataset)
+test_dataloader = val_dataloader
+# training settings
+default_hooks = dict(
+    param_scheduler=dict(
+        scheduler_type='linear',
+        lr_factor=0.01,
+        max_epochs=max_epochs),
+    checkpoint=dict(
+        max_keep_ckpts=-1,
+        save_best=None,
+        interval=save_epoch_intervals))
+custom_hooks = [
+    dict(
+        type='EMAHook',
+        ema_type='ExpMomentumEMA',
+        momentum=0.0001,
+        update_buffers=True,
+        strict_load=False,
+        priority=49),
+    dict(
+        type='mmdet.PipelineSwitchHook',
+        switch_epoch=max_epochs - close_mosaic_epochs,
+        switch_pipeline=train_pipeline_stage2)
+]
+train_cfg = dict(
+    max_epochs=max_epochs,
+    val_interval=5,
+    dynamic_intervals=[((max_epochs - close_mosaic_epochs),
+                        _base_.val_interval_stage2)])
+optim_wrapper = dict(
+    optimizer=dict(
+        _delete_=True,
+        type='AdamW',
+        lr=base_lr,
+        weight_decay=weight_decay,
+        batch_size_per_gpu=train_batch_size_per_gpu),
+    paramwise_cfg=dict(
+        custom_keys={'backbone.text_model': dict(lr_mult=0.01),
+                     'logit_scale': dict(weight_decay=0.0)}),
+    constructor='YOLOWv5OptimizerConstructor')
+# evaluation settings
+val_evaluator = dict(
+    _delete_=True,
+    type='mmdet.CocoMetric',
+    proposal_nums=(100, 1, 10),
+    ann_file='data/coco/annotations/instances_val2017.json',
+    metric='bbox')

configs/finetune_coco/yolo_world_l_efficient_neck_2e-4_80e_8gpus_mask-refine_finetune_coco.py

+_base_ = ('../../third_party/mmyolo/configs/yolov8/'
+          'yolov8_l_mask-refine_syncbn_fast_8xb16-500e_coco.py')
+custom_imports = dict(imports=['yolo_world'], allow_failed_imports=False)
+
+# hyper-parameters
+num_classes = 80
+num_training_classes = 80
+max_epochs = 80  # Maximum training epochs
+close_mosaic_epochs = 10
+save_epoch_intervals = 5
+text_channels = 512
+neck_embed_channels = [128, 256, _base_.last_stage_out_channels // 2]
+neck_num_heads = [4, 8, _base_.last_stage_out_channels // 2 // 32]
+base_lr = 2e-4
+weight_decay = 0.05
+train_batch_size_per_gpu = 16
+load_from = 'pretrained_models/yolo_world_l_clip_base_dual_vlpan_2e-3adamw_32xb16_100e_o365_goldg_train_pretrained-0e566235.pth'
+# huggingface text model
+text_model_name = 'openai/clip-vit-base-patch32'
+persistent_workers = False
+
+# model settings
+model = dict(
+    type='YOLOWorldDetector',
+    mm_neck=True,
+    num_train_classes=num_training_classes,
+    num_test_classes=num_classes,
+    data_preprocessor=dict(type='YOLOWDetDataPreprocessor'),
+    backbone=dict(
+        _delete_=True,
+        type='MultiModalYOLOBackbone',
+        image_model={{_base_.model.backbone}},
+        text_model=dict(
+            type='HuggingCLIPLanguageBackbone',
+            model_name=text_model_name,
+            frozen_modules=['all'])),
+    neck=dict(type='YOLOWorldPAFPN',
+              guide_channels=text_channels,
+              embed_channels=neck_embed_channels,
+              num_heads=neck_num_heads,
+              block_cfg=dict(type='EfficientCSPLayerWithTwoConv')),
+    bbox_head=dict(type='YOLOWorldHead',
+                   head_module=dict(type='YOLOWorldHeadModule',
+                                    embed_dims=text_channels,
+                                    num_classes=num_training_classes)),
+    train_cfg=dict(assigner=dict(num_classes=num_training_classes)))
+
+# dataset settings
+text_transform = [
+    dict(type='RandomLoadText',
+         num_neg_samples=(num_classes, num_classes),
+         max_num_samples=num_training_classes,
+         padding_to_max=True,
+         padding_value=''),
+    dict(type='mmdet.PackDetInputs',
+         meta_keys=('img_id', 'img_path', 'ori_shape', 'img_shape', 'flip',
+                    'flip_direction', 'texts'))
+]
+mosaic_affine_transform = [
+    dict(type='MultiModalMosaic',
+         img_scale=_base_.img_scale,
+         pad_val=114.0,
+         pre_transform=_base_.pre_transform),
+    dict(type='YOLOv5CopyPaste', prob=_base_.copypaste_prob),
+    dict(
+        type='YOLOv5RandomAffine',
+        max_rotate_degree=0.0,
+        max_shear_degree=0.0,
+        max_aspect_ratio=100.,
+        scaling_ratio_range=(1 - _base_.affine_scale, 1 + _base_.affine_scale),
+        # img_scale is (width, height)
+        border=(-_base_.img_scale[0] // 2, -_base_.img_scale[1] // 2),
+        border_val=(114, 114, 114),
+        min_area_ratio=_base_.min_area_ratio,
+        use_mask_refine=_base_.use_mask2refine)
+]
+train_pipeline = [
+    *_base_.pre_transform, *mosaic_affine_transform,
+    dict(type='YOLOv5MultiModalMixUp',
+         prob=_base_.mixup_prob,
+         pre_transform=[*_base_.pre_transform, *mosaic_affine_transform]),
+    *_base_.last_transform[:-1], *text_transform
+]
+train_pipeline_stage2 = [*_base_.train_pipeline_stage2[:-1], *text_transform]
+coco_train_dataset = dict(_delete_=True,
+                          type='MultiModalDataset',
+                          dataset=dict(
+                              type='YOLOv5CocoDataset',
+                              data_root='data/coco',
+                              ann_file='annotations/instances_train2017.json',
+                              data_prefix=dict(img='train2017/'),
+                              filter_cfg=dict(filter_empty_gt=False,
+                                              min_size=32)),
+                          class_text_path='data/texts/coco_class_texts.json',
+                          pipeline=train_pipeline)
+
+train_dataloader = dict(persistent_workers=persistent_workers,
+                        batch_size=train_batch_size_per_gpu,
+                        collate_fn=dict(type='yolow_collate'),
+                        dataset=coco_train_dataset)
+test_pipeline = [
+    *_base_.test_pipeline[:-1],
+    dict(type='LoadText'),
+    dict(type='mmdet.PackDetInputs',
+         meta_keys=('img_id', 'img_path', 'ori_shape', 'img_shape',
+                    'scale_factor', 'pad_param', 'texts'))
+]
+coco_val_dataset = dict(
+    _delete_=True,
+    type='MultiModalDataset',
+    dataset=dict(type='YOLOv5CocoDataset',
+                 data_root='data/coco',
+                 ann_file='annotations/instances_val2017.json',
+                 data_prefix=dict(img='val2017/'),
+                 filter_cfg=dict(filter_empty_gt=False, min_size=32)),
+    class_text_path='data/texts/coco_class_texts.json',
+    pipeline=test_pipeline)
+val_dataloader = dict(dataset=coco_val_dataset)
+test_dataloader = val_dataloader
+# training settings
+default_hooks = dict(param_scheduler=dict(scheduler_type='linear',
+                                          lr_factor=0.01,
+                                          max_epochs=max_epochs),
+                     checkpoint=dict(max_keep_ckpts=-1,
+                                     save_best=None,
+                                     interval=save_epoch_intervals))
+custom_hooks = [
+    dict(type='EMAHook',
+         ema_type='ExpMomentumEMA',
+         momentum=0.0001,
+         update_buffers=True,
+         strict_load=False,
+         priority=49),
+    dict(type='mmdet.PipelineSwitchHook',
+         switch_epoch=max_epochs - close_mosaic_epochs,
+         switch_pipeline=train_pipeline_stage2)
+]
+train_cfg = dict(max_epochs=max_epochs,
+                 val_interval=5,
+                 dynamic_intervals=[((max_epochs - close_mosaic_epochs),
+                                     _base_.val_interval_stage2)])
+optim_wrapper = dict(
+    optimizer=dict(
+        _delete_=True,
+        type='AdamW',
+        lr=base_lr,
+        weight_decay=weight_decay,
+        batch_size_per_gpu=train_batch_size_per_gpu),
+    paramwise_cfg=dict(
+        custom_keys={'backbone.text_model': dict(lr_mult=0.01),
+                     'logit_scale': dict(weight_decay=0.0)}),
+    constructor='YOLOWv5OptimizerConstructor')
+
+# evaluation settings
+val_evaluator = dict(_delete_=True,
+                     type='mmdet.CocoMetric',
+                     proposal_nums=(100, 1, 10),
+                     ann_file='data/coco/annotations/instances_val2017.json',
+                     metric='bbox')

configs/finetune_coco/yolo_world_v2_l_efficient_neck_2e-4_80e_8gpus_mask-refine_finetune_coco.py

+_base_ = (
+    '../../third_party/mmyolo/configs/yolov8/'
+    'yolov8_l_mask-refine_syncbn_fast_8xb16-500e_coco.py')
+custom_imports = dict(
+    imports=['yolo_world'],
+    allow_failed_imports=False)
+
+# hyper-parameters
+num_classes = 80
+num_training_classes = 80
+max_epochs = 80  # Maximum training epochs
+close_mosaic_epochs = 10
+save_epoch_intervals = 5
+text_channels = 512
+neck_embed_channels = [128, 256, _base_.last_stage_out_channels // 2]
+neck_num_heads = [4, 8, _base_.last_stage_out_channels // 2 // 32]
+base_lr = 2e-4
+weight_decay = 0.05
+train_batch_size_per_gpu = 16
+load_from = 'pretrained_models/yolo_world_l_clip_t2i_bn_2e-3adamw_32xb16-100e_obj365v1_goldg_cc3mlite_train-ca93cd1f.pth'
+text_model_name = '../pretrained_models/clip-vit-base-patch32-projection'
+# text_model_name = 'openai/clip-vit-base-patch32'
+persistent_workers = False
+
+# model settings
+model = dict(
+    type='YOLOWorldDetector',
+    mm_neck=True,
+    num_train_classes=num_training_classes,
+    num_test_classes=num_classes,
+    data_preprocessor=dict(type='YOLOWDetDataPreprocessor'),
+    backbone=dict(
+        _delete_=True,
+        type='MultiModalYOLOBackbone',
+        image_model={{_base_.model.backbone}},
+        text_model=dict(
+            type='HuggingCLIPLanguageBackbone',
+            model_name=text_model_name,
+            frozen_modules=['all'])),
+    neck=dict(type='YOLOWorldPAFPN',
+              guide_channels=text_channels,
+              embed_channels=neck_embed_channels,
+              num_heads=neck_num_heads,
+              block_cfg=dict(type='EfficientCSPLayerWithTwoConv')),
+    bbox_head=dict(type='YOLOWorldHead',
+                   head_module=dict(type='YOLOWorldHeadModule',
+                                    use_bn_head=True,
+                                    embed_dims=text_channels,
+                                    num_classes=num_training_classes)),
+    train_cfg=dict(assigner=dict(num_classes=num_training_classes)))
+
+# dataset settings
+text_transform = [
+    dict(type='RandomLoadText',
+         num_neg_samples=(num_classes, num_classes),
+         max_num_samples=num_training_classes,
+         padding_to_max=True,
+         padding_value=''),
+    dict(type='mmdet.PackDetInputs',
+         meta_keys=('img_id', 'img_path', 'ori_shape', 'img_shape', 'flip',
+                    'flip_direction', 'texts'))
+]
+mosaic_affine_transform = [
+    dict(
+        type='MultiModalMosaic',
+        img_scale=_base_.img_scale,
+        pad_val=114.0,
+        pre_transform=_base_.pre_transform),
+    dict(type='YOLOv5CopyPaste', prob=_base_.copypaste_prob),
+    dict(
+        type='YOLOv5RandomAffine',
+        max_rotate_degree=0.0,
+        max_shear_degree=0.0,
+        max_aspect_ratio=100.,
+        scaling_ratio_range=(1 - _base_.affine_scale,
+                             1 + _base_.affine_scale),
+        # img_scale is (width, height)
+        border=(-_base_.img_scale[0] // 2, -_base_.img_scale[1] // 2),
+        border_val=(114, 114, 114),
+        min_area_ratio=_base_.min_area_ratio,
+        use_mask_refine=_base_.use_mask2refine)
+]
+train_pipeline = [
+    *_base_.pre_transform,
+    *mosaic_affine_transform,
+    dict(
+        type='YOLOv5MultiModalMixUp',
+        prob=_base_.mixup_prob,
+        pre_transform=[*_base_.pre_transform,
+                       *mosaic_affine_transform]),
+    *_base_.last_transform[:-1],
+    *text_transform
+]
+train_pipeline_stage2 = [
+    *_base_.train_pipeline_stage2[:-1],
+    *text_transform
+]
+coco_train_dataset = dict(
+    _delete_=True,
+    type='MultiModalDataset',
+    dataset=dict(
+        type='YOLOv5CocoDataset',
+        data_root='data/coco',
+        ann_file='annotations/instances_train2017.json',
+        data_prefix=dict(img='train2017/'),
+        filter_cfg=dict(filter_empty_gt=False, min_size=32)),
+    class_text_path='data/texts/coco_class_texts.json',
+    pipeline=train_pipeline)
+
+train_dataloader = dict(
+    persistent_workers=persistent_workers,
+    batch_size=train_batch_size_per_gpu,
+    collate_fn=dict(type='yolow_collate'),
+    dataset=coco_train_dataset)
+test_pipeline = [
+    *_base_.test_pipeline[:-1],
+    dict(type='LoadText'),
+    dict(
+        type='mmdet.PackDetInputs',
+        meta_keys=('img_id', 'img_path', 'ori_shape', 'img_shape',
+                   'scale_factor', 'pad_param', 'texts'))
+]
+coco_val_dataset = dict(
+    _delete_=True,
+    type='MultiModalDataset',
+    dataset=dict(
+        type='YOLOv5CocoDataset',
+        data_root='data/coco',
+        ann_file='annotations/instances_val2017.json',
+        data_prefix=dict(img='val2017/'),
+        filter_cfg=dict(filter_empty_gt=False, min_size=32)),
+    class_text_path='data/texts/coco_class_texts.json',
+    pipeline=test_pipeline)
+val_dataloader = dict(dataset=coco_val_dataset)
+test_dataloader = val_dataloader
+# training settings
+default_hooks = dict(
+    param_scheduler=dict(
+        scheduler_type='linear',
+        lr_factor=0.01,
+        max_epochs=max_epochs),
+    checkpoint=dict(
+        max_keep_ckpts=-1,
+        save_best=None,
+        interval=save_epoch_intervals))
+custom_hooks = [
+    dict(
+        type='EMAHook',
+        ema_type='ExpMomentumEMA',
+        momentum=0.0001,
+        update_buffers=True,
+        strict_load=False,
+        priority=49),
+    dict(
+        type='mmdet.PipelineSwitchHook',
+        switch_epoch=max_epochs - close_mosaic_epochs,
+        switch_pipeline=train_pipeline_stage2)
+]
+train_cfg = dict(
+    max_epochs=max_epochs,
+    val_interval=5,
+    dynamic_intervals=[((max_epochs - close_mosaic_epochs),
+                        _base_.val_interval_stage2)])
+optim_wrapper = dict(
+    optimizer=dict(
+        _delete_=True,
+        type='AdamW',
+        lr=base_lr,
+        weight_decay=weight_decay,
+        batch_size_per_gpu=train_batch_size_per_gpu),
+    paramwise_cfg=dict(
+        custom_keys={'backbone.text_model': dict(lr_mult=0.01),
+                     'logit_scale': dict(weight_decay=0.0)}),
+    constructor='YOLOWv5OptimizerConstructor')
+
+# evaluation settings
+val_evaluator = dict(
+    _delete_=True,
+    type='mmdet.CocoMetric',
+    proposal_nums=(100, 1, 10),
+    ann_file='data/coco/annotations/instances_val2017.json',
+    metric='bbox')

configs/finetune_coco/yolo_world_v2_l_vlpan_bn_2e-4_80e_8gpus_mask-refine_finetune_coco.py

+_base_ = (
+    '../../third_party/mmyolo/configs/yolov8/'
+    'yolov8_l_mask-refine_syncbn_fast_8xb16-500e_coco.py')
+custom_imports = dict(
+    imports=['yolo_world'],
+    allow_failed_imports=False)
+
+# hyper-parameters
+num_classes = 80
+num_training_classes = 80
+max_epochs = 80  # Maximum training epochs
+close_mosaic_epochs = 10
+save_epoch_intervals = 5
+text_channels = 512
+neck_embed_channels = [128, 256, _base_.last_stage_out_channels // 2]
+neck_num_heads = [4, 8, _base_.last_stage_out_channels // 2 // 32]
+base_lr = 2e-4
+weight_decay = 0.05
+train_batch_size_per_gpu = 16
+load_from = 'pretrained_models/yolo_world_l_clip_t2i_bn_2e-3adamw_32xb16-100e_obj365v1_goldg_cc3mlite_train-ca93cd1f.pth'
+# text_model_name = '../pretrained_models/clip-vit-base-patch32-projection'
+text_model_name = 'openai/clip-vit-base-patch32'
+persistent_workers = False
+
+# model settings
+model = dict(
+    type='YOLOWorldDetector',
+    mm_neck=True,
+    num_train_classes=num_training_classes,
+    num_test_classes=num_classes,
+    data_preprocessor=dict(type='YOLOWDetDataPreprocessor'),
+    backbone=dict(
+        _delete_=True,
+        type='MultiModalYOLOBackbone',
+        image_model={{_base_.model.backbone}},
+        text_model=dict(
+            type='HuggingCLIPLanguageBackbone',
+            model_name=text_model_name,
+            frozen_modules=['all'])),
+    neck=dict(type='YOLOWorldPAFPN',
+              guide_channels=text_channels,
+              embed_channels=neck_embed_channels,
+              num_heads=neck_num_heads,
+              block_cfg=dict(type='MaxSigmoidCSPLayerWithTwoConv')),
+    bbox_head=dict(type='YOLOWorldHead',
+                   head_module=dict(type='YOLOWorldHeadModule',
+                                    use_bn_head=True,
+                                    embed_dims=text_channels,
+                                    num_classes=num_training_classes)),
+    train_cfg=dict(assigner=dict(num_classes=num_training_classes)))
+
+# dataset settings
+text_transform = [
+    dict(type='RandomLoadText',
+         num_neg_samples=(num_classes, num_classes),
+         max_num_samples=num_training_classes,
+         padding_to_max=True,
+         padding_value=''),
+    dict(type='mmdet.PackDetInputs',
+         meta_keys=('img_id', 'img_path', 'ori_shape', 'img_shape', 'flip',
+                    'flip_direction', 'texts'))
+]
+mosaic_affine_transform = [
+    dict(
+        type='MultiModalMosaic',
+        img_scale=_base_.img_scale,
+        pad_val=114.0,
+        pre_transform=_base_.pre_transform),
+    dict(type='YOLOv5CopyPaste', prob=_base_.copypaste_prob),
+    dict(
+        type='YOLOv5RandomAffine',
+        max_rotate_degree=0.0,
+        max_shear_degree=0.0,
+        max_aspect_ratio=100.,
+        scaling_ratio_range=(1 - _base_.affine_scale,
+                             1 + _base_.affine_scale),
+        # img_scale is (width, height)
+        border=(-_base_.img_scale[0] // 2, -_base_.img_scale[1] // 2),
+        border_val=(114, 114, 114),
+        min_area_ratio=_base_.min_area_ratio,
+        use_mask_refine=_base_.use_mask2refine)
+]
+train_pipeline = [
+    *_base_.pre_transform,
+    *mosaic_affine_transform,
+    dict(
+        type='YOLOv5MultiModalMixUp',
+        prob=_base_.mixup_prob,
+        pre_transform=[*_base_.pre_transform,
+                       *mosaic_affine_transform]),
+    *_base_.last_transform[:-1],
+    *text_transform
+]
+train_pipeline_stage2 = [
+    *_base_.train_pipeline_stage2[:-1],
+    *text_transform
+]
+coco_train_dataset = dict(
+    _delete_=True,
+    type='MultiModalDataset',
+    dataset=dict(
+        type='YOLOv5CocoDataset',
+        data_root='data/coco',
+        ann_file='annotations/instances_train2017.json',
+        data_prefix=dict(img='train2017/'),
+        filter_cfg=dict(filter_empty_gt=False, min_size=32)),
+    class_text_path='data/texts/coco_class_texts.json',
+    pipeline=train_pipeline)
+
+train_dataloader = dict(
+    persistent_workers=persistent_workers,
+    batch_size=train_batch_size_per_gpu,
+    collate_fn=dict(type='yolow_collate'),
+    dataset=coco_train_dataset)
+test_pipeline = [
+    *_base_.test_pipeline[:-1],
+    dict(type='LoadText'),
+    dict(
+        type='mmdet.PackDetInputs',
+        meta_keys=('img_id', 'img_path', 'ori_shape', 'img_shape',
+                   'scale_factor', 'pad_param', 'texts'))
+]
+coco_val_dataset = dict(
+    _delete_=True,
+    type='MultiModalDataset',
+    dataset=dict(
+        type='YOLOv5CocoDataset',
+        data_root='data/coco',
+        ann_file='annotations/instances_val2017.json',
+        data_prefix=dict(img='val2017/'),
+        filter_cfg=dict(filter_empty_gt=False, min_size=32)),
+    class_text_path='data/texts/coco_class_texts.json',
+    pipeline=test_pipeline)
+val_dataloader = dict(dataset=coco_val_dataset)
+test_dataloader = val_dataloader
+# training settings
+default_hooks = dict(
+    param_scheduler=dict(
+        scheduler_type='linear',
+        lr_factor=0.01,
+        max_epochs=max_epochs),
+    checkpoint=dict(
+        max_keep_ckpts=-1,
+        save_best=None,
+        interval=save_epoch_intervals))
+custom_hooks = [
+    dict(
+        type='EMAHook',
+        ema_type='ExpMomentumEMA',
+        momentum=0.0001,
+        update_buffers=True,
+        strict_load=False,
+        priority=49),
+    dict(
+        type='mmdet.PipelineSwitchHook',
+        switch_epoch=max_epochs - close_mosaic_epochs,
+        switch_pipeline=train_pipeline_stage2)
+]
+train_cfg = dict(
+    max_epochs=max_epochs,
+    val_interval=5,
+    dynamic_intervals=[((max_epochs - close_mosaic_epochs),
+                        _base_.val_interval_stage2)])
+optim_wrapper = dict(
+    optimizer=dict(
+        _delete_=True,
+        type='AdamW',
+        lr=base_lr,
+        weight_decay=weight_decay,
+        batch_size_per_gpu=train_batch_size_per_gpu),
+    paramwise_cfg=dict(
+        custom_keys={'backbone.text_model': dict(lr_mult=0.01),
+                     'logit_scale': dict(weight_decay=0.0)}),
+    constructor='YOLOWv5OptimizerConstructor')
+# evaluation settings
+val_evaluator = dict(
+    _delete_=True,
+    type='mmdet.CocoMetric',
+    proposal_nums=(100, 1, 10),
+    ann_file='data/coco/annotations/instances_val2017.json',
+    metric='bbox')

configs/finetune_coco/yolo_world_v2_l_vlpan_bn_sgd_1e-3_40e_8gpus_finetune_coco.py

+_base_ = ('../../third_party/mmyolo/configs/yolov8/'
+          'yolov8_l_syncbn_fast_8xb16-500e_coco.py')
+custom_imports = dict(imports=['yolo_world'], allow_failed_imports=False)
+
+# hyper-parameters
+num_classes = 80
+num_training_classes = 80
+max_epochs = 40  # Maximum training epochs
+close_mosaic_epochs = 30
+save_epoch_intervals = 5
+text_channels = 512
+neck_embed_channels = [128, 256, _base_.last_stage_out_channels // 2]
+neck_num_heads = [4, 8, _base_.last_stage_out_channels // 2 // 32]
+base_lr = 1e-3
+weight_decay = 0.0005
+train_batch_size_per_gpu = 16
+load_from = 'pretrained_models/yolo_world_l_clip_t2i_bn_2e-3adamw_32xb16-100e_obj365v1_goldg_cc3mlite_train-ca93cd1f.pth'
+text_model_name = '../pretrained_models/clip-vit-base-patch32-projection'
+# text_model_name = 'openai/clip-vit-base-patch32'
+persistent_workers = False
+
+# model settings
+model = dict(type='YOLOWorldDetector',
+             mm_neck=True,
+             num_train_classes=num_training_classes,
+             num_test_classes=num_classes,
+             data_preprocessor=dict(type='YOLOWDetDataPreprocessor'),
+             backbone=dict(_delete_=True,
+                           type='MultiModalYOLOBackbone',
+                           image_model={{_base_.model.backbone}},
+                           text_model=dict(type='HuggingCLIPLanguageBackbone',
+                                           model_name=text_model_name,
+                                           frozen_modules=['all'])),
+             neck=dict(type='YOLOWorldPAFPN',
+                       guide_channels=text_channels,
+                       embed_channels=neck_embed_channels,
+                       num_heads=neck_num_heads,
+                       block_cfg=dict(type='MaxSigmoidCSPLayerWithTwoConv')),
+             bbox_head=dict(type='YOLOWorldHead',
+                            head_module=dict(
+                                type='YOLOWorldHeadModule',
+                                use_bn_head=True,
+                                embed_dims=text_channels,
+                                num_classes=num_training_classes)),
+             train_cfg=dict(assigner=dict(num_classes=num_training_classes)))
+
+# dataset settings
+text_transform = [
+    dict(type='RandomLoadText',
+         num_neg_samples=(num_classes, num_classes),
+         max_num_samples=num_training_classes,
+         padding_to_max=True,
+         padding_value=''),
+    dict(type='mmdet.PackDetInputs',
+         meta_keys=('img_id', 'img_path', 'ori_shape', 'img_shape', 'flip',
+                    'flip_direction', 'texts'))
+]
+mosaic_affine_transform = [
+    dict(type='MultiModalMosaic',
+         img_scale=_base_.img_scale,
+         pad_val=114.0,
+         pre_transform=_base_.pre_transform),
+    dict(
+        type='YOLOv5RandomAffine',
+        max_rotate_degree=0.0,
+        max_shear_degree=0.0,
+        max_aspect_ratio=100.,
+        scaling_ratio_range=(1 - _base_.affine_scale, 1 + _base_.affine_scale),
+        # img_scale is (width, height)
+        border=(-_base_.img_scale[0] // 2, -_base_.img_scale[1] // 2),
+        border_val=(114, 114, 114))
+]
+
+train_pipeline = [
+    *_base_.pre_transform, *mosaic_affine_transform,
+    dict(type='YOLOv5MultiModalMixUp',
+         prob=_base_.mixup_prob,
+         pre_transform=[*_base_.pre_transform, *mosaic_affine_transform]),
+    *_base_.last_transform[:-1], *text_transform
+]
+train_pipeline_stage2 = [*_base_.train_pipeline_stage2[:-1], *text_transform]
+
+coco_train_dataset = dict(_delete_=True,
+                          type='MultiModalDataset',
+                          dataset=dict(
+                              type='YOLOv5CocoDataset',
+                              data_root='data/coco',
+                              ann_file='annotations/instances_train2017.json',
+                              data_prefix=dict(img='train2017/'),
+                              filter_cfg=dict(filter_empty_gt=False,
+                                              min_size=32)),
+                          class_text_path='data/texts/coco_class_texts.json',
+                          pipeline=train_pipeline)
+
+train_dataloader = dict(persistent_workers=persistent_workers,
+                        batch_size=train_batch_size_per_gpu,
+                        collate_fn=dict(type='yolow_collate'),
+                        dataset=coco_train_dataset)
+test_pipeline = [
+    *_base_.test_pipeline[:-1],
+    dict(type='LoadText'),
+    dict(type='mmdet.PackDetInputs',
+         meta_keys=('img_id', 'img_path', 'ori_shape', 'img_shape',
+                    'scale_factor', 'pad_param', 'texts'))
+]
+coco_val_dataset = dict(
+    _delete_=True,
+    type='MultiModalDataset',
+    dataset=dict(type='YOLOv5CocoDataset',
+                 data_root='data/coco',
+                 ann_file='annotations/instances_val2017.json',
+                 data_prefix=dict(img='val2017/'),
+                 filter_cfg=dict(filter_empty_gt=False, min_size=32)),
+    class_text_path='data/texts/coco_class_texts.json',
+    pipeline=test_pipeline)
+val_dataloader = dict(dataset=coco_val_dataset)
+test_dataloader = val_dataloader
+# training settings
+default_hooks = dict(param_scheduler=dict(scheduler_type='linear',
+                                          lr_factor=0.01,
+                                          max_epochs=max_epochs),
+                     checkpoint=dict(max_keep_ckpts=-1,
+                                     save_best=None,
+                                     interval=save_epoch_intervals))
+custom_hooks = [
+    dict(type='EMAHook',
+         ema_type='ExpMomentumEMA',
+         momentum=0.0001,
+         update_buffers=True,
+         strict_load=False,
+         priority=49),
+    dict(type='mmdet.PipelineSwitchHook',
+         switch_epoch=max_epochs - close_mosaic_epochs,
+         switch_pipeline=train_pipeline_stage2)
+]
+train_cfg = dict(max_epochs=max_epochs,
+                 val_interval=5,
+                 dynamic_intervals=[((max_epochs - close_mosaic_epochs),
+                                     _base_.val_interval_stage2)])
+optim_wrapper = dict(optimizer=dict(
+    _delete_=True,
+    type='SGD',
+    lr=base_lr,
+    momentum=0.937,
+    nesterov=True,
+    weight_decay=weight_decay,
+    batch_size_per_gpu=train_batch_size_per_gpu),
+                     paramwise_cfg=dict(
+                         custom_keys={
+                             'backbone.text_model': dict(lr_mult=0.01),
+                             'logit_scale': dict(weight_decay=0.0)
+                         }),
+                     constructor='YOLOWv5OptimizerConstructor')
+
+# evaluation settings
+val_evaluator = dict(_delete_=True,
+                     type='mmdet.CocoMetric',
+                     proposal_nums=(100, 1, 10),
+                     ann_file='data/coco/annotations/instances_val2017.json',
+                     metric='bbox')

configs/finetune_coco/yolo_world_v2_l_vlpan_bn_sgd_1e-3_80e_8gpus_mask-refine_finetune_coco.py

+_base_ = ('../../third_party/mmyolo/configs/yolov8/'
+          'yolov8_l_mask-refine_syncbn_fast_8xb16-500e_coco.py')
+custom_imports = dict(imports=['yolo_world'], allow_failed_imports=False)
+
+# hyper-parameters
+num_classes = 80
+num_training_classes = 80
+max_epochs = 80  # Maximum training epochs
+close_mosaic_epochs = 10
+save_epoch_intervals = 5
+text_channels = 512
+neck_embed_channels = [128, 256, _base_.last_stage_out_channels // 2]
+neck_num_heads = [4, 8, _base_.last_stage_out_channels // 2 // 32]
+base_lr = 1e-3
+weight_decay = 0.0005
+train_batch_size_per_gpu = 16
+load_from = 'pretrained_models/yolo_world_l_clip_t2i_bn_2e-3adamw_32xb16-100e_obj365v1_goldg_cc3mlite_train-ca93cd1f.pth'
+text_model_name = '../pretrained_models/clip-vit-base-patch32-projection'
+# text_model_name = 'openai/clip-vit-base-patch32'
+persistent_workers = False
+
+# model settings
+model = dict(type='YOLOWorldDetector',
+             mm_neck=True,
+             num_train_classes=num_training_classes,
+             num_test_classes=num_classes,
+             data_preprocessor=dict(type='YOLOWDetDataPreprocessor'),
+             backbone=dict(_delete_=True,
+                           type='MultiModalYOLOBackbone',
+                           image_model={{_base_.model.backbone}},
+                           text_model=dict(type='HuggingCLIPLanguageBackbone',
+                                           model_name=text_model_name,
+                                           frozen_modules=['all'])),
+             neck=dict(type='YOLOWorldPAFPN',
+                       guide_channels=text_channels,
+                       embed_channels=neck_embed_channels,
+                       num_heads=neck_num_heads,
+                       block_cfg=dict(type='MaxSigmoidCSPLayerWithTwoConv')),
+             bbox_head=dict(type='YOLOWorldHead',
+                            head_module=dict(
+                                type='YOLOWorldHeadModule',
+                                use_bn_head=True,
+                                embed_dims=text_channels,
+                                num_classes=num_training_classes)),
+             train_cfg=dict(assigner=dict(num_classes=num_training_classes)))
+
+# dataset settings
+text_transform = [
+    dict(type='RandomLoadText',
+         num_neg_samples=(num_classes, num_classes),
+         max_num_samples=num_training_classes,
+         padding_to_max=True,
+         padding_value=''),
+    dict(type='mmdet.PackDetInputs',
+         meta_keys=('img_id', 'img_path', 'ori_shape', 'img_shape', 'flip',
+                    'flip_direction', 'texts'))
+]
+mosaic_affine_transform = [
+    dict(type='MultiModalMosaic',
+         img_scale=_base_.img_scale,
+         pad_val=114.0,
+         pre_transform=_base_.pre_transform),
+    dict(type='YOLOv5CopyPaste', prob=_base_.copypaste_prob),
+    dict(
+        type='YOLOv5RandomAffine',
+        max_rotate_degree=0.0,
+        max_shear_degree=0.0,
+        max_aspect_ratio=100.,
+        scaling_ratio_range=(1 - _base_.affine_scale, 1 + _base_.affine_scale),
+        # img_scale is (width, height)
+        border=(-_base_.img_scale[0] // 2, -_base_.img_scale[1] // 2),
+        border_val=(114, 114, 114),
+        min_area_ratio=_base_.min_area_ratio,
+        use_mask_refine=_base_.use_mask2refine)
+]
+train_pipeline = [
+    *_base_.pre_transform, *mosaic_affine_transform,
+    dict(type='YOLOv5MultiModalMixUp',
+         prob=_base_.mixup_prob,
+         pre_transform=[*_base_.pre_transform, *mosaic_affine_transform]),
+    *_base_.last_transform[:-1], *text_transform
+]
+train_pipeline_stage2 = [*_base_.train_pipeline_stage2[:-1], *text_transform]
+coco_train_dataset = dict(_delete_=True,
+                          type='MultiModalDataset',
+                          dataset=dict(
+                              type='YOLOv5CocoDataset',
+                              data_root='data/coco',
+                              ann_file='annotations/instances_train2017.json',
+                              data_prefix=dict(img='train2017/'),
+                              filter_cfg=dict(filter_empty_gt=False,
+                                              min_size=32)),
+                          class_text_path='data/texts/coco_class_texts.json',
+                          pipeline=train_pipeline)
+
+train_dataloader = dict(persistent_workers=persistent_workers,
+                        batch_size=train_batch_size_per_gpu,
+                        collate_fn=dict(type='yolow_collate'),
+                        dataset=coco_train_dataset)
+test_pipeline = [
+    *_base_.test_pipeline[:-1],
+    dict(type='LoadText'),
+    dict(type='mmdet.PackDetInputs',
+         meta_keys=('img_id', 'img_path', 'ori_shape', 'img_shape',
+                    'scale_factor', 'pad_param', 'texts'))
+]
+coco_val_dataset = dict(
+    _delete_=True,
+    type='MultiModalDataset',
+    dataset=dict(type='YOLOv5CocoDataset',
+                 data_root='data/coco',
+                 ann_file='annotations/instances_val2017.json',
+                 data_prefix=dict(img='val2017/'),
+                 filter_cfg=dict(filter_empty_gt=False, min_size=32)),
+    class_text_path='data/texts/coco_class_texts.json',
+    pipeline=test_pipeline)
+val_dataloader = dict(dataset=coco_val_dataset)
+test_dataloader = val_dataloader
+# training settings
+default_hooks = dict(param_scheduler=dict(scheduler_type='linear',
+                                          lr_factor=0.01,
+                                          max_epochs=max_epochs),
+                     checkpoint=dict(max_keep_ckpts=-1,
+                                     save_best=None,
+                                     interval=save_epoch_intervals))
+custom_hooks = [
+    dict(type='EMAHook',
+         ema_type='ExpMomentumEMA',
+         momentum=0.0001,
+         update_buffers=True,
+         strict_load=False,
+         priority=49),
+    dict(type='mmdet.PipelineSwitchHook',
+         switch_epoch=max_epochs - close_mosaic_epochs,
+         switch_pipeline=train_pipeline_stage2)
+]
+train_cfg = dict(max_epochs=max_epochs,
+                 val_interval=5,
+                 dynamic_intervals=[((max_epochs - close_mosaic_epochs),
+                                     _base_.val_interval_stage2)])
+optim_wrapper = dict(optimizer=dict(
+    _delete_=True,
+    type='SGD',
+    lr=base_lr,
+    momentum=0.937,
+    nesterov=True,
+    weight_decay=weight_decay,
+    batch_size_per_gpu=train_batch_size_per_gpu),
+                     paramwise_cfg=dict(
+                         custom_keys={
+                             'backbone.text_model': dict(lr_mult=0.01),
+                             'logit_scale': dict(weight_decay=0.0)
+                         }),
+                     constructor='YOLOWv5OptimizerConstructor')
+
+# evaluation settings
+val_evaluator = dict(_delete_=True,
+                     type='mmdet.CocoMetric',
+                     proposal_nums=(100, 1, 10),
+                     ann_file='data/coco/annotations/instances_val2017.json',
+                     metric='bbox')

configs/finetune_coco/yolo_world_v2_m_vlpan_bn_2e-4_80e_8gpus_mask-refine_finetune_coco.py

+_base_ = (
+    '../../third_party/mmyolo/configs/yolov8/'
+    'yolov8_m_mask-refine_syncbn_fast_8xb16-500e_coco.py')
+custom_imports = dict(
+    imports=['yolo_world'],
+    allow_failed_imports=False)
+
+# hyper-parameters
+num_classes = 80
+num_training_classes = 80
+max_epochs = 80  # Maximum training epochs
+close_mosaic_epochs = 10
+save_epoch_intervals = 5
+text_channels = 512
+neck_embed_channels = [128, 256, _base_.last_stage_out_channels // 2]
+neck_num_heads = [4, 8, _base_.last_stage_out_channels // 2 // 32]
+base_lr = 2e-4
+weight_decay = 0.05
+train_batch_size_per_gpu = 16
+load_from = 'pretrained_models/yolo_world_m_clip_t2i_bn_2e-3adamw_32xb16-100e_obj365v1_goldg_train-c6237d5b.pth'
+# text_model_name = '../pretrained_models/clip-vit-base-patch32-projection'
+text_model_name = 'openai/clip-vit-base-patch32'
+persistent_workers = False
+
+# model settings
+model = dict(
+    type='YOLOWorldDetector',
+    mm_neck=True,
+    num_train_classes=num_training_classes,
+    num_test_classes=num_classes,
+    data_preprocessor=dict(type='YOLOWDetDataPreprocessor'),
+    backbone=dict(
+        _delete_=True,
+        type='MultiModalYOLOBackbone',
+        image_model={{_base_.model.backbone}},
+        text_model=dict(
+            type='HuggingCLIPLanguageBackbone',
+            model_name=text_model_name,
+            frozen_modules=['all'])),
+    neck=dict(type='YOLOWorldPAFPN',
+              guide_channels=text_channels,
+              embed_channels=neck_embed_channels,
+              num_heads=neck_num_heads,
+              block_cfg=dict(type='MaxSigmoidCSPLayerWithTwoConv')),
+    bbox_head=dict(type='YOLOWorldHead',
+                   head_module=dict(type='YOLOWorldHeadModule',
+                                    use_bn_head=True,
+                                    embed_dims=text_channels,
+                                    num_classes=num_training_classes)),
+    train_cfg=dict(assigner=dict(num_classes=num_training_classes)))
+
+# dataset settings
+text_transform = [
+    dict(type='RandomLoadText',
+         num_neg_samples=(num_classes, num_classes),
+         max_num_samples=num_training_classes,
+         padding_to_max=True,
+         padding_value=''),
+    dict(type='mmdet.PackDetInputs',
+         meta_keys=('img_id', 'img_path', 'ori_shape', 'img_shape', 'flip',
+                    'flip_direction', 'texts'))
+]
+mosaic_affine_transform = [
+    dict(
+        type='MultiModalMosaic',
+        img_scale=_base_.img_scale,
+        pad_val=114.0,
+        pre_transform=_base_.pre_transform),
+    dict(type='YOLOv5CopyPaste', prob=_base_.copypaste_prob),
+    dict(
+        type='YOLOv5RandomAffine',
+        max_rotate_degree=0.0,
+        max_shear_degree=0.0,
+        max_aspect_ratio=100.,
+        scaling_ratio_range=(1 - _base_.affine_scale,
+                             1 + _base_.affine_scale),
+        # img_scale is (width, height)
+        border=(-_base_.img_scale[0] // 2, -_base_.img_scale[1] // 2),
+        border_val=(114, 114, 114),
+        min_area_ratio=_base_.min_area_ratio,
+        use_mask_refine=_base_.use_mask2refine)
+]
+train_pipeline = [
+    *_base_.pre_transform,
+    *mosaic_affine_transform,
+    dict(
+        type='YOLOv5MultiModalMixUp',
+        prob=_base_.mixup_prob,
+        pre_transform=[*_base_.pre_transform,
+                       *mosaic_affine_transform]),
+    *_base_.last_transform[:-1],
+    *text_transform
+]
+train_pipeline_stage2 = [
+    *_base_.train_pipeline_stage2[:-1],
+    *text_transform
+]
+coco_train_dataset = dict(
+    _delete_=True,
+    type='MultiModalDataset',
+    dataset=dict(
+        type='YOLOv5CocoDataset',
+        data_root='data/coco',
+        ann_file='annotations/instances_train2017.json',
+        data_prefix=dict(img='train2017/'),
+        filter_cfg=dict(filter_empty_gt=False, min_size=32)),
+    class_text_path='data/texts/coco_class_texts.json',
+    pipeline=train_pipeline)
+
+train_dataloader = dict(
+    persistent_workers=persistent_workers,
+    batch_size=train_batch_size_per_gpu,
+    collate_fn=dict(type='yolow_collate'),
+    dataset=coco_train_dataset)
+test_pipeline = [
+    *_base_.test_pipeline[:-1],
+    dict(type='LoadText'),
+    dict(
+        type='mmdet.PackDetInputs',
+        meta_keys=('img_id', 'img_path', 'ori_shape', 'img_shape',
+                   'scale_factor', 'pad_param', 'texts'))
+]
+coco_val_dataset = dict(
+    _delete_=True,
+    type='MultiModalDataset',
+    dataset=dict(
+        type='YOLOv5CocoDataset',
+        data_root='data/coco',
+        ann_file='annotations/instances_val2017.json',
+        data_prefix=dict(img='val2017/'),
+        filter_cfg=dict(filter_empty_gt=False, min_size=32)),
+    class_text_path='data/texts/coco_class_texts.json',
+    pipeline=test_pipeline)
+val_dataloader = dict(dataset=coco_val_dataset)
+test_dataloader = val_dataloader
+# training settings
+default_hooks = dict(
+    param_scheduler=dict(
+        scheduler_type='linear',
+        lr_factor=0.01,
+        max_epochs=max_epochs),
+    checkpoint=dict(
+        max_keep_ckpts=-1,
+        save_best=None,
+        interval=save_epoch_intervals))
+custom_hooks = [
+    dict(
+        type='EMAHook',
+        ema_type='ExpMomentumEMA',
+        momentum=0.0001,
+        update_buffers=True,
+        strict_load=False,
+        priority=49),
+    dict(
+        type='mmdet.PipelineSwitchHook',
+        switch_epoch=max_epochs - close_mosaic_epochs,
+        switch_pipeline=train_pipeline_stage2)
+]
+train_cfg = dict(
+    max_epochs=max_epochs,
+    val_interval=5,
+    dynamic_intervals=[((max_epochs - close_mosaic_epochs),
+                        _base_.val_interval_stage2)])
+optim_wrapper = dict(
+    optimizer=dict(
+        _delete_=True,
+        type='AdamW',
+        lr=base_lr,
+        weight_decay=weight_decay,
+        batch_size_per_gpu=train_batch_size_per_gpu),
+    paramwise_cfg=dict(
+        custom_keys={'backbone.text_model': dict(lr_mult=0.01),
+                     'logit_scale': dict(weight_decay=0.0)}),
+    constructor='YOLOWv5OptimizerConstructor')
+
+# evaluation settings
+val_evaluator = dict(
+    _delete_=True,
+    type='mmdet.CocoMetric',
+    proposal_nums=(100, 1, 10),
+    ann_file='data/coco/annotations/instances_val2017.json',
+    metric='bbox')