Skip to content

GitLab

Commit 6fcdec70 authored by chenych's avatar chenych
Browse files

modify the data and eval in README

parent a0e0b906
Hide whitespace changes
Inline Side-by-side
Showing with 106 additions and 390 deletions
README.md
View file @ 6fcdec70
...@@ -26,7 +26,7 @@ TORCH_MINOR = int(torch.__version__.split('.')[1]) ...@@ -26,7 +26,7 @@ TORCH_MINOR = int(torch.__version__.split('.')[1])
if TORCH_MAJOR == 1 and TORCH_MINOR < 8: if TORCH_MAJOR == 1 and TORCH_MINOR < 8:
from torch._six import container_abcs from torch._six import container_abcs
else: else:
import collections.abc as container_abcs import collections.abc as container_abcs
``` ```
### Docker（方法一） ### Docker（方法一）
...@@ -87,231 +87,10 @@ python -m pip install -e detectron2 ...@@ -87,231 +87,10 @@ python -m pip install -e detectron2
``` ```
## 数据集 ## 数据集
本项目所需数据集较多, 所以提供了项目数据集需求较多, 可以使用提供的[a toy training dataset](https://huggingface.co/BAAI/Painter/blob/main/toy_datasets.tar)数据集来验证功能, 数据集由每个类别中各10个类别组成. 将数据集放置于 `$Painter_ROOT/toy_datasets` 路径下, 并设置`$Painter_ROOT/train_painter_vit_large.sh` 中 `DATA_PATH=toy_datasets`即可, 其他参数请参考训练章节的介绍. 本项目所需数据集较多, 所以提供了项目数据集需求较多, 可以使用提供的[a toy training dataset](https://huggingface.co/BAAI/Painter/blob/main/toy_datasets.tar)数据集来验证部分功能, 数据集由每个类别中各10个类别组成. 将数据集放置于 `$Painter_ROOT/toy_datasets` 路径下, 并设置`$Painter_ROOT/train_painter_vit_large.sh` 中 `DATA_PATH=toy_datasets`即可, 其他参数请参考训练章节的介绍.
如需完整数据集, 请参考以下步骤： 如需完整数据集, 请参考[data instructions](docs/DATA.md).
### 完整数据集所需环境配置 完整数据集的目录结构如下：
#### COCO Panoptic Segmentation
```bash
pip install openmim #（0.3.9）
mim install mmcv-full # 注意版本是不是1.7.1
pip install mmdet==2.26.0 # 对应 mmcv-1.7.1
```
#### COCO Pose Estimation
pip install mmcv==1.3.9
pip install mmpose==0.29.0
或者也可以直接采用源码安装mmpose
```bash
# choose commit id `8c58a18b`
git clone https://github.com/open-mmlab/mmpose.git
cd mmpose
pip install -r requirements.txt
pip install -v -e .
```
### 完整数据集下载
#### NYU Depth V2
首先, 下载数据集[here](https://drive.google.com/file/d/1AysroWpfISmm-yRFGBgFTrLy6FjQwvwP/view?usp=sharing). 确保将下载的数据集存放到 `$Painter_ROOT/datasets/nyu_depth_v2/sync.zip`
接下来准备NYU_Depth_V2测试集[NYU Depth V2 test](https://cs.nyu.edu/~silberman/datasets/nyu_depth_v2.html).
```bash
# 下载原始 NYU Depth V2 split file
wget -P datasets/nyu_depth_v2/ http://horatio.cs.nyu.edu/mit/silberman/nyu_depth_v2/nyu_depth_v2_labeled.mat
# 将 mat 数据转换成 image files
python data/depth/extract_official_train_test_set_from_mat.py datasets/nyu_depth_v2/nyu_depth_v2_labeled.mat data/depth/splits.mat datasets/nyu_depth_v2/official_splits/
```
最后, 准备训练和验证所需json数据, 生成的json数据将会默认保存到 `$Painter_ROOT/datasets/nyu_depth_v2/` 路径下.
```bash
python data/depth/gen_json_nyuv2_depth.py --split sync
python data/depth/gen_json_nyuv2_depth.py --split test
```
#### ADE20k Semantic Segmentation
首先, 下载数据集 [official website](https://groups.csail.mit.edu/vision/datasets/ADE20K/), 将下载的数据集存放到 `$Painter_ROOT/datasets/`.
接下来, 解压 zip 文件并重命名为`ade20k`. 完成后的 ade20k 文件结构如下所示:
```bash
ade20k/
images/
annotations/
```
第二, 执行下面的命令准备训练和验证所需的 annotations, 生成的 annotations 将会默认保存到 `$Painter_ROOT/datasets/ade20k/annotations_with_color/` 路径下.
```bash
python data/ade20k/gen_color_ade20k_sem.py --split training
python data/ade20k/gen_color_ade20k_sem.py --split validation
```
第三, 准备训练和验证所需json文件, 生成的json数据将会默认保存到 `$Painter_ROOT/datasets/ade20k/` 路径下.
```bash
python data/ade20k/gen_json_ade20k_sem.py --split training
python data/ade20k/gen_json_ade20k_sem.py --split validation
```
最后, 为了确认能通过 detectron2 进行验证, 创建 `$Painter_ROOT/datasets/ade20k` to `$Painter_ROOT/datasets/ADEChallengeData2016` 的软连接, 然后执行下面的操作:
```bash
# 创建软连接
# ln -s $Painter_ROOT/datasets/ade20k datasets/ADEChallengeData2016
# 执行
python data/prepare_ade20k_sem_seg.py
```
#### COCO Panoptic Segmentation
下载COCO2017数据和相应的全视分割标注. 完成后的文件结构如下所示:
```
coco/
train2017/
val2017/
annotations/
instances_train2017.json
instances_val2017.json
panoptic_train2017.json
panoptic_val2017.json
panoptic_train2017/
panoptic_val2017/
```
1. 准备 COCO Semantic Segmentation
准备训练所需的annotations, 生成的annotations默认保存到 `$Painter_ROOT/datasets/coco/pano_sem_seg/` 路径下.
```bash
python data/coco_semseg/gen_color_coco_panoptic_segm.py --split train2017
python data/coco_semseg/gen_color_coco_panoptic_segm.py --split val2017
```
准备训练和验证所需的json数据, 生成的json数据默认保存到 `$Painter_ROOT/datasets/coco/pano_sem_seg/` 路径下.
```bash
python data/coco_semseg/gen_json_coco_panoptic_segm.py --split train2017
python data/coco_semseg/gen_json_coco_panoptic_segm.py --split val2017
```
2. 准备 COCO Class-Agnostic Instance Segmentation
第一步, 通过下面的命令对数据进行预处理, 生成的 painted ground truth 将会默认保存到 `$Painter_ROOT/datasets/coco/pano_ca_inst` 路径下.
```bash
cd $Painter_ROOT/data/mmdet_custom
# 为实例分割生成使用通用数据增强的训练数据, 注意我们通过在configs/coco_panoptic_ca_inst_gen_augg.py中交替生成30个副本train_aug{idx}
./tools/dist_train.sh configs/coco_panoptic_ca_inst_gen_aug.py 1
# 仅使用水平翻转增强生成训练数据
./tools/dist_train.sh configs/coco_panoptic_ca_inst_gen_orgflip.py 1
# 生成无数据增强的训练数据
./tools/dist_train.sh configs/coco_panoptic_ca_inst_gen_org.py 1
# 生成验证数据(无数据增强)
./tools/dist_test.sh configs/coco_panoptic_ca_inst_gen_org.py none 1 --eval segm
```
然后, 准备训练和验证所需json文件. 生成的json文件将会默认保存到 `$Painter_ROOT/datasets/coco/pano_ca_inst` 路径下.
```bash
cd $Painter_ROOT
python data/mmdet_custom/gen_json_coco_panoptic_inst.py --split train
python data/mmdet_custom/gen_json_coco_panoptic_inst.py --split val
```
最后, 为了确保使用detectron2进行验证, 创建`$Painter_ROOT/datasets/coco/annotations/panoptic_val2017` to `$Painter_ROOT/datasets/coco/panoptic_val2017` 的软连接并运行:
```bash
# 创建软连接
# ln -s $Painter_ROOT/datasets/coco/annotations/panoptic_val2017 datasets/coco/panoptic_val2017
# 执行
python data/prepare_coco_semantic_annos_from_panoptic_annos.py
```
#### COCO Human Pose Estimation
首先, 下载COCO val2017的行人检测结果 [google drive](https://drive.google.com/drive/folders/1fRUDNUDxe9fjqcRZ2bnF_TKMlO0nB_dk), 将下载的数据放入 `$Painter_ROOT/datasets/coco_pose/` 路径下
然后, 通过下面的命令对数据进行预处理, 得到的 painted ground truth 默认保存到 `$Painter_ROOT/datasets/coco_pose/` 路径下.
```bash
cd $Painter_ROOT/data/mmpose_custom
# 生成用于姿态估计的通用数据增强的训练数据, 本项目生成20个副本用于训练, 需要对coco_256x192_gendata.py中52行的aug_idx参数进行对应数量修改,当前默认为0
./tools/dist_train.sh configs/coco_256x192_gendata.py 1
# 生成训练期间验证的数据
./tools/dist_test.sh configs/coco_256x192_gendata.py none 1
# 生成用于测试的数据(使用离线盒子)
./tools/dist_test.sh configs/coco_256x192_gendata_test.py none 1
# 生成用于测试的数据(使用离线盒子+翻转)
./tools/dist_test.sh configs/coco_256x192_gendata_testflip.py none 1
```
接着, 准备训练和验证所需json文件. 生成的json文件将会默认保存到 `datasets/pano_ca_inst/` 路径下.
```bash
cd $Painter_ROOT
python data/mmpose_custom/gen_json_coco_pose.py --split train
python data/mmpose_custom/gen_json_coco_pose.py --split val
```
#### Low-level Vision Tasks
##### Deraining
参考[MPRNet](https://github.com/swz30/MPRNet) 进行deraining的数据准备.
跟随[MPRNet](https://github.com/swz30/MPRNet/blob/main/Deraining/Datasets/README.md)的指令步骤下载数据集, 将下载的数据集保存到 `$Painter_ROOT/datasets/derain/`. 完成后的 Derain 文件结构如下所示:
```
derain/
train/
input/
target/
test/
Rain100H/
Rain100L/
Test100/
Test1200/
Test2800/
```
接着, 通过下面的命令, 准备训练和验证所需json文件. 生成的json文件将保存到 `datasets/derain/` 路径下.
```bash
python data/derain/gen_json_rain.py --split train
python data/derain/gen_json_rain.py --split val
```
### Denoising
参考[Uformer](https://github.com/ZhendongWang6/Uformer)准备SIDD denoising数据集.
针对训练用的SIDD数据集, 可从[official url](https://www.eecs.yorku.ca/~kamel/sidd/dataset.php)中下载SIDD-Medium dataset数据;
针对验证用的SIDD数据集. 可以从[here](https://mailustceducn-my.sharepoint.com/:f:/g/personal/zhendongwang_mail_ustc_edu_cn/Ev832uKaw2JJhwROKqiXGfMBttyFko_zrDVzfSbFFDoi4Q?e=S3p5hQ)下载.
接下来, 使用以下命令生成用于训练的图像补丁:
```bash
python data/sidd/generate_patches_SIDD.py --src_dir datasets/denoise/SIDD_Medium_Srgb/Data --tar_dir datasets/denoise/train
```
最后, 准备训练和验证所需json文件, 生成的json文件将保存在 `datasets/denoise/` 路径下.
```bash
python data/sidd/gen_json_sidd.py --split train
python data/sidd/gen_json_sidd.py --split val
```
### Low-Light Image Enhancement
首先, 下载 LOL 数据集 [google drive](https://drive.google.com/file/d/157bjO1_cFuSd0HWDUuAmcHRJDVyWpOxB/view), 将下载的数据集存放到 `$Painter_ROOT/datasets/light_enhance/` 路径下. 完成后的 LOL 文件结构如下所示:
```
light_enhance/
our485/
low/
high/
eval15/
low/
high/
```
准备训练和验证所需json文件, 生成的json文件将保存在 `$Painter_ROOTdatasets/light_enhance/` 路径下.
```
python data/lol/gen_json_lol.py --split train
python data/lol/gen_json_lol.py --split val
```
数据集的目录结构如下：
``` ```
├── nyu_depth_v2/ ├── nyu_depth_v2/
...@@ -415,38 +194,46 @@ bash train_painter_vit_large.sh ...@@ -415,38 +194,46 @@ bash train_painter_vit_large.sh
``` ```
## 推理 ## 推理
下载推理模型[🤗 Hugging Face Models](https://huggingface.co/BAAI/Painter/blob/main/painter_vit_large.pth), 或者准备好自己的待测试模型, 各个任务的推理方法如下： 1. 下载推理模型[🤗 Hugging Face Models](https://huggingface.co/BAAI/Painter/blob/main/painter_vit_large.pth), 或者准备好自己的待测试模型
2. 部分测试无法使用toy_datasets进行验证，如果使用toy_datasets数据集进行推理, 需确认默认图片是否存在, 如不存在请修改对应参数, 如
`eval/nyuv2_depth/eval.sh` 中的 `PROMPT="study_room_0005b/rgb_00094"`, rgb_00094图片不存在于toy_datasets数据集中，请对应修改为toy_datasets中的图片名，如`PROMPT="study_room_0005b/rgb_00092"`
各个数据集推理方法如下:
### NYU Depth V2 ### NYU Depth V2
首先设置 `$Painter_ROOT/eval/nyuv2_depth/eval.sh` 文件里的 `JOB_NAME` 和 `DATA_DIR` 参数, 然后执行下面的命令: 设置 `$Painter_ROOT/eval/nyuv2_depth/eval.sh` 文件里的 `JOB_NAME`、`PROMPT`、`CKPT_FILE` 和 `DATA_DIR` 参数:
```bash ```bash
bash eval/nyuv2_depth/eval.sh bash eval/nyuv2_depth/eval.sh
``` ```
### ADE20k Semantic Segmentation ### ADE20k Semantic Segmentation
首先设置 `$Painter_ROOT/eval/ade20k_semantic/eval.sh` 文件里的 `JOB_NAME` 参数, 然后执行下面的命令: 1. 无法使用toy_datasets进行验证;
2. 设置 `$Painter_ROOT/eval/ade20k_semantic/eval.sh` 文件里的 `JOB_NAME`、`PROMPT` 参数, 执行下面的命令:
```bash ```bash
bash eval/ade20k_semantic/eval.sh bash eval/ade20k_semantic/eval.sh
``` ```
### COCO Panoptic Segmentation ### COCO Panoptic Segmentation
首先设置 `$Painter_ROOT/eval/coco_panoptic/eval.sh` 文件里的 `JOB_NAME` 参数, 然后执行下面的命令: 1. 无法使用toy_datasets进行验证;
2. 设置 `$Painter_ROOT/eval/coco_panoptic/eval.sh` 文件里的 `JOB_NAME`、`PROMPT` 参数, 然后执行下面的命令:
```bash ```bash
bash eval/coco_panoptic/eval.sh bash eval/coco_panoptic/eval.sh
``` ```
### COCO Human Pose Estimation ### COCO Human Pose Estimation
为了评估Painter对COCO姿态的估计, 首先生成验证所需的图像: 1. 无法使用toy_datasets进行验证;
2. 生成验证所需的图像:
```bash ```bash
python -m torch.distributed.launch --nproc_per_node=8 --master_port=29500 --use_env eval/mmpose_custom/painter_inference_pose.py --ckpt_path models/painter_vit_large/painter_vit_large.pth python -m torch.distributed.launch --nproc_per_node=4 --master_port=29500 --use_env eval/mmpose_custom/painter_inference_pose.py --ckpt_path models/painter_vit_large/painter_vit_large.pth
python -m torch.distributed.launch --nproc_per_node=8 --master_port=29500 --use_env eval/mmpose_custom/painter_inference_pose.py --ckpt_path models/painter_vit_large/painter_vit_large.pth --flip_test python -m torch.distributed.launch --nproc_per_node=4 --master_port=29500 --use_env eval/mmpose_custom/painter_inference_pose.py --ckpt_path models/painter_vit_large/painter_vit_large.pth --flip_test
``` ```
接下来, 修改 `$Painter_ROOT/eval/mmpose_custom/configs/coco_256x192_test_offline.py` 文件中的 `job_name`、`data_root`、`bbox_file` 和 `ckpt_file` 参数, 执行: 3. 修改 `$Painter_ROOT/eval/mmpose_custom/configs/coco_256x192_test_offline.py` 文件中的 `job_name`、`data_root`、`bbox_file` 和 `ckpt_file` 参数, 执行:
```bash ```bash
cd $Painter_ROOT/eval/mmpose_custom cd $Painter_ROOT/eval/mmpose_custom
./tools/dist_test.sh configs/coco_256x192_test_offline.py none 1 --eval mAP ./tools/dist_test.sh configs/coco_256x192_test_offline.py none 1 --eval mAP
...@@ -454,12 +241,12 @@ cd $Painter_ROOT/eval/mmpose_custom ...@@ -454,12 +241,12 @@ cd $Painter_ROOT/eval/mmpose_custom
### Low-Light Image Enhancement ### Low-Light Image Enhancement
To evaluate Painter on LoL image enhancement: 执行
```bash ```bash
python eval/lol/painter_inference_lol.py --ckpt_path models/path/of/painter_vit_large.pth --data_dir path/of/datasets python eval/lol/painter_inference_lol.py --ckpt_path models/path/of/painter_vit_large.pth --data_dir path/of/datasets
Example: Example:
python eval/lol/painter_inference_lol.py --ckpt_path models/painter_vit_large.pth --data_dir toy_datasets python eval/lol/painter_inference_lol.py --ckpt_path models/painter_vit_large.pth --data_dir datasets
``` ```
## result ## result
......
docs/DATA.md
View file @ 6fcdec70
# Prepare datasets for Painter # 准备Painter所需数据集
The training of our model uses [COCO](https://cocodataset.org/), [ADE20K](https://groups.csail.mit.edu/vision/datasets/ADE20K/), [NYUDepthV2](https://cs.nyu.edu/~silberman/datasets/nyu_depth_v2.html), [Synthetic Rain Datasets](https://paperswithcode.com/dataset/synthetic-rain-datasets), [SIDD](https://www.eecs.yorku.ca/~kamel/sidd/), and [LoL](https://daooshee.github.io/BMVC2018website/) datasets. 训练所需的数据集有: [COCO](https://cocodataset.org/), [ADE20K-官方](https://groups.csail.mit.edu/vision/datasets/ADE20K/), [NYUDepthV2](https://cs.nyu.edu/~silberman/datasets/nyu_depth_v2.html), [Synthetic Rain Datasets](https://paperswithcode.com/dataset/synthetic-rain-datasets), [SIDD](https://www.eecs.yorku.ca/~kamel/sidd/), 和 [LoL](https://daooshee.github.io/BMVC2018website/).
After processing, the datasets should look like:
### 数据集准备所需环境配置
#### COCO Panoptic Segmentation
```bash
pip install openmim #（0.3.9）
mim install mmcv-full # 注意版本是不是1.7.1
pip install mmdet==2.26.0 # 对应 mmcv-1.7.1
``` ```
$Painter_ROOT/datasets/
nyu_depth_v2/
sync/
official_splits/
nyu_depth_v2_labeled.mat
datasets/nyu_depth_v2/
nyuv2_sync_image_depth.json # generated
nyuv2_test_image_depth.json # generated
ade20k/
images/
annotations/
annotations_detectron2/ # generated
annotations_with_color/ # generated
ade20k_training_image_semantic.json # generated
ade20k_validation_image_semantic.json # generated
ADEChallengeData2016/ # sim-link to $Painter_ROOT/datasets/ade20k
coco/
train2017/
val2017/
annotations/
instances_train2017.json
instances_val2017.json
person_keypoints_val2017.json
panoptic_train2017.json
panoptic_val2017.json
panoptic_train2017/
panoptic_val2017/
panoptic_semseg_val2017/ # generated
panoptic_val2017/ # sim-link to $Painter_ROOT/datasets/coco/annotations/panoptic_val2017
pano_sem_seg/ # generated
panoptic_segm_train2017_with_color
panoptic_segm_val2017_with_color
coco_train2017_image_panoptic_sem_seg.json
coco_val2017_image_panoptic_sem_seg.json
pano_ca_inst/ # generated
train_aug0/
train_aug1/
...
train_aug29/
train_org/
train_flip/
val_org/
coco_train_image_panoptic_inst.json
coco_val_image_panoptic_inst.json
coco_pose/
person_detection_results/
COCO_val2017_detections_AP_H_56_person.json
data_pair/ # generated
train_256x192_aug0/
train_256x192_aug1/
...
train_256x192_aug19/
val_256x192/
test_256x192/
test_256x192_flip/
coco_pose_256x192_train.json # generated
coco_pose_256x192_val.json # generated
derain/
train/
input/
target/
test/
Rain100H/
Rain100L/
Test100/
Test1200/
Test2800/
derain_train.json
derain_test_rain100h.json
denoise/
SIDD_Medium_Srgb/
train/
val/
denoise_ssid_train.json # generated
denoise_ssid_val.json # generated
light_enhance/
our485/
low/
high/
eval15/
low/
high/
enhance_lol_train.json # generated
enhance_lol_val.json # generated
#### COCO Pose Estimation
pip install mmcv==1.3.9
pip install mmpose==0.29.0
或者也可以直接采用源码安装mmpose
```bash
# choose commit id `8c58a18b`
git clone https://github.com/open-mmlab/mmpose.git
cd mmpose
pip install -r requirements.txt
pip install -v -e .
``` ```
Please follow the following instruction to pre-process individual datasets.
### 完整数据集下载
#### NYU Depth V2
## NYU Depth V2 首先, 下载数据集[here](https://drive.google.com/file/d/1AysroWpfISmm-yRFGBgFTrLy6FjQwvwP/view?usp=sharing). 确保将下载的数据集存放到 `$Painter_ROOT/datasets/nyu_depth_v2/sync.zip`
First, download the dataset from [here](https://drive.google.com/file/d/1AysroWpfISmm-yRFGBgFTrLy6FjQwvwP/view?usp=sharing). Please make sure to locate the downloaded file to `$Painter_ROOT/datasets/nyu_depth_v2/sync.zip` 接下来准备NYU_Depth_V2测试集[NYU Depth V2 test](https://cs.nyu.edu/~silberman/datasets/nyu_depth_v2.html).
Next, prepare [NYU Depth V2](https://cs.nyu.edu/~silberman/datasets/nyu_depth_v2.html) test set.
```bash ```bash
# get official NYU Depth V2 split file # 下载原始 NYU Depth V2 split file
wget -P datasets/nyu_depth_v2/ http://horatio.cs.nyu.edu/mit/silberman/nyu_depth_v2/nyu_depth_v2_labeled.mat wget -P datasets/nyu_depth_v2/ http://horatio.cs.nyu.edu/mit/silberman/nyu_depth_v2/nyu_depth_v2_labeled.mat
# convert mat file to image files # 将 mat 数据转换成 image files
python data/depth/extract_official_train_test_set_from_mat.py datasets/nyu_depth_v2/nyu_depth_v2_labeled.mat data/depth/splits.mat datasets/nyu_depth_v2/official_splits/ python data/depth/extract_official_train_test_set_from_mat.py datasets/nyu_depth_v2/nyu_depth_v2_labeled.mat data/depth/splits.mat datasets/nyu_depth_v2/official_splits/
``` ```
Lastly, prepare json files for training and evaluation. The generated json files will be saved at `$Painter_ROOT/datasets/nyu_depth_v2/`. 最后, 准备训练和验证所需json数据, 生成的json数据将会默认保存到 `$Painter_ROOT/datasets/nyu_depth_v2/` 路径下.
```bash ```bash
python data/depth/gen_json_nyuv2_depth.py --split sync python data/depth/gen_json_nyuv2_depth.py --split sync
python data/depth/gen_json_nyuv2_depth.py --split test python data/depth/gen_json_nyuv2_depth.py --split test
``` ```
## ADE20k Semantic Segmentation #### ADE20k Semantic Segmentation
First, download the dataset from the [official website](https://groups.csail.mit.edu/vision/datasets/ADE20K/), and put it in `$Painter_ROOT/datasets/`. Afterward, unzip the zip file and rename the target folder as `ade20k`. The ADE20k folder should look like: 1. 下载数据集: [ADEChallengeData2016](https://aistudio.baidu.com/datasetdetail/54455);
```
2. 将下载的数据集存放到 `$Painter_ROOT/datasets/`;
3. 解压文件并重命名为`ade20k`. 完成后的 ade20k 文件结构如下所示:
```bash
ade20k/ ade20k/
images/ images/
annotations/ annotations/
``` ```
Second, prepare annotations for training using the following command. The generated annotations will be saved at `$Painter_ROOT/datasets/ade20k/annotations_with_color/`. 4. 执行下面的命令准备训练和验证所需的 annotations, 生成的 annotations 将会默认保存到 `$Painter_ROOT/datasets/ade20k/annotations_with_color/` 路径下.
```bash ```bash
python data/ade20k/gen_color_ade20k_sem.py --split training python data/ade20k/gen_color_ade20k_sem.py --split training
python data/ade20k/gen_color_ade20k_sem.py --split validation python data/ade20k/gen_color_ade20k_sem.py --split validation
``` ```
Third, prepare json files for training and evaluation. The generated json files will be saved at `$Painter_ROOT/datasets/ade20k/`. 5. 准备训练和验证所需json文件, 生成的json数据将会默认保存到 `$Painter_ROOT/datasets/ade20k/` 路径下.
```bash ```bash
python data/ade20k/gen_json_ade20k_sem.py --split training python data/ade20k/gen_json_ade20k_sem.py --split training
python data/ade20k/gen_json_ade20k_sem.py --split validation python data/ade20k/gen_json_ade20k_sem.py --split validation
``` ```
Lastly, to enable evaluation with detectron2, link `$Painter_ROOT/datasets/ade20k` to `$Painter_ROOT/datasets/ADEChallengeData2016` and run: 6. 为了确认能通过 detectron2 进行验证, 创建 `$Painter_ROOT/datasets/ade20k` to `$Painter_ROOT/datasets/ADEChallengeData2016` 的软连接, 然后执行下面的操作:
```bash ```bash
# 创建软连接, 注意, 一定是datasets下面创建ADEChallengeData2016
# ln -s $Painter_ROOT/datasets/ade20k datasets/ADEChallengeData2016 # ln -s $Painter_ROOT/datasets/ade20k datasets/ADEChallengeData2016
# 执行
python data/prepare_ade20k_sem_seg.py python data/prepare_ade20k_sem_seg.py
``` ```
## COCO Panoptic Segmentation #### COCO Panoptic Segmentation
Download the COCO2017 dataset and the corresponding panoptic segmentation annotation. The COCO folder should look like: 下载COCO2017数据和相应的全视分割标注. 完成后的文件结构如下所示:
``` ```
coco/ coco/
train2017/ train2017/
...@@ -154,88 +94,84 @@ coco/ ...@@ -154,88 +94,84 @@ coco/
panoptic_val2017/ panoptic_val2017/
``` ```
### Prepare Data for COCO Semantic Segmentation 1. 准备 COCO Semantic Segmentation
Prepare annotations for training using the following command. The generated annotations will be saved at `$Painter_ROOT/datasets/coco/pano_sem_seg/`. 准备训练所需的annotations, 生成的annotations默认保存到 `$Painter_ROOT/datasets/coco/pano_sem_seg/` 路径下.
```bash ```bash
python data/coco_semseg/gen_color_coco_panoptic_segm.py --split train2017 python data/coco_semseg/gen_color_coco_panoptic_segm.py --split train2017
python data/coco_semseg/gen_color_coco_panoptic_segm.py --split val2017 python data/coco_semseg/gen_color_coco_panoptic_segm.py --split val2017
``` ```
Prepare json files for training and evaluation. The generated json files will be saved at `$Painter_ROOT/datasets/coco/pano_sem_seg/`. 准备训练和验证所需的json数据, 生成的json数据默认保存到 `$Painter_ROOT/datasets/coco/pano_sem_seg/` 路径下.
```bash ```bash
python data/coco_semseg/gen_json_coco_panoptic_segm.py --split train2017 python data/coco_semseg/gen_json_coco_panoptic_segm.py --split train2017
python data/coco_semseg/gen_json_coco_panoptic_segm.py --split val2017 python data/coco_semseg/gen_json_coco_panoptic_segm.py --split val2017
``` ```
### Prepare Data for COCO Class-Agnostic Instance Segmentation 2. 准备 COCO Class-Agnostic Instance Segmentation
First, pre-process the dataset using the following command, the painted ground truth will be saved to `$Painter_ROOT/datasets/coco/pano_ca_inst`. 第一步, 通过下面的命令对数据进行预处理, 生成的 painted ground truth 将会默认保存到 `$Painter_ROOT/datasets/coco/pano_ca_inst` 路径下.
```bash ```bash
cd $Painter_ROOT/data/mmdet_custom cd $Painter_ROOT/data/mmdet_custom
# generate training data with common data augmentation for instance segmentation, # 为实例分割生成使用通用数据增强的训练数据, 注意我们通过在configs/coco_panoptic_ca_inst_gen_augg.py中交替生成30个副本train_aug{idx}
# note we generate 30 copies by alternating train_aug{idx} in configs/coco_panoptic_ca_inst_gen_aug.py
./tools/dist_train.sh configs/coco_panoptic_ca_inst_gen_aug.py 1 ./tools/dist_train.sh configs/coco_panoptic_ca_inst_gen_aug.py 1
# generate training data with only horizontal flip augmentation # 仅使用水平翻转增强生成训练数据
./tools/dist_train.sh configs/coco_panoptic_ca_inst_gen_orgflip.py 1 ./tools/dist_train.sh configs/coco_panoptic_ca_inst_gen_orgflip.py 1
# generate training data w/o data augmentation # 生成无数据增强的训练数据
./tools/dist_train.sh configs/coco_panoptic_ca_inst_gen_org.py 1 ./tools/dist_train.sh configs/coco_panoptic_ca_inst_gen_org.py 1
# 生成验证数据(无数据增强)
# generate validation data (w/o data augmentation)
./tools/dist_test.sh configs/coco_panoptic_ca_inst_gen_org.py none 1 --eval segm ./tools/dist_test.sh configs/coco_panoptic_ca_inst_gen_org.py none 1 --eval segm
``` ```
Next, prepare json files for training and evaluation. The generated json files will be saved at `$Painter_ROOT/datasets/coco/pano_ca_inst`. 然后, 准备训练和验证所需json文件. 生成的json文件将会默认保存到 `$Painter_ROOT/datasets/coco/pano_ca_inst` 路径下.
```bash ```bash
cd $Painter_ROOT cd $Painter_ROOT
python data/mmdet_custom/gen_json_coco_panoptic_inst.py --split train python data/mmdet_custom/gen_json_coco_panoptic_inst.py --split train
python data/mmdet_custom/gen_json_coco_panoptic_inst.py --split val python data/mmdet_custom/gen_json_coco_panoptic_inst.py --split val
``` ```
Lastly, to enable evaluation with detectron2, link `$Painter_ROOT/datasets/coco/annotations/panoptic_val2017` to `$Painter_ROOT/datasets/coco/panoptic_val2017` and run: 最后, 为了确保使用detectron2进行验证, 创建`$Painter_ROOT/datasets/coco/annotations/panoptic_val2017` to `$Painter_ROOT/datasets/coco/panoptic_val2017` 的软连接并运行:
```bash ```bash
# 创建软连接
# ln -s $Painter_ROOT/datasets/coco/annotations/panoptic_val2017 datasets/coco/panoptic_val2017 # ln -s $Painter_ROOT/datasets/coco/annotations/panoptic_val2017 datasets/coco/panoptic_val2017
# 执行
python data/prepare_coco_semantic_annos_from_panoptic_annos.py python data/prepare_coco_semantic_annos_from_panoptic_annos.py
``` ```
#### COCO Human Pose Estimation
## COCO Human Pose Estimation 1. 下载COCO val2017的行人检测结果 [google drive](https://drive.google.com/drive/folders/1fRUDNUDxe9fjqcRZ2bnF_TKMlO0nB_dk), 将下载的数据放入 `$Painter_ROOT/datasets/coco_pose/` 路径下
First, download person detection result of COCO val2017 from [google drive](https://drive.google.com/drive/folders/1fRUDNUDxe9fjqcRZ2bnF_TKMlO0nB_dk), and put it in `$Painter_ROOT/datasets/coco_pose/`
First, pre-process the dataset using the following command, the painted ground truth will be saved to `$Painter_ROOT/datasets/coco_pose/`. 2. 通过下面的命令对数据进行预处理, 得到的 painted ground truth 默认保存到 `$Painter_ROOT/datasets/coco_pose/` 路径下.
```bash ```bash
cd $Painter_ROOT/data/mmpose_custom cd $Painter_ROOT/data/mmpose_custom
# generate training data with common data augmentation for pose estimation, note we generate 20 copies for training # 生成用于姿态估计的通用数据增强的训练数据, 本项目生成20个副本用于训练, 需要对coco_256x192_gendata.py中52行的aug_idx参数进行对应数量修改,当前默认为0
./tools/dist_train.sh configs/coco_256x192_gendata.py 1 ./tools/dist_train.sh configs/coco_256x192_gendata.py 1
# genearte data for eval during training # 生成训练期间验证的数据
./tools/dist_test.sh configs/coco_256x192_gendata.py none 1 ./tools/dist_test.sh configs/coco_256x192_gendata.py none 1
# 生成用于测试的数据(使用离线盒子)
# generate data for testing (using offline boxes)
./tools/dist_test.sh configs/coco_256x192_gendata_test.py none 1 ./tools/dist_test.sh configs/coco_256x192_gendata_test.py none 1
# generate data for testing (using offline boxes & with flip) # 生成用于测试的数据(使用离线盒子+翻转)
./tools/dist_test.sh configs/coco_256x192_gendata_testflip.py none 1 ./tools/dist_test.sh configs/coco_256x192_gendata_testflip.py none 1
``` ```
Next, prepare json files for training and evaluation. The generated json files will be saved at `datasets/pano_ca_inst/`. 接着, 准备训练和验证所需json文件. 生成的json文件将会默认保存到 `datasets/pano_ca_inst/` 路径下.
```bash ```bash
cd $Painter_ROOT cd $Painter_ROOT
python data/mmpose_custom/gen_json_coco_pose.py --split train python data/mmpose_custom/gen_json_coco_pose.py --split train
python data/mmpose_custom/gen_json_coco_pose.py --split val python data/mmpose_custom/gen_json_coco_pose.py --split val
``` ```
#### Low-level Vision Tasks
##### Deraining
参考[MPRNet](https://github.com/swz30/MPRNet) 进行deraining的数据准备.
## Low-level Vision Tasks 跟随[MPRNet](https://github.com/swz30/MPRNet/blob/main/Deraining/Datasets/README.md)的指令步骤下载数据集, 将下载的数据集保存到 `$Painter_ROOT/datasets/derain/`. 完成后的 Derain 文件结构如下所示:
```
### Deraining
We follow [MPRNet](https://github.com/swz30/MPRNet) to prepare the data for deraining.
Download the dataset following the instructions in [MPRNet](https://github.com/swz30/MPRNet/blob/main/Deraining/Datasets/README.md), and put it in `$Painter_ROOT/datasets/derain/`. The folder should look like:
```bash
derain/ derain/
train/ train/
input/ input/
...@@ -248,34 +184,34 @@ derain/ ...@@ -248,34 +184,34 @@ derain/
Test2800/ Test2800/
``` ```
Next, prepare json files for training and evaluation. The generated json files will be saved at `datasets/derain/`. 接着, 通过下面的命令, 准备训练和验证所需json文件. 生成的json文件将保存到 `datasets/derain/` 路径下.
```bash ```bash
python data/derain/gen_json_rain.py --split train python data/derain/gen_json_rain.py --split train
python data/derain/gen_json_rain.py --split val python data/derain/gen_json_rain.py --split val
``` ```
### Denoising ### Denoising
We follow [Uformer](https://github.com/ZhendongWang6/Uformer) to prepare the data for SIDD denoising dataset. 参考[Uformer](https://github.com/ZhendongWang6/Uformer)准备SIDD denoising数据集.
针对训练用的SIDD数据集, 可从[official url](https://www.eecs.yorku.ca/~kamel/sidd/dataset.php)中下载SIDD-Medium dataset数据;
For training data of SIDD, you can download the SIDD-Medium dataset from the [official url](https://www.eecs.yorku.ca/~kamel/sidd/dataset.php). For evaluation on SIDD, you can download data from [here](https://mailustceducn-my.sharepoint.com/:f:/g/personal/zhendongwang_mail_ustc_edu_cn/Ev832uKaw2JJhwROKqiXGfMBttyFko_zrDVzfSbFFDoi4Q?e=S3p5hQ). 针对验证用的SIDD数据集. 可以从[here](https://mailustceducn-my.sharepoint.com/:f:/g/personal/zhendongwang_mail_ustc_edu_cn/Ev832uKaw2JJhwROKqiXGfMBttyFko_zrDVzfSbFFDoi4Q?e=S3p5hQ)下载.
Next, generate image patches for training by the following command: 接下来, 使用以下命令生成用于训练的图像补丁:
```bash ```bash
python data/sidd/generate_patches_SIDD.py --src_dir datasets/denoise/SIDD_Medium_Srgb/Data --tar_dir datasets/denoise/train python data/sidd/generate_patches_SIDD.py --src_dir datasets/denoise/SIDD_Medium_Srgb/Data --tar_dir datasets/denoise/train
``` ```
Lastly, prepare json files for training and evaluation. The generated json files will be saved at `datasets/denoise/`. 最后, 准备训练和验证所需json文件, 生成的json文件将保存在 `datasets/denoise/` 路径下.
```bash ```bash
python data/sidd/gen_json_sidd.py --split train python data/sidd/gen_json_sidd.py --split train
python data/sidd/gen_json_sidd.py --split val python data/sidd/gen_json_sidd.py --split val
``` ```
### Low-Light Image Enhancement ### Low-Light Image Enhancement
首先, 下载 LOL 数据集 [google drive](https://drive.google.com/file/d/157bjO1_cFuSd0HWDUuAmcHRJDVyWpOxB/view), 将下载的数据集存放到 `$Painter_ROOT/datasets/light_enhance/` 路径下. 完成后的 LOL 文件结构如下所示:
First, download images of LOL dataset from [google drive](https://drive.google.com/file/d/157bjO1_cFuSd0HWDUuAmcHRJDVyWpOxB/view) and put it in `$Painter_ROOT/datasets/light_enhance/`. The folder should look like: ```
look like:
```bash
light_enhance/ light_enhance/
our485/ our485/
low/ low/
...@@ -285,9 +221,8 @@ light_enhance/ ...@@ -285,9 +221,8 @@ light_enhance/
high/ high/
``` ```
Next, prepare json files for training and evaluation. The generated json files will be saved at `$Painter_ROOTdatasets/light_enhance/`. 准备训练和验证所需json文件, 生成的json文件将保存在 `$Painter_ROOTdatasets/light_enhance/` 路径下.
```bash ```
python data/lol/gen_json_lol.py --split train python data/lol/gen_json_lol.py --split train
python data/lol/gen_json_lol.py --split val python data/lol/gen_json_lol.py --split val
``` ```
eval/ade20k_semantic/eval.sh
View file @ 6fcdec70
...@@ -12,13 +12,11 @@ MODEL="painter_vit_large_patch16_input896x448_win_dec64_8glb_sl1" ...@@ -12,13 +12,11 @@ MODEL="painter_vit_large_patch16_input896x448_win_dec64_8glb_sl1"
CKPT_PATH="models/${JOB_NAME}/${CKPT_FILE}" CKPT_PATH="models/${JOB_NAME}/${CKPT_FILE}"
DST_DIR="models_inference/${JOB_NAME}/ade20k_semseg_inference_${CKPT_FILE}_${PROMPT}_size${SIZE}" DST_DIR="models_inference/${JOB_NAME}/ade20k_semseg_inference_${CKPT_FILE}_${PROMPT}_size${SIZE}"
DATA_DIR="datasets"
# inference # inference
python -m torch.distributed.launch --nproc_per_node=${NUM_GPUS} --master_port=29504 --use_env \ python -m torch.distributed.launch --nproc_per_node=${NUM_GPUS} --master_port=29504 --use_env \
eval/ade20k_semantic/painter_inference_segm.py \ eval/ade20k_semantic/painter_inference_segm.py \
--model ${MODEL} --prompt ${PROMPT} \ --model ${MODEL} --prompt ${PROMPT} \
--data_dir ${DATA_DIR} \
--ckpt_path ${CKPT_PATH} --input_size ${SIZE} --ckpt_path ${CKPT_PATH} --input_size ${SIZE}
# postprocessing and eval # postprocessing and eval
......
eval/ade20k_semantic/painter_inference_segm.py
View file @ 6fcdec70
...@@ -43,7 +43,6 @@ def get_args_parser(): ...@@ -43,7 +43,6 @@ def get_args_parser():
parser.add_argument('--prompt', type=str, help='prompt image in train set', parser.add_argument('--prompt', type=str, help='prompt image in train set',
default='ADE_train_00014165') default='ADE_train_00014165')
parser.add_argument('--input_size', type=int, default=448) parser.add_argument('--input_size', type=int, default=448)
parser.add_argument('--data_dir', type=str, default="datasets")
# distributed training parameters # distributed training parameters
parser.add_argument('--world_size', default=1, type=int, parser.add_argument('--world_size', default=1, type=int,
help='number of distributed processes') help='number of distributed processes')
...@@ -120,15 +119,15 @@ if __name__ == '__main__': ...@@ -120,15 +119,15 @@ if __name__ == '__main__':
device = torch.device("cuda") device = torch.device("cuda")
model_painter.to(device) model_painter.to(device)
img_src_dir = "{}/ade20k/images/validation".format(args.data_dir) img_src_dir = "datasets/ade20k/images/validation"
# img_path_list = glob.glob(os.path.join(img_src_dir, "*.jpg")) # img_path_list = glob.glob(os.path.join(img_src_dir, "*.jpg"))
dataset_val = DatasetTest(img_src_dir, input_size, ext_list=('*.jpg',)) dataset_val = DatasetTest(img_src_dir, input_size, ext_list=('*.jpg',))
sampler_val = DistributedSampler(dataset_val, shuffle=False) sampler_val = DistributedSampler(dataset_val, shuffle=False)
data_loader_val = DataLoader(dataset_val, batch_size=1, sampler=sampler_val, data_loader_val = DataLoader(dataset_val, batch_size=1, sampler=sampler_val,
drop_last=False, collate_fn=ddp_utils.collate_fn, num_workers=2) drop_last=False, collate_fn=ddp_utils.collate_fn, num_workers=2)
img2_path = "{}/ade20k/images/training/{}.jpg".format(args.data_dir, prompt) img2_path = "datasets/ade20k/images/training/{}.jpg".format(prompt)
tgt2_path = "{}/ade20k/annotations_with_color/training/{}.png".format(args.data_dir, prompt) tgt2_path = "datasets/ade20k/annotations_with_color/training/{}.png".format(prompt)
# load the shared prompt image pair # load the shared prompt image pair
img2 = Image.open(img2_path).convert("RGB") img2 = Image.open(img2_path).convert("RGB")
......
eval/nyuv2_depth/eval_with_pngs.py
View file @ 6fcdec70
...@@ -223,6 +223,3 @@ def main(): ...@@ -223,6 +223,3 @@ def main():
if __name__ == '__main__': if __name__ == '__main__':
main() main()
val.sh
View file @ 6fcdec70
...@@ -4,4 +4,4 @@ export HSA_FORCE_FINE_GRAIN_PCIE=1 ...@@ -4,4 +4,4 @@ export HSA_FORCE_FINE_GRAIN_PCIE=1
export USE_MIOPEN_BATCHNORM=1 export USE_MIOPEN_BATCHNORM=1
# LOL datasets # LOL datasets
python eval/lol/painter_inference_lol.py --ckpt_path pretrained_models/painter_vit_large.pth --data_dir toy_datasets python eval/lol/painter_inference_lol.py --ckpt_path pretrained_models/painter_vit_large.pth --data_dir toy_datasets --prompt 193
\ No newline at end of file \ No newline at end of file
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment