Merge branch 'hepj_work' into 'main'

增加conformer代码

See merge request dcutoolkit/deeplearing/dlexamples_new!7

PyTorch/NLP/Conformer-main/mmdetection/configs/mask_rcnn/mask_rcnn_x101_64x4d_fpn_2x_coco.py

+_base_ = './mask_rcnn_x101_32x4d_fpn_2x_coco.py'
+model = dict(
+    pretrained='open-mmlab://resnext101_64x4d',
+    backbone=dict(
+        type='ResNeXt',
+        depth=101,
+        groups=64,
+        base_width=4,
+        num_stages=4,
+        out_indices=(0, 1, 2, 3),
+        frozen_stages=1,
+        norm_cfg=dict(type='BN', requires_grad=True),
+        style='pytorch'))

PyTorch/NLP/Conformer-main/mmdetection/configs/paa/README.md

+# Probabilistic Anchor Assignment with IoU Prediction for Object Detection
+
+[ALGORITHM]
+
+## Results and Models
+
+We provide config files to reproduce the object detection results in the
+ECCV 2020 paper for Probabilistic Anchor Assignment with IoU
+Prediction for Object Detection.
+
+| Backbone    | Lr schd | Mem (GB) | Score voting | box AP | Config | Download |
+|:-----------:|:-------:|:--------:|:------------:|:------:|:------:|:--------:|
+| R-50-FPN    | 12e     | 3.7     | True          | 40.4   | [config](https://github.com/open-mmlab/mmdetection/tree/master/configs/paa/paa_r50_fpn_1x_coco.py) | [model](http://download.openmmlab.com/mmdetection/v2.0/paa/paa_r50_fpn_1x_coco/paa_r50_fpn_1x_coco_20200821-936edec3.pth) | [log](http://download.openmmlab.com/mmdetection/v2.0/paa/paa_r50_fpn_1x_coco/paa_r50_fpn_1x_coco_20200821-936edec3.log.json) |
+| R-50-FPN    | 12e     | 3.7     | False         | 40.2   | - |
+| R-50-FPN    | 18e     | 3.7     | True          | 41.4   | [config](https://github.com/open-mmlab/mmdetection/tree/master/configs/paa/paa_r50_fpn_1.5x_coco.py) | [model](http://download.openmmlab.com/mmdetection/v2.0/paa/paa_r50_fpn_1.5x_coco/paa_r50_fpn_1.5x_coco_20200823-805d6078.pth) | [log](http://download.openmmlab.com/mmdetection/v2.0/paa/paa_r50_fpn_1.5x_coco/paa_r50_fpn_1.5x_coco_20200823-805d6078.log.json) |
+| R-50-FPN    | 18e     | 3.7     | False         | 41.2   | - |
+| R-50-FPN    | 24e     | 3.7     | True          | 41.6   | [config](https://github.com/open-mmlab/mmdetection/tree/master/configs/paa/paa_r50_fpn_2x_coco.py) | [model](http://download.openmmlab.com/mmdetection/v2.0/paa/paa_r50_fpn_2x_coco/paa_r50_fpn_2x_coco_20200821-c98bfc4e.pth) | [log](http://download.openmmlab.com/mmdetection/v2.0/paa/paa_r50_fpn_2x_coco/paa_r50_fpn_2x_coco_20200821-c98bfc4e.log.json) |
+| R-101-FPN   | 12e     | 6.2     | True          | 42.6   | [config](https://github.com/open-mmlab/mmdetection/tree/master/configs/paa/paa_r101_fpn_1x_coco.py) | [model](http://download.openmmlab.com/mmdetection/v2.0/paa/paa_r101_fpn_1x_coco/paa_r101_fpn_1x_coco_20200821-0a1825a4.pth) | [log](http://download.openmmlab.com/mmdetection/v2.0/paa/paa_r101_fpn_1x_coco/paa_r101_fpn_1x_coco_20200821-0a1825a4.log.json) |
+| R-101-FPN   | 12e     | 6.2     | False         | 42.4   | - |
+| R-101-FPN   | 24e     | 6.2     | True          | 43.5   | [config](https://github.com/open-mmlab/mmdetection/tree/master/configs/paa/paa_r101_fpn_2x_coco.py) | [model](http://download.openmmlab.com/mmdetection/v2.0/paa/paa_r101_fpn_2x_coco/paa_r101_fpn_2x_coco_20200821-6829f96b.pth) | [log](http://download.openmmlab.com/mmdetection/v2.0/paa/paa_r101_fpn_2x_coco/paa_r101_fpn_2x_coco_20200821-6829f96b.log.json) |
+
+**Note**:
+
+1. We find that the performance is unstable with 1x setting and may fluctuate by about 0.2 mAP. We report the best results.

PyTorch/NLP/Conformer-main/mmdetection/configs/paa/paa_conformer_small_patch32_fpn_1x_coco.py

+_base_ = [
+    '../_base_/default_runtime.py'
+]
+model = dict(
+    type='PAA',
+    pretrained=None,
+    backbone=dict(
+        type='Conformer',
+        embed_dim=384,
+        depth=12,
+        patch_size=32,
+        channel_ratio=4,
+        num_heads=6,
+        mlp_ratio=4,
+        qkv_bias=True,
+        norm_eval=True,
+        frozen_stages=1
+    ),
+    neck=dict(
+        type='FPN',
+        in_channels=[256, 512, 1024, 1024],
+        out_channels=256,
+        start_level=1,
+        add_extra_convs='on_output',
+        num_outs=5),
+    bbox_head=dict(
+        type='PAAHead',
+        reg_decoded_bbox=True,
+        score_voting=True,
+        topk=9,
+        num_classes=80,
+        in_channels=256,
+        stacked_convs=4,
+        feat_channels=256,
+        anchor_generator=dict(
+            type='AnchorGenerator',
+            ratios=[1.0],
+            octave_base_scale=8,
+            scales_per_octave=1,
+            strides=[8, 16, 32, 64, 128]),
+        bbox_coder=dict(
+            type='DeltaXYWHBBoxCoder',
+            target_means=[.0, .0, .0, .0],
+            target_stds=[0.1, 0.1, 0.2, 0.2]),
+        loss_cls=dict(
+            type='FocalLoss',
+            use_sigmoid=True,
+            gamma=2.0,
+            alpha=0.25,
+            loss_weight=1.0),
+        loss_bbox=dict(type='GIoULoss', loss_weight=1.3),
+        loss_centerness=dict(
+            type='CrossEntropyLoss', use_sigmoid=True, loss_weight=0.5)),
+    # training and testing settings
+    train_cfg=dict(
+        assigner=dict(
+            type='MaxIoUAssigner',
+            pos_iou_thr=0.1,
+            neg_iou_thr=0.1,
+            min_pos_iou=0,
+            ignore_iof_thr=-1),
+        allowed_border=-1,
+        pos_weight=-1,
+        debug=False),
+    test_cfg=dict(
+        nms_pre=1000,
+        min_bbox_size=0,
+        score_thr=0.05,
+        nms=dict(type='nms', iou_threshold=0.6),
+        max_per_img=100))
+
+
+dataset_type = 'CocoDataset'
+data_root = 'data/coco/'
+
+img_norm_cfg = dict(
+    mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True)
+
+train_pipeline = [
+    dict(type='LoadImageFromFile'),
+    dict(type='LoadAnnotations', with_bbox=True),
+    dict(type='Resize', img_scale=(1344, 800), keep_ratio=True),
+    dict(type='RandomFlip', flip_ratio=0.5),
+    dict(type='Normalize', **img_norm_cfg),
+    dict(type='Pad', size_divisor=32),
+    dict(type='DefaultFormatBundle'),
+    dict(type='Collect', keys=['img', 'gt_bboxes', 'gt_labels']),
+]
+test_pipeline = [
+    dict(type='LoadImageFromFile'),
+    dict(
+        type='MultiScaleFlipAug',
+        img_scale=(1344, 800),
+        flip=False,
+        transforms=[
+            dict(type='Resize', keep_ratio=True),
+            dict(type='RandomFlip'),
+            dict(type='Normalize', **img_norm_cfg),
+            dict(type='Pad', size_divisor=32),
+            dict(type='ImageToTensor', keys=['img']),
+            dict(type='Collect', keys=['img']),
+        ])
+]
+data = dict(
+    samples_per_gpu=2,
+    workers_per_gpu=2,
+    train=dict(
+        type=dataset_type,
+        ann_file=data_root + 'annotations/instances_train2017.json',
+        img_prefix=data_root + 'train2017/',
+        pipeline=train_pipeline),
+    val=dict(
+        type=dataset_type,
+        ann_file=data_root + 'annotations/instances_val2017.json',
+        img_prefix=data_root + 'val2017/',
+        pipeline=test_pipeline),
+    test=dict(
+        type=dataset_type,
+        ann_file=data_root + 'annotations/instances_val2017.json',
+        img_prefix=data_root + 'val2017/',
+        pipeline=test_pipeline))
+evaluation = dict(interval=1, metric='bbox')
+
+# optimizer
+optimizer = dict(
+    type='AdamW',
+    lr=0.0001,
+    weight_decay=0.0001,
+)
+optimizer_config = dict(grad_clip=None)
+# learning policy
+lr_config = dict(
+    policy='step',
+    warmup='linear',
+    warmup_iters=500,
+    warmup_ratio=0.001,
+    step=[8, 11])
+total_epochs = 12
+
+# optimizer_config = dict(grad_clip=dict(max_norm=0.1, norm_type=2))
+
+find_unused_parameters=True

PyTorch/NLP/Conformer-main/mmdetection/configs/paa/paa_r101_fpn_1x_coco.py

+_base_ = './paa_r50_fpn_1x_coco.py'
+model = dict(pretrained='torchvision://resnet101', backbone=dict(depth=101))
+lr_config = dict(step=[16, 22])
+total_epochs = 24

PyTorch/NLP/Conformer-main/mmdetection/configs/paa/paa_r101_fpn_2x_coco.py

+_base_ = './paa_r101_fpn_1x_coco.py'
+lr_config = dict(step=[16, 22])
+total_epochs = 24

PyTorch/NLP/Conformer-main/mmdetection/configs/paa/paa_r50_fpn_1.5x_coco.py

+_base_ = './paa_r50_fpn_1x_coco.py'
+lr_config = dict(step=[12, 16])
+total_epochs = 18

PyTorch/NLP/Conformer-main/mmdetection/configs/paa/paa_r50_fpn_1x_coco.py

+_base_ = [
+    '../_base_/datasets/coco_detection.py',
+    '../_base_/schedules/schedule_1x.py', '../_base_/default_runtime.py'
+]
+model = dict(
+    type='PAA',
+    pretrained='torchvision://resnet50',
+    backbone=dict(
+        type='ResNet',
+        depth=50,
+        num_stages=4,
+        out_indices=(0, 1, 2, 3),
+        frozen_stages=1,
+        norm_cfg=dict(type='BN', requires_grad=True),
+        norm_eval=True,
+        style='pytorch'),
+    neck=dict(
+        type='FPN',
+        in_channels=[256, 512, 1024, 2048],
+        out_channels=256,
+        start_level=1,
+        add_extra_convs='on_output',
+        num_outs=5),
+    bbox_head=dict(
+        type='PAAHead',
+        reg_decoded_bbox=True,
+        score_voting=True,
+        topk=9,
+        num_classes=80,
+        in_channels=256,
+        stacked_convs=4,
+        feat_channels=256,
+        anchor_generator=dict(
+            type='AnchorGenerator',
+            ratios=[1.0],
+            octave_base_scale=8,
+            scales_per_octave=1,
+            strides=[8, 16, 32, 64, 128]),
+        bbox_coder=dict(
+            type='DeltaXYWHBBoxCoder',
+            target_means=[.0, .0, .0, .0],
+            target_stds=[0.1, 0.1, 0.2, 0.2]),
+        loss_cls=dict(
+            type='FocalLoss',
+            use_sigmoid=True,
+            gamma=2.0,
+            alpha=0.25,
+            loss_weight=1.0),
+        loss_bbox=dict(type='GIoULoss', loss_weight=1.3),
+        loss_centerness=dict(
+            type='CrossEntropyLoss', use_sigmoid=True, loss_weight=0.5)),
+    # training and testing settings
+    train_cfg=dict(
+        assigner=dict(
+            type='MaxIoUAssigner',
+            pos_iou_thr=0.1,
+            neg_iou_thr=0.1,
+            min_pos_iou=0,
+            ignore_iof_thr=-1),
+        allowed_border=-1,
+        pos_weight=-1,
+        debug=False),
+    test_cfg=dict(
+        nms_pre=1000,
+        min_bbox_size=0,
+        score_thr=0.05,
+        nms=dict(type='nms', iou_threshold=0.6),
+        max_per_img=100))
+# optimizer
+optimizer = dict(type='SGD', lr=0.01, momentum=0.9, weight_decay=0.0001)

PyTorch/NLP/Conformer-main/mmdetection/configs/paa/paa_r50_fpn_2x_coco.py

+_base_ = './paa_r50_fpn_1x_coco.py'
+lr_config = dict(step=[16, 22])
+total_epochs = 24

PyTorch/NLP/Conformer-main/mmdetection/demo/image_demo.py

+from argparse import ArgumentParser
+
+from mmdet.apis import inference_detector, init_detector, show_result_pyplot
+
+
+def main():
+    parser = ArgumentParser()
+    parser.add_argument('img', help='Image file')
+    parser.add_argument('config', help='Config file')
+    parser.add_argument('checkpoint', help='Checkpoint file')
+    parser.add_argument(
+        '--device', default='cuda:0', help='Device used for inference')
+    parser.add_argument(
+        '--score-thr', type=float, default=0.3, help='bbox score threshold')
+    args = parser.parse_args()
+
+    # build the model from a config file and a checkpoint file
+    model = init_detector(args.config, args.checkpoint, device=args.device)
+    # test a single image
+    result = inference_detector(model, args.img)
+    # show the results
+    show_result_pyplot(model, args.img, result, score_thr=args.score_thr)
+
+
+if __name__ == '__main__':
+    main()

PyTorch/NLP/Conformer-main/mmdetection/demo/webcam_demo.py

+import argparse
+
+import cv2
+import torch
+
+from mmdet.apis import inference_detector, init_detector
+
+
+def parse_args():
+    parser = argparse.ArgumentParser(description='MMDetection webcam demo')
+    parser.add_argument('config', help='test config file path')
+    parser.add_argument('checkpoint', help='checkpoint file')
+    parser.add_argument(
+        '--device', type=str, default='cuda:0', help='CPU/CUDA device option')
+    parser.add_argument(
+        '--camera-id', type=int, default=0, help='camera device id')
+    parser.add_argument(
+        '--score-thr', type=float, default=0.5, help='bbox score threshold')
+    args = parser.parse_args()
+    return args
+
+
+def main():
+    args = parse_args()
+
+    device = torch.device(args.device)
+
+    model = init_detector(args.config, args.checkpoint, device=device)
+
+    camera = cv2.VideoCapture(args.camera_id)
+
+    print('Press "Esc", "q" or "Q" to exit.')
+    while True:
+        ret_val, img = camera.read()
+        result = inference_detector(model, img)
+
+        ch = cv2.waitKey(1)
+        if ch == 27 or ch == ord('q') or ch == ord('Q'):
+            break
+
+        model.show_result(
+            img, result, score_thr=args.score_thr, wait_time=1, show=True)
+
+
+if __name__ == '__main__':
+    main()

PyTorch/NLP/Conformer-main/mmdetection/docker/Dockerfile

+ARG PYTORCH="1.6.0"
+ARG CUDA="10.1"
+ARG CUDNN="7"
+
+FROM pytorch/pytorch:${PYTORCH}-cuda${CUDA}-cudnn${CUDNN}-devel
+
+ENV TORCH_CUDA_ARCH_LIST="6.0 6.1 7.0+PTX"
+ENV TORCH_NVCC_FLAGS="-Xfatbin -compress-all"
+ENV CMAKE_PREFIX_PATH="$(dirname $(which conda))/../"
+
+RUN apt-get update && apt-get install -y ffmpeg libsm6 libxext6 git ninja-build libglib2.0-0 libsm6 libxrender-dev libxext6 \
+    && apt-get clean \
+    && rm -rf /var/lib/apt/lists/*
+
+# Install MMCV
+RUN pip install mmcv-full==latest+torch1.6.0+cu101 -f https://openmmlab.oss-accelerate.aliyuncs.com/mmcv/dist/index.html
+
+# Install MMDetection
+RUN conda clean --all
+RUN git clone https://github.com/open-mmlab/mmdetection.git /mmdetection
+WORKDIR /mmdetection
+ENV FORCE_CUDA="1"
+RUN pip install -r requirements/build.txt
+RUN pip install --no-cache-dir -e .

PyTorch/NLP/Conformer-main/mmdetection/docs/1_exist_data_model.md

+# 1: Inference and train with existing models and standard datasets
+
+MMDetection provides hundreds of existing and existing detection models in [Model Zoo](https://mmdetection.readthedocs.io/en/latest/model_zoo.html)), and supports multiple standard datasets, including Pascal VOC, COCO, CityScapes, LVIS, etc. This note will show how to perform common tasks on these existing models and standard datasets, including:
+
+- Use existing models to inference on given images.
+- Test existing models on standard datasets.
+- Train predefined models on standard datasets.
+
+## Inference with existing models
+
+By inference, we mean using trained models to detect objects on images. In MMDetection, a model is defined by a configuration file and existing model parameters are save in a checkpoint file.
+
+To start with, we recommend [Faster RCNN](https://github.com/open-mmlab/mmdetection/tree/master/configs/faster_rcnn) with this [configuration file](https://github.com/open-mmlab/mmdetection/blob/master/configs/faster_rcnn/faster_rcnn_r50_fpn_1x_coco.py) and this [checkpoint file](http://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_1x_coco/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth). It is recommended to download the checkpoint file to `checkpoints` directory.
+
+### High-level APIs for inference
+
+MMDetection provide high-level Python APIs for inference on images. Here is an example of building the model and inference on given images or videos.
+
+```python
+from mmdet.apis import init_detector, inference_detector
+import mmcv
+
+# Specify the path to model config and checkpoint file
+config_file = 'configs/faster_rcnn/faster_rcnn_r50_fpn_1x_coco.py'
+checkpoint_file = 'checkpoints/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth'
+
+# build the model from a config file and a checkpoint file
+model = init_detector(config_file, checkpoint_file, device='cuda:0')
+
+# test a single image and show the results
+img = 'test.jpg'  # or img = mmcv.imread(img), which will only load it once
+result = inference_detector(model, img)
+# visualize the results in a new window
+model.show_result(img, result)
+# or save the visualization results to image files
+model.show_result(img, result, out_file='result.jpg')
+
+# test a video and show the results
+video = mmcv.VideoReader('video.mp4')
+for frame in video:
+    result = inference_detector(model, frame)
+    model.show_result(frame, result, wait_time=1)
+```
+
+A notebook demo can be found in [demo/inference_demo.ipynb](https://github.com/open-mmlab/mmdetection/blob/master/demo/inference_demo.ipynb).
+
+Note:  `inference_detector` only supports single-image inference for now.
+
+### Asynchronous interface - supported for Python 3.7+
+
+For Python 3.7+, MMDetection also supports async interfaces.
+By utilizing CUDA streams, it allows not to block CPU on GPU bound inference code and enables better CPU/GPU utilization for single-threaded application. Inference can be done concurrently either between different input data samples or between different models of some inference pipeline.
+
+See `tests/async_benchmark.py` to compare the speed of synchronous and asynchronous interfaces.
+
+```python
+import asyncio
+import torch
+from mmdet.apis import init_detector, async_inference_detector
+from mmdet.utils.contextmanagers import concurrent
+
+async def main():
+    config_file = 'configs/faster_rcnn/faster_rcnn_r50_fpn_1x_coco.py'
+    checkpoint_file = 'checkpoints/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth'
+    device = 'cuda:0'
+    model = init_detector(config_file, checkpoint=checkpoint_file, device=device)
+
+    # queue is used for concurrent inference of multiple images
+    streamqueue = asyncio.Queue()
+    # queue size defines concurrency level
+    streamqueue_size = 3
+
+    for _ in range(streamqueue_size):
+        streamqueue.put_nowait(torch.cuda.Stream(device=device))
+
+    # test a single image and show the results
+    img = 'test.jpg'  # or img = mmcv.imread(img), which will only load it once
+
+    async with concurrent(streamqueue):
+        result = await async_inference_detector(model, img)
+
+    # visualize the results in a new window
+    model.show_result(img, result)
+    # or save the visualization results to image files
+    model.show_result(img, result, out_file='result.jpg')
+
+
+asyncio.run(main())
+
+```
+
+### Demos
+
+We also provide two demo scripts, implemented with high-level APIs and supporting functionality codes.
+Source codes are available [here](https://github.com/open-mmlab/mmdetection/tree/master/demo).
+
+#### Image demo
+
+This script performs inference on a single image.
+
+```shell
+python demo/image_demo.py \
+    ${IMAGE_FILE} \
+    ${CONFIG_FILE} \
+    ${CHECKPOINT_FILE} \
+    [--device ${GPU_ID}] \
+    [--score-thr ${SCORE_THR}]
+```
+
+Examples:
+
+```shell
+python demo/image_demo.py demo/demo.jpg \
+    configs/faster_rcnn_r50_fpn_1x_coco.py \
+    checkpoints/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth \
+    --device cpu
+```
+
+#### Webcam demo
+
+This is a live demo from a webcam.
+
+```shell
+python demo/webcam_demo.py \
+    ${CONFIG_FILE} \
+    ${CHECKPOINT_FILE} \
+    [--device ${GPU_ID}] \
+    [--camera-id ${CAMERA-ID}] \
+    [--score-thr ${SCORE_THR}]
+```
+
+Examples:
+
+```shell
+python demo/webcam_demo.py \
+    configs/faster_rcnn_r50_fpn_1x_coco.py \
+    checkpoints/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth
+```
+
+## Test existing models on standard datasets
+
+To evaluate a model's accuracy, one usually tests the model on some standard datasets.
+MMDetection supports multiple public datasets including COCO, Pascal VOC, CityScapes, and [more](https://github.com/open-mmlab/mmdetection/tree/master/configs/_base_/datasets).
+This section will show how to test existing models on supported datasets.
+
+### Prepare datasets
+
+Public datasets like [Pascal VOC](http://host.robots.ox.ac.uk/pascal/VOC/index.html) or mirror and [COCO](https://cocodataset.org/#download) are available from official websites or mirrors. Note: In the detection task, Pascal VOC 2012 is an extension of Pascal VOC 2007 without overlap, and we usually use them together.
+It is recommended to download and extract the dataset somewhere outside the project directory and symlink the dataset root to `$MMDETECTION/data` as below.
+If your folder structure is different, you may need to change the corresponding paths in config files.
+
+```plain
+mmdetection
+├── mmdet
+├── tools
+├── configs
+├── data
+│   ├── coco
+│   │   ├── annotations
+│   │   ├── train2017
+│   │   ├── val2017
+│   │   ├── test2017
+│   ├── cityscapes
+│   │   ├── annotations
+│   │   ├── leftImg8bit
+│   │   │   ├── train
+│   │   │   ├── val
+│   │   ├── gtFine
+│   │   │   ├── train
+│   │   │   ├── val
+│   ├── VOCdevkit
+│   │   ├── VOC2007
+│   │   ├── VOC2012
+
+```
+
+The [cityscapes](https://www.cityscapes-dataset.com/) annotations need to be converted into the coco format using `tools/dataset_converters/cityscapes.py`:
+
+```shell
+pip install cityscapesscripts
+
+python tools/dataset_converters/cityscapes.py \
+    ./data/cityscapes \
+    --nproc 8 \
+    --out-dir ./data/cityscapes/annotations
+```
+
+TODO: CHANGE TO THE NEW PATH
+
+### Test existing models
+
+We provide testing scripts for evaluating an existing model on the whole dataset (COCO, PASCAL VOC, Cityscapes, etc.).
+The following testing environments are supported:
+
+- single GPU
+- single node multiple GPUs
+- multiple nodes
+
+Choose the proper script to perform testing depending on the testing environment.
+
+```shell
+# single-gpu testing
+python tools/test.py \
+    ${CONFIG_FILE} \
+    ${CHECKPOINT_FILE} \
+    [--out ${RESULT_FILE}] \
+    [--eval ${EVAL_METRICS}] \
+    [--show]
+
+# multi-gpu testing
+bash tools/dist_test.sh \
+    ${CONFIG_FILE} \
+    ${CHECKPOINT_FILE} \
+    ${GPU_NUM} \
+    [--out ${RESULT_FILE}] \
+    [--eval ${EVAL_METRICS}]
+```
+
+`tools/dist_test.sh` also supports multi-node testing, but relies on PyTorch's [launch utility](https://pytorch.org/docs/stable/distributed.html#launch-utility).
+
+Optional arguments:
+
+- `RESULT_FILE`: Filename of the output results in pickle format. If not specified, the results will not be saved to a file.
+- `EVAL_METRICS`: Items to be evaluated on the results. Allowed values depend on the dataset, e.g., `proposal_fast`, `proposal`, `bbox`, `segm` are available for COCO, `mAP`, `recall` for PASCAL VOC. Cityscapes could be evaluated by `cityscapes` as well as all COCO metrics.
+- `--show`: If specified, detection results will be plotted on the images and shown in a new window. It is only applicable to single GPU testing and used for debugging and visualization. Please make sure that GUI is available in your environment. Otherwise, you may encounter an error like `cannot connect to X server`.
+- `--show-dir`: If specified, detection results will be plotted on the images and saved to the specified directory. It is only applicable to single GPU testing and used for debugging and visualization. You do NOT need a GUI available in your environment for using this option.
+- `--show-score-thr`: If specified, detections with scores below this threshold will be removed.
+- `--cfg-options`:  if specified, the key-value pair optional cfg will be merged into config file
+- `--eval-options`: if specified, the key-value pair optional eval cfg will be kwargs for dataset.evaluate() function, it's only for evaluation
+
+### Examples
+
+Assume that you have already downloaded the checkpoints to the directory `checkpoints/`.
+
+1. Test Faster R-CNN and visualize the results. Press any key for the next image.
+   Config and checkpoint files are available [here](https://github.com/open-mmlab/mmdetection/tree/master/configs/faster_rcnn).
+
+   ```shell
+   python tools/test.py \
+       configs/faster_rcnn/faster_rcnn_r50_fpn_1x_coco.py \
+       checkpoints/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth \
+       --show
+   ```
+
+2. Test Faster R-CNN and save the painted images for future visualization.
+   Config and checkpoint files are available [here](https://github.com/open-mmlab/mmdetection/tree/master/configs/faster_rcnn).
+
+   ```shell
+   python tools/test.py \
+       configs/faster_rcnn/faster_rcnn_r50_fpn_1x.py \
+       checkpoints/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth \
+       --show-dir faster_rcnn_r50_fpn_1x_results
+   ```
+
+3. Test Faster R-CNN on PASCAL VOC (without saving the test results) and evaluate the mAP.
+   Config and checkpoint files are available [here](https://github.com/open-mmlab/mmdetection/tree/master/configs/pascal_voc).
+
+   ```shell
+   python tools/test.py \
+       configs/pascal_voc/faster_rcnn_r50_fpn_1x_voc.py \
+       checkpoints/faster_rcnn_r50_fpn_1x_voc0712_20200624-c9895d40.pth \
+       --eval mAP
+   ```
+
+4. Test Mask R-CNN with 8 GPUs, and evaluate the bbox and mask AP.
+   Config and checkpoint files are available [here](https://github.com/open-mmlab/mmdetection/tree/master/configs/mask_rcnn).
+
+   ```shell
+   ./tools/dist_test.sh \
+       configs/mask_rcnn_r50_fpn_1x_coco.py \
+       checkpoints/mask_rcnn_r50_fpn_1x_coco_20200205-d4b0c5d6.pth \
+       8 \
+       --out results.pkl \
+       --eval bbox segm
+   ```
+
+5. Test Mask R-CNN with 8 GPUs, and evaluate the **classwise** bbox and mask AP.
+   Config and checkpoint files are available [here](https://github.com/open-mmlab/mmdetection/tree/master/configs/mask_rcnn).
+
+   ```shell
+   ./tools/dist_test.sh \
+       configs/mask_rcnn/mask_rcnn_r50_fpn_1x_coco.py \
+       checkpoints/mask_rcnn_r50_fpn_1x_coco_20200205-d4b0c5d6.pth \
+       8 \
+       --out results.pkl \
+       --eval bbox segm \
+       --options "classwise=True"
+   ```
+
+6. Test Mask R-CNN on COCO test-dev with 8 GPUs, and generate JSON files for submitting to the official evaluation server.
+   Config and checkpoint files are available [here](https://github.com/open-mmlab/mmdetection/tree/master/configs/mask_rcnn).
+
+   ```shell
+   ./tools/dist_test.sh \
+       configs/mask_rcnn/mask_rcnn_r50_fpn_1x_coco.py \
+       checkpoints/mask_rcnn_r50_fpn_1x_coco_20200205-d4b0c5d6.pth \
+       8 \
+       -format-only \
+       --options "jsonfile_prefix=./mask_rcnn_test-dev_results"
+   ```
+
+   This command generates two JSON files `mask_rcnn_test-dev_results.bbox.json` and `mask_rcnn_test-dev_results.segm.json`.
+
+7. Test Mask R-CNN on Cityscapes test with 8 GPUs, and generate txt and png files for submitting to the official evaluation server.
+   Config and checkpoint files are available [here](https://github.com/open-mmlab/mmdetection/tree/master/configs/cityscapes).
+
+   ```shell
+   ./tools/dist_test.sh \
+       configs/cityscapes/mask_rcnn_r50_fpn_1x_cityscapes.py \
+       checkpoints/mask_rcnn_r50_fpn_1x_cityscapes_20200227-afe51d5a.pth \
+       8 \
+       --format-only \
+       --options "txtfile_prefix=./mask_rcnn_cityscapes_test_results"
+   ```
+
+   The generated png and txt would be under `./mask_rcnn_cityscapes_test_results` directory.
+
+### Batch Inference
+
+MMDetection supports inference with a single image or batched images in test mode. By default, we use single-image inference and you can use batch inference by modifying `samples_per_gpu` in the config of test data. You can do that either by modifying the config as below.
+
+```shell
+data = dict(train=dict(...), val=dict(...), test=dict(samples_per_gpu=2, ...))
+```
+
+Or you can set it through `--cfg-options` as `--cfg-options data.test.samples_per_gpu=2`
+
+### Deprecated ImageToTensor
+
+In test mode,  `ImageToTensor`  pipeline is deprecated, it's replaced by `DefaultFormatBundle` that recommended to manually replace it in the test data pipeline in your config file.  examples:
+
+```python
+# use ImageToTensor (deprecated)
+pipelines = [
+   dict(type='LoadImageFromFile'),
+   dict(
+       type='MultiScaleFlipAug',
+       img_scale=(1333, 800),
+       flip=False,
+       transforms=[
+           dict(type='Resize', keep_ratio=True),
+           dict(type='RandomFlip'),
+           dict(type='Normalize', mean=[0, 0, 0], std=[1, 1, 1]),
+           dict(type='Pad', size_divisor=32),
+           dict(type='ImageToTensor', keys=['img']),
+           dict(type='Collect', keys=['img']),
+       ])
+   ]
+
+# manually replace ImageToTensor to DefaultFormatBundle (recommended)
+pipelines = [
+   dict(type='LoadImageFromFile'),
+   dict(
+       type='MultiScaleFlipAug',
+       img_scale=(1333, 800),
+       flip=False,
+       transforms=[
+           dict(type='Resize', keep_ratio=True),
+           dict(type='RandomFlip'),
+           dict(type='Normalize', mean=[0, 0, 0], std=[1, 1, 1]),
+           dict(type='Pad', size_divisor=32),
+           dict(type='DefaultFormatBundle'),
+           dict(type='Collect', keys=['img']),
+       ])
+   ]
+```
+
+## Train predefined models on standard datasets
+
+MMDetection also provides out-of-the-box tools for training detection models.
+This section will show how to train _predefined_ models (under [configs](https://github.com/open-mmlab/mmdetection/tree/master/configs)) on standard datasets i.e. COCO.
+
+**Important**: The default learning rate in config files is for 8 GPUs and 2 img/gpu (batch size = 8\*2 = 16).
+According to the [linear scaling rule](https://arxiv.org/abs/1706.02677), you need to set the learning rate proportional to the batch size if you use different GPUs or images per GPU, e.g., `lr=0.01` for 4 GPUs \* 2 imgs/gpu and `lr=0.08` for 16 GPUs \* 4 imgs/gpu.
+
+### Prepare datasets
+
+Training requires preparing datasets too. See section [Prepare datasets](#prepare-datasets) above for details.
+
+**Note**:
+Currently, the config files under `configs/cityscapes` use COCO pretrained weights to initialize.
+You could download the existing models in advance if the network connection is unavailable or slow. Otherwise, it would cause errors at the beginning of training.
+
+### Training on a single GPU
+
+We provide `tools/train.py` to launch training jobs on a single GPU.
+The basic usage is as follows.
+
+```shell
+python tools/train.py \
+    ${CONFIG_FILE} \
+    [optional arguments]
+```
+
+During training, log files and checkpoints will be saved to the working directory, which is specified by `work_dir` in the config file or via CLI argument `--work-dir`.
+
+By default, the model is evaluated on the validation set every epoch, the evaluation interval can be specified in the config file as shown below.
+
+```python
+# evaluate the model every 12 epoch.
+evaluation = dict(interval=12)
+```
+
+This tool accepts several optional arguments, including:
+
+- `--no-validate` (**not suggested**): Disable evaluation during training.
+- `--work-dir ${WORK_DIR}`: Override the working directory.
+- `--resume-from ${CHECKPOINT_FILE}`: Resume from a previous checkpoint file.
+- `--options 'Key=value'`: Overrides other settings in the used config.
+
+**Note**:
+
+Difference between `resume-from` and `load-from`:
+
+`resume-from` loads both the model weights and optimizer status, and the epoch is also inherited from the specified checkpoint. It is usually used for resuming the training process that is interrupted accidentally.
+`load-from` only loads the model weights and the training epoch starts from 0. It is usually used for finetuning.
+
+### Training on multiple GPUs
+
+We provide `tools/dist_train.sh` to launch training on multiple GPUs.
+The basic usage is as follows.
+
+```shell
+bash ./tools/dist_train.sh \
+    ${CONFIG_FILE} \
+    ${GPU_NUM} \
+    [optional arguments]
+```
+
+Optional arguments remain the same as stated [above](#train-with-a-single-GPU).
+
+#### Launch multiple jobs simultaneously
+
+If you would like to launch multiple jobs on a single machine, e.g., 2 jobs of 4-GPU training on a machine with 8 GPUs,
+you need to specify different ports (29500 by default) for each job to avoid communication conflict.
+
+If you use `dist_train.sh` to launch training jobs, you can set the port in commands.
+
+```shell
+CUDA_VISIBLE_DEVICES=0,1,2,3 PORT=29500 ./tools/dist_train.sh ${CONFIG_FILE} 4
+CUDA_VISIBLE_DEVICES=4,5,6,7 PORT=29501 ./tools/dist_train.sh ${CONFIG_FILE} 4
+```
+
+### Training on multiple nodes
+
+MMDetection relies on `torch.distributed` package for distributed training.
+Thus, as a basic usage, one can launch distributed training via PyTorch's [launch utility](https://pytorch.org/docs/stable/distributed.html#launch-utility).
+
+### Manage jobs with Slurm
+
+[Slurm](https://slurm.schedmd.com/) is a good job scheduling system for computing clusters.
+On a cluster managed by Slurm, you can use `slurm_train.sh` to spawn training jobs. It supports both single-node and multi-node training.
+
+The basic usage is as follows.
+
+```shell
+[GPUS=${GPUS}] ./tools/slurm_train.sh ${PARTITION} ${JOB_NAME} ${CONFIG_FILE} ${WORK_DIR}
+```
+
+Below is an example of using 16 GPUs to train Mask R-CNN on a Slurm partition named _dev_, and set the work-dir to some shared file systems.
+
+```shell
+GPUS=16 ./tools/slurm_train.sh dev mask_r50_1x configs/mask_rcnn_r50_fpn_1x_coco.py /nfs/xxxx/mask_rcnn_r50_fpn_1x
+```
+
+You can check [the source code](https://github.com/open-mmlab/mmdetection/blob/master/tools/slurm_train.sh) to review full arguments and environment variables.
+
+When using Slurm, the port option need to be set in one of the following ways:
+
+1. Set the port through `--options`. This is more recommended since it does not change the original configs.
+
+   ```shell
+   CUDA_VISIBLE_DEVICES=0,1,2,3 GPUS=4 ./tools/slurm_train.sh ${PARTITION} ${JOB_NAME} config1.py ${WORK_DIR} --options 'dist_params.port=29500'
+   CUDA_VISIBLE_DEVICES=4,5,6,7 GPUS=4 ./tools/slurm_train.sh ${PARTITION} ${JOB_NAME} config2.py ${WORK_DIR} --options 'dist_params.port=29501'
+   ```
+
+2. Modify the config files to set different communication ports.
+
+   In `config1.py`, set
+
+   ```python
+   dist_params = dict(backend='nccl', port=29500)
+   ```
+
+   In `config2.py`, set
+
+   ```python
+   dist_params = dict(backend='nccl', port=29501)
+   ```
+
+   Then you can launch two jobs with `config1.py` and `config2.py`.
+
+   ```shell
+   CUDA_VISIBLE_DEVICES=0,1,2,3 GPUS=4 ./tools/slurm_train.sh ${PARTITION} ${JOB_NAME} config1.py ${WORK_DIR}
+   CUDA_VISIBLE_DEVICES=4,5,6,7 GPUS=4 ./tools/slurm_train.sh ${PARTITION} ${JOB_NAME} config2.py ${WORK_DIR}
+   ```

PyTorch/NLP/Conformer-main/mmdetection/docs/2_new_data_model.md

+# 2: Train with customized datasets
+
+In this note, you will know how to inference, test, and train predefined models with customized datasets. We use the [ballon dataset](https://github.com/matterport/Mask_RCNN/tree/master/samples/balloon) as an example to describe the whole process.
+
+The basic steps are as below:
+
+1. Prepare the customized dataset
+2. Prepare a config
+3. Train, test, inference models on the customized dataset.
+
+## Prepare the customized dataset
+
+There are three ways to support a new dataset in MMDetection:
+
+1. reorganize the dataset into COCO format.
+2. reorganize the dataset into a middle format.
+3. implement a new dataset.
+
+Usually we recommend to use the first two methods which are usually easier than the third.
+
+In this note, we give an example for converting the data into COCO format.
+
+**Note**: MMDetection only supports evaluating mask AP of dataset in COCO format for now.
+So for instance segmentation task users should convert the data into coco format.
+
+### COCO annotation format
+
+The necessary keys of COCO format for instance segmentation is as below, for the complete details, please refer [here](https://cocodataset.org/#format-data).
+
+```json
+{
+    "images": [image],
+    "annotations": [annotation],
+    "categories": [category]
+}
+
+
+image = {
+    "id": int,
+    "width": int,
+    "height": int,
+    "file_name": str,
+}
+
+annotation = {
+    "id": int,
+    "image_id": int,
+    "category_id": int,
+    "segmentation": RLE or [polygon],
+    "area": float,
+    "bbox": [x,y,width,height],
+    "iscrowd": 0 or 1,
+}
+
+categories = [{
+    "id": int,
+    "name": str,
+    "supercategory": str,
+}]
+```
+
+Assume we use the ballon dataset.
+After downloading the data, we need to implement a function to convert the annotation format into the COCO format. Then we can use implemented COCODataset to load the data and perform training and evaluation.
+
+If you take a look at the dataset, you will find the dataset format is as below:
+
+```json
+{'base64_img_data': '',
+ 'file_attributes': {},
+ 'filename': '34020010494_e5cb88e1c4_k.jpg',
+ 'fileref': '',
+ 'regions': {'0': {'region_attributes': {},
+   'shape_attributes': {'all_points_x': [1020,
+     1000,
+     994,
+     1003,
+     1023,
+     1050,
+     1089,
+     1134,
+     1190,
+     1265,
+     1321,
+     1361,
+     1403,
+     1428,
+     1442,
+     1445,
+     1441,
+     1427,
+     1400,
+     1361,
+     1316,
+     1269,
+     1228,
+     1198,
+     1207,
+     1210,
+     1190,
+     1177,
+     1172,
+     1174,
+     1170,
+     1153,
+     1127,
+     1104,
+     1061,
+     1032,
+     1020],
+    'all_points_y': [963,
+     899,
+     841,
+     787,
+     738,
+     700,
+     663,
+     638,
+     621,
+     619,
+     643,
+     672,
+     720,
+     765,
+     800,
+     860,
+     896,
+     942,
+     990,
+     1035,
+     1079,
+     1112,
+     1129,
+     1134,
+     1144,
+     1153,
+     1166,
+     1166,
+     1150,
+     1136,
+     1129,
+     1122,
+     1112,
+     1084,
+     1037,
+     989,
+     963],
+    'name': 'polygon'}}},
+ 'size': 1115004}
+```
+
+The annotation is a JSON file where each key indicates an image's all annotations.
+The code to convert the ballon dataset into coco format is as below.
+
+```python
+import os.path as osp
+
+def convert_balloon_to_coco(ann_file, out_file, image_prefix):
+    data_infos = mmcv.load(ann_file)
+
+    annotations = []
+    images = []
+    obj_count = 0
+    for idx, v in enumerate(mmcv.track_iter_progress(data_infos.values())):
+        filename = v['filename']
+        img_path = osp.join(image_prefix, filename)
+        height, width = mmcv.imread(img_path).shape[:2]
+
+        images.append(dict(
+            id=idx,
+            file_name=filename,
+            height=height,
+            width=width))
+
+        bboxes = []
+        labels = []
+        masks = []
+        for _, obj in v['regions'].items():
+            assert not obj['region_attributes']
+            obj = obj['shape_attributes']
+            px = obj['all_points_x']
+            py = obj['all_points_y']
+            poly = [(x + 0.5, y + 0.5) for x, y in zip(px, py)]
+            poly = [p for x in poly for p in x]
+
+            x_min, y_min, x_max, y_max = (
+                min(px), min(py), max(px), max(py))
+
+
+            data_anno = dict(
+                image_id=idx,
+                id=obj_count,
+                category_id=0,
+                bbox=[x_min, y_min, x_max - x_min, y_max - y_min],
+                area=(x_max - x_min) * (y_max - y_min),
+                segmentation=[poly],
+                iscrowd=0)
+            annotations.append(data_anno)
+            obj_count += 1
+
+    coco_format_json = dict(
+        images=images,
+        annotations=annotations,
+        categories=[{'id':0, 'name': 'balloon'}])
+    mmcv.dump(coco_format_json, out_file)
+
+```
+
+Using the function above, users can successfully convert the annotation file into json format, then we can use `CocoDataset` to train and evaluate the model.
+
+## Prepare a config
+
+The second step is to prepare a config thus the dataset could be successfully loaded. Assume that we want to use Mask R-CNN with FPN, the config to train the detector on ballon dataset is as below. Assume the config is under directory `configs/ballon/` and named as `mask_rcnn_r50_caffe_fpn_mstrain-poly_1x_balloon.py`, the config is as below.
+
+```python
+# The new config inherits a base config to highlight the necessary modification
+_base_ = 'mask_rcnn/mask_rcnn_r50_caffe_fpn_mstrain-poly_1x_coco.py'
+
+# We also need to change the num_classes in head to match the dataset's annotation
+model = dict(
+    roi_head=dict(
+        bbox_head=dict(num_classes=1),
+        mask_head=dict(num_classes=1)))
+
+# Modify dataset related settings
+dataset_type = 'COCODataset'
+classes = ('balloon',)
+data = dict(
+    train=dict(
+        img_prefix='balloon/train/',
+        classes=classes,
+        ann_file='balloon/train/annotation_coco.json'),
+    val=dict(
+        img_prefix='balloon/val/',
+        classes=classes,
+        ann_file='balloon/val/annotation_coco.json'),
+    test=dict(
+        img_prefix='balloon/val/',
+        classes=classes,
+        ann_file='balloon/val/annotation_coco.json'))
+
+# We can use the pre-trained Mask RCNN model to obtain higher performance
+load_from = 'checkpoints/mask_rcnn_r50_caffe_fpn_mstrain-poly_3x_coco_bbox_mAP-0.408__segm_mAP-0.37_20200504_163245-42aa3d00.pth'
+```
+
+## Train a new model
+
+To train a model with the new config, you can simply run
+
+```shell
+python tools/train.py configs/ballon/mask_rcnn_r50_caffe_fpn_mstrain-poly_1x_balloon.py
+```
+
+For more detailed usages, please refer to the [Case 1](1_exist_data_model.md).
+
+## Test and inference
+
+To test the trained model, you can simply run
+
+```shell
+python tools/test.py configs/ballon/mask_rcnn_r50_caffe_fpn_mstrain-poly_1x_balloon.py work_dirs/mask_rcnn_r50_caffe_fpn_mstrain-poly_1x_balloon.py/latest.pth --eval bbox segm
+```
+
+For more detailed usages, please refer to the [Case 1](1_exist_data_model.md).

PyTorch/NLP/Conformer-main/mmdetection/docs/3_exist_data_new_model.md

+# 3: Train with customized models and standard datasets
+
+In this note, you will know how to train, test and inference your own customized models under standard datasets. We use the cityscapes dataset to train a customized Cascade Mask R-CNN R50 model as an example to demonstrate the whole process, which using [`AugFPN`](https://github.com/Gus-Guo/AugFPN) to replace the defalut `FPN` as neck, and add `Rotate` or `Translate` as training-time auto augmentation.
+
+The basic steps are as below:
+
+1. Prepare the standard dataset
+2. Prepare your own customized model
+3. Prepare a config
+4. Train, test, and inference models on the standard dataset.
+
+## Prepare the standard dataset
+
+In this note, as we use the standard cityscapes dataset as an example.
+
+It is recommended to symlink the dataset root to `$MMDETECTION/data`.
+If your folder structure is different, you may need to change the corresponding paths in config files.
+
+```none
+mmdetection
+├── mmdet
+├── tools
+├── configs
+├── data
+│   ├── coco
+│   │   ├── annotations
+│   │   ├── train2017
+│   │   ├── val2017
+│   │   ├── test2017
+│   ├── cityscapes
+│   │   ├── annotations
+│   │   ├── leftImg8bit
+│   │   │   ├── train
+│   │   │   ├── val
+│   │   ├── gtFine
+│   │   │   ├── train
+│   │   │   ├── val
+│   ├── VOCdevkit
+│   │   ├── VOC2007
+│   │   ├── VOC2012
+
+```
+
+The cityscapes annotations have to be converted into the coco format using `tools/dataset_converters/cityscapes.py`:
+
+```shell
+pip install cityscapesscripts
+python tools/dataset_converters/cityscapes.py ./data/cityscapes --nproc 8 --out-dir ./data/cityscapes/annotations
+```
+
+Currently the config files in `cityscapes` use COCO pre-trained weights to initialize.
+You could download the pre-trained models in advance if network is unavailable or slow, otherwise it would cause errors at the beginning of training.
+
+## Prepare your own customized model
+
+The second step is to use your own module or training setting. Assume that we want to implement a new neck called `AugFPN` to replace with the default `FPN` under the existing detector Cascade Mask R-CNN R50. The following implements`AugFPN` under MMDetection.
+
+### 1. Define a new neck (e.g. AugFPN)
+
+Firstly create a new file `mmdet/models/necks/augfpn.py`.
+
+```python
+from ..builder import NECKS
+
+@NECKS.register_module()
+class AugFPN(nn.Module):
+
+    def __init__(self,
+                in_channels,
+                out_channels,
+                num_outs,
+                start_level=0,
+                end_level=-1,
+                add_extra_convs=False):
+        pass
+
+    def forward(self, inputs):
+        # implementation is ignored
+        pass
+```
+
+### 2. Import the module
+
+You can either add the following line to `mmdet/models/necks/__init__.py`,
+
+```python
+from .augfpn import AugFPN
+```
+
+or alternatively add
+
+```python
+custom_imports = dict(
+    imports=['mmdet.models.necks.augfpn.py'],
+    allow_failed_imports=False)
+```
+
+to the config file and avoid modifying the original code.
+
+### 3. Modify the config file
+
+```python
+neck=dict(
+    type='AugFPN',
+    in_channels=[256, 512, 1024, 2048],
+    out_channels=256,
+    num_outs=5)
+```
+
+For more detailed usages about customize your own models (e.g. implement a new backbone, head, loss, etc) and runtime training settings (e.g. define a new optimizer, use gradient clip, customize training schedules and hooks, etc), please refer to the guideline [Customize Models](tutorials/customize_models.md) and [Customize Runtime Settings](tutorials/customize_runtime.md) respectively.
+
+## Prepare a config
+
+The third step is to prepare a config for your own training setting. Assume that we want to add `AugFPN` and `Rotate` or `Translate` augmentation to existing Cascade Mask R-CNN R50 to train the cityscapes dataset, and assume the config is under directory `configs/cityscapes/` and named as `cascade_mask_rcnn_r50_augfpn_autoaug_10e_cityscapes.py`, the config is as below.
+
+```python
+# The new config inherits the base configs to highlight the necessary modification
+_base_ = [
+    '../_base_/models/cascade_mask_rcnn_r50_fpn.py',
+    '../_base_/datasets/cityscapes_instance.py', '../_base_/default_runtime.py'
+]
+
+model = dict(
+    # set None to avoid loading ImageNet pretrained backbone,
+    # instead here we set `load_from` to load from COCO pretrained detectors.
+    pretrained=None,
+    # replace neck from defaultly `FPN` to our new implemented module `AugFPN`
+    neck=dict(
+        type='AugFPN',
+        in_channels=[256, 512, 1024, 2048],
+        out_channels=256,
+        num_outs=5),
+    # We also need to change the num_classes in head from 80 to 8, to match the
+    # cityscapes dataset's annotation. This modification involves `bbox_head` and `mask_head`.
+    roi_head=dict(
+        bbox_head=[
+            dict(
+                type='Shared2FCBBoxHead',
+                in_channels=256,
+                fc_out_channels=1024,
+                roi_feat_size=7,
+                # change the number of classes from defaultly COCO to cityscapes
+                num_classes=8,
+                bbox_coder=dict(
+                    type='DeltaXYWHBBoxCoder',
+                    target_means=[0., 0., 0., 0.],
+                    target_stds=[0.1, 0.1, 0.2, 0.2]),
+                reg_class_agnostic=True,
+                loss_cls=dict(
+                    type='CrossEntropyLoss',
+                    use_sigmoid=False,
+                    loss_weight=1.0),
+                loss_bbox=dict(type='SmoothL1Loss', beta=1.0,
+                               loss_weight=1.0)),
+            dict(
+                type='Shared2FCBBoxHead',
+                in_channels=256,
+                fc_out_channels=1024,
+                roi_feat_size=7,
+                # change the number of classes from defaultly COCO to cityscapes
+                num_classes=8,
+                bbox_coder=dict(
+                    type='DeltaXYWHBBoxCoder',
+                    target_means=[0., 0., 0., 0.],
+                    target_stds=[0.05, 0.05, 0.1, 0.1]),
+                reg_class_agnostic=True,
+                loss_cls=dict(
+                    type='CrossEntropyLoss',
+                    use_sigmoid=False,
+                    loss_weight=1.0),
+                loss_bbox=dict(type='SmoothL1Loss', beta=1.0,
+                               loss_weight=1.0)),
+            dict(
+                type='Shared2FCBBoxHead',
+                in_channels=256,
+                fc_out_channels=1024,
+                roi_feat_size=7,
+                # change the number of classes from defaultly COCO to cityscapes
+                num_classes=8,
+                bbox_coder=dict(
+                    type='DeltaXYWHBBoxCoder',
+                    target_means=[0., 0., 0., 0.],
+                    target_stds=[0.033, 0.033, 0.067, 0.067]),
+                reg_class_agnostic=True,
+                loss_cls=dict(
+                    type='CrossEntropyLoss',
+                    use_sigmoid=False,
+                    loss_weight=1.0),
+                loss_bbox=dict(type='SmoothL1Loss', beta=1.0, loss_weight=1.0))
+        ],
+        mask_head=dict(
+            type='FCNMaskHead',
+            num_convs=4,
+            in_channels=256,
+            conv_out_channels=256,
+            # change the number of classes from defaultly COCO to cityscapes
+            num_classes=8,
+            loss_mask=dict(
+                type='CrossEntropyLoss', use_mask=True, loss_weight=1.0))))
+
+# over-write `train_pipeline` for new added `AutoAugment` training setting
+img_norm_cfg = dict(
+    mean=[123.675, 116.28, 103.53], std=[58.395, 57.12, 57.375], to_rgb=True)
+train_pipeline = [
+    dict(type='LoadImageFromFile'),
+    dict(type='LoadAnnotations', with_bbox=True, with_mask=True),
+    dict(
+        type='AutoAugment',
+        policies=[
+            [dict(
+                 type='Rotate',
+                 level=5,
+                 img_fill_val=(124, 116, 104),
+                 prob=0.5,
+                 scale=1)
+            ],
+            [dict(type='Rotate', level=7, img_fill_val=(124, 116, 104)),
+             dict(
+                 type='Translate',
+                 level=5,
+                 prob=0.5,
+                 img_fill_val=(124, 116, 104))
+            ],
+        ]),
+    dict(
+        type='Resize', img_scale=[(2048, 800), (2048, 1024)], keep_ratio=True),
+    dict(type='RandomFlip', flip_ratio=0.5),
+    dict(type='Normalize', **img_norm_cfg),
+    dict(type='Pad', size_divisor=32),
+    dict(type='DefaultFormatBundle'),
+    dict(type='Collect', keys=['img', 'gt_bboxes', 'gt_labels', 'gt_masks']),
+]
+
+# set batch_size per gpu, and set new training pipeline
+data = dict(
+    samples_per_gpu=1,
+    workers_per_gpu=3,
+    # over-write `pipeline` with new training pipeline setting
+    train=dict(dataset=dict(pipeline=train_pipeline)))
+
+# Set optimizer
+optimizer = dict(type='SGD', lr=0.01, momentum=0.9, weight_decay=0.0001)
+optimizer_config = dict(grad_clip=None)
+# Set customized learning policy
+lr_config = dict(
+    policy='step',
+    warmup='linear',
+    warmup_iters=500,
+    warmup_ratio=0.001,
+    step=[8])
+total_epochs = 10
+
+# We can use the COCO pretrained Cascade Mask R-CNN R50 model for more stable performance initialization
+load_from = 'http://download.openmmlab.com/mmdetection/v2.0/cascade_rcnn/cascade_mask_rcnn_r50_fpn_1x_coco/cascade_mask_rcnn_r50_fpn_1x_coco_20200203-9d4dcb24.pth'
+```
+
+## Train a new model
+
+To train a model with the new config, you can simply run
+
+```shell
+python tools/train.py configs/cityscapes/cascade_mask_rcnn_r50_augfpn_autoaug_10e_cityscapes.py
+```
+
+For more detailed usages, please refer to the [Case 1](1_exist_data_model.md).
+
+## Test and inference
+
+To test the trained model, you can simply run
+
+```shell
+python tools/test.py configs/cityscapes/cascade_mask_rcnn_r50_augfpn_autoaug_10e_cityscapes.py work_dirs/cascade_mask_rcnn_r50_augfpn_autoaug_10e_cityscapes.py/latest.pth --eval bbox segm
+```
+
+For more detailed usages, please refer to the [Case 1](1_exist_data_model.md).

PyTorch/NLP/Conformer-main/mmdetection/docs/Makefile

+# Minimal makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line, and also
+# from the environment for the first two.
+SPHINXOPTS    ?=
+SPHINXBUILD   ?= sphinx-build
+SOURCEDIR     = .
+BUILDDIR      = _build
+
+# Put it first so that "make" without argument is like "make help".
+help:
+	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+.PHONY: help Makefile
+
+# Catch-all target: route all unknown targets to Sphinx using the new
+# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
+%: Makefile
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

PyTorch/NLP/Conformer-main/mmdetection/docs/api.rst

+API Reference
+=================
+
+mmdet.apis
+--------------
+.. automodule:: mmdet.apis
+    :members:
+
+mmdet.core
+--------------
+
+anchor
+^^^^^^^^^^
+.. automodule:: mmdet.core.anchor
+    :members:
+
+bbox
+^^^^^^^^^^
+.. automodule:: mmdet.core.bbox
+    :members:
+
+export
+^^^^^^^^^^
+.. automodule:: mmdet.core.export
+    :members:
+
+mask
+^^^^^^^^^^
+.. automodule:: mmdet.core.mask
+    :members:
+
+evaluation
+^^^^^^^^^^
+.. automodule:: mmdet.core.evaluation
+    :members:
+
+post_processing
+^^^^^^^^^^^^^^^
+.. automodule:: mmdet.core.post_processing
+    :members:
+
+optimizer
+^^^^^^^^^^
+.. automodule:: mmdet.core.optimizer
+    :members:
+
+utils
+^^^^^^^^^^
+.. automodule:: mmdet.core.utils
+    :members:
+
+mmdet.datasets
+--------------
+
+datasets
+^^^^^^^^^^
+.. automodule:: mmdet.datasets
+    :members:
+
+pipelines
+^^^^^^^^^^
+.. automodule:: mmdet.datasets.pipelines
+    :members:
+
+mmdet.models
+--------------
+
+detectors
+^^^^^^^^^^
+.. automodule:: mmdet.models.detectors
+    :members:
+
+backbones
+^^^^^^^^^^
+.. automodule:: mmdet.models.backbones
+    :members:
+
+necks
+^^^^^^^^^^^^
+.. automodule:: mmdet.models.necks
+    :members:
+
+dense_heads
+^^^^^^^^^^^^
+.. automodule:: mmdet.models.dense_heads
+    :members:
+
+roi_heads
+^^^^^^^^^^
+.. automodule:: mmdet.models.roi_heads
+    :members:
+
+losses
+^^^^^^^^^^
+.. automodule:: mmdet.models.losses
+    :members:
+
+utils
+^^^^^^^^^^
+.. automodule:: mmdet.models.utils
+    :members:

PyTorch/NLP/Conformer-main/mmdetection/docs/changelog.md

+## Changelog
+
+### v2.8.0 (04/01/2021)
+
+#### Highlights
+
+- Support new methods: [Cascade RPN](https://arxiv.org/abs/1909.06720), [TridentNet](https://arxiv.org/abs/1901.01892)
+
+#### New Features
+
+- Support [Cascade RPN](https://arxiv.org/abs/1909.06720) (#1900)
+- Support [TridentNet](https://arxiv.org/abs/1901.01892) (#3313)
+
+#### Bug Fixes
+
+- Fix bug of show result in async_benchmark (#4367)
+- Fix scale factor in MaskTestMixin (#4366)
+- Fix but when returning indices in `multiclass_nms` (#4362)
+- Fix bug of empirical attention in resnext backbone error (#4300)
+- Fix bug of `img_norm_cfg` in FCOS-HRNet models with updated performance and models (#4250)
+- Fix invalid checkpoint and log in Mask R-CNN models on Cityscapes dataset (#4287)
+- Fix bug in distributed sampler when dataset is too small (#4257)
+- Fix bug of 'PAFPN has no attribute extra_convs_on_inputs' (#4235)
+
+#### Improvements
+
+- Update model url from aws to aliyun (#4349)
+- Update ATSS for PyTorch 1.6+ (#4359)
+- Update script to install ruby in pre-commit installation (#4360)
+- Delete deprecated `mmdet.ops` (#4325)
+- Refactor hungarian assigner for more general usage in Sparse R-CNN (#4259)
+- Handle scipy import in DETR to reduce package dependencies (#4339)
+- Update documentation of usages for config options after MMCV (1.2.3) supports overriding list in config (#4326)
+- Update pre-train models of faster rcnn trained on COCO subsets (#4307)
+- Avoid zero or too small value for beta in Dynamic R-CNN (#4303)
+- Add doccumentation for Pytorch2ONNX (#4271)
+- Add deprecated warning FPN arguments (#4264)
+- Support returning indices of kept bboxes when using nms (#4251)
+- Update type and device requirements when creating tensors `GFLHead` (#4210)
+- Update device requirements when creating tensors in `CrossEntropyLoss` (#4224)
+
+### v2.7.0 (30/11/2020)
+
+- Support new method: [DETR](https://arxiv.org/abs/2005.12872), [ResNest](https://arxiv.org/abs/2004.08955), Faster R-CNN DC5.
+- Support YOLO, Mask R-CNN, and Cascade R-CNN models exportable to ONNX.
+
+#### New Features
+
+- Support [DETR](https://arxiv.org/abs/2005.12872) (#4201, #4206)
+- Support to link the best checkpoint in training (#3773)
+- Support to override config through options in inference.py (#4175)
+- Support YOLO, Mask R-CNN, and Cascade R-CNN models exportable to ONNX (#4087, #4083)
+- Support [ResNeSt](https://arxiv.org/abs/2004.08955) backbone (#2959)
+- Support unclip border bbox regression (#4076)
+- Add tpfp func in evaluating AP (#4069)
+- Support mixed precision training of SSD detector with other backbones (#4081)
+- Add Faster R-CNN DC5 models (#4043)
+
+#### Bug Fixes
+
+- Fix bug of `gpu_id` in distributed training mode (#4163)
+- Support Albumentations with version higher than 0.5 (#4032)
+- Fix num_classes bug in faster rcnn config (#4088)
+- Update code in docs/2_new_data_model.md (#4041)
+
+#### Improvements
+
+- Ensure DCN offset to have similar type as features in VFNet (#4198)
+- Add config links in README files of models (#4190)
+- Add tutorials for loss conventions (#3818)
+- Add solution to installation issues in 30-series GPUs (#4176)
+- Update docker version in get_started.md (#4145)
+- Add model statistics and polish some titles in configs README (#4140)
+- Clamp neg probability in FreeAnchor (#4082)
+- Speed up expanding large images (#4089)
+- Fix Pytorch 1.7 incompatibility issues (#4103)
+- Update trouble shooting page to resolve segmentation fault (#4055)
+- Update aLRP-Loss in project page (#4078)
+- Clean duplicated `reduce_mean` function (#4056)
+- Refactor Q&A (#4045)
+
+### v2.6.0 (1/11/2020)
+
+- Support new method: [VarifocalNet](https://arxiv.org/abs/2008.13367).
+- Refactored documentation with more tutorials.
+
+#### New Features
+
+- Support GIoU calculation in `BboxOverlaps2D`, and re-implement `giou_loss` using `bbox_overlaps` (#3936)
+- Support random sampling in CPU mode (#3948)
+- Support VarifocalNet (#3666, #4024)
+
+#### Bug Fixes
+
+- Fix SABL validating bug in Cascade R-CNN (#3913)
+- Avoid division by zero in PAA head when num_pos=0 (#3938)
+- Fix temporary directory bug of multi-node testing error (#4034, #4017)
+- Fix `--show-dir` option in test script (#4025)
+- Fix GA-RetinaNet r50 model url (#3983)
+- Update code in docs and fix broken urls (#3947)
+
+#### Improvements
+
+- Refactor pytorch2onnx API into `mmdet.core.export` and use `generate_inputs_and_wrap_model` for pytorch2onnx (#3857, #3912)
+- Update RPN upgrade scripts for v2.5.0 compatibility (#3986)
+- Use mmcv `tensor2imgs` (#4010)
+- Update test robustness (#4000)
+- Update trouble shooting page (#3994)
+- Accelerate PAA training speed (#3985)
+- Support batch_size > 1 in validation (#3966)
+- Use RoIAlign implemented in MMCV for inference in CPU mode (#3930)
+- Documentation refactoring (#4031)
+
+### v2.5.0 (5/10/2020)
+
+#### Highlights
+
+- Support new methods: [YOLACT](https://arxiv.org/abs/1904.02689), [CentripetalNet](https://arxiv.org/abs/2003.09119).
+- Add more documentations for easier and more clear usage.
+
+#### Backwards Incompatible Changes
+
+**FP16 related methods are imported from mmcv instead of mmdet. (#3766, #3822)**
+Mixed precision training utils in `mmdet.core.fp16` are moved to `mmcv.runner`, including `force_fp32`, `auto_fp16`, `wrap_fp16_model`, and `Fp16OptimizerHook`. A deprecation warning will be raised if users attempt to import those methods from `mmdet.core.fp16`, and will be finally removed in V2.8.0.
+
+**[0, N-1] represents foreground classes and N indicates background classes for all models. (#3221)**
+Before v2.5.0, the background label for RPN is 0, and N for other heads. Now the behavior is consistent for all models. Thus `self.background_labels` in `dense_heads` is removed and all heads use `self.num_classes` to indicate the class index of background labels.
+This change has no effect on the pre-trained models in the v2.x model zoo, but will affect the training of all models with RPN heads. Two-stage detectors whose RPN head uses softmax will be affected because the order of categories is changed.
+
+**Only call `get_subset_by_classes` when `test_mode=True` and `self.filter_empty_gt=True` (#3695)**
+Function `get_subset_by_classes` in dataset is refactored and only filters out images when `test_mode=True` and `self.filter_empty_gt=True`.
+    In the original implementation, `get_subset_by_classes` is not related to the flag `self.filter_empty_gt` and will only be called when the classes is set during initialization no matter `test_mode` is `True` or `False`. This brings ambiguous behavior and potential bugs in many cases. After v2.5.0, if `filter_empty_gt=False`, no matter whether the classes are specified in a dataset, the dataset will use all the images in the annotations. If `filter_empty_gt=True` and `test_mode=True`, no matter whether the classes are specified, the dataset will call ``get_subset_by_classes` to check the images and filter out images containing no GT boxes. Therefore, the users should be responsible for the data filtering/cleaning process for the test dataset.
+
+#### New Features
+
+- Test time augmentation for single stage detectors (#3844, #3638)
+- Support to show the name of experiments during training (#3764)
+- Add `Shear`, `Rotate`, `Translate` Augmentation (#3656, #3619, #3687)
+- Add image-only transformations including `Constrast`, `Equalize`, `Color`, and `Brightness`. (#3643)
+- Support [YOLACT](https://arxiv.org/abs/1904.02689) (#3456)
+- Support [CentripetalNet](https://arxiv.org/abs/2003.09119) (#3390)
+- Support PyTorch 1.6 in docker (#3905)
+
+#### Bug Fixes
+
+- Fix the bug of training ATSS when there is no ground truth boxes (#3702)
+- Fix the bug of using Focal Loss when there is `num_pos` is 0 (#3702)
+- Fix the label index mapping in dataset browser (#3708)
+- Fix Mask R-CNN training stuck problem when ther is no positive rois (#3713)
+- Fix the bug of `self.rpn_head.test_cfg` in `RPNTestMixin` by using `self.rpn_head` in rpn head (#3808)
+- Fix deprecated `Conv2d` from mmcv.ops (#3791)
+- Fix device bug in RepPoints (#3836)
+- Fix SABL validating bug (#3849)
+- Use `https://download.openmmlab.com/mmcv/dist/index.html` for installing MMCV (#3840)
+- Fix nonzero in NMS for PyTorch 1.6.0 (#3867)
+- Fix the API change bug of PAA (#3883)
+- Fix typo in bbox_flip (#3886)
+- Fix cv2 import error of ligGL.so.1 in Dockerfile (#3891)
+
+#### Improvements
+
+- Change to use `mmcv.utils.collect_env` for collecting environment information to avoid duplicate codes (#3779)
+- Update checkpoint file names to v2.0 models in documentation (#3795)
+- Update tutorials for changing runtime settings (#3778), modifing loss (#3777)
+- Improve the function of `simple_test_bboxes` in SABL (#3853)
+- Convert mask to bool before using it as img's index for robustness and speedup (#3870)
+- Improve documentation of modules and dataset customization (#3821)
+
+### v2.4.0 (5/9/2020)
+
+**Highlights**
+
+- Fix lots of issues/bugs and reorganize the trouble shooting page
+- Support new methods [SABL](https://arxiv.org/abs/1912.04260), [YOLOv3](https://arxiv.org/abs/1804.02767), and [PAA Assign](https://arxiv.org/abs/2007.08103)
+- Support Batch Inference
+- Start to publish `mmdet` package to PyPI since v2.3.0
+- Switch model zoo to download.openmmlab.com
+
+**Backwards Incompatible Changes**
+
+- Support Batch Inference (#3564, #3686, #3705): Since v2.4.0, MMDetection could inference model with multiple images in a single GPU.
+  This change influences all the test APIs in MMDetection and downstream codebases. To help the users migrate their code, we use `replace_ImageToTensor` (#3686) to convert legacy test data pipelines during dataset initialization.
+- Support RandomFlip with horizontal/vertical/diagonal direction (#3608): Since v2.4.0, MMDetection supports horizontal/vertical/diagonal flip in the data augmentation. This influences bounding box, mask, and image transformations in data augmentation process and the process that will map those data back to the original format.
+- Migrate to use `mmlvis` and `mmpycocotools` for COCO and LVIS dataset (#3727). The APIs are fully compatible with the original `lvis` and `pycocotools`. Users need to uninstall the existing pycocotools and lvis packages in their environment first and install `mmlvis` & `mmpycocotools`.
+
+**Bug Fixes**
+
+- Fix default mean/std for onnx (#3491)
+- Fix coco evaluation and add metric items (#3497)
+- Fix typo for install.md (#3516)
+- Fix atss when sampler per gpu is 1 (#3528)
+- Fix import of fuse_conv_bn (#3529)
+- Fix bug of gaussian_target, update unittest of heatmap (#3543)
+- Fixed VOC2012 evaluate (#3553)
+- Fix scale factor bug of rescale (#3566)
+- Fix with_xxx_attributes in base detector (#3567)
+- Fix boxes scaling when number is 0 (#3575)
+- Fix rfp check when neck config is a list (#3591)
+- Fix import of fuse conv bn in benchmark.py (#3606)
+- Fix webcam demo (#3634)
+- Fix typo and itemize issues in tutorial (#3658)
+- Fix error in distributed training when some levels of FPN are not assigned with bounding boxes (#3670)
+- Fix the width and height orders of stride in valid flag generation (#3685)
+- Fix weight initialization bug in Res2Net DCN (#3714)
+- Fix bug in OHEMSampler (#3677)
+
+**New Features**
+
+- Support Cutout augmentation (#3521)
+- Support evaluation on multiple datasets through ConcatDataset (#3522)
+- Support [PAA assign](https://arxiv.org/abs/2007.08103) #(3547)
+- Support eval metric with pickle results (#3607)
+- Support [YOLOv3](https://arxiv.org/abs/1804.02767) (#3083)
+- Support [SABL](https://arxiv.org/abs/1912.04260) (#3603)
+- Support to publish to Pypi in github-action (#3510)
+- Support custom imports (#3641)
+
+**Improvements**
+
+- Refactor common issues in documentation (#3530)
+- Add pytorch 1.6 to CI config (#3532)
+- Add config to runner meta (#3534)
+- Add eval-option flag for testing (#3537)
+- Add init_eval to evaluation hook (#3550)
+- Add include_bkg in ClassBalancedDataset (#3577)
+- Using config's loading in inference_detector (#3611)
+- Add ATSS ResNet-101 models in model zoo (#3639)
+- Update urls to download.openmmlab.com (#3665)
+- Support non-mask training for CocoDataset (#3711)
+
+### v2.3.0 (5/8/2020)
+
+**Highlights**
+
+- The CUDA/C++ operators have been moved to `mmcv.ops`. For backward compatibility `mmdet.ops` is kept as warppers of `mmcv.ops`.
+- Support new methods [CornerNet](https://arxiv.org/abs/1808.01244), [DIOU](https://arxiv.org/abs/1911.08287)/[CIOU](https://arxiv.org/abs/2005.03572) loss, and new dataset: [LVIS V1](https://arxiv.org/abs/1908.03195)
+- Provide more detailed colab training tutorials and more complete documentation.
+- Support to convert RetinaNet from Pytorch to ONNX.
+
+**Bug Fixes**
+
+- Fix the model initialization bug of DetectoRS (#3187)
+- Fix the bug of module names in NASFCOSHead (#3205)
+- Fix the filename bug in publish_model.py (#3237)
+- Fix the dimensionality bug when `inside_flags.any()` is `False` in dense heads (#3242)
+- Fix the bug of forgetting to pass flip directions in `MultiScaleFlipAug` (#3262)
+- Fixed the bug caused by default value of `stem_channels` (#3333)
+- Fix the bug of model checkpoint loading for CPU inference (#3318, #3316)
+- Fix topk bug when box number is smaller than the expected topk number in ATSSAssigner (#3361)
+- Fix the gt priority bug in center_region_assigner.py (#3208)
+- Fix NaN issue of iou calculation in iou_loss.py (#3394)
+- Fix the bug that `iou_thrs` is not actually used during evaluation in coco.py (#3407)
+- Fix test-time augmentation of RepPoints (#3435)
+- Fix runtimeError caused by incontiguous tensor in Res2Net+DCN (#3412)
+
+**New Features**
+
+- Support [CornerNet](https://arxiv.org/abs/1808.01244) (#3036)
+- Support [DIOU](https://arxiv.org/abs/1911.08287)/[CIOU](https://arxiv.org/abs/2005.03572) loss (#3151)
+- Support [LVIS V1](https://arxiv.org/abs/1908.03195) dataset (#)
+- Support customized hooks in training (#3395)
+- Support fp16 training of generalized focal loss (#3410)
+- Support to convert RetinaNet from Pytorch to ONNX (#3075)
+
+**Improvements**
+
+- Support to process ignore boxes in ATSS assigner (#3082)
+- Allow to crop images without ground truth in `RandomCrop` (#3153)
+- Enable the the `Accuracy` module to set threshold (#3155)
+- Refactoring unit tests (#3206)
+- Unify the training settings of `to_float32` and `norm_cfg` in RegNets configs (#3210)
+- Add colab training tutorials for beginners (#3213, #3273)
+- Move CUDA/C++ operators into `mmcv.ops` and keep `mmdet.ops` as warppers for backward compatibility (#3232)(#3457)
+- Update installation scripts in documentation (#3290) and dockerfile (#3320)
+- Support to set image resize backend (#3392)
+- Remove git hash in version file (#3466)
+- Check mmcv version to force version compatibility (#3460)
+
+### v2.2.0 (1/7/2020)
+
+**Highlights**
+
+- Support new methods: [DetectoRS](https://arxiv.org/abs/2006.02334), [PointRend](https://arxiv.org/abs/1912.08193), [Generalized Focal Loss](https://arxiv.org/abs/2006.04388), [Dynamic R-CNN](https://arxiv.org/abs/2004.06002)
+
+**Bug Fixes**
+
+- Fix FreeAnchor when no gt in image (#3176)
+- Clean up deprecated usage of `register_module()` (#3092, #3161)
+- Fix pretrain bug in NAS FCOS (#3145)
+- Fix `num_classes` in SSD (#3142)
+- Fix FCOS warmup (#3119)
+- Fix `rstrip` in `tools/publish_model.py`
+- Fix `flip_ratio` default value in RandomFLip pipeline (#3106)
+- Fix cityscapes eval with ms_rcnn (#3112)
+- Fix RPN softmax (#3056)
+- Fix filename of LVIS@v0.5 (#2998)
+- Fix nan loss by filtering out-of-frame gt_bboxes in COCO (#2999)
+- Fix bug in FSAF (#3018)
+- Add FocalLoss `num_classes` check (#2964)
+- Fix PISA Loss when there are no gts (#2992)
+- Avoid nan in `iou_calculator` (#2975)
+- Prevent possible bugs in loading and transforms caused by shallow copy (#2967)
+
+**New Features**
+
+- Add DetectoRS (#3064)
+- Support Generalize Focal Loss (#3097)
+- Support PointRend (#2752)
+- Support Dynamic R-CNN (#3040)
+- Add DeepFashion dataset (#2968)
+- Implement FCOS training tricks (#2935)
+- Use BaseDenseHead as base class for anchor-base heads (#2963)
+- Add `with_cp` for BasicBlock (#2891)
+- Add `stem_channles` argument for ResNet (#2954)
+
+**Improvements**
+
+- Add anchor free base head (#2867)
+- Migrate to github action (#3137)
+- Add docstring for datasets, pipelines, core modules and methods (#3130, #3125, #3120)
+- Add VOC benchmark (#3060)
+- Add `concat` mode in GRoI (#3098)
+- Remove cmd arg `autorescale-lr` (#3080)
+- Use `len(data['img_metas'])` to indicate `num_samples` (#3073, #3053)
+- Switch to EpochBasedRunner (#2976)
+
+### v2.1.0 (8/6/2020)
+
+**Highlights**
+
+- Support new backbones: [RegNetX](https://arxiv.org/abs/2003.13678), [Res2Net](https://arxiv.org/abs/1904.01169)
+- Support new methods: [NASFCOS](https://arxiv.org/abs/1906.04423), [PISA](https://arxiv.org/abs/1904.04821), [GRoIE](https://arxiv.org/abs/2004.13665)
+- Support new dataset: [LVIS](https://arxiv.org/abs/1908.03195)
+
+**Bug Fixes**
+
+- Change the CLI argument `--validate` to `--no-validate` to enable validation after training epochs by default. (#2651)
+- Add missing cython to docker file (#2713)
+- Fix bug in nms cpu implementation (#2754)
+- Fix bug when showing mask results (#2763)
+- Fix gcc requirement (#2806)
+- Fix bug in async test (#2820)
+- Fix mask encoding-decoding bugs in test API (#2824)
+- Fix bug in test time augmentation (#2858, #2921, #2944)
+- Fix a typo in comment of apis/train (#2877)
+- Fix the bug of returning None when no gt bboxes are in the original image in `RandomCrop`. Fix the bug that misses to handle `gt_bboxes_ignore`, `gt_label_ignore`, and `gt_masks_ignore` in `RandomCrop`, `MinIoURandomCrop` and `Expand` modules. (#2810)
+- Fix bug of `base_channels` of regnet (#2917)
+- Fix the bug of logger when loading pre-trained weights in base detector (#2936)
+
+**New Features**
+
+- Add IoU models (#2666)
+- Add colab demo for inference
+- Support class agnostic nms (#2553)
+- Add benchmark gathering scripts for development only (#2676)
+- Add mmdet-based project links (#2736, #2767, #2895)
+- Add config dump in training (#2779)
+- Add ClassBalancedDataset (#2721)
+- Add res2net backbone (#2237)
+- Support RegNetX models (#2710)
+- Use `mmcv.FileClient` to support different storage backends (#2712)
+- Add ClassBalancedDataset (#2721)
+- Code Release: Prime Sample Attention in Object Detection (CVPR 2020) (#2626)
+- Implement NASFCOS (#2682)
+- Add class weight in CrossEntropyLoss (#2797)
+- Support LVIS dataset (#2088)
+- Support GRoIE (#2584)
+
+**Improvements**
+
+- Allow different x and y strides in anchor heads. (#2629)
+- Make FSAF loss more robust to no gt (#2680)
+- Compute pure inference time instead (#2657) and update inference speed (#2730)
+- Avoided the possibility that a patch with 0 area is cropped. (#2704)
+- Add warnings when deprecated `imgs_per_gpu` is used. (#2700)
+- Add a mask rcnn example for config (#2645)
+- Update model zoo (#2762, #2866, #2876, #2879, #2831)
+- Add `ori_filename` to img_metas and use it in test show-dir (#2612)
+- Use `img_fields` to handle multiple images during image transform (#2800)
+- Add upsample_cfg support in FPN (#2787)
+- Add `['img']` as default `img_fields` for back compatibility (#2809)
+- Rename the pretrained model from `open-mmlab://resnet50_caffe` and `open-mmlab://resnet50_caffe_bgr` to `open-mmlab://detectron/resnet50_caffe` and `open-mmlab://detectron2/resnet50_caffe`. (#2832)
+- Added sleep(2) in test.py to reduce hanging problem (#2847)
+- Support `c10::half` in CARAFE (#2890)
+- Improve documentations (#2918, #2714)
+- Use optimizer constructor in mmcv and clean the original implementation in `mmdet.core.optimizer` (#2947)
+
+### v2.0.0 (6/5/2020)
+
+In this release, we made lots of major refactoring and modifications.
+
+1. **Faster speed**. We optimize the training and inference speed for common models, achieving up to 30% speedup for training and 25% for inference. Please refer to [model zoo](model_zoo.md#comparison-with-detectron2) for details.
+
+2. **Higher performance**. We change some default hyperparameters with no additional cost, which leads to a gain of performance for most models. Please refer to [compatibility](compatibility.md#training-hyperparameters) for details.
+
+3. **More documentation and tutorials**. We add a bunch of documentation and tutorials to help users get started more smoothly. Read it [here](https://mmdetection.readthedocs.io/en/latest/).
+
+4. **Support PyTorch 1.5**. The support for 1.1 and 1.2 is dropped, and we switch to some new APIs.
+
+5. **Better configuration system**. Inheritance is supported to reduce the redundancy of configs.
+
+6. **Better modular design**. Towards the goal of simplicity and flexibility, we simplify some encapsulation while add more other configurable modules like BBoxCoder, IoUCalculator, OptimizerConstructor, RoIHead. Target computation is also included in heads and the call hierarchy is simpler.
+
+7. Support new methods: [FSAF](https://arxiv.org/abs/1903.00621) and PAFPN (part of [PAFPN](https://arxiv.org/abs/1803.01534)).
+
+**Breaking Changes**
+Models training with MMDetection 1.x are not fully compatible with 2.0, please refer to the [compatibility doc](compatibility.md) for the details and how to migrate to the new version.
+
+**Improvements**
+
+- Unify cuda and cpp API for custom ops. (#2277)
+- New config files with inheritance. (#2216)
+- Encapsulate the second stage into RoI heads. (#1999)
+- Refactor GCNet/EmpericalAttention into plugins. (#2345)
+- Set low quality match as an option in IoU-based bbox assigners. (#2375)
+- Change the codebase's coordinate system. (#2380)
+- Refactor the category order in heads. 0 means the first positive class instead of background now. (#2374)
+- Add bbox sampler and assigner registry. (#2419)
+- Speed up the inference of RPN. (#2420)
+- Add `train_cfg` and `test_cfg` as class members in all anchor heads. (#2422)
+- Merge target computation methods into heads. (#2429)
+- Add bbox coder to support different bbox encoding and losses. (#2480)
+- Unify the API for regression loss. (#2156)
+- Refactor Anchor Generator. (#2474)
+- Make `lr` an optional argument for optimizers. (#2509)
+- Migrate to modules and methods in MMCV. (#2502, #2511, #2569, #2572)
+- Support PyTorch 1.5. (#2524)
+- Drop the support for Python 3.5 and use F-string in the codebase. (#2531)
+
+**Bug Fixes**
+
+- Fix the scale factors for resized images without keep the aspect ratio. (#2039)
+- Check if max_num > 0 before slicing in NMS. (#2486)
+- Fix Deformable RoIPool when there is no instance. (#2490)
+- Fix the default value of assigned labels. (#2536)
+- Fix the evaluation of Cityscapes. (#2578)
+
+**New Features**
+
+- Add deep_stem and avg_down option to ResNet, i.e., support ResNetV1d. (#2252)
+- Add L1 loss. (#2376)
+- Support both polygon and bitmap for instance masks. (#2353, #2540)
+- Support CPU mode for inference. (#2385)
+- Add optimizer constructor for complicated configuration of optimizers. (#2397, #2488)
+- Implement PAFPN. (#2392)
+- Support empty tensor input for some modules. (#2280)
+- Support for custom dataset classes without overriding it. (#2408, #2443)
+- Support to train subsets of coco dataset. (#2340)
+- Add iou_calculator to potentially support more IoU calculation methods. (2405)
+- Support class wise mean AP (was removed in the last version). (#2459)
+- Add option to save the testing result images. (#2414)
+- Support MomentumUpdaterHook. (#2571)
+- Add a demo to inference a single image. (#2605)
+
+### v1.1.0 (24/2/2020)
+
+**Highlights**
+
+- Dataset evaluation is rewritten with a unified api, which is used by both evaluation hooks and test scripts.
+- Support new methods: [CARAFE](https://arxiv.org/abs/1905.02188).
+
+**Breaking Changes**
+
+- The new MMDDP inherits from the official DDP, thus the `__init__` api is changed to be the same as official DDP.
+- The `mask_head` field in HTC config files is modified.
+- The evaluation and testing script is updated.
+- In all transforms, instance masks are stored as a numpy array shaped (n, h, w) instead of a list of (h, w) arrays, where n is the number of instances.
+
+**Bug Fixes**
+
+- Fix IOU assigners when ignore_iof_thr > 0 and there is no pred boxes. (#2135)
+- Fix mAP evaluation when there are no ignored boxes. (#2116)
+- Fix the empty RoI input for Deformable RoI Pooling. (#2099)
+- Fix the dataset settings for multiple workflows. (#2103)
+- Fix the warning related to `torch.uint8` in PyTorch 1.4. (#2105)
+- Fix the inference demo on devices other than gpu:0. (#2098)
+- Fix Dockerfile. (#2097)
+- Fix the bug that `pad_val` is unused in Pad transform. (#2093)
+- Fix the albumentation transform when there is no ground truth bbox. (#2032)
+
+**Improvements**
+
+- Use torch instead of numpy for random sampling. (#2094)
+- Migrate to the new MMDDP implementation in MMCV v0.3. (#2090)
+- Add meta information in logs. (#2086)
+- Rewrite Soft NMS with pytorch extension and remove cython as a dependency. (#2056)
+- Rewrite dataset evaluation. (#2042, #2087, #2114, #2128)
+- Use numpy array for masks in transforms. (#2030)
+
+**New Features**
+
+- Implement "CARAFE: Content-Aware ReAssembly of FEatures". (#1583)
+- Add `worker_init_fn()` in data_loader when seed is set. (#2066, #2111)
+- Add logging utils. (#2035)
+
+### v1.0.0 (30/1/2020)
+
+This release mainly improves the code quality and add more docstrings.
+
+**Highlights**
+
+- Documentation is online now: https://mmdetection.readthedocs.io.
+- Support new models: [ATSS](https://arxiv.org/abs/1912.02424).
+- DCN is now available with the api `build_conv_layer` and `ConvModule` like the normal conv layer.
+- A tool to collect environment information is available for trouble shooting.
+
+**Bug Fixes**
+
+- Fix the incompatibility of the latest numpy and pycocotools. (#2024)
+- Fix the case when distributed package is unavailable, e.g., on Windows. (#1985)
+- Fix the dimension issue for `refine_bboxes()`. (#1962)
+- Fix the typo when `seg_prefix` is a list. (#1906)
+- Add segmentation map cropping to RandomCrop. (#1880)
+- Fix the return value of `ga_shape_target_single()`. (#1853)
+- Fix the loaded shape of empty proposals. (#1819)
+- Fix the mask data type when using albumentation. (#1818)
+
+**Improvements**
+
+- Enhance AssignResult and SamplingResult. (#1995)
+- Add ability to overwrite existing module in Registry. (#1982)
+- Reorganize requirements and make albumentations and imagecorruptions optional. (#1969)
+- Check NaN in `SSDHead`. (#1935)
+- Encapsulate the DCN in ResNe(X)t into a ConvModule & Conv_layers. (#1894)
+- Refactoring for mAP evaluation and support multiprocessing and logging. (#1889)
+- Init the root logger before constructing Runner to log more information. (#1865)
+- Split `SegResizeFlipPadRescale` into different existing transforms. (#1852)
+- Move `init_dist()` to MMCV. (#1851)
+- Documentation and docstring improvements. (#1971, #1938, #1869, #1838)
+- Fix the color of the same class for mask visualization. (#1834)
+- Remove the option `keep_all_stages` in HTC and Cascade R-CNN. (#1806)
+
+**New Features**
+
+- Add two test-time options `crop_mask` and `rle_mask_encode` for mask heads. (#2013)
+- Support loading grayscale images as single channel. (#1975)
+- Implement "Bridging the Gap Between Anchor-based and Anchor-free Detection via Adaptive Training Sample Selection". (#1872)
+- Add sphinx generated docs. (#1859, #1864)
+- Add GN support for flops computation. (#1850)
+- Collect env info for trouble shooting. (#1812)
+
+### v1.0rc1 (13/12/2019)
+
+The RC1 release mainly focuses on improving the user experience, and fixing bugs.
+
+**Highlights**
+
+- Support new models: [FoveaBox](https://arxiv.org/abs/1904.03797), [RepPoints](https://arxiv.org/abs/1904.11490) and [FreeAnchor](https://arxiv.org/abs/1909.02466).
+- Add a Dockerfile.
+- Add a jupyter notebook demo and a webcam demo.
+- Setup the code style and CI.
+- Add lots of docstrings and unit tests.
+- Fix lots of bugs.
+
+**Breaking Changes**
+
+- There was a bug for computing COCO-style mAP w.r.t different scales (AP_s, AP_m, AP_l), introduced by #621. (#1679)
+
+**Bug Fixes**
+
+- Fix a sampling interval bug in Libra R-CNN. (#1800)
+- Fix the learning rate in SSD300 WIDER FACE. (#1781)
+- Fix the scaling issue when `keep_ratio=False`. (#1730)
+- Fix typos. (#1721, #1492, #1242, #1108, #1107)
+- Fix the shuffle argument in `build_dataloader`. (#1693)
+- Clip the proposal when computing mask targets. (#1688)
+- Fix the "index out of range" bug for samplers in some corner cases. (#1610, #1404)
+- Fix the NMS issue on devices other than GPU:0. (#1603)
+- Fix SSD Head and GHM Loss on CPU. (#1578)
+- Fix the OOM error when there are too many gt bboxes. (#1575)
+- Fix the wrong keyword argument `nms_cfg` in HTC. (#1573)
+- Process masks and semantic segmentation in Expand and MinIoUCrop transforms. (#1550, #1361)
+- Fix a scale bug in the Non Local op. (#1528)
+- Fix a bug in transforms when `gt_bboxes_ignore` is None. (#1498)
+- Fix a bug when `img_prefix` is None. (#1497)
+- Pass the device argument to `grid_anchors` and `valid_flags`. (#1478)
+- Fix the data pipeline for test_robustness. (#1476)
+- Fix the argument type of deformable pooling. (#1390)
+- Fix the coco_eval when there are only two classes. (#1376)
+- Fix a bug in Modulated DeformableConv when deformable_group>1. (#1359)
+- Fix the mask cropping in RandomCrop. (#1333)
+- Fix zero outputs in DeformConv when not running on cuda:0. (#1326)
+- Fix the type issue in Expand. (#1288)
+- Fix the inference API. (#1255)
+- Fix the inplace operation in Expand. (#1249)
+- Fix the from-scratch training config. (#1196)
+- Fix inplace add in RoIExtractor which cause an error in PyTorch 1.2. (#1160)
+- Fix FCOS when input images has no positive sample. (#1136)
+- Fix recursive imports. (#1099)
+
+**Improvements**
+
+- Print the config file and mmdet version in the log. (#1721)
+- Lint the code before compiling in travis CI. (#1715)
+- Add a probability argument for the `Expand` transform. (#1651)
+- Update the PyTorch and CUDA version in the docker file. (#1615)
+- Raise a warning when specifying `--validate` in non-distributed training. (#1624, #1651)
+- Beautify the mAP printing. (#1614)
+- Add pre-commit hook. (#1536)
+- Add the argument `in_channels` to backbones. (#1475)
+- Add lots of docstrings and unit tests, thanks to [@Erotemic](https://github.com/Erotemic). (#1603, #1517, #1506, #1505, #1491, #1479, #1477, #1475, #1474)
+- Add support for multi-node distributed test when there is no shared storage. (#1399)
+- Optimize Dockerfile to reduce the image size. (#1306)
+- Update new results of HRNet. (#1284, #1182)
+- Add an argument `no_norm_on_lateral` in FPN. (#1240)
+- Test the compiling in CI. (#1235)
+- Move docs to a separate folder. (#1233)
+- Add a jupyter notebook demo. (#1158)
+- Support different type of dataset for training. (#1133)
+- Use int64_t instead of long in cuda kernels. (#1131)
+- Support unsquare RoIs for bbox and mask heads. (#1128)
+- Manually add type promotion to make compatible to PyTorch 1.2. (#1114)
+- Allowing validation dataset for computing validation loss. (#1093)
+- Use `.scalar_type()` instead of `.type()` to suppress some warnings. (#1070)
+
+**New Features**
+
+- Add an option `--with_ap` to compute the AP for each class. (#1549)
+- Implement "FreeAnchor: Learning to Match Anchors for Visual Object Detection". (#1391)
+- Support [Albumentations](https://github.com/albumentations-team/albumentations) for augmentations in the data pipeline. (#1354)
+- Implement "FoveaBox: Beyond Anchor-based Object Detector". (#1339)
+- Support horizontal and vertical flipping. (#1273, #1115)
+- Implement "RepPoints: Point Set Representation for Object Detection". (#1265)
+- Add test-time augmentation to HTC and Cascade R-CNN. (#1251)
+- Add a COCO result analysis tool. (#1228)
+- Add Dockerfile. (#1168)
+- Add a webcam demo. (#1155, #1150)
+- Add FLOPs counter. (#1127)
+- Allow arbitrary layer order for ConvModule. (#1078)
+
+### v1.0rc0 (27/07/2019)
+
+- Implement lots of new methods and components (Mixed Precision Training, HTC, Libra R-CNN, Guided Anchoring, Empirical Attention, Mask Scoring R-CNN, Grid R-CNN (Plus), GHM, GCNet, FCOS, HRNet, Weight Standardization, etc.). Thank all collaborators!
+- Support two additional datasets: WIDER FACE and Cityscapes.
+- Refactoring for loss APIs and make it more flexible to adopt different losses and related hyper-parameters.
+- Speed up multi-gpu testing.
+- Integrate all compiling and installing in a single script.
+
+### v0.6.0 (14/04/2019)
+
+- Up to 30% speedup compared to the model zoo.
+- Support both PyTorch stable and nightly version.
+- Replace NMS and SigmoidFocalLoss with Pytorch CUDA extensions.
+
+### v0.6rc0(06/02/2019)
+
+- Migrate to PyTorch 1.0.
+
+### v0.5.7 (06/02/2019)
+
+- Add support for Deformable ConvNet v2. (Many thanks to the authors and [@chengdazhi](https://github.com/chengdazhi))
+- This is the last release based on PyTorch 0.4.1.
+
+### v0.5.6 (17/01/2019)
+
+- Add support for Group Normalization.
+- Unify RPNHead and single stage heads (RetinaHead, SSDHead) with AnchorHead.
+
+### v0.5.5 (22/12/2018)
+
+- Add SSD for COCO and PASCAL VOC.
+- Add ResNeXt backbones and detection models.
+- Refactoring for Samplers/Assigners and add OHEM.
+- Add VOC dataset and evaluation scripts.
+
+### v0.5.4 (27/11/2018)
+
+- Add SingleStageDetector and RetinaNet.
+
+### v0.5.3 (26/11/2018)
+
+- Add Cascade R-CNN and Cascade Mask R-CNN.
+- Add support for Soft-NMS in config files.
+
+### v0.5.2 (21/10/2018)
+
+- Add support for custom datasets.
+- Add a script to convert PASCAL VOC annotations to the expected format.
+
+### v0.5.1 (20/10/2018)
+
+- Add BBoxAssigner and BBoxSampler, the `train_cfg` field in config files are restructured.
+- `ConvFCRoIHead` / `SharedFCRoIHead` are renamed to `ConvFCBBoxHead` / `SharedFCBBoxHead` for consistency.