README.md 8.8 KB
Newer Older
pangjm's avatar
pangjm committed
1

Kai Chen's avatar
Kai Chen committed
2
# mmdetection
Dahua Lin's avatar
Dahua Lin committed
3

Kai Chen's avatar
Kai Chen committed
4
## Introduction
Dahua Lin's avatar
Dahua Lin committed
5

Kai Chen's avatar
Kai Chen committed
6
7
8
The master branch works with **PyTorch 1.0**. If you would like to use PyTorch 0.4.1,
please checkout to the [pytorch-0.4.1](https://github.com/open-mmlab/mmdetection/tree/pytorch-0.4.1) branch.

Kai Chen's avatar
Kai Chen committed
9
10
mmdetection is an open source object detection toolbox based on PyTorch. It is
a part of the open-mmlab project developed by [Multimedia Laboratory, CUHK](http://mmlab.ie.cuhk.edu.hk/).
Dahua Lin's avatar
Dahua Lin committed
11

Kai Chen's avatar
Kai Chen committed
12
13
![demo image](demo/coco_test_12510.jpg)

Kai Chen's avatar
Kai Chen committed
14
### Major features
Dahua Lin's avatar
Dahua Lin committed
15
16
17

- **Modular Design**

pangjm's avatar
pangjm committed
18
19
  One can easily construct a customized object detection framework by combining different components.

Dahua Lin's avatar
Dahua Lin committed
20
21
- **Support of multiple frameworks out of box**

Kai Chen's avatar
Kai Chen committed
22
  The toolbox directly supports popular detection frameworks, *e.g.* Faster RCNN, Mask RCNN, RetinaNet, etc.
Kai Chen's avatar
Kai Chen committed
23
24
25
26
27

- **Efficient**

  All basic bbox and mask operations run on GPUs now.
  The training speed is about 5% ~ 20% faster than Detectron for different models.
pangjm's avatar
pangjm committed
28

Dahua Lin's avatar
Dahua Lin committed
29
30
- **State of the art**

pangjm's avatar
pangjm committed
31
  This was the codebase of the *MMDet* team, who won the [COCO Detection 2018 challenge](http://cocodataset.org/#detection-leaderboard).
Kai Chen's avatar
Kai Chen committed
32

Kai Chen's avatar
Kai Chen committed
33
34
Apart from mmdetection, we also released a library [mmcv](https://github.com/open-mmlab/mmcv) for computer vision research,
which is heavily depended on by this toolbox.
Kai Chen's avatar
Kai Chen committed
35
36
37

## License

Kai Chen's avatar
Kai Chen committed
38
This project is released under the [Apache 2.0 license](LICENSE).
Kai Chen's avatar
Kai Chen committed
39

Kai Chen's avatar
Kai Chen committed
40
41
## Updates

Kai Chen's avatar
Kai Chen committed
42
43
44
v0.6rc0(06/02/2019)
- Migrate to PyTorch 1.0.

Kai Chen's avatar
Kai Chen committed
45
46
47
48
v0.5.7 (06/02/2019)
- Add support for Deformable ConvNet v2. (Many thanks to the authors and [@chengdazhi](https://github.com/chengdazhi))
- This is the last release based on PyTorch 0.4.1.

Kai Chen's avatar
Kai Chen committed
49
50
51
52
v0.5.6 (17/01/2019)
- Add support for Group Normalization.
- Unify RPNHead and single stage heads (RetinaHead, SSDHead) with AnchorHead.

Kai Chen's avatar
Kai Chen committed
53
54
55
56
57
58
v0.5.5 (22/12/2018)
- Add SSD for COCO and PASCAL VOC.
- Add ResNeXt backbones and detection models.
- Refactoring for Samplers/Assigners and add OHEM.
- Add VOC dataset and evaluation scripts.

Kai Chen's avatar
Kai Chen committed
59
60
61
v0.5.4 (27/11/2018)
- Add SingleStageDetector and RetinaNet.

Kai Chen's avatar
Kai Chen committed
62
63
v0.5.3 (26/11/2018)
- Add Cascade R-CNN and Cascade Mask R-CNN.
Kai Chen's avatar
Kai Chen committed
64
- Add support for Soft-NMS in config files.
Kai Chen's avatar
Kai Chen committed
65

Kai Chen's avatar
Kai Chen committed
66
67
68
69
v0.5.2 (21/10/2018)
- Add support for custom datasets.
- Add a script to convert PASCAL VOC annotations to the expected format.

Kai Chen's avatar
Kai Chen committed
70
71
72
73
v0.5.1 (20/10/2018)
- Add BBoxAssigner and BBoxSampler, the `train_cfg` field in config files are restructured.
- `ConvFCRoIHead` / `SharedFCRoIHead` are renamed to `ConvFCBBoxHead` / `SharedFCBBoxHead` for consistency.

Kai Chen's avatar
Kai Chen committed
74
75
## Benchmark and model zoo

Kai Chen's avatar
Kai Chen committed
76
77
78
79
80
Supported methods and backbones are shown in the below table.
Results and models are available in the [Model zoo](MODEL_ZOO.md).

|                    | ResNet   | ResNeXt  | SENet    | VGG      |
|--------------------|:--------:|:--------:|:--------:|:--------:|
pangjm's avatar
pangjm committed
81
82
83
84
85
86
| RPN                | ✓        | ✓        | ☐        | ✗        |
| Fast R-CNN         | ✓        | ✓        | ☐        | ✗        |
| Faster R-CNN       | ✓        | ✓        | ☐        | ✗        |
| Mask R-CNN         | ✓        | ✓        | ☐        | ✗        |
| Cascade R-CNN      | ✓        | ✓        | ☐        | ✗        |
| Cascade Mask R-CNN | ✓        | ✓        | ☐        | ✗        |
Kai Chen's avatar
Kai Chen committed
87
| SSD                | ✗        | ✗        | ✗        | ✓        |
pangjm's avatar
pangjm committed
88
| RetinaNet          | ✓        | ✓        | ☐        | ✗        |
Kai Chen's avatar
Kai Chen committed
89

Kai Chen's avatar
Kai Chen committed
90
Other features
Kai Chen's avatar
Kai Chen committed
91
- [x] DCNv2
Kai Chen's avatar
Kai Chen committed
92
93
94
95
96
- [x] Group Normalization
- [x] OHEM
- [x] Soft-NMS


Kai Chen's avatar
Kai Chen committed
97
98
## Installation

Kai Chen's avatar
Kai Chen committed
99
Please refer to [INSTALL.md](INSTALL.md) for installation and dataset preparation.
Kai Chen's avatar
Kai Chen committed
100

Kai Chen's avatar
Kai Chen committed
101
102
103

## Inference with pretrained models

Kai Chen's avatar
Kai Chen committed
104
105
106
107
108
109
110
111
112
113
114
115
### Test a dataset

- [x] single GPU testing
- [x] multiple GPU testing
- [x] visualize detection results

We allow to run one or multiple processes on each GPU, e.g. 8 processes on 8 GPU
or 16 processes on 8 GPU. When the GPU workload is not very heavy for a single
process, running multiple processes will accelerate the testing, which is specified
with the argument `--proc_per_gpu <PROCESS_NUM>`.


Kai Chen's avatar
Kai Chen committed
116
117
118
119
120
121
122
To test a dataset and save the results.

```shell
python tools/test.py <CONFIG_FILE> <CHECKPOINT_FILE> --gpus <GPU_NUM> --out <OUT_FILE>
```

To perform evaluation after testing, add `--eval <EVAL_TYPES>`. Supported types are:
Kai Chen's avatar
Kai Chen committed
123
124
125
`[proposal_fast, proposal, bbox, segm, keypoints]`.
`proposal_fast` denotes evaluating proposal recalls with our own implementation,
others denote evaluating the corresponding metric with the official coco api.
Kai Chen's avatar
Kai Chen committed
126

Kai Chen's avatar
Kai Chen committed
127
For example, to evaluate Mask R-CNN with 8 GPUs and save the result as `results.pkl`.
Kai Chen's avatar
Kai Chen committed
128
129
130
131
132

```shell
python tools/test.py configs/mask_rcnn_r50_fpn_1x.py <CHECKPOINT_FILE> --gpus 8 --out results.pkl --eval bbox segm
```

Kai Chen's avatar
Kai Chen committed
133
It is also convenient to visualize the results during testing by adding an argument `--show`.
Kai Chen's avatar
Kai Chen committed
134
135
136
137
138

```shell
python tools/test.py <CONFIG_FILE> <CHECKPOINT_FILE> --show
```

Kai Chen's avatar
Kai Chen committed
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
### Test image(s)

We provide some high-level apis (experimental) to test an image.

```python
import mmcv
from mmcv.runner import load_checkpoint
from mmdet.models import build_detector
from mmdet.apis import inference_detector, show_result

cfg = mmcv.Config.fromfile('configs/faster_rcnn_r50_fpn_1x.py')
cfg.model.pretrained = None

# construct the model and load checkpoint
model = build_detector(cfg.model, test_cfg=cfg.test_cfg)
_ = load_checkpoint(model, 'https://s3.ap-northeast-2.amazonaws.com/open-mmlab/mmdetection/models/faster_rcnn_r50_fpn_1x_20181010-3d1b3351.pth')

# test a single image
img = mmcv.imread('test.jpg')
result = inference_detector(model, img, cfg)
show_result(img, result)

# test a list of images
imgs = ['test1.jpg', 'test2.jpg']
for i, result in enumerate(inference_detector(model, imgs, cfg, device='cuda:0')):
    print(i, imgs[i])
    show_result(imgs[i], result)
```

Kai Chen's avatar
Kai Chen committed
168
169
170

## Train a model

Kai Chen's avatar
Kai Chen committed
171
mmdetection implements distributed training and non-distributed training,
Kai Chen's avatar
Kai Chen committed
172
173
which uses `MMDistributedDataParallel` and `MMDataParallel` respectively.

Kai Chen's avatar
Kai Chen committed
174
### Distributed training (Single or Multiples machines)
Kai Chen's avatar
Kai Chen committed
175

Kai Chen's avatar
Kai Chen committed
176
mmdetection potentially supports multiple launch methods, e.g., PyTorch’s built-in launch utility, slurm and MPI.
Kai Chen's avatar
Kai Chen committed
177
178
179
180
181
182
183
184
185
186

We provide a training script using the launch utility provided by PyTorch.

```shell
./tools/dist_train.sh <CONFIG_FILE> <GPU_NUM> [optional arguments]
```

Supported arguments are:

- --validate: perform evaluation every k (default=1) epochs during the training.
youkaichao's avatar
youkaichao committed
187
- --work_dir <WORK_DIR>: if specified, the path in config file will be replaced.
Kai Chen's avatar
Kai Chen committed
188
189
190
191
192
193
194

Expected results in WORK_DIR:

- log file
- saved checkpoints (every k epochs, defaults=1)
- a symbol link to the latest checkpoint

Kai Chen's avatar
Kai Chen committed
195
196
197
198
199
200
**Important**: The default learning rate is for 8 GPUs. If you use less or more than 8 GPUs, you need to set the learning rate proportional to the GPU num. E.g., modify lr to 0.01 for 4 GPUs or 0.04 for 16 GPUs.

### Non-distributed training

Please refer to `tools/train.py` for non-distributed training, which is not recommended
and left for debugging. Even on a single machine, distributed training is preferred.
Kai Chen's avatar
Kai Chen committed
201

Kai Chen's avatar
Kai Chen committed
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
### Train on custom datasets

We define a simple annotation format.

The annotation of a dataset is a list of dict, each dict corresponds to an image.
There are 3 field `filename` (relative path), `width`, `height` for testing,
and an additional field `ann` for training. `ann` is also a dict containing at least 2 fields:
`bboxes` and `labels`, both of which are numpy arrays. Some datasets may provide
annotations like crowd/difficult/ignored bboxes, we use `bboxes_ignore` and `labels_ignore`
to cover them.

Here is an example.
```
[
    {
        'filename': 'a.jpg',
        'width': 1280,
        'height': 720,
        'ann': {
            'bboxes': <np.ndarray> (n, 4),
            'labels': <np.ndarray> (n, ),
            'bboxes_ignore': <np.ndarray> (k, 4),
Kai Chen's avatar
Kai Chen committed
224
            'labels_ignore': <np.ndarray> (k, ) (optional field)
Kai Chen's avatar
Kai Chen committed
225
226
227
228
229
230
231
232
233
234
235
        }
    },
    ...
]
```

There are two ways to work with custom datasets.

- online conversion

  You can write a new Dataset class inherited from `CustomDataset`, and overwrite two methods
Kai Chen's avatar
Kai Chen committed
236
  `load_annotations(self, ann_file)` and `get_ann_info(self, idx)`, like [CocoDataset](mmdet/datasets/coco.py) and [VOCDataset](mmdet/datasets/voc.py).
Kai Chen's avatar
Kai Chen committed
237
238
239
240

- offline conversion

  You can convert the annotation format to the expected format above and save it to
Kai Chen's avatar
Kai Chen committed
241
  a pickle or json file, like [pascal_voc.py](tools/convert_datasets/pascal_voc.py).
Kai Chen's avatar
Kai Chen committed
242
243
  Then you can simply use `CustomDataset`.

Kai Chen's avatar
Kai Chen committed
244
## Technical details
Kai Chen's avatar
Kai Chen committed
245

pangjm's avatar
pangjm committed
246
Some implementation details and project structures are described in the [technical details](TECHNICAL_DETAILS.md).
Kai Chen's avatar
Kai Chen committed
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262

## Citation

If you use our codebase or models in your research, please cite this project.
We will release a paper or technical report later.

```
@misc{mmdetection2018,
  author =       {Kai Chen and Jiangmiao Pang and Jiaqi Wang and Yu Xiong and Xiaoxiao Li
                  and Shuyang Sun and Wansen Feng and Ziwei Liu and Jianping Shi and
                  Wanli Ouyang and Chen Change Loy and Dahua Lin},
  title =        {mmdetection},
  howpublished = {\url{https://github.com/open-mmlab/mmdetection}},
  year =         {2018}
}
```