data_pipeline.md 5.94 KB
Newer Older
1
# Customize Data Pipelines
zhangwenwei's avatar
zhangwenwei committed
2

zhangwenwei's avatar
Doc  
zhangwenwei committed
3
## Design of Data pipelines
zhangwenwei's avatar
zhangwenwei committed
4
5
6
7

Following typical conventions, we use `Dataset` and `DataLoader` for data loading
with multiple workers. `Dataset` returns a dict of data items corresponding
the arguments of models' forward method.
8
Since the data in object detection may not be the same size (point number, gt bbox size, etc.),
zhangwenwei's avatar
zhangwenwei committed
9
10
11
12
13
14
15
16
17
we introduce a new `DataContainer` type in MMCV to help collect and distribute
data of different size.
See [here](https://github.com/open-mmlab/mmcv/blob/master/mmcv/parallel/data_container.py) for more details.

The data preparation pipeline and the dataset is decomposed. Usually a dataset
defines how to process the annotations and a data pipeline defines all the steps to prepare a data dict.
A pipeline consists of a sequence of operations. Each operation takes a dict as input and also output a dict for the next transform.

We present a classical pipeline in the following figure. The blue blocks are pipeline operations. With the pipeline going on, each operator can add new keys (marked as green) to the result dict or update the existing keys (marked as orange).
18

19
![](../../../resources/data_pipeline.png)
zhangwenwei's avatar
zhangwenwei committed
20
21
22

The operations are categorized into data loading, pre-processing, formatting and test-time augmentation.

wangtai's avatar
wangtai committed
23
24
Here is an pipeline example for PointPillars.

zhangwenwei's avatar
zhangwenwei committed
25
26
```python
train_pipeline = [
wangtai's avatar
wangtai committed
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
    dict(
        type='LoadPointsFromFile',
        load_dim=5,
        use_dim=5,
        file_client_args=file_client_args),
    dict(
        type='LoadPointsFromMultiSweeps',
        sweeps_num=10,
        file_client_args=file_client_args),
    dict(type='LoadAnnotations3D', with_bbox_3d=True, with_label_3d=True),
    dict(
        type='GlobalRotScaleTrans',
        rot_range=[-0.3925, 0.3925],
        scale_ratio_range=[0.95, 1.05],
        translation_std=[0, 0, 0]),
    dict(type='RandomFlip3D', flip_ratio_bev_horizontal=0.5),
    dict(type='PointsRangeFilter', point_cloud_range=point_cloud_range),
    dict(type='ObjectRangeFilter', point_cloud_range=point_cloud_range),
    dict(type='ObjectNameFilter', classes=class_names),
    dict(type='PointShuffle'),
    dict(type='DefaultFormatBundle3D', class_names=class_names),
    dict(type='Collect3D', keys=['points', 'gt_bboxes_3d', 'gt_labels_3d'])
zhangwenwei's avatar
zhangwenwei committed
49
50
]
test_pipeline = [
wangtai's avatar
wangtai committed
51
52
53
54
55
56
57
58
59
    dict(
        type='LoadPointsFromFile',
        load_dim=5,
        use_dim=5,
        file_client_args=file_client_args),
    dict(
        type='LoadPointsFromMultiSweeps',
        sweeps_num=10,
        file_client_args=file_client_args),
zhangwenwei's avatar
zhangwenwei committed
60
61
62
    dict(
        type='MultiScaleFlipAug',
        img_scale=(1333, 800),
wangtai's avatar
wangtai committed
63
        pts_scale_ratio=1.0,
zhangwenwei's avatar
zhangwenwei committed
64
        flip=False,
wangtai's avatar
wangtai committed
65
66
        pcd_horizontal_flip=False,
        pcd_vertical_flip=False,
zhangwenwei's avatar
zhangwenwei committed
67
        transforms=[
wangtai's avatar
wangtai committed
68
69
70
71
72
73
74
75
76
77
78
79
80
            dict(
                type='GlobalRotScaleTrans',
                rot_range=[0, 0],
                scale_ratio_range=[1., 1.],
                translation_std=[0, 0, 0]),
            dict(type='RandomFlip3D'),
            dict(
                type='PointsRangeFilter', point_cloud_range=point_cloud_range),
            dict(
                type='DefaultFormatBundle3D',
                class_names=class_names,
                with_label=False),
            dict(type='Collect3D', keys=['points'])
zhangwenwei's avatar
zhangwenwei committed
81
82
83
84
85
86
87
88
        ])
]
```

For each operation, we list the related dict fields that are added/updated/removed.

### Data loading

wangtai's avatar
wangtai committed
89
`LoadPointsFromFile`
90

wangtai's avatar
wangtai committed
91
- add: points
zhangwenwei's avatar
zhangwenwei committed
92

wangtai's avatar
wangtai committed
93
`LoadPointsFromMultiSweeps`
94

wangtai's avatar
wangtai committed
95
- update: points
zhangwenwei's avatar
zhangwenwei committed
96

97
`LoadAnnotations3D`
98

99
- add: gt_bboxes_3d, gt_labels_3d, gt_bboxes, gt_labels, pts_instance_mask, pts_semantic_mask, bbox3d_fields, pts_mask_fields, pts_seg_fields
zhangwenwei's avatar
zhangwenwei committed
100
101
102

### Pre-processing

wangtai's avatar
wangtai committed
103
`GlobalRotScaleTrans`
104

wangtai's avatar
wangtai committed
105
- add: pcd_trans, pcd_rotation, pcd_scale_factor
106
- update: points, \*bbox3d_fields
zhangwenwei's avatar
zhangwenwei committed
107

wangtai's avatar
wangtai committed
108
`RandomFlip3D`
109

wangtai's avatar
wangtai committed
110
- add: flip, pcd_horizontal_flip, pcd_vertical_flip
111
- update: points, \*bbox3d_fields
zhangwenwei's avatar
zhangwenwei committed
112

wangtai's avatar
wangtai committed
113
`PointsRangeFilter`
114

wangtai's avatar
wangtai committed
115
- update: points
zhangwenwei's avatar
zhangwenwei committed
116

wangtai's avatar
wangtai committed
117
`ObjectRangeFilter`
118

wangtai's avatar
wangtai committed
119
- update: gt_bboxes_3d, gt_labels_3d
zhangwenwei's avatar
zhangwenwei committed
120

wangtai's avatar
wangtai committed
121
`ObjectNameFilter`
122

wangtai's avatar
wangtai committed
123
- update: gt_bboxes_3d, gt_labels_3d
zhangwenwei's avatar
zhangwenwei committed
124

wangtai's avatar
wangtai committed
125
`PointShuffle`
126

wangtai's avatar
wangtai committed
127
- update: points
zhangwenwei's avatar
zhangwenwei committed
128

wangtai's avatar
wangtai committed
129
`PointsRangeFilter`
130

wangtai's avatar
wangtai committed
131
- update: points
zhangwenwei's avatar
zhangwenwei committed
132
133
134

### Formatting

wangtai's avatar
wangtai committed
135
`DefaultFormatBundle3D`
136

137
- update: points, gt_bboxes_3d, gt_labels_3d, gt_bboxes, gt_labels
zhangwenwei's avatar
zhangwenwei committed
138

wangtai's avatar
wangtai committed
139
`Collect3D`
140

zhangwenwei's avatar
zhangwenwei committed
141
- add: img_meta (the keys of img_meta is specified by `meta_keys`)
zhangwenwei's avatar
zhangwenwei committed
142
143
144
145
- remove: all other keys except for those specified by `keys`

### Test time augmentation

zhangwenwei's avatar
zhangwenwei committed
146
`MultiScaleFlipAug`
147

wangtai's avatar
wangtai committed
148
- update: scale, pcd_scale_factor, flip, flip_direction, pcd_horizontal_flip, pcd_vertical_flip with list of augmented data with these specific parameters
zhangwenwei's avatar
zhangwenwei committed
149

zhangwenwei's avatar
Doc  
zhangwenwei committed
150
## Extend and use custom pipelines
zhangwenwei's avatar
zhangwenwei committed
151

zhangwenwei's avatar
Doc  
zhangwenwei committed
152
1. Write a new pipeline in any file, e.g., `my_pipeline.py`. It takes a dict as input and return a dict.
zhangwenwei's avatar
zhangwenwei committed
153

154
155
   ```python
   from mmdet.datasets import PIPELINES
zhangwenwei's avatar
zhangwenwei committed
156

157
158
   @PIPELINES.register_module()
   class MyTransform:
zhangwenwei's avatar
zhangwenwei committed
159

160
161
162
163
       def __call__(self, results):
           results['dummy'] = True
           return results
   ```
zhangwenwei's avatar
zhangwenwei committed
164

zhangwenwei's avatar
Doc  
zhangwenwei committed
165
2. Import the new class.
zhangwenwei's avatar
zhangwenwei committed
166

167
168
169
   ```python
   from .my_pipeline import MyTransform
   ```
zhangwenwei's avatar
zhangwenwei committed
170

zhangwenwei's avatar
Doc  
zhangwenwei committed
171
3. Use it in config files.
zhangwenwei's avatar
zhangwenwei committed
172

173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
   ```python
   train_pipeline = [
       dict(
           type='LoadPointsFromFile',
           load_dim=5,
           use_dim=5,
           file_client_args=file_client_args),
       dict(
           type='LoadPointsFromMultiSweeps',
           sweeps_num=10,
           file_client_args=file_client_args),
       dict(type='LoadAnnotations3D', with_bbox_3d=True, with_label_3d=True),
       dict(
           type='GlobalRotScaleTrans',
           rot_range=[-0.3925, 0.3925],
           scale_ratio_range=[0.95, 1.05],
           translation_std=[0, 0, 0]),
       dict(type='RandomFlip3D', flip_ratio_bev_horizontal=0.5),
       dict(type='PointsRangeFilter', point_cloud_range=point_cloud_range),
       dict(type='ObjectRangeFilter', point_cloud_range=point_cloud_range),
       dict(type='ObjectNameFilter', classes=class_names),
       dict(type='MyTransform'),
       dict(type='PointShuffle'),
       dict(type='DefaultFormatBundle3D', class_names=class_names),
       dict(type='Collect3D', keys=['points', 'gt_bboxes_3d', 'gt_labels_3d'])
   ]
   ```