benchmarks.md 12.6 KB
Newer Older
zhangwenwei's avatar
zhangwenwei committed
1
2
3
# Benchmarks

Here we benchmark the training and testing speed of models in MMDetection3D,
zhangwenwei's avatar
zhangwenwei committed
4
with some other open source 3D detection codebases.
zhangwenwei's avatar
zhangwenwei committed
5
6
7

## Settings

8
9
10
11
- Hardwares: 8 NVIDIA Tesla V100 (32G) GPUs, Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
- Software: Python 3.7, CUDA 10.1, cuDNN 7.6.5, PyTorch 1.3, numba 0.48.0.
- Model: Since all the other codebases implements different models, we compare the corresponding models including SECOND, PointPillars, Part-A2, and VoteNet with them separately.
- Metrics: We use the average throughput in iterations of the entire training run and skip the first 50 iterations of each epoch to skip GPU warmup time.
zhangwenwei's avatar
zhangwenwei committed
12
13
14

## Main Results

Wenwei Zhang's avatar
Wenwei Zhang committed
15
We compare the training speed (samples/s) with other codebases if they implement the similar models. The results are as below, the greater the numbers in the table, the faster of the training process. The models that are not supported by other codebases are marked by `×`.
zhangwenwei's avatar
zhangwenwei committed
16

17
18
19
20
21
22
23
|       Methods       | MMDetection3D | OpenPCDet | votenet | Det3D |
| :-----------------: | :-----------: | :-------: | :-----: | :---: |
|       VoteNet       |      358      |     ×     |   77    |   ×   |
|  PointPillars-car   |      141      |     ×     |    ×    |  140  |
| PointPillars-3class |      107      |    44     |    ×    |   ×   |
|       SECOND        |      40       |    30     |    ×    |   ×   |
|       Part-A2       |      17       |    14     |    ×    |   ×   |
zhangwenwei's avatar
zhangwenwei committed
24
25
26

## Details of Comparison

27
28
### Modification for Calculating Speed

29
30
31
- __MMDetection3D__: We try to use as similar settings as those of other codebases as possible using [benchmark configs](https://github.com/open-mmlab/MMDetection3D/blob/master/configs/benchmark).

- __Det3D__: For comparison with Det3D, we use the commit [519251e](https://github.com/poodarchu/Det3D/tree/519251e72a5c1fdd58972eabeac67808676b9bb7).
32

33
- __OpenPCDet__: For comparison with OpenPCDet, we use the commit [b32fbddb](https://github.com/open-mmlab/OpenPCDet/tree/b32fbddbe06183507bad433ed99b407cbc2175c2).
zhangwenwei's avatar
zhangwenwei committed
34

35
  For training speed, we add code to record the running time in the file `./tools/train_utils/train_utils.py`. We calculate the speed of each epoch, and report the average speed of all the epochs.
liyinhao's avatar
liyinhao committed
36

37
  <details>
liyinhao's avatar
liyinhao committed
38
39
40
41
    <summary>
    (diff to make it use the same method for benchmarking speed - click to expand)
    </summary>

42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
  ```diff
  diff --git a/tools/train_utils/train_utils.py b/tools/train_utils/train_utils.py
  index 91f21dd..021359d 100644
  --- a/tools/train_utils/train_utils.py
  +++ b/tools/train_utils/train_utils.py
  @@ -2,6 +2,7 @@ import torch
   import os
   import glob
   import tqdm
  +import datetime
   from torch.nn.utils import clip_grad_norm_


  @@ -13,7 +14,10 @@ def train_one_epoch(model, optimizer, train_loader, model_func, lr_scheduler, ac
       if rank == 0:
           pbar = tqdm.tqdm(total=total_it_each_epoch, leave=leave_pbar, desc='train', dynamic_ncols=True)

  +    start_time = None
       for cur_it in range(total_it_each_epoch):
  +        if cur_it > 49 and start_time is None:
  +            start_time = datetime.datetime.now()
           try:
               batch = next(dataloader_iter)
           except StopIteration:
  @@ -55,9 +59,11 @@ def train_one_epoch(model, optimizer, train_loader, model_func, lr_scheduler, ac
                   tb_log.add_scalar('learning_rate', cur_lr, accumulated_iter)
                   for key, val in tb_dict.items():
                       tb_log.add_scalar('train_' + key, val, accumulated_iter)
  +    endtime = datetime.datetime.now()
  +    speed = (endtime - start_time).seconds / (total_it_each_epoch - 50)
       if rank == 0:
           pbar.close()
  -    return accumulated_iter
  +    return accumulated_iter, speed


   def train_model(model, optimizer, train_loader, model_func, lr_scheduler, optim_cfg,
  @@ -65,6 +71,7 @@ def train_model(model, optimizer, train_loader, model_func, lr_scheduler, optim_
                   lr_warmup_scheduler=None, ckpt_save_interval=1, max_ckpt_save_num=50,
                   merge_all_iters_to_one_epoch=False):
       accumulated_iter = start_iter
  +    speeds = []
       with tqdm.trange(start_epoch, total_epochs, desc='epochs', dynamic_ncols=True, leave=(rank == 0)) as tbar:
           total_it_each_epoch = len(train_loader)
           if merge_all_iters_to_one_epoch:
  @@ -82,7 +89,7 @@ def train_model(model, optimizer, train_loader, model_func, lr_scheduler, optim_
                   cur_scheduler = lr_warmup_scheduler
               else:
                   cur_scheduler = lr_scheduler
  -            accumulated_iter = train_one_epoch(
  +            accumulated_iter, speed = train_one_epoch(
                   model, optimizer, train_loader, model_func,
                   lr_scheduler=cur_scheduler,
                   accumulated_iter=accumulated_iter, optim_cfg=optim_cfg,
  @@ -91,7 +98,7 @@ def train_model(model, optimizer, train_loader, model_func, lr_scheduler, optim_
                   total_it_each_epoch=total_it_each_epoch,
                   dataloader_iter=dataloader_iter
               )
  -
  +            speeds.append(speed)
               # save trained model
               trained_epoch = cur_epoch + 1
               if trained_epoch % ckpt_save_interval == 0 and rank == 0:
  @@ -107,6 +114,8 @@ def train_model(model, optimizer, train_loader, model_func, lr_scheduler, optim_
                   save_checkpoint(
                       checkpoint_state(model, optimizer, trained_epoch, accumulated_iter), filename=ckpt_name,
                   )
  +            print(speed)
  +    print(f'*******{sum(speeds) / len(speeds)}******')


   def model_state_to_cpu(model_state):
  ```

  </details>
liyinhao's avatar
liyinhao committed
117

zhangwenwei's avatar
zhangwenwei committed
118
119
### VoteNet

120
- __MMDetection3D__: With release v0.1.0, run
liyinhao's avatar
liyinhao committed
121
122

  ```bash
123
  ./tools/dist_train.sh configs/votenet/votenet_8xb16_sunrgbd-3d.py 8 --no-validate
liyinhao's avatar
liyinhao committed
124
125
  ```

126
- __votenet__: At commit [2f6d6d3](https://github.com/facebookresearch/votenet/tree/2f6d6d36ff98d96901182e935afe48ccee82d566), run
liyinhao's avatar
liyinhao committed
127
128

  ```bash
zhangwenwei's avatar
zhangwenwei committed
129
  python train.py --dataset sunrgbd --batch_size 16
liyinhao's avatar
liyinhao committed
130
131
  ```

wuyuefeng's avatar
wuyuefeng committed
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
  Then benchmark the test speed by running

  ```bash
  python eval.py --dataset sunrgbd --checkpoint_path log_sunrgbd/checkpoint.tar --batch_size 1 --dump_dir eval_sunrgbd --cluster_sampling seed_fps --use_3d_nms --use_cls_nms --per_class_proposal
  ```

  Note that eval.py is modified to compute inference time.

  <details>
  <summary>
  (diff to benchmark the similar models - click to expand)
  </summary>

  ```diff
  diff --git a/eval.py b/eval.py
    index c0b2886..04921e9 100644
    --- a/eval.py
    +++ b/eval.py
    @@ -10,6 +10,7 @@ import os
     import sys
     import numpy as np
     from datetime import datetime
    +import time
     import argparse
     import importlib
     import torch
    @@ -28,7 +29,7 @@ parser.add_argument('--checkpoint_path', default=None, help='Model checkpoint pa
     parser.add_argument('--dump_dir', default=None, help='Dump dir to save sample outputs [default: None]')
     parser.add_argument('--num_point', type=int, default=20000, help='Point Number [default: 20000]')
     parser.add_argument('--num_target', type=int, default=256, help='Point Number [default: 256]')
    -parser.add_argument('--batch_size', type=int, default=8, help='Batch Size during training [default: 8]')
    +parser.add_argument('--batch_size', type=int, default=1, help='Batch Size during training [default: 8]')
     parser.add_argument('--vote_factor', type=int, default=1, help='Number of votes generated from each seed [default: 1]')
     parser.add_argument('--cluster_sampling', default='vote_fps', help='Sampling strategy for vote clusters: vote_fps, seed_fps, random [default: vote_fps]')
     parser.add_argument('--ap_iou_thresholds', default='0.25,0.5', help='A list of AP IoU thresholds [default: 0.25,0.5]')
    @@ -132,6 +133,7 @@ CONFIG_DICT = {'remove_empty_box': (not FLAGS.faster_eval), 'use_3d_nms': FLAGS.
     # ------------------------------------------------------------------------- GLOBAL CONFIG END

     def evaluate_one_epoch():
    +    time_list = list()
         stat_dict = {}
         ap_calculator_list = [APCalculator(iou_thresh, DATASET_CONFIG.class2type) \
             for iou_thresh in AP_IOU_THRESHOLDS]
    @@ -144,6 +146,8 @@ def evaluate_one_epoch():

             # Forward pass
             inputs = {'point_clouds': batch_data_label['point_clouds']}
    +        torch.cuda.synchronize()
    +        start_time = time.perf_counter()
             with torch.no_grad():
                 end_points = net(inputs)

    @@ -161,6 +165,12 @@ def evaluate_one_epoch():

             batch_pred_map_cls = parse_predictions(end_points, CONFIG_DICT)
             batch_gt_map_cls = parse_groundtruths(end_points, CONFIG_DICT)
    +        torch.cuda.synchronize()
    +        elapsed = time.perf_counter() - start_time
    +        time_list.append(elapsed)
    +
    +        if len(time_list==200):
    +            print("average inference time: %4f"%(sum(time_list[5:])/len(time_list[5:])))
             for ap_calculator in ap_calculator_list:
                 ap_calculator.step(batch_pred_map_cls, batch_gt_map_cls)

  ```

199
### PointPillars-car
zhangwenwei's avatar
zhangwenwei committed
200

201
- __MMDetection3D__: With release v0.1.0, run
liyinhao's avatar
liyinhao committed
202
203

  ```bash
wuyuefeng's avatar
wuyuefeng committed
204
  ./tools/dist_train.sh configs/benchmark/hv_pointpillars_secfpn_3x8_100e_det3d_kitti-3d-car.py 8 --no-validate
ZwwWayne's avatar
ZwwWayne committed
205
  ```
liyinhao's avatar
liyinhao committed
206

207
- __Det3D__: At commit [519251e](https://github.com/poodarchu/Det3D/tree/519251e72a5c1fdd58972eabeac67808676b9bb7), use `kitti_point_pillars_mghead_syncbn.py` and run
liyinhao's avatar
liyinhao committed
208
209

  ```bash
ZwwWayne's avatar
ZwwWayne committed
210
211
  ./tools/scripts/train.sh --launcher=slurm --gpus=8
  ```
liyinhao's avatar
liyinhao committed
212

ZwwWayne's avatar
ZwwWayne committed
213
214
215
216
  Note that the config in train.sh is modified to train point pillars.

  <details>
  <summary>
liyinhao's avatar
liyinhao committed
217
  (diff to benchmark the similar models - click to expand)
ZwwWayne's avatar
ZwwWayne committed
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
  </summary>

  ```diff
  diff --git a/tools/scripts/train.sh b/tools/scripts/train.sh
  index 3a93f95..461e0ea 100755
  --- a/tools/scripts/train.sh
  +++ b/tools/scripts/train.sh
  @@ -16,9 +16,9 @@ then
   fi

   # Voxelnet
  -python -m torch.distributed.launch --nproc_per_node=8 ./tools/train.py examples/second/configs/  kitti_car_vfev3_spmiddlefhd_rpn1_mghead_syncbn.py --work_dir=$SECOND_WORK_DIR
  +# python -m torch.distributed.launch --nproc_per_node=8 ./tools/train.py examples/second/configs/  kitti_car_vfev3_spmiddlefhd_rpn1_mghead_syncbn.py --work_dir=$SECOND_WORK_DIR
   # python -m torch.distributed.launch --nproc_per_node=8 ./tools/train.py examples/cbgs/configs/  nusc_all_vfev3_spmiddleresnetfhd_rpn2_mghead_syncbn.py --work_dir=$NUSC_CBGS_WORK_DIR
   # python -m torch.distributed.launch --nproc_per_node=8 ./tools/train.py examples/second/configs/  lyft_all_vfev3_spmiddleresnetfhd_rpn2_mghead_syncbn.py --work_dir=$LYFT_CBGS_WORK_DIR

   # PointPillars
  -# python -m torch.distributed.launch --nproc_per_node=8 ./tools/train.py ./examples/point_pillars/configs/  original_pp_mghead_syncbn_kitti.py --work_dir=$PP_WORK_DIR
  +python -m torch.distributed.launch --nproc_per_node=8 ./tools/train.py ./examples/point_pillars/configs/  kitti_point_pillars_mghead_syncbn.py
  ```
wuyuefeng's avatar
wuyuefeng committed
238

ZwwWayne's avatar
ZwwWayne committed
239
  </details>
zhangwenwei's avatar
zhangwenwei committed
240

241
### PointPillars-3class
zhangwenwei's avatar
zhangwenwei committed
242

243
- __MMDetection3D__: With release v0.1.0, run
zhangwenwei's avatar
zhangwenwei committed
244
245
246
247
248

  ```bash
  ./tools/dist_train.sh configs/benchmark/hv_pointpillars_secfpn_4x8_80e_pcdet_kitti-3d-3class.py 8 --no-validate
  ```

249
- __OpenPCDet__: At commit [b32fbddb](https://github.com/open-mmlab/OpenPCDet/tree/b32fbddbe06183507bad433ed99b407cbc2175c2), run
zhangwenwei's avatar
zhangwenwei committed
250
251
252

  ```bash
  cd tools
253
  sh scripts/slurm_train.sh ${PARTITION} ${JOB_NAME} 8  --cfg_file ./cfgs/kitti_models/pointpillar.yaml --batch_size 32  --workers 32 --epochs 80
zhangwenwei's avatar
zhangwenwei committed
254
255
  ```

zhangwenwei's avatar
zhangwenwei committed
256
257
### SECOND

Wenwei Zhang's avatar
Wenwei Zhang committed
258
259
For SECOND, we mean the [SECONDv1.5](https://github.com/traveller59/second.pytorch/blob/master/second/configs/all.fhd.config) that was first implemented in [second.Pytorch](https://github.com/traveller59/second.pytorch). Det3D's implementation of SECOND uses its self-implemented Multi-Group Head, so its speed is not compatible with other codebases.

260
- __MMDetection3D__: With release v0.1.0, run
zhangwenwei's avatar
zhangwenwei committed
261

liyinhao's avatar
liyinhao committed
262
263
264
  ```bash
  ./tools/dist_train.sh configs/benchmark/hv_second_secfpn_4x8_80e_pcdet_kitti-3d-3class.py 8 --no-validate
  ```
zhangwenwei's avatar
zhangwenwei committed
265

266
- __OpenPCDet__: At commit [b32fbddb](https://github.com/open-mmlab/OpenPCDet/tree/b32fbddbe06183507bad433ed99b407cbc2175c2), run
liyinhao's avatar
liyinhao committed
267
268
269

  ```bash
  cd tools
270
  sh ./scripts/slurm_train.sh ${PARTITION} ${JOB_NAME} 8  --cfg_file ./cfgs/kitti_models/second.yaml --batch_size 32  --workers 32 --epochs 80
liyinhao's avatar
liyinhao committed
271
  ```
zhangwenwei's avatar
zhangwenwei committed
272
273
274

### Part-A2

275
- __MMDetection3D__: With release v0.1.0, run
zhangwenwei's avatar
zhangwenwei committed
276

liyinhao's avatar
liyinhao committed
277
278
279
280
  ```bash
  ./tools/dist_train.sh configs/benchmark/hv_PartA2_secfpn_4x8_cyclic_80e_pcdet_kitti-3d-3class.py 8 --no-validate
  ```

281
- __OpenPCDet__: At commit [b32fbddb](https://github.com/open-mmlab/OpenPCDet/tree/b32fbddbe06183507bad433ed99b407cbc2175c2), train the model by running
liyinhao's avatar
liyinhao committed
282
283
284

  ```bash
  cd tools
285
  sh ./scripts/slurm_train.sh ${PARTITION} ${JOB_NAME} 8  --cfg_file ./cfgs/kitti_models/PartA2.yaml --batch_size 32 --workers 32 --epochs 80
liyinhao's avatar
liyinhao committed
286
  ```