train_test.md 13.3 KB
Newer Older
1
# Inference and train with existing models and standard datasets
twang's avatar
twang committed
2
3
4
5
6

## Inference with existing models

Here we provide testing scripts to evaluate a whole dataset (SUNRGBD, ScanNet, KITTI, etc.).

7
For high-level apis easier to integrated into other projects and basic demos, please refer to Verification/Demo under [Get Started](https://mmdetection3d.readthedocs.io/en/dev-1.x/inference.html).
twang's avatar
twang committed
8
9
10
11

### Test existing models on standard datasets

- single GPU
12
- CPU
twang's avatar
twang committed
13
14
15
16
17
18
19
- single node multiple GPU
- multiple node

You can use the following commands to test a dataset.

```shell
# single-gpu testing
20
python tools/test.py ${CONFIG_FILE} ${CHECKPOINT_FILE} [--cfg-options test_evaluator.pklfile_prefix=${RESULT_FILE}]  [--show] [--show-dir ${SHOW_DIR}]
twang's avatar
twang committed
21

22
23
# CPU: disable GPUs and run single-gpu testing script (experimental)
export CUDA_VISIBLE_DEVICES=-1
24
python tools/test.py ${CONFIG_FILE} ${CHECKPOINT_FILE} [--cfg-options test_evaluator.pklfile_prefix=${RESULT_FILE}]  [--show] [--show-dir ${SHOW_DIR}]
25

twang's avatar
twang committed
26
# multi-gpu testing
27
./tools/dist_test.sh ${CONFIG_FILE} ${CHECKPOINT_FILE} ${GPU_NUM} [--cfg-options test_evaluator.pklfile_prefix=${RESULT_FILE}]  [--show] [--show-dir ${SHOW_DIR}]
twang's avatar
twang committed
28
29
```

30
31
32
33
**Note**:

For now, CPU testing is only supported for SMOKE.

twang's avatar
twang committed
34
Optional arguments:
35

twang's avatar
twang committed
36
- `--show`: If specified, detection results will be plotted in the silient mode. It is only applicable to single GPU testing and used for debugging and visualization. This should be used with `--show-dir`.
37
- `--show-dir`: If specified, detection results will be plotted on the `***_points.obj` and `***_pred.obj` files in the specified directory. It is only applicable to single GPU testing and used for debugging and visualization. You do NOT need a GUI available in your environment for using this option.
twang's avatar
twang committed
38

39
40
41
42
43
44
45
46
All evaluation related arguments are set in the `test_evaluator` in corresponding dataset configuration. such as
`test_evaluator = dict(type='KittiMetric', ann_file=data_root + 'kitti_infos_val.pkl', pklfile_prefix=None, submission_prefix=None)`

The arguments:

- `type`: The name of the corresponding metric, usually associated with the dataset.
- `ann_file`: The path of annotation file.
- `pklfile_prefix`: An optional argument. The filename of the output results in pickle format. If not specified, the results will not be saved to a file.
47
- `submission_prefix`: An optional argument. The results will be saved to a file then you can upload it to do the official evaluation.
48

twang's avatar
twang committed
49
50
51
52
Examples:

Assume that you have already downloaded the checkpoints to the directory `checkpoints/`.

53
1. Test VoteNet on ScanNet and save the points and prediction visualization results.
twang's avatar
twang committed
54
55

   ```shell
56
   python tools/test.py configs/votenet/votenet_8xb8_scannet-3d.py \
twang's avatar
twang committed
57
58
59
60
       checkpoints/votenet_8x8_scannet-3d-18class_20200620_230238-2cea9c3a.pth \
       --show --show-dir ./data/scannet/show_results
   ```

61
2. Test VoteNet on ScanNet, save the points, prediction, groundtruth visualization results, and evaluate the mAP.
twang's avatar
twang committed
62
63

   ```shell
64
   python tools/test.py configs/votenet/votenet_8xb8_scannet-3d.py \
twang's avatar
twang committed
65
       checkpoints/votenet_8x8_scannet-3d-18class_20200620_230238-2cea9c3a.pth \
66
       --show --show-dir ./data/scannet/show_results
twang's avatar
twang committed
67
68
   ```

69
3. Test VoteNet on ScanNet (without saving the test results) and evaluate the mAP.
twang's avatar
twang committed
70
71

   ```shell
72
   python tools/test.py configs/votenet/votenet_8xb8_scannet-3d.py \
73
       checkpoints/votenet_8x8_scannet-3d-18class_20200620_230238-2cea9c3a.pth
twang's avatar
twang committed
74
75
   ```

76
4. Test SECOND on KITTI with 8 GPUs, and evaluate the mAP.
twang's avatar
twang committed
77
78

   ```shell
79
80
   ./tools/slurm_test.sh ${PARTITION} ${JOB_NAME} configs/second/second_hv_secfpn_8xb6-80e_kitti-3d-3class.py \
       checkpoints/hv_second_secfpn_6x8_80e_kitti-3d-3class_20200620_230238-9208083a.pth
twang's avatar
twang committed
81
82
   ```

83
5. Test PointPillars on nuScenes with 8 GPUs, and generate the json file to be submit to the official evaluation server.
twang's avatar
twang committed
84
85

   ```shell
86
   ./tools/slurm_test.sh ${PARTITION} ${JOB_NAME} configs/pointpillars/pointpillars_hv_secfpn_sbn-all_8xb4-2x_nus-3d.py \
twang's avatar
twang committed
87
       checkpoints/hv_pointpillars_fpn_sbn-all_4x8_2x_nus-3d_20200620_230405-2fa62f3d.pth \
88
      --cfg-options 'test_evaluator.jsonfile_prefix=./pointpillars_nuscenes_results'
twang's avatar
twang committed
89
90
91
92
   ```

   The generated results be under `./pointpillars_nuscenes_results` directory.

93
6. Test SECOND on KITTI with 8 GPUs, and generate the pkl files and submission data to be submit to the official evaluation server.
twang's avatar
twang committed
94
95

   ```shell
96
   ./tools/slurm_test.sh ${PARTITION} ${JOB_NAME} configs/second/second_hv_secfpn_8xb6-80e_kitti-3d-3class.py \
twang's avatar
twang committed
97
       checkpoints/hv_second_secfpn_6x8_80e_kitti-3d-3class_20200620_230238-9208083a.pth \
98
       --cfg-options 'test_evaluator.pklfile_prefix=./second_kitti_results' 'submission_prefix=./second_kitti_results'
twang's avatar
twang committed
99
100
101
102
103
104
105
106
   ```

   The generated results be under `./second_kitti_results` directory.

7. Test PointPillars on Lyft with 8 GPUs, generate the pkl files and make a submission to the leaderboard.

   ```shell
   ./tools/slurm_test.sh ${PARTITION} ${JOB_NAME} configs/pointpillars/hv_pointpillars_fpn_sbn-2x8_2x_lyft-3d.py \
107
108
109
110
       checkpoints/hv_pointpillars_fpn_sbn-2x8_2x_lyft-3d_latest.pth \
       --cfg-options 'test_evaluator.jsonfile_prefix=results/pp_lyft/results_challenge' \
       'test_evaluator.csv_savepath=results/pp_lyft/results_challenge.csv' \
       'test_evaluator.pklfile_prefix=results/pp_lyft/results_challenge.pkl'
twang's avatar
twang committed
111
112
   ```

113
   **Notice**: To generate submissions on Lyft, `csv_savepath` must be given in the `--cfg-options`. After generating the csv file, you can make a submission with kaggle commands given on the [website](https://www.kaggle.com/c/3d-object-detection-for-autonomous-vehicles/submit).
twang's avatar
twang committed
114

115
   Note that in the [config of Lyft dataset](../../configs/_base_/datasets/lyft-3d.py), the value of `ann_file` keyword in `test` is `'lyft_infos_test.pkl'`, which is the official test set of Lyft without annotation. To test on the validation set, please change this to `'lyft_infos_val.pkl'`.
116
117

8. Test PointPillars on waymo with 8 GPUs, and evaluate the mAP with waymo metrics.
twang's avatar
twang committed
118
119

   ```shell
120
121
122
123
   ./tools/slurm_test.sh ${PARTITION} ${JOB_NAME} configs/pointpillars/pointpillars_hv_secfpn_sbn-all_16xb2-2x_waymo-3d-car.py  \
       checkpoints/hv_pointpillars_secfpn_sbn-2x16_2x_waymo-3d-car_latest.pth \
       --cfg-options 'test_evaluator.pklfile_prefix=results/waymo-car/kitti_results' \
       'test_evaluator.submission_prefix=results/waymo-car/kitti_results'
twang's avatar
twang committed
124
125
   ```

126
   **Notice**: For evaluation on waymo, please follow the [instruction](https://github.com/waymo-research/waymo-open-dataset/blob/master/docs/quick_start.md/) to build the binary file `compute_detection_metrics_main` for metrics computation and put it into `mmdet3d/core/evaluation/waymo_utils/`.(Sometimes when using bazel to build `compute_detection_metrics_main`, an error `'round' is not a member of 'std'` may appear. We just need to remove the `std::` before `round` in that file.) `pklfile_prefix` should be given in the `--eval-options` for the bin file generation. For metrics, `waymo` is the recommended official evaluation prototype. Currently, evaluating with choice `kitti` is adapted from KITTI and the results for each difficulty are not exactly the same as the definition of KITTI. Instead, most of objects are marked with difficulty 0 currently, which will be fixed in the future. The reasons of its instability include the large computation for evaluation, the lack of occlusion and truncation in the converted data, different definition of difficulty and different methods of computing average precision.
twang's avatar
twang committed
127

128
9. Test PointPillars on waymo with 8 GPUs, generate the bin files and make a submission to the leaderboard.
twang's avatar
twang committed
129
130

   ```shell
131
132
133
134
   ./tools/slurm_test.sh ${PARTITION} ${JOB_NAME} configs/pointpillars/pointpillars_hv_secfpn_sbn-all_16xb2-2x_waymo-3d-car.py  \
       checkpoints/hv_pointpillars_secfpn_sbn-2x16_2x_waymo-3d-car_latest.pth \
       --cfg-options 'test_evaluator.pklfile_prefix=results/waymo-car/kitti_results' \
       'test_evaluator.submission_prefix=results/waymo-car/kitti_results'
twang's avatar
twang committed
135
136
   ```

137
   **Notice**: After generating the bin file, you can simply build the binary file `create_submission` and use them to create a submission file by following the [instruction](https://github.com/waymo-research/waymo-open-dataset/blob/master/docs/quick_start.md/). For evaluation on the validation set with the eval server, you can also use the same way to generate a submission.
twang's avatar
twang committed
138
139
140

## Train predefined models on standard datasets

141
MMDetection3D implements distributed training and non-distributed training,
twang's avatar
twang committed
142
143
144
145
146
147
which uses `MMDistributedDataParallel` and `MMDataParallel` respectively.

All outputs (log files and checkpoints) will be saved to the working directory,
which is specified by `work_dir` in the config file.

By default we evaluate the model on the validation set after each epoch, you can change the evaluation interval by adding the interval argument in the training config.
Ziyi Wu's avatar
Ziyi Wu committed
148

twang's avatar
twang committed
149
```python
150
train_cfg = dict(type='EpochBasedTrainLoop', val_interval=1)  # This evaluate the model per 12 epoch.
twang's avatar
twang committed
151
152
```

153
**Important**: The default learning rate in config files is for 8 GPUs and the exact batch size is marked by the config's file name, e.g. '2xb8' means 2 samples per GPU using 8 GPUs.
twang's avatar
twang committed
154
155
156
157
158
159
160
161
According to the [Linear Scaling Rule](https://arxiv.org/abs/1706.02677), you need to set the learning rate proportional to the batch size if you use different GPUs or images per GPU, e.g., lr=0.01 for 4 GPUs * 2 img/gpu and lr=0.08 for 16 GPUs * 4 img/gpu. However, since most of the models in this repo use ADAM rather than SGD for optimization, the rule may not hold and users need to tune the learning rate by themselves.

### Train with a single GPU

```shell
python tools/train.py ${CONFIG_FILE} [optional arguments]
```

162
If you want to specify the working directory in the command, you can add an argument `--work-dir ${YOUR_WORK_DIR}`.
twang's avatar
twang committed
163

164
165
166
167
168
169
170
171
172
173
174
175
176
177
### Training with CPU (experimental)

The process of training on the CPU is consistent with single GPU training. We just need to disable GPUs before the training process.

```shell
export CUDA_VISIBLE_DEVICES=-1
```

And then run the script of train with a single GPU.

**Note**:

For now, most of the point cloud related algorithms rely on 3D CUDA op, which can not be trained on CPU. Some monocular 3D object detection algorithms, like FCOS3D and SMOKE can be trained on CPU. We do not recommend users to use CPU for training because it is too slow. We support this feature to allow users to debug certain models on machines without GPU for convenience.

twang's avatar
twang committed
178
179
180
181
182
183
184
185
### Train with multiple GPUs

```shell
./tools/dist_train.sh ${CONFIG_FILE} ${GPU_NUM} [optional arguments]
```

Optional arguments are:

186
- `--cfg-options 'Key=value'`: Override some settings in the used config.
twang's avatar
twang committed
187
188
189

### Train with multiple machines

190
If you run MMDetection3D on a cluster managed with [slurm](https://slurm.schedmd.com/), you can use the script `slurm_train.sh`. (This script also supports single machine training.)
twang's avatar
twang committed
191
192
193
194
195
196
197
198

```shell
[GPUS=${GPUS}] ./tools/slurm_train.sh ${PARTITION} ${JOB_NAME} ${CONFIG_FILE} ${WORK_DIR}
```

Here is an example of using 16 GPUs to train Mask R-CNN on the dev partition.

```shell
199
GPUS=16 ./tools/slurm_train.sh dev pp_kitti_3class configs/pointpillars/pointpillars_hv_secfpn_8xb6-160e_kitti-3d-3class.py /nfs/xxxx/pp_kitti_3class
twang's avatar
twang committed
200
201
202
203
```

You can check [slurm_train.sh](https://github.com/open-mmlab/mmdetection/blob/master/tools/slurm_train.sh) for full arguments and environment variables.

204
205
206
207
208
209
210
211
212
213
214
215
216
217
If you launch with multiple machines simply connected with ethernet, you can simply run following commands:

On the first machine:

```shell
NNODES=2 NODE_RANK=0 PORT=$MASTER_PORT MASTER_ADDR=$MASTER_ADDR ./tools/dist_train.sh $CONFIG $GPUS
```

On the second machine:

```shell
NNODES=2 NODE_RANK=1 PORT=$MASTER_PORT MASTER_ADDR=$MASTER_ADDR ./tools/dist_train.sh $CONFIG $GPUS
```

twang's avatar
twang committed
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
Usually it is slow if you do not have high speed networking like InfiniBand.

### Launch multiple jobs on a single machine

If you launch multiple jobs on a single machine, e.g., 2 jobs of 4-GPU training on a machine with 8 GPUs,
you need to specify different ports (29500 by default) for each job to avoid communication conflict.

If you use `dist_train.sh` to launch training jobs, you can set the port in commands.

```shell
CUDA_VISIBLE_DEVICES=0,1,2,3 PORT=29500 ./tools/dist_train.sh ${CONFIG_FILE} 4
CUDA_VISIBLE_DEVICES=4,5,6,7 PORT=29501 ./tools/dist_train.sh ${CONFIG_FILE} 4
```

If you use launch training jobs with Slurm, there are two ways to specify the ports.

234
1. Set the port through `--cfg-options`. This is more recommended since it does not change the original configs.
twang's avatar
twang committed
235
236

   ```shell
237
238
   CUDA_VISIBLE_DEVICES=0,1,2,3 GPUS=4 ./tools/slurm_train.sh ${PARTITION} ${JOB_NAME} config1.py ${WORK_DIR} --cfg-options 'env_cfg.dist_cfg.port=29500'
   CUDA_VISIBLE_DEVICES=4,5,6,7 GPUS=4 ./tools/slurm_train.sh ${PARTITION} ${JOB_NAME} config2.py ${WORK_DIR} --cfg-options 'env_cfg.dist_cfg.port=29501'
twang's avatar
twang committed
239
240
241
242
243
244
245
   ```

2. Modify the config files (usually the 6th line from the bottom in config files) to set different communication ports.

   In `config1.py`,

   ```python
246
247
248
   env_cfg = dict(
       dist_cfg=dict(backend='nccl', port=29500)
   )
twang's avatar
twang committed
249
250
251
252
253
   ```

   In `config2.py`,

   ```python
254
255
256
   env_cfg = dict(
       dist_cfg=dict(backend='nccl', port=29501)
   )
twang's avatar
twang committed
257
258
   ```

259
   Then you can launch two jobs with `config1.py` and `config2.py`.
twang's avatar
twang committed
260
261
262
263
264

   ```shell
   CUDA_VISIBLE_DEVICES=0,1,2,3 GPUS=4 ./tools/slurm_train.sh ${PARTITION} ${JOB_NAME} config1.py ${WORK_DIR}
   CUDA_VISIBLE_DEVICES=4,5,6,7 GPUS=4 ./tools/slurm_train.sh ${PARTITION} ${JOB_NAME} config2.py ${WORK_DIR}
   ```