useful_tools.md 9.32 KB
Newer Older
twang's avatar
twang committed
1
2
3
4
5
6
7
8
9
We provide lots of useful tools under `tools/` directory.

# Log Analysis

You can plot loss/mAP curves given a training log file. Run `pip install seaborn` first to install the dependency.

![loss curve image](../resources/loss_curve.png)

```shell
Ziyi Wu's avatar
Ziyi Wu committed
10
python tools/analysis_tools/analyze_logs.py plot_curve [--keys ${KEYS}] [--title ${TITLE}] [--legend ${LEGEND}] [--backend ${BACKEND}] [--style ${STYLE}] [--out ${OUT_FILE}] [--mode ${MODE}] [--interval ${INTERVAL}]
twang's avatar
twang committed
11
12
13
14
```

Examples:

Ziyi Wu's avatar
Ziyi Wu committed
15
-   Plot the classification loss of some run.
twang's avatar
twang committed
16

Ziyi Wu's avatar
Ziyi Wu committed
17
18
19
    ```shell
    python tools/analysis_tools/analyze_logs.py plot_curve log.json --keys loss_cls --legend loss_cls
    ```
twang's avatar
twang committed
20

Ziyi Wu's avatar
Ziyi Wu committed
21
-   Plot the classification and regression loss of some run, and save the figure to a pdf.
twang's avatar
twang committed
22

Ziyi Wu's avatar
Ziyi Wu committed
23
24
25
    ```shell
    python tools/analysis_tools/analyze_logs.py plot_curve log.json --keys loss_cls loss_bbox --out losses.pdf
    ```
twang's avatar
twang committed
26

Ziyi Wu's avatar
Ziyi Wu committed
27
-   Compare the bbox mAP of two runs in the same figure.
twang's avatar
twang committed
28

Ziyi Wu's avatar
Ziyi Wu committed
29
30
31
32
33
34
    ```shell
    # evaluate PartA2 and second on KITTI according to Car_3D_moderate_strict
    python tools/analysis_tools/analyze_logs.py plot_curve tools/logs/PartA2.log.json tools/logs/second.log.json --keys KITTI/Car_3D_moderate_strict --legend PartA2 second --mode eval --interval 1
    # evaluate PointPillars for car and 3 classes on KITTI according to Car_3D_moderate_strict
    python tools/analysis_tools/analyze_logs.py plot_curve tools/logs/pp-3class.log.json tools/logs/pp.log.json --keys KITTI/Car_3D_moderate_strict --legend pp-3class pp --mode eval --interval 2
    ```
twang's avatar
twang committed
35
36
37
38

You can also compute the average training speed.

```shell
Ziyi Wu's avatar
Ziyi Wu committed
39
python tools/analysis_tools/analyze_logs.py cal_train_time log.json [--include-outliers]
twang's avatar
twang committed
40
41
42
43
44
45
46
47
48
49
50
51
```

The output is expected to be like the following.

```
-----Analyze train time of work_dirs/some_exp/20190611_192040.log.json-----
slowest epoch 11, average time is 1.2024
fastest epoch 1, average time is 1.1909
time std over epochs is 0.0028
average iter time: 1.1959 s/iter
```

52
53
 

twang's avatar
twang committed
54
55
# Visualization

56
57
## Results

twang's avatar
twang committed
58
59
60
61
62
63
To see the SUNRGBD, ScanNet or KITTI points and detection results, you can run the following command

```bash
python tools/test.py ${CONFIG_FILE} ${CKPT_PATH} --show --show-dir ${SHOW_DIR}
```

64
Aftering running this command, plotted results **_\_points.obj and _**\_pred.obj files in `${SHOW_DIR}`.
twang's avatar
twang committed
65
66

To see the points, detection results and ground truth of SUNRGBD, ScanNet or KITTI during evaluation time, you can run the following command
Ziyi Wu's avatar
Ziyi Wu committed
67

twang's avatar
twang committed
68
69
70
```bash
python tools/test.py ${CONFIG_FILE} ${CKPT_PATH} --eval 'mAP' --options 'show=True' 'out_dir=${SHOW_DIR}'
```
Ziyi Wu's avatar
Ziyi Wu committed
71

72
After running this command, you will obtain **_\_points.obj, _**\_pred.obj files and \*\*\*\_gt.obj in `${SHOW_DIR}`. When `show` is enabled, [Open3D](http://www.open3d.org/) will be used to visualize the results online. You need to set `show=False` while running test in remote server withou GUI.
twang's avatar
twang committed
73

74
75
As for offline visualization, you will have two options.
To visualize the results with `Open3D` backend, you can run the following command
Ziyi Wu's avatar
Ziyi Wu committed
76

77
```bash
Ziyi Wu's avatar
Ziyi Wu committed
78
python tools/misc/visualize_results.py ${CONFIG_FILE} --result ${RESULTS_PATH} --show-dir ${SHOW_DIR}'
79
```
Ziyi Wu's avatar
Ziyi Wu committed
80

81
82
![Open3D_visualization](../resources/open3d_visual.gif)

83
Or you can use 3D visualization software such as the [MeshLab](http://www.meshlab.net/) to open the these files under `${SHOW_DIR}` to see the 3D detection output. Specifically, open `***_points.obj` to see the input point cloud and open `***_pred.obj` to see the predicted 3D bounding boxes. This allows the inference and results generation be done in remote server and the users can open them on their host with GUI.
twang's avatar
twang committed
84
85
86

**Notice**: The visualization API is a little unstable since we plan to refactor these parts together with MMDetection in the future.

87
88
## Dataset

89
We also provide scripts to visualize the dataset without inference. You can use `tools/misc/browse_dataset.py` to show loaded data and ground-truth online and save them on the disk. Currently we support single-modality 3D detection and 3D segmentation on all the datasets, as well as multi-modality 3D detection on KITTI and SUN RGB-D. To browse the KITTI dataset, you can run the following command
90
91

```shell
92
python tools/misc/browse_dataset.py configs/_base_/datasets/kitti-3d-3class.py --output-dir ${OUTPUT_DIR} --online
93
94
```

95
**Notice**: Once specifying `--output-dir`, the images of views specified by users will be saved when pressing `_ESC_` in open3d window. If you don't have a monitor, you can remove the `--online` flag to only save the visualization results and browse them offline.
96

97
If you also want to show 2D images with 3D bounding boxes projected onto them, you need to find a config that supports multi-modality data loading, and then add the `--multi-modality` flag to the command. An example is showed below
98
99

```shell
100
python tools/misc/browse_dataset.py configs/mvxnet/dv_mvx-fpn_second_secfpn_adamw_2x8_80e_kitti-3d-3class.py --output-dir ${OUTPUT_DIR} --online --multi-modality
101
102
```

103
104
105
106
107
108
109
110
111
![Open3D_visualization](../resources/browse_dataset_multi_modality.png)

You can simply browse different datasets using different configs, e.g. visualizing the ScanNet dataset in 3D semantic segmentation task

```shell
python tools/misc/browse_dataset.py configs/_base_/datasets/scannet_seg-3d-20class.py --output-dir ${OUTPUT_DIR} --online
```

![Open3D_visualization](../resources/browse_dataset_seg.png)
112
113
114

 

twang's avatar
twang committed
115
116
# Model Complexity

Ziyi Wu's avatar
Ziyi Wu committed
117
You can use `tools/analysis_tools/get_flops.py` in MMDetection, a script adapted from [flops-counter.pytorch](https://github.com/sovrasov/flops-counter.pytorch), to compute the FLOPs and params of a given model.
twang's avatar
twang committed
118
119

```shell
Ziyi Wu's avatar
Ziyi Wu committed
120
python tools/analysis_tools/get_flops.py ${CONFIG_FILE} [--shape ${INPUT_SHAPE}]
twang's avatar
twang committed
121
122
123
124
125
126
127
128
129
130
131
132
133
```

You will get the results like this.

```text
==============================
Input shape: (3, 1280, 800)
Flops: 239.32 GFLOPs
Params: 37.74 M
==============================
```

**Note**: This tool is still experimental and we do not guarantee that the
Ziyi Wu's avatar
Ziyi Wu committed
134
135
number is absolutely correct. You may well use the result for simple
comparisons, but double check it before you adopt it in technical reports or papers.
twang's avatar
twang committed
136
137

1. FLOPs are related to the input shape while parameters are not. The default
Ziyi Wu's avatar
Ziyi Wu committed
138
   input shape is (1, 3, 1280, 800).
twang's avatar
twang committed
139
140
141
2. Some operators are not counted into FLOPs like GN and custom operators. Refer to [`mmcv.cnn.get_model_complexity_info()`](https://github.com/open-mmlab/mmcv/blob/master/mmcv/cnn/utils/flops_counter.py) for details.
3. The FLOPs of two-stage detectors is dependent on the number of proposals.

142
143
 

twang's avatar
twang committed
144
145
146
147
# Model Conversion

## RegNet model to MMDetection

Ziyi Wu's avatar
Ziyi Wu committed
148
149
`tools/model_converters/regnet2mmdet.py` convert keys in pycls pretrained RegNet models to
MMDetection style.
twang's avatar
twang committed
150
151

```shell
Ziyi Wu's avatar
Ziyi Wu committed
152
python tools/model_converters/regnet2mmdet.py ${SRC} ${DST} [-h]
twang's avatar
twang committed
153
154
155
156
157
```

## Detectron ResNet to Pytorch

`tools/detectron2pytorch.py` in MMDetection could convert keys in the original detectron pretrained
Ziyi Wu's avatar
Ziyi Wu committed
158
ResNet models to PyTorch style.
twang's avatar
twang committed
159
160
161
162
163
164
165

```shell
python tools/detectron2pytorch.py ${SRC} ${DST} ${DEPTH} [-h]
```

## Prepare a model for publishing

Ziyi Wu's avatar
Ziyi Wu committed
166
`tools/model_converters/publish_model.py` helps users to prepare their model for publishing.
twang's avatar
twang committed
167
168
169
170
171
172

Before you upload a model to AWS, you may want to

1. convert model weights to CPU tensors
2. delete the optimizer states and
3. compute the hash of the checkpoint file and append the hash id to the
Ziyi Wu's avatar
Ziyi Wu committed
173
   filename.
twang's avatar
twang committed
174
175

```shell
Ziyi Wu's avatar
Ziyi Wu committed
176
python tools/model_converters/publish_model.py ${INPUT_FILENAME} ${OUTPUT_FILENAME}
twang's avatar
twang committed
177
178
179
180
181
```

E.g.,

```shell
Ziyi Wu's avatar
Ziyi Wu committed
182
python tools/model_converters/publish_model.py work_dirs/faster_rcnn/latest.pth faster_rcnn_r50_fpn_1x_20190801.pth
twang's avatar
twang committed
183
184
185
186
```

The final output filename will be `faster_rcnn_r50_fpn_1x_20190801-{hash id}.pth`.

187
188
 

twang's avatar
twang committed
189
190
191
192
193
194
195
196
197
198
199
# Dataset Conversion

`tools/data_converter/` contains tools to convert datasets to other formats. Most of them convert datasets to pickle based info files, like kitti, nuscenes and lyft. Waymo converter is used to reorganize waymo raw data like KITTI style. Users could refer to them for our approach to converting data format. It is also convenient to modify them to use as scripts like nuImages converter.

To convert the nuImages dataset into COCO format, please use the command below:

```shell
python -u tools/data_converter/nuimage_converter.py --data-root ${DATA_ROOT} --version ${VERIONS} \
                                                    --out-dir ${OUT_DIR} --nproc ${NUM_WORKERS} --extra-tag ${TAG}
```

Ziyi Wu's avatar
Ziyi Wu committed
200
201
202
203
204
-   `--data-root`: the root of the dataset, defaults to `./data/nuimages`.
-   `--version`: the version of the dataset, defaults to `v1.0-mini`. To get the full dataset, please use `--version v1.0-train v1.0-val v1.0-mini`
-   `--out-dir`: the output directory of annotations and semantic masks, defaults to `./data/nuimages/annotations/`.
-   `--nproc`: number of workers for data preparation, defaults to `4`. Larger number could reduce the preparation time as images are processed in parallel.
-   `--extra-tag`: extra tag of the annotations, defaults to `nuimages`. This can be used to separate different annotations processed in different time for study.
twang's avatar
twang committed
205

206
More details could be referred to the [doc](https://mmdetection3d.readthedocs.io/en/latest/data_preparation.html) for dataset preparation and [README](https://github.com/open-mmlab/mmdetection3d/blob/master/configs/nuimages/README.md/) for nuImages dataset.
twang's avatar
twang committed
207

208
209
 

twang's avatar
twang committed
210
211
212
213
# Miscellaneous

## Print the entire config

Ziyi Wu's avatar
Ziyi Wu committed
214
215
`tools/misc/print_config.py` prints the whole config verbatim, expanding all its
imports.
twang's avatar
twang committed
216
217

```shell
Ziyi Wu's avatar
Ziyi Wu committed
218
python tools/misc/print_config.py ${CONFIG} [-h] [--options ${OPTIONS [OPTIONS...]}]
twang's avatar
twang committed
219
```