README.md 10.6 KB
Newer Older
Shaoshuai Shi's avatar
Shaoshuai Shi committed
1
2
3
<img src="docs/open_mmlab.png" align="right" width="30%">

# OpenPCDet
Shaoshuai Shi's avatar
Shaoshuai Shi committed
4

Shaoshuai Shi's avatar
Shaoshuai Shi committed
5
`OpenPCDet` is a clear, simple, self-contained open source project for LiDAR-based 3D object detection. 
Shaoshuai Shi's avatar
Shaoshuai Shi committed
6

Shaoshuai Shi's avatar
Shaoshuai Shi committed
7
It is also the official code release of [`[PointRCNN]`](https://arxiv.org/abs/1812.04244), [`[Part-A^2 net]`](https://arxiv.org/abs/1907.03670) and [`[PV-RCNN]`](https://arxiv.org/abs/1912.13192). 
Shaoshuai Shi's avatar
Shaoshuai Shi committed
8
9
10
11
12


## Overview
- [Changelog](#changelog)
- [Design Pattern](#openpcdet-design-pattern)
Shaoshuai Shi's avatar
Shaoshuai Shi committed
13
- [Model Zoo](#model-zoo)
14
15
16
- [Installation](docs/INSTALL.md)
- [Quick Demo](docs/DEMO.md)
- [Getting Started](docs/GETTING_STARTED.md)
Shaoshuai Shi's avatar
Shaoshuai Shi committed
17
18
19
20
- [Citation](#citation)


## Changelog
21
22
[2021-05-14] Added support for the monocular 3D object detection model [`CaDDN`](#KITTI-3D-Object-Detection-Baselines)

23
24
25
26
[2020-11-27] **Bugfixed:** Please re-prepare the validation infos of Waymo dataset (version 1.2) if you would like to 
use our provided Waymo evaluation tool (see [PR](https://github.com/open-mmlab/OpenPCDet/pull/383)). 
Note that you do not need to re-prepare the training data and ground-truth database. 

27
28
[2020-11-10] **NEW:** The [Waymo Open Dataset](#waymo-open-dataset-baselines) has been supported with state-of-the-art results. Currently we provide the 
configs and results of `SECOND`, `PartA2` and `PV-RCNN` on the Waymo Open Dataset, and more models could be easily supported by modifying their dataset configs. 
29

Shaoshuai Shi's avatar
Shaoshuai Shi committed
30
31
32
[2020-08-10] Bugfixed: The provided NuScenes models have been updated to fix the loading bugs. Please redownload it if you need to use the pretrained NuScenes models.

[2020-07-30] `OpenPCDet` v0.3.0 is released with the following features:
Shaoshuai Shi's avatar
Shaoshuai Shi committed
33
34
   * The Point-based and Anchor-Free models ([`PointRCNN`](#KITTI-3D-Object-Detection-Baselines), [`PartA2-Free`](#KITTI-3D-Object-Detection-Baselines)) are supported now.
   * The NuScenes dataset is supported with strong baseline results ([`SECOND-MultiHead (CBGS)`](#NuScenes-3D-Object-Detection-Baselines) and [`PointPillar-MultiHead`](#NuScenes-3D-Object-Detection-Baselines)).
Shaoshuai Shi's avatar
Shaoshuai Shi committed
35
   * High efficiency than last version, support **PyTorch 1.1~1.7** and **spconv 1.0~1.2** simultaneously.
Shaoshuai Shi's avatar
Shaoshuai Shi committed
36
   
Shaoshuai Shi's avatar
Shaoshuai Shi committed
37
38
39
40
41
42
43
44
[2020-07-17]  Add simple visualization codes and a quick demo to test with custom data. 

[2020-06-24] `OpenPCDet` v0.2.0 is released with pretty new structures to support more models and datasets. 

[2020-03-16] `OpenPCDet` v0.1.0 is released. 


## Introduction
Shaoshuai Shi's avatar
Shaoshuai Shi committed
45

46

Shaoshuai Shi's avatar
Shaoshuai Shi committed
47
### What does `OpenPCDet` toolbox do?
Shaoshuai Shi's avatar
Shaoshuai Shi committed
48

Gus-Guo's avatar
Gus-Guo committed
49
Note that we have upgrated `PCDet` from `v0.1` to `v0.2` with pretty new structures to support various datasets and models.
50

Shaoshuai Shi's avatar
Shaoshuai Shi committed
51
`OpenPCDet` is a general PyTorch-based codebase for 3D object detection from point cloud. 
Shaoshuai Shi's avatar
Shaoshuai Shi committed
52
It currently supports multiple state-of-the-art 3D object detection methods with highly refactored codes for both one-stage and two-stage 3D detection frameworks.
Shaoshuai Shi's avatar
Shaoshuai Shi committed
53

Shaoshuai Shi's avatar
Shaoshuai Shi committed
54
Based on `OpenPCDet` toolbox, we win the Waymo Open Dataset challenge in [3D Detection](https://waymo.com/open/challenges/3d-detection/), 
Shaoshuai Shi's avatar
Shaoshuai Shi committed
55
[3D Tracking](https://waymo.com/open/challenges/3d-tracking/), [Domain Adaptation](https://waymo.com/open/challenges/domain-adaptation/) 
Shaoshuai Shi's avatar
Shaoshuai Shi committed
56
three tracks among all LiDAR-only methods, and the Waymo related models will be released to `OpenPCDet` soon.    
Shaoshuai Shi's avatar
Shaoshuai Shi committed
57

Shaoshuai Shi's avatar
Shaoshuai Shi committed
58
59
60
We are actively updating this repo currently, and more datasets and models will be supported soon. 
Contributions are also welcomed. 

Shaoshuai Shi's avatar
Shaoshuai Shi committed
61
### `OpenPCDet` design pattern
Shaoshuai Shi's avatar
Shaoshuai Shi committed
62

Shaoshuai Shi's avatar
Shaoshuai Shi committed
63
* Data-Model separation with unified point cloud coordinate for easily extending to custom datasets:
Shaoshuai Shi's avatar
Shaoshuai Shi committed
64
65
66
67
<p align="center">
  <img src="docs/dataset_vs_model.png" width="95%" height="320">
</p>

Shaoshuai Shi's avatar
Shaoshuai Shi committed
68
69
* Unified 3D box definition: (x, y, z, dx, dy, dz, heading).

Shaoshuai Shi's avatar
Shaoshuai Shi committed
70
71
72
73
74
* Flexible and clear model structure to easily support various 3D detection models: 
<p align="center">
  <img src="docs/model_framework.png" width="95%">
</p>

Shaoshuai Shi's avatar
Shaoshuai Shi committed
75
76
77
78
* Support various models within one framework as: 
<p align="center">
  <img src="docs/multiple_models_demo.png" width="95%">
</p>
Shaoshuai Shi's avatar
Shaoshuai Shi committed
79

Shaoshuai Shi's avatar
Shaoshuai Shi committed
80

Shaoshuai Shi's avatar
Shaoshuai Shi committed
81
### Currently Supported Features
Shaoshuai Shi's avatar
Shaoshuai Shi committed
82

Shaoshuai Shi's avatar
Shaoshuai Shi committed
83
- [x] Support both one-stage and two-stage 3D object detection frameworks
Shaoshuai Shi's avatar
Shaoshuai Shi committed
84
85
86
87
88
89
90
- [x] Support distributed training & testing with multiple GPUs and multiple machines
- [x] Support multiple heads on different scales to detect different classes
- [x] Support stacked version set abstraction to encode various number of points in different scenes
- [x] Support Adaptive Training Sample Selection (ATSS) for target assignment
- [x] Support RoI-aware point cloud pooling & RoI-grid point cloud pooling
- [x] Support GPU version 3D IoU calculation and rotated NMS 

Shaoshuai Shi's avatar
Shaoshuai Shi committed
91

Shaoshuai Shi's avatar
Shaoshuai Shi committed
92
93
94
## Model Zoo

### KITTI 3D Object Detection Baselines
95
Selected supported methods are shown in the below table. The results are the 3D detection performance of moderate difficulty on the *val* set of KITTI dataset.
Shaoshuai Shi's avatar
Shaoshuai Shi committed
96
97
* All models are trained with 8 GTX 1080Ti GPUs and are available for download. 
* The training time is measured with 8 TITAN XP GPUs and PyTorch 1.5.
Shaoshuai Shi's avatar
Shaoshuai Shi committed
98

Shaoshuai Shi's avatar
Shaoshuai Shi committed
99
|                                             | training time | Car@R11 | Pedestrian@R11 | Cyclist@R11  | download | 
Shaoshuai Shi's avatar
Shaoshuai Shi committed
100
|---------------------------------------------|----------:|:-------:|:-------:|:-------:|:---------:|
Shaoshuai Shi's avatar
Shaoshuai Shi committed
101
102
| [PointPillar](tools/cfgs/kitti_models/pointpillar.yaml) |~1.2 hours| 77.28 | 52.29 | 62.68 | [model-18M](https://drive.google.com/file/d/1wMxWTpU1qUoY3DsCH31WJmvJxcjFXKlm/view?usp=sharing) | 
| [SECOND](tools/cfgs/kitti_models/second.yaml)       |  ~1.7 hours  | 78.62 | 52.98 | 67.15 | [model-20M](https://drive.google.com/file/d/1-01zsPOsqanZQqIIyy7FpNXStL3y4jdR/view?usp=sharing) |
103
| [SECOND-IoU](tools/cfgs/kitti_models/second_iou.yaml)       | -  | 79.09 | 55.74 | 71.31 | [model](https://drive.google.com/file/d/1AQkeNs4bxhvhDQ-5sEo_yvQUlfo73lsW/view?usp=sharing) |
Shaoshuai Shi's avatar
Shaoshuai Shi committed
104
105
| [PointRCNN](tools/cfgs/kitti_models/pointrcnn.yaml) | ~3 hours | 78.70 | 54.41 | 72.11 | [model-16M](https://drive.google.com/file/d/1BCX9wMn-GYAfSOPpyxf6Iv6fc0qKLSiU/view?usp=sharing)| 
| [PointRCNN-IoU](tools/cfgs/kitti_models/pointrcnn_iou.yaml) | ~3 hours | 78.75 | 58.32 | 71.34 | [model-16M](https://drive.google.com/file/d/1V0vNZ3lAHpEEt0MlT80eL2f41K2tHm_D/view?usp=sharing)|
Shaoshuai Shi's avatar
Shaoshuai Shi committed
106
107
108
| [Part-A^2-Free](tools/cfgs/kitti_models/PartA2_free.yaml)   | ~3.8 hours| 78.72 | 65.99 | 74.29 | [model-226M](https://drive.google.com/file/d/1lcUUxF8mJgZ_e-tZhP1XNQtTBuC-R0zr/view?usp=sharing) |
| [Part-A^2-Anchor](tools/cfgs/kitti_models/PartA2.yaml)    | ~4.3 hours| 79.40 | 60.05 | 69.90 | [model-244M](https://drive.google.com/file/d/10GK1aCkLqxGNeX3lVu8cLZyE0G8002hY/view?usp=sharing) |
| [PV-RCNN](tools/cfgs/kitti_models/pv_rcnn.yaml) | ~5 hours| 83.61 | 57.90 | 70.47 | [model-50M](https://drive.google.com/file/d/1lIOq4Hxr0W3qsX83ilQv0nk1Cls6KAr-/view?usp=sharing) |
djiajunustc's avatar
djiajunustc committed
109
| [Voxel R-CNN (Car)](tools/cfgs/kitti_models/voxel_rcnn_car.yaml) | ~2.2 hours| 84.54 | - | - | [model-28M](https://drive.google.com/file/d/19_jiAeGLz7V0wNjSJw4cKmMjdm5EW5By/view?usp=sharing) |
110
| [CaDDN](tools/cfgs/kitti_models/CaDDN.yaml) |~15 hours| 21.38 | 13.02 | 9.76 | [model-774M](https://drive.google.com/file/d/1OQTO2PtXT8GGr35W9m2GZGuqgb6fyU1V/view?usp=sharing) |
Shaoshuai Shi's avatar
Shaoshuai Shi committed
111
112

### NuScenes 3D Object Detection Baselines
113
114
All models are trained with 8 GTX 1080Ti GPUs and are available for download.

Shaoshuai Shi's avatar
Shaoshuai Shi committed
115
116
|                                             | mATE | mASE | mAOE | mAVE | mAAE | mAP | NDS | download | 
|---------------------------------------------|----------:|:-------:|:-------:|:-------:|:---------:|:-------:|:-------:|:---------:|
117
118
| [PointPillar-MultiHead](tools/cfgs/nuscenes_models/cbgs_pp_multihead.yaml) | 33.87	| 26.00 | 32.07	| 28.74 | 20.15 | 44.63 | 58.23	 | [model-23M](https://drive.google.com/file/d/1p-501mTWsq0G9RzroTWSXreIMyTUUpBM/view?usp=sharing) | 
| [SECOND-MultiHead (CBGS)](tools/cfgs/nuscenes_models/cbgs_second_multihead.yaml) | 31.15 |	25.51 |	26.64 | 26.26 | 20.46 | 50.59 | 62.29 | [model-35M](https://drive.google.com/file/d/1bNzcOnE3u9iooBFMk2xK7HqhdeQ_nwTq/view?usp=sharing) |
Shaoshuai Shi's avatar
Shaoshuai Shi committed
119

Shaoshuai Shi's avatar
Shaoshuai Shi committed
120
### Waymo Open Dataset Baselines
Shaoshuai Shi's avatar
Shaoshuai Shi committed
121
We provide the setting of [`DATA_CONFIG.SAMPLED_INTERVAL`](tools/cfgs/dataset_configs/waymo_dataset.yaml) on the Waymo Open Dataset (WOD) to subsample partial samples for training and evaluation, 
Shaoshuai Shi's avatar
Shaoshuai Shi committed
122
so you could also play with WOD by setting a smaller `DATA_CONFIG.SAMPLED_INTERVAL` even if you only have limited GPU resources. 
Shaoshuai Shi's avatar
Shaoshuai Shi committed
123

Shaoshuai Shi's avatar
Shaoshuai Shi committed
124
By default, all models are trained with **20% data (~32k frames)** of all the training samples on 8 GTX 1080Ti GPUs, and the results of each cell here are mAP/mAPH calculated by the official Waymo evaluation metrics on the **whole** validation set (version 1.2).    
Shaoshuai Shi's avatar
Shaoshuai Shi committed
125
126
127
128

|                                             | Vec_L1 | Vec_L2 | Ped_L1 | Ped_L2 | Cyc_L1 | Cyc_L2 |  
|---------------------------------------------|----------:|:-------:|:-------:|:-------:|:-------:|:-------:|
| [SECOND](tools/cfgs/waymo_models/second.yaml) | 68.03/67.44	| 59.57/59.04 | 61.14/50.33	| 53.00/43.56 | 54.66/53.31 | 52.67/51.37 | 
Shaoshuai Shi's avatar
Shaoshuai Shi committed
129
| [Part-A^2-Anchor](tools/cfgs/waymo_models/PartA2.yaml) | 71.82/71.29 | 64.33/63.82 | 63.15/54.96 | 54.24/47.11 | 65.23/63.92 | 62.61/61.35 |
Shaoshuai Shi's avatar
Shaoshuai Shi committed
130
131
| [PV-RCNN](tools/cfgs/waymo_models/pv_rcnn.yaml) | 74.06/73.38 | 64.99/64.38 |	62.66/52.68 | 53.80/45.14 |	63.32/61.71	| 60.72/59.18 | 

132
133
We could not provide the above pretrained models due to [Waymo Dataset License Agreement](https://waymo.com/open/terms/), 
but you could easily achieve similar performance by training with the default configs.
Shaoshuai Shi's avatar
Shaoshuai Shi committed
134
135


Shaoshuai Shi's avatar
Shaoshuai Shi committed
136

Shaoshuai Shi's avatar
Shaoshuai Shi committed
137
138
139
### Other datasets
More datasets are on the way. 

Shaoshuai Shi's avatar
Shaoshuai Shi committed
140
141
## Installation

Shaoshuai Shi's avatar
Shaoshuai Shi committed
142
Please refer to [INSTALL.md](docs/INSTALL.md) for the installation of `OpenPCDet`.
Shaoshuai Shi's avatar
Shaoshuai Shi committed
143
144


145
146
147
148
## Quick Demo
Please refer to [DEMO.md](docs/DEMO.md) for a quick demo to test with a pretrained model and 
visualize the predicted results on your custom data or the original KITTI data.

Shaoshuai Shi's avatar
Shaoshuai Shi committed
149
## Getting Started
Shaoshuai Shi's avatar
Shaoshuai Shi committed
150
151
152

Please refer to [GETTING_STARTED.md](docs/GETTING_STARTED.md) to learn more usage about this project.

153

Shaoshuai Shi's avatar
Shaoshuai Shi committed
154
155
## License

Shaoshuai Shi's avatar
Shaoshuai Shi committed
156
`OpenPCDet` is released under the [Apache 2.0 license](LICENSE).
Shaoshuai Shi's avatar
Shaoshuai Shi committed
157
158

## Acknowledgement
Shaoshuai Shi's avatar
Shaoshuai Shi committed
159
`OpenPCDet` is an open source project for LiDAR-based 3D scene perception that supports multiple
Shaoshuai Shi's avatar
Shaoshuai Shi committed
160
161
162
163
164
165
166
167
168
169
170
LiDAR-based perception models as shown above. Some parts of `PCDet` are learned from the official released codes of the above supported methods. 
We would like to thank for their proposed methods and the official implementation.   

We hope that this repo could serve as a strong and flexible codebase to benefit the research community by speeding up the process of reimplementing previous works and/or developing new methods.


## Citation 
If you find this project useful in your research, please consider cite:


```
Shaoshuai Shi's avatar
Shaoshuai Shi committed
171
@misc{openpcdet2020,
Shaoshuai Shi's avatar
Shaoshuai Shi committed
172
173
174
175
    title={OpenPCDet: An Open-source Toolbox for 3D Object Detection from Point Clouds},
    author={OpenPCDet Development Team},
    howpublished = {\url{https://github.com/open-mmlab/OpenPCDet}},
    year={2020}
Shaoshuai Shi's avatar
Shaoshuai Shi committed
176
}
Shaoshuai Shi's avatar
Shaoshuai Shi committed
177
```
Shaoshuai Shi's avatar
Shaoshuai Shi committed
178

Shaoshuai Shi's avatar
Shaoshuai Shi committed
179
180
## Contribution
Welcome to be a member of the OpenPCDet development team by contributing to this repo, and feel free to contact us for any potential contributions. 
Shaoshuai Shi's avatar
Shaoshuai Shi committed
181

Shaoshuai Shi's avatar
Shaoshuai Shi committed
182