# PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection
> [PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection](https://arxiv.org/abs/1912.13192)
<!-- [ALGORITHM] -->
## Introduction
3D object detection has been receiving increasing attention from both industry and academia thanks to its wide applications in various fields such as autonomous driving and robotics. LiDAR sensors are widely adopted in autonomous driving vehicles and robots for capturing 3D scene information as sparse and irregular point clouds, which provide vital cues for 3D scene perception and understanding. In this paper, we propose to achieve high performance 3D object detection by designing novel point-voxel integrated networks to learn better 3D features from irregular point clouds.
@@ -16,16 +16,39 @@ The ideal situation is that we can reorganize the customized raw data and conver
...
@@ -16,16 +16,39 @@ The ideal situation is that we can reorganize the customized raw data and conver
#### Point cloud Format
#### Point cloud Format
Currently, we only support '.bin' format point cloud for training and inference. Before training on your own datasets, you need to convert your point cloud files with other formats to '.bin' files. The common point cloud data formats include `.pcd` and `.las`, we list some open-source tools for reference.
Currently, we only support `.bin` format point cloud for training and inference. Before training on your own datasets, you need to convert your point cloud files with other formats to `.bin` files. The common point cloud data formats include `.pcd` and `.las`, we list some open-source tools for reference.
1. Convert pcd to bin: https://github.com/leofansq/Tools_RosBag2KITTI
1. Convert `.pcd` to `.bin`: https://github.com/DanielPollithy/pypcd
2. Convert las to bin: The common conversion path is las -> pcd -> bin, and the conversion from las -> pcd can be achieved through [this tool](https://github.com/Hitachi-Automotive-And-Industry-Lab/semantic-segmentation-editor).
- You can install `pypcd` with the following command:
2. Convert `.las` to `.bin`: The common conversion path is `.las -> .pcd -> .bin`, and the conversion from `.las -> .pcd` can be achieved through [this tool](https://github.com/Hitachi-Automotive-And-Industry-Lab/semantic-segmentation-editor).
#### Label Format
#### Label Format
The most basic information: 3D bounding box and category label of each scene need to be contained in the annotation `.txt` file. Each line represents a 3D box in a certain scene as follow:
The most basic information: 3D bounding box and category label of each scene need to be contained in the annotation `.txt` file. Each line represents a 3D box in a certain scene as follow:
```python
```
# format: [x, y, z, dx, dy, dz, yaw, category_name]
# format: [x, y, z, dx, dy, dz, yaw, category_name]
1.23 1.42 0.23 3.96 1.65 1.55 1.56 Car
1.23 1.42 0.23 3.96 1.65 1.55 1.56 Car
3.51 2.15 0.42 1.05 0.87 1.86 1.23 Pedestrian
3.51 2.15 0.42 1.05 0.87 1.86 1.23 Pedestrian
...
@@ -83,7 +106,7 @@ mmdetection3d
...
@@ -83,7 +106,7 @@ mmdetection3d
#### Vision-Based 3D Detection
#### Vision-Based 3D Detection
The raw data for vision-based 3D object detection are typically organized as follows, where `ImageSets` contains split files indicating which files belong to training/validation set, `images` contains the images from different cameras, for example, images from `camera_x` need to be placed in `images\images_x`. `calibs` contains calibration information files which store the camera intrinsic matrix of each camera, and `labels` includes label files for 3D detection.
The raw data for vision-based 3D object detection are typically organized as follows, where `ImageSets` contains split files indicating which files belong to training/validation set, `images` contains the images from different cameras, for example, images from `camera_x` need to be placed in `images/images_x`. `calibs` contains calibration information files which store the camera intrinsic matrix of each camera, and `labels` includes label files for 3D detection.
```
```
mmdetection3d
mmdetection3d
...
@@ -197,22 +220,17 @@ class MyDataset(Det3DDataset):
...
@@ -197,22 +220,17 @@ class MyDataset(Det3DDataset):
# replace with all the classes in customized pkl info file
# replace with all the classes in customized pkl info file
METAINFO={
METAINFO={
'CLASSES':('Pedestrian','Cyclist','Car')
'classes':('Pedestrian','Cyclist','Car')
}
}
defparse_ann_info(self,info):
defparse_ann_info(self,info):
"""Get annotation info according to the given index.
"""Process the `instances` in data info to `ann_info`
Args:
Args:
info (dict): Data information of single data sample.
info (dict): Info dict.
Returns:
Returns:
dict: annotation information consists of the following keys:
dict | None: Processed `ann_info`
- gt_bboxes_3d (:obj:`LiDARInstance3DBoxes`):
3D ground truth bboxes.
- bbox_labels_3d (np.ndarray): Labels of ground truths.
"""
"""
ann_info=super().parse_ann_info(info)
ann_info=super().parse_ann_info(info)
ifann_infoisNone:
ifann_infoisNone:
...
@@ -250,7 +268,7 @@ data_root = 'data/custom/'
...
@@ -250,7 +268,7 @@ data_root = 'data/custom/'
class_names=['Pedestrian','Cyclist','Car']# replace with your dataset class
class_names=['Pedestrian','Cyclist','Car']# replace with your dataset class
point_cloud_range=[0,-40,-3,70.4,40,1]# adjust according to your dataset
point_cloud_range=[0,-40,-3,70.4,40,1]# adjust according to your dataset
To valiate whether your prepared data and config are correct, it's highly recommended to use `tools/browse_dataest.py` script
To valiate whether your prepared data and config are correct, it's highly recommended to use `tools/misc/browse_dataest.py` script
to visualize your dataset and annotations before training and validation, more details refer to the [visualization](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/docs/en/user_guides/visualization.md/) doc.
to visualize your dataset and annotations before training and validation, more details refer to the [visualization](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/docs/en/user_guides/visualization.md) doc.
