Unverified Commit aa753ec0 authored by Shaoshuai Shi's avatar Shaoshuai Shi Committed by GitHub
Browse files

Update to OpenPCDet v0.6 #1087

 Merge pull request #1087 from sshaoshuai/dev_v0.6
parents 7e8bbe26 beb249e5
...@@ -4,10 +4,10 @@ ...@@ -4,10 +4,10 @@
`OpenPCDet` is a clear, simple, self-contained open source project for LiDAR-based 3D object detection. `OpenPCDet` is a clear, simple, self-contained open source project for LiDAR-based 3D object detection.
It is also the official code release of [`[PointRCNN]`](https://arxiv.org/abs/1812.04244), [`[Part-A2-Net]`](https://arxiv.org/abs/1907.03670), [`[PV-RCNN]`](https://arxiv.org/abs/1912.13192), [`[Voxel R-CNN]`](https://arxiv.org/abs/2012.15712) and [`[PV-RCNN++]`](https://arxiv.org/abs/2102.00463). It is also the official code release of [`[PointRCNN]`](https://arxiv.org/abs/1812.04244), [`[Part-A2-Net]`](https://arxiv.org/abs/1907.03670), [`[PV-RCNN]`](https://arxiv.org/abs/1912.13192), [`[Voxel R-CNN]`](https://arxiv.org/abs/2012.15712), [`[PV-RCNN++]`](https://arxiv.org/abs/2102.00463) and [`[MPPNet]`](https://arxiv.org/abs/2205.05979).
**Highlights**: **Highlights**:
* `OpenPCDet` has been updated to `v0.5.2` (Jan. 2022). * `OpenPCDet` has been updated to `v0.6.0` (Sep. 2022).
* The codes of PV-RCNN++ has been supported. * The codes of PV-RCNN++ has been supported.
## Overview ## Overview
...@@ -21,6 +21,12 @@ It is also the official code release of [`[PointRCNN]`](https://arxiv.org/abs/18 ...@@ -21,6 +21,12 @@ It is also the official code release of [`[PointRCNN]`](https://arxiv.org/abs/18
## Changelog ## Changelog
[2022-09-02] **NEW:** Update `OpenPCDet` to v0.6.0:
* Official code release of [MPPNet](https://arxiv.org/abs/2205.05979) for temporal 3D object detection, which supports long-term multi-frame 3D object detection and ranks 1st place on 3D detection learderboard of Waymo Open Dataset (see the [guideline](docs/guidelines_of_approaches/mppnet.md) on how to train/test with MPPNet).
* Support multi-frame training/testing on Waymo Open Dataset (see the [change log](docs/changelog.md) for more details on how to process data).
* Support to save changing training details (e.g., loss, iter, epoch) to file (previous tqdm progress bar is still supported by using `--use_tqdm_to_record`). Please use `pip install gpustat` if you also want to log the GPU related information.
* Support to save latest model every 5 mintues, so you can restore the model training from latest status instead of previous epoch.
[2022-08-22] Added support for [custom dataset tutorial and template](docs/CUSTOM_DATASET_TUTORIAL.md) [2022-08-22] Added support for [custom dataset tutorial and template](docs/CUSTOM_DATASET_TUTORIAL.md)
[2022-07-05] Added support for the 3D object detection backbone network [`Focals Conv`](https://openaccess.thecvf.com/content/CVPR2022/papers/Chen_Focal_Sparse_Convolutional_Networks_for_3D_Object_Detection_CVPR_2022_paper.pdf). [2022-07-05] Added support for the 3D object detection backbone network [`Focals Conv`](https://openaccess.thecvf.com/content/CVPR2022/papers/Chen_Focal_Sparse_Convolutional_Networks_for_3D_Object_Detection_CVPR_2022_paper.pdf).
......
...@@ -74,11 +74,15 @@ OpenPCDet ...@@ -74,11 +74,15 @@ OpenPCDet
| | |── waymo_processed_data_v0_5_0 | | |── waymo_processed_data_v0_5_0
│ │ │ │── segment-xxxxxxxx/ │ │ │ │── segment-xxxxxxxx/
| | | |── ... | | | |── ...
│ │ │── waymo_processed_data_v0_5_0_gt_database_train_sampled_1/ │ │ │── waymo_processed_data_v0_5_0_gt_database_train_sampled_1/ (old, for single-frame)
│ │ │── waymo_processed_data_v0_5_0_waymo_dbinfos_train_sampled_1.pkl │ │ │── waymo_processed_data_v0_5_0_waymo_dbinfos_train_sampled_1.pkl (old, for single-frame)
│ │ │── waymo_processed_data_v0_5_0_gt_database_train_sampled_1_global.npy (optional) │ │ │── waymo_processed_data_v0_5_0_gt_database_train_sampled_1_global.npy (optional, old, for single-frame)
│ │ │── waymo_processed_data_v0_5_0_infos_train.pkl (optional) │ │ │── waymo_processed_data_v0_5_0_infos_train.pkl (optional)
│ │ │── waymo_processed_data_v0_5_0_infos_val.pkl (optional) │ │ │── waymo_processed_data_v0_5_0_infos_val.pkl (optional)
| | |── waymo_processed_data_v0_5_0_gt_database_train_sampled_1_multiframe_-4_to_0 (new, for single/multi-frame)
│ │ │── waymo_processed_data_v0_5_0_waymo_dbinfos_train_sampled_1_multiframe_-4_to_0.pkl (new, for single/multi-frame)
│ │ │── waymo_processed_data_v0_5_0_gt_database_train_sampled_1_multiframe_-4_to_0_global.np (new, for single/multi-frame)
├── pcdet ├── pcdet
├── tools ├── tools
``` ```
...@@ -92,8 +96,13 @@ pip3 install waymo-open-dataset-tf-2-5-0 --user ...@@ -92,8 +96,13 @@ pip3 install waymo-open-dataset-tf-2-5-0 --user
* Extract point cloud data from tfrecord and generate data infos by running the following command (it takes several hours, * Extract point cloud data from tfrecord and generate data infos by running the following command (it takes several hours,
and you could refer to `data/waymo/waymo_processed_data_v0_5_0` to see how many records that have been processed): and you could refer to `data/waymo/waymo_processed_data_v0_5_0` to see how many records that have been processed):
```python ```python
# only for single-frame setting
python -m pcdet.datasets.waymo.waymo_dataset --func create_waymo_infos \ python -m pcdet.datasets.waymo.waymo_dataset --func create_waymo_infos \
--cfg_file tools/cfgs/dataset_configs/waymo_dataset.yaml --cfg_file tools/cfgs/dataset_configs/waymo_dataset.yaml
# for single-frame or multi-frame setting
python -m pcdet.datasets.waymo.waymo_dataset --func create_waymo_infos \
--cfg_file tools/cfgs/dataset_configs/waymo_dataset_multiframe.yaml
# Ignore 'CUDA_ERROR_NO_DEVICE' error as this process does not require GPU. # Ignore 'CUDA_ERROR_NO_DEVICE' error as this process does not require GPU.
``` ```
......
# Changelog and Guidelines
### [2022-09-02] Update to v0.6.0:
* How to process data to support multi-frame training/testing on Waymo Open Dataset?
* If you never use the OpenPCDet, you can directly follow the [GETTING_STARTED.md](GETTING_STARTED.md)
* If you have been using previous OpenPCDet (`v0.5`), then you need to follow the following steps to update your data:
* Update your waymo infos (the `*.pkl` files for each sequence) by adding argument `--update_info_only`:
```
python -m pcdet.datasets.waymo.waymo_dataset --func create_waymo_infos --cfg_file tools/cfgs/dataset_configs/waymo_dataset.yaml --update_info_only
```
* Generate multi-frame GT database for copy-paste augmentation of multi-frame training. There is also a faster version with parallel data generation by adding `--use_parallel`, but you need to read the codes and rename the file after getting the results.
```
python -m pcdet.datasets.waymo.waymo_dataset --func create_waymo_gt_database --cfg_file tools/cfgs/dataset_configs/waymo_dataset_multiframe.yaml
```
This will generate the new files like the following (the last three lines under `data/waymo`):
```
OpenPCDet
├── data
│ ├── waymo
│ │ │── ImageSets
│ │ │── raw_data
│ │ │ │── segment-xxxxxxxx.tfrecord
| | | |── ...
| | |── waymo_processed_data_v0_5_0
│ │ │ │── segment-xxxxxxxx/
| | | |── ...
│ │ │── waymo_processed_data_v0_5_0_gt_database_train_sampled_1/
│ │ │── waymo_processed_data_v0_5_0_waymo_dbinfos_train_sampled_1.pkl
│ │ │── waymo_processed_data_v0_5_0_gt_database_train_sampled_1_global.npy (optional)
│ │ │── waymo_processed_data_v0_5_0_infos_train.pkl (optional)
│ │ │── waymo_processed_data_v0_5_0_infos_val.pkl (optional)
| | |── waymo_processed_data_v0_5_0_gt_database_train_sampled_1_multiframe_-4_to_0 (new)
│ │ │── waymo_processed_data_v0_5_0_waymo_dbinfos_train_sampled_1_multiframe_-4_to_0.pkl (new)
│ │ │── waymo_processed_data_v0_5_0_gt_database_train_sampled_1_multiframe_-4_to_0_global.np (new, optional)
├── pcdet
├── tools
```
# Will be available soon
\ No newline at end of file
...@@ -126,60 +126,6 @@ def random_image_flip_horizontal(image, depth_map, gt_boxes, calib): ...@@ -126,60 +126,6 @@ def random_image_flip_horizontal(image, depth_map, gt_boxes, calib):
return aug_image, aug_depth_map, aug_gt_boxes return aug_image, aug_depth_map, aug_gt_boxes
def random_translation_along_x(gt_boxes, points, offset_std):
"""
Args:
gt_boxes: (N, 7), [x, y, z, dx, dy, dz, heading, [vx], [vy]]
points: (M, 3 + C),
offset_std: float
Returns:
"""
offset = np.random.normal(0, offset_std, 1)
points[:, 0] += offset
gt_boxes[:, 0] += offset
# if gt_boxes.shape[1] > 7:
# gt_boxes[:, 7] += offset
return gt_boxes, points
def random_translation_along_y(gt_boxes, points, offset_std):
"""
Args:
gt_boxes: (N, 7), [x, y, z, dx, dy, dz, heading, [vx], [vy]]
points: (M, 3 + C),
offset_std: float
Returns:
"""
offset = np.random.normal(0, offset_std, 1)
points[:, 1] += offset
gt_boxes[:, 1] += offset
# if gt_boxes.shape[1] > 8:
# gt_boxes[:, 8] += offset
return gt_boxes, points
def random_translation_along_z(gt_boxes, points, offset_std):
"""
Args:
gt_boxes: (N, 7), [x, y, z, dx, dy, dz, heading, [vx], [vy]]
points: (M, 3 + C),
offset_std: float
Returns:
"""
offset = np.random.normal(0, offset_std, 1)
points[:, 2] += offset
gt_boxes[:, 2] += offset
return gt_boxes, points
def random_local_translation_along_x(gt_boxes, points, offset_range): def random_local_translation_along_x(gt_boxes, points, offset_range):
""" """
Args: Args:
......
...@@ -105,15 +105,16 @@ class DataAugmentor(object): ...@@ -105,15 +105,16 @@ class DataAugmentor(object):
if data_dict is None: if data_dict is None:
return partial(self.random_world_translation, config=config) return partial(self.random_world_translation, config=config)
noise_translate_std = config['NOISE_TRANSLATE_STD'] noise_translate_std = config['NOISE_TRANSLATE_STD']
if noise_translate_std == 0: assert len(noise_translate_std) == 3
return data_dict noise_translate = np.array([
gt_boxes, points = data_dict['gt_boxes'], data_dict['points'] np.random.normal(0, noise_translate_std[0], 1),
for cur_axis in config['ALONG_AXIS_LIST']: np.random.normal(0, noise_translate_std[1], 1),
assert cur_axis in ['x', 'y', 'z'] np.random.normal(0, noise_translate_std[2], 1),
gt_boxes, points = getattr(augmentor_utils, 'random_translation_along_%s' % cur_axis)( ], dtype=np.float32).T
gt_boxes, points, noise_translate_std,
)
gt_boxes, points = data_dict['gt_boxes'], data_dict['points']
points[:, :3] += noise_translate
gt_boxes[:, :3] += noise_translate
data_dict['gt_boxes'] = gt_boxes data_dict['gt_boxes'] = gt_boxes
data_dict['points'] = points data_dict['points'] = points
return data_dict return data_dict
......
...@@ -30,6 +30,13 @@ class DataBaseSampler(object): ...@@ -30,6 +30,13 @@ class DataBaseSampler(object):
for db_info_path in sampler_cfg.DB_INFO_PATH: for db_info_path in sampler_cfg.DB_INFO_PATH:
db_info_path = self.root_path.resolve() / db_info_path db_info_path = self.root_path.resolve() / db_info_path
if not db_info_path.exists():
assert len(sampler_cfg.DB_INFO_PATH) == 1
sampler_cfg.DB_INFO_PATH[0] = sampler_cfg.BACKUP_DB_INFO['DB_INFO_PATH']
sampler_cfg.DB_DATA_PATH[0] = sampler_cfg.BACKUP_DB_INFO['DB_DATA_PATH']
db_info_path = self.root_path.resolve() / sampler_cfg.DB_INFO_PATH[0]
sampler_cfg.NUM_POINT_FEATURES = sampler_cfg.BACKUP_DB_INFO['NUM_POINT_FEATURES']
with open(str(db_info_path), 'rb') as f: with open(str(db_info_path), 'rb') as f:
infos = pickle.load(f) infos = pickle.load(f)
[self.db_infos[cur_class].extend(infos[cur_class]) for cur_class in class_names] [self.db_infos[cur_class].extend(infos[cur_class]) for cur_class in class_names]
...@@ -403,11 +410,23 @@ class DataBaseSampler(object): ...@@ -403,11 +410,23 @@ class DataBaseSampler(object):
obj_points = np.concatenate(obj_points_list, axis=0) obj_points = np.concatenate(obj_points_list, axis=0)
sampled_gt_names = np.array([x['name'] for x in total_valid_sampled_dict]) sampled_gt_names = np.array([x['name'] for x in total_valid_sampled_dict])
if self.sampler_cfg.get('FILTER_OBJ_POINTS_BY_TIMESTAMP', False) or obj_points.shape[-1] != points.shape[-1]:
if self.sampler_cfg.get('FILTER_OBJ_POINTS_BY_TIMESTAMP', False):
min_time = min(self.sampler_cfg.TIME_RANGE[0], self.sampler_cfg.TIME_RANGE[1])
max_time = max(self.sampler_cfg.TIME_RANGE[0], self.sampler_cfg.TIME_RANGE[1])
else:
assert obj_points.shape[-1] == points.shape[-1] + 1
# transform multi-frame GT points to single-frame GT points
min_time = max_time = 0.0
time_mask = np.logical_and(obj_points[:, -1] < max_time + 1e-6, obj_points[:, -1] > min_time - 1e-6)
obj_points = obj_points[time_mask]
large_sampled_gt_boxes = box_utils.enlarge_box3d( large_sampled_gt_boxes = box_utils.enlarge_box3d(
sampled_gt_boxes[:, 0:7], extra_width=self.sampler_cfg.REMOVE_EXTRA_WIDTH sampled_gt_boxes[:, 0:7], extra_width=self.sampler_cfg.REMOVE_EXTRA_WIDTH
) )
points = box_utils.remove_points_in_boxes3d(points, large_sampled_gt_boxes) points = box_utils.remove_points_in_boxes3d(points, large_sampled_gt_boxes)
points = np.concatenate([obj_points, points], axis=0) points = np.concatenate([obj_points[:, :points.shape[-1]], points], axis=0)
gt_names = np.concatenate([gt_names, sampled_gt_names], axis=0) gt_names = np.concatenate([gt_names, sampled_gt_names], axis=0)
gt_boxes = np.concatenate([gt_boxes, sampled_gt_boxes], axis=0) gt_boxes = np.concatenate([gt_boxes, sampled_gt_boxes], axis=0)
data_dict['gt_boxes'] = gt_boxes data_dict['gt_boxes'] = gt_boxes
...@@ -462,7 +481,7 @@ class DataBaseSampler(object): ...@@ -462,7 +481,7 @@ class DataBaseSampler(object):
valid_sampled_dict = [sampled_dict[x] for x in valid_mask] valid_sampled_dict = [sampled_dict[x] for x in valid_mask]
valid_sampled_boxes = sampled_boxes[valid_mask] valid_sampled_boxes = sampled_boxes[valid_mask]
existed_boxes = np.concatenate((existed_boxes, valid_sampled_boxes), axis=0) existed_boxes = np.concatenate((existed_boxes, valid_sampled_boxes[:, :existed_boxes.shape[-1]]), axis=0)
total_valid_sampled_dict.extend(valid_sampled_dict) total_valid_sampled_dict.extend(valid_sampled_dict)
sampled_gt_boxes = existed_boxes[gt_boxes.shape[0]:, :] sampled_gt_boxes = existed_boxes[gt_boxes.shape[0]:, :]
......
...@@ -57,14 +57,13 @@ class DatasetTemplate(torch_data.Dataset): ...@@ -57,14 +57,13 @@ class DatasetTemplate(torch_data.Dataset):
def __setstate__(self, d): def __setstate__(self, d):
self.__dict__.update(d) self.__dict__.update(d)
@staticmethod def generate_prediction_dicts(self, batch_dict, pred_dicts, class_names, output_path=None):
def generate_prediction_dicts(batch_dict, pred_dicts, class_names, output_path=None):
""" """
Args: Args:
batch_dict: batch_dict:
frame_id: frame_id:
pred_dicts: list of pred_dicts pred_dicts: list of pred_dicts
pred_boxes: (N, 7), Tensor pred_boxes: (N, 7 or 9), Tensor
pred_scores: (N), Tensor pred_scores: (N), Tensor
pred_labels: (N), Tensor pred_labels: (N), Tensor
class_names: class_names:
...@@ -75,9 +74,10 @@ class DatasetTemplate(torch_data.Dataset): ...@@ -75,9 +74,10 @@ class DatasetTemplate(torch_data.Dataset):
""" """
def get_template_prediction(num_samples): def get_template_prediction(num_samples):
box_dim = 9 if self.dataset_cfg.get('TRAIN_WITH_SPEED', False) else 7
ret_dict = { ret_dict = {
'name': np.zeros(num_samples), 'score': np.zeros(num_samples), 'name': np.zeros(num_samples), 'score': np.zeros(num_samples),
'boxes_lidar': np.zeros([num_samples, 7]), 'pred_labels': np.zeros(num_samples) 'boxes_lidar': np.zeros([num_samples, box_dim]), 'pred_labels': np.zeros(num_samples)
} }
return ret_dict return ret_dict
......
...@@ -85,7 +85,8 @@ class DataProcessor(object): ...@@ -85,7 +85,8 @@ class DataProcessor(object):
if data_dict.get('gt_boxes', None) is not None and config.REMOVE_OUTSIDE_BOXES and self.training: if data_dict.get('gt_boxes', None) is not None and config.REMOVE_OUTSIDE_BOXES and self.training:
mask = box_utils.mask_boxes_outside_range_numpy( mask = box_utils.mask_boxes_outside_range_numpy(
data_dict['gt_boxes'], self.point_cloud_range, min_num_corners=config.get('min_num_corners', 1) data_dict['gt_boxes'], self.point_cloud_range, min_num_corners=config.get('min_num_corners', 1),
use_center_to_filter=config.get('USE_CENTER_TO_FILTER', True)
) )
data_dict['gt_boxes'] = data_dict['gt_boxes'][mask] data_dict['gt_boxes'] = data_dict['gt_boxes'][mask]
return data_dict return data_dict
......
...@@ -45,6 +45,7 @@ class PointFeatureEncoder(object): ...@@ -45,6 +45,7 @@ class PointFeatureEncoder(object):
num_output_features = len(self.used_feature_list) num_output_features = len(self.used_feature_list)
return num_output_features return num_output_features
assert points.shape[-1] == len(self.src_feature_list)
point_feature_list = [points[:, 0:3]] point_feature_list = [points[:, 0:3]]
for x in self.used_feature_list: for x in self.used_feature_list:
if x in ['x', 'y', 'z']: if x in ['x', 'y', 'z']:
......
...@@ -13,6 +13,8 @@ import SharedArray ...@@ -13,6 +13,8 @@ import SharedArray
import torch.distributed as dist import torch.distributed as dist
from tqdm import tqdm from tqdm import tqdm
from pathlib import Path from pathlib import Path
from functools import partial
from ...ops.roiaware_pool3d import roiaware_pool3d_utils from ...ops.roiaware_pool3d import roiaware_pool3d_utils
from ...utils import box_utils, common_utils from ...utils import box_utils, common_utils
from ..dataset import DatasetTemplate from ..dataset import DatasetTemplate
...@@ -29,7 +31,7 @@ class WaymoDataset(DatasetTemplate): ...@@ -29,7 +31,7 @@ class WaymoDataset(DatasetTemplate):
self.sample_sequence_list = [x.strip() for x in open(split_dir).readlines()] self.sample_sequence_list = [x.strip() for x in open(split_dir).readlines()]
self.infos = [] self.infos = []
self.include_waymo_data(self.mode) self.seq_name_to_infos = self.include_waymo_data(self.mode)
self.use_shared_memory = self.dataset_cfg.get('USE_SHARED_MEMORY', False) and self.training self.use_shared_memory = self.dataset_cfg.get('USE_SHARED_MEMORY', False) and self.training
if self.use_shared_memory: if self.use_shared_memory:
...@@ -45,11 +47,12 @@ class WaymoDataset(DatasetTemplate): ...@@ -45,11 +47,12 @@ class WaymoDataset(DatasetTemplate):
split_dir = self.root_path / 'ImageSets' / (self.split + '.txt') split_dir = self.root_path / 'ImageSets' / (self.split + '.txt')
self.sample_sequence_list = [x.strip() for x in open(split_dir).readlines()] self.sample_sequence_list = [x.strip() for x in open(split_dir).readlines()]
self.infos = [] self.infos = []
self.include_waymo_data(self.mode) self.seq_name_to_infos = self.include_waymo_data(self.mode)
def include_waymo_data(self, mode): def include_waymo_data(self, mode):
self.logger.info('Loading Waymo dataset') self.logger.info('Loading Waymo dataset')
waymo_infos = [] waymo_infos = []
seq_name_to_infos = {}
num_skipped_infos = 0 num_skipped_infos = 0
for k in range(len(self.sample_sequence_list)): for k in range(len(self.sample_sequence_list)):
...@@ -63,6 +66,8 @@ class WaymoDataset(DatasetTemplate): ...@@ -63,6 +66,8 @@ class WaymoDataset(DatasetTemplate):
infos = pickle.load(f) infos = pickle.load(f)
waymo_infos.extend(infos) waymo_infos.extend(infos)
seq_name_to_infos[infos[0]['point_cloud']['lidar_sequence']] = infos
self.infos.extend(waymo_infos[:]) self.infos.extend(waymo_infos[:])
self.logger.info('Total skipped info %s' % num_skipped_infos) self.logger.info('Total skipped info %s' % num_skipped_infos)
self.logger.info('Total samples for Waymo dataset: %d' % (len(waymo_infos))) self.logger.info('Total samples for Waymo dataset: %d' % (len(waymo_infos)))
...@@ -74,6 +79,11 @@ class WaymoDataset(DatasetTemplate): ...@@ -74,6 +79,11 @@ class WaymoDataset(DatasetTemplate):
self.infos = sampled_waymo_infos self.infos = sampled_waymo_infos
self.logger.info('Total sampled samples for Waymo dataset: %d' % len(self.infos)) self.logger.info('Total sampled samples for Waymo dataset: %d' % len(self.infos))
use_sequence_data = self.dataset_cfg.get('SEQUENCE_CONFIG', None) is not None and self.dataset_cfg.SEQUENCE_CONFIG.ENABLED
if not use_sequence_data:
seq_name_to_infos = None
return seq_name_to_infos
def load_data_to_shared_memory(self): def load_data_to_shared_memory(self):
self.logger.info(f'Loading training data to shared memory (file limit={self.shared_memory_file_limit})') self.logger.info(f'Loading training data to shared memory (file limit={self.shared_memory_file_limit})')
...@@ -134,21 +144,21 @@ class WaymoDataset(DatasetTemplate): ...@@ -134,21 +144,21 @@ class WaymoDataset(DatasetTemplate):
sequence_file = found_sequence_file sequence_file = found_sequence_file
return sequence_file return sequence_file
def get_infos(self, raw_data_path, save_path, num_workers=multiprocessing.cpu_count(), has_label=True, sampled_interval=1): def get_infos(self, raw_data_path, save_path, num_workers=multiprocessing.cpu_count(), has_label=True, sampled_interval=1, update_info_only=False):
from functools import partial
from . import waymo_utils from . import waymo_utils
print('---------------The waymo sample interval is %d, total sequecnes is %d-----------------' print('---------------The waymo sample interval is %d, total sequecnes is %d-----------------'
% (sampled_interval, len(self.sample_sequence_list))) % (sampled_interval, len(self.sample_sequence_list)))
process_single_sequence = partial( process_single_sequence = partial(
waymo_utils.process_single_sequence, waymo_utils.process_single_sequence,
save_path=save_path, sampled_interval=sampled_interval, has_label=has_label save_path=save_path, sampled_interval=sampled_interval, has_label=has_label, update_info_only=update_info_only
) )
sample_sequence_file_list = [ sample_sequence_file_list = [
self.check_sequence_name_with_all_version(raw_data_path / sequence_file) self.check_sequence_name_with_all_version(raw_data_path / sequence_file)
for sequence_file in self.sample_sequence_list for sequence_file in self.sample_sequence_list
] ]
# process_single_sequence(sample_sequence_file_list[0])
with multiprocessing.Pool(num_workers) as p: with multiprocessing.Pool(num_workers) as p:
sequence_infos = list(tqdm(p.imap(process_single_sequence, sample_sequence_file_list), sequence_infos = list(tqdm(p.imap(process_single_sequence, sample_sequence_file_list),
total=len(sample_sequence_file_list))) total=len(sample_sequence_file_list)))
...@@ -166,6 +176,63 @@ class WaymoDataset(DatasetTemplate): ...@@ -166,6 +176,63 @@ class WaymoDataset(DatasetTemplate):
points_all[:, 3] = np.tanh(points_all[:, 3]) points_all[:, 3] = np.tanh(points_all[:, 3])
return points_all return points_all
def get_sequence_data(self, info, points, sequence_name, sample_idx, sequence_cfg):
"""
Args:
info:
points:
sequence_name:
sample_idx:
sequence_cfg:
Returns:
"""
def remove_ego_points(points, center_radius=1.0):
mask = ~((np.abs(points[:, 0]) < center_radius) & (np.abs(points[:, 1]) < center_radius))
return points[mask]
pose_cur = info['pose'].reshape((4, 4))
num_pts_cur = points.shape[0]
sample_idx_pre_list = np.clip(sample_idx + np.arange(
sequence_cfg.SAMPLE_OFFSET[0], sequence_cfg.SAMPLE_OFFSET[1]), 0, 0x7FFFFFFF)
if sequence_cfg.get('ONEHOT_TIMESTAMP', False):
onehot_cur = np.zeros((points.shape[0], len(sample_idx_pre_list) + 1)).astype(points.dtype)
onehot_cur[:, 0] = 1
points = np.hstack([points, onehot_cur])
else:
points = np.hstack([points, np.zeros((points.shape[0], 1)).astype(points.dtype)])
points_pre_all = []
num_points_pre = []
sequence_info = self.seq_name_to_infos[sequence_name]
for i, sample_idx_pre in enumerate(sample_idx_pre_list):
if sample_idx == sample_idx_pre:
continue
points_pre = self.get_lidar(sequence_name, sample_idx_pre)
pose_pre = sequence_info[sample_idx_pre]['pose'].reshape((4, 4))
expand_points_pre = np.concatenate([points_pre[:, :3], np.ones((points_pre.shape[0], 1))], axis=-1)
points_pre_global = np.dot(expand_points_pre, pose_pre.T)[:, :3]
expand_points_pre_global = np.concatenate([points_pre_global,
np.ones((points_pre_global.shape[0], 1))], axis=-1)
points_pre2cur = np.dot(expand_points_pre_global, np.linalg.inv(pose_cur.T))[:, :3]
points_pre = np.concatenate([points_pre2cur, points_pre[:, 3:]], axis=-1)
if sequence_cfg.get('ONEHOT_TIMESTAMP', False):
onehot_vector = np.zeros((points_pre.shape[0], len(sample_idx_pre_list) + 1))
onehot_vector[:, i + 1] = 1
points_pre = np.hstack([points_pre, onehot_vector])
else:
# add timestamp
points_pre = np.hstack([points_pre, 0.1 * (sample_idx - sample_idx_pre)
* np.ones((points_pre.shape[0], 1)).astype(points_pre.dtype)]) # one frame 0.1s
points_pre = remove_ego_points(points_pre, 1.0)
points_pre_all.append(points_pre)
num_points_pre.append(points_pre.shape[0])
points = np.concatenate([points] + points_pre_all, axis=0).astype(np.float32)
num_points_all = np.array([num_pts_cur] + num_points_pre).astype(np.int32)
return points, num_points_all, sample_idx_pre_list
def __len__(self): def __len__(self):
if self._merge_all_iters_to_one_epoch: if self._merge_all_iters_to_one_epoch:
return len(self.infos) * self.total_epochs return len(self.infos) * self.total_epochs
...@@ -187,6 +254,11 @@ class WaymoDataset(DatasetTemplate): ...@@ -187,6 +254,11 @@ class WaymoDataset(DatasetTemplate):
else: else:
points = self.get_lidar(sequence_name, sample_idx) points = self.get_lidar(sequence_name, sample_idx)
if self.dataset_cfg.get('SEQUENCE_CONFIG', None) is not None and self.dataset_cfg.SEQUENCE_CONFIG.ENABLED:
points, num_points_all, sample_idx_pre_list = self.get_sequence_data(
info, points, sequence_name, sample_idx, self.dataset_cfg.SEQUENCE_CONFIG
)
input_dict = { input_dict = {
'points': points, 'points': points,
'frame_id': info['frame_id'], 'frame_id': info['frame_id'],
...@@ -201,6 +273,11 @@ class WaymoDataset(DatasetTemplate): ...@@ -201,6 +273,11 @@ class WaymoDataset(DatasetTemplate):
else: else:
gt_boxes_lidar = annos['gt_boxes_lidar'] gt_boxes_lidar = annos['gt_boxes_lidar']
if self.dataset_cfg.get('TRAIN_WITH_SPEED', False):
assert gt_boxes_lidar.shape[-1] == 9
else:
gt_boxes_lidar = gt_boxes_lidar[:, 0:7]
if self.training and self.dataset_cfg.get('FILTER_EMPTY_BOXES_FOR_TRAIN', False): if self.training and self.dataset_cfg.get('FILTER_EMPTY_BOXES_FOR_TRAIN', False):
mask = (annos['num_points_in_gt'] > 0) # filter empty boxes mask = (annos['num_points_in_gt'] > 0) # filter empty boxes
annos['name'] = annos['name'][mask] annos['name'] = annos['name'][mask]
...@@ -273,9 +350,21 @@ class WaymoDataset(DatasetTemplate): ...@@ -273,9 +350,21 @@ class WaymoDataset(DatasetTemplate):
def create_groundtruth_database(self, info_path, save_path, used_classes=None, split='train', sampled_interval=10, def create_groundtruth_database(self, info_path, save_path, used_classes=None, split='train', sampled_interval=10,
processed_data_tag=None): processed_data_tag=None):
use_sequence_data = self.dataset_cfg.get('SEQUENCE_CONFIG', None) is not None and self.dataset_cfg.SEQUENCE_CONFIG.ENABLED
if use_sequence_data:
st_frame, ed_frame = self.dataset_cfg.SEQUENCE_CONFIG.SAMPLE_OFFSET[0], self.dataset_cfg.SEQUENCE_CONFIG.SAMPLE_OFFSET[1]
self.dataset_cfg.SEQUENCE_CONFIG.SAMPLE_OFFSET[0] = min(-4, st_frame) # at least we use 5 frames for generating gt database to support various sequence configs (<= 5 frames)
st_frame = self.dataset_cfg.SEQUENCE_CONFIG.SAMPLE_OFFSET[0]
database_save_path = save_path / ('%s_gt_database_%s_sampled_%d_multiframe_%s_to_%s' % (processed_data_tag, split, sampled_interval, st_frame, ed_frame))
db_info_save_path = save_path / ('%s_waymo_dbinfos_%s_sampled_%d_multiframe_%s_to_%s.pkl' % (processed_data_tag, split, sampled_interval, st_frame, ed_frame))
db_data_save_path = save_path / ('%s_gt_database_%s_sampled_%d_multiframe_%s_to_%s_global.npy' % (processed_data_tag, split, sampled_interval, st_frame, ed_frame))
else:
database_save_path = save_path / ('%s_gt_database_%s_sampled_%d' % (processed_data_tag, split, sampled_interval)) database_save_path = save_path / ('%s_gt_database_%s_sampled_%d' % (processed_data_tag, split, sampled_interval))
db_info_save_path = save_path / ('%s_waymo_dbinfos_%s_sampled_%d.pkl' % (processed_data_tag, split, sampled_interval)) db_info_save_path = save_path / ('%s_waymo_dbinfos_%s_sampled_%d.pkl' % (processed_data_tag, split, sampled_interval))
db_data_save_path = save_path / ('%s_gt_database_%s_sampled_%d_global.npy' % (processed_data_tag, split, sampled_interval)) db_data_save_path = save_path / ('%s_gt_database_%s_sampled_%d_global.npy' % (processed_data_tag, split, sampled_interval))
database_save_path.mkdir(parents=True, exist_ok=True) database_save_path.mkdir(parents=True, exist_ok=True)
all_db_infos = {} all_db_infos = {}
with open(info_path, 'rb') as f: with open(info_path, 'rb') as f:
...@@ -283,8 +372,8 @@ class WaymoDataset(DatasetTemplate): ...@@ -283,8 +372,8 @@ class WaymoDataset(DatasetTemplate):
point_offset_cnt = 0 point_offset_cnt = 0
stacked_gt_points = [] stacked_gt_points = []
for k in range(0, len(infos), sampled_interval): for k in tqdm(range(0, len(infos), sampled_interval)):
print('gt_database sample: %d/%d' % (k + 1, len(infos))) # print('gt_database sample: %d/%d' % (k + 1, len(infos)))
info = infos[k] info = infos[k]
pc_info = info['point_cloud'] pc_info = info['point_cloud']
...@@ -292,6 +381,11 @@ class WaymoDataset(DatasetTemplate): ...@@ -292,6 +381,11 @@ class WaymoDataset(DatasetTemplate):
sample_idx = pc_info['sample_idx'] sample_idx = pc_info['sample_idx']
points = self.get_lidar(sequence_name, sample_idx) points = self.get_lidar(sequence_name, sample_idx)
if use_sequence_data:
points, num_points_all, sample_idx_pre_list = self.get_sequence_data(
info, points, sequence_name, sample_idx, self.dataset_cfg.SEQUENCE_CONFIG
)
annos = info['annos'] annos = info['annos']
names = annos['name'] names = annos['name']
difficulty = annos['difficulty'] difficulty = annos['difficulty']
...@@ -325,6 +419,8 @@ class WaymoDataset(DatasetTemplate): ...@@ -325,6 +419,8 @@ class WaymoDataset(DatasetTemplate):
gt_points[:, :3] -= gt_boxes[i, :3] gt_points[:, :3] -= gt_boxes[i, :3]
if (used_classes is None) or names[i] in used_classes: if (used_classes is None) or names[i] in used_classes:
gt_points = gt_points.astype(np.float32)
assert gt_points.dtype == np.float32
with open(filepath, 'w') as f: with open(filepath, 'w') as f:
gt_points.tofile(f) gt_points.tofile(f)
...@@ -352,10 +448,148 @@ class WaymoDataset(DatasetTemplate): ...@@ -352,10 +448,148 @@ class WaymoDataset(DatasetTemplate):
stacked_gt_points = np.concatenate(stacked_gt_points, axis=0) stacked_gt_points = np.concatenate(stacked_gt_points, axis=0)
np.save(db_data_save_path, stacked_gt_points) np.save(db_data_save_path, stacked_gt_points)
def create_gt_database_of_single_scene(self, info_with_idx, database_save_path=None, use_sequence_data=False, used_classes=None,
total_samples=0, use_cuda=False, crop_gt_with_tail=False):
info, info_idx = info_with_idx
print('gt_database sample: %d/%d' % (info_idx, total_samples))
all_db_infos = {}
pc_info = info['point_cloud']
sequence_name = pc_info['lidar_sequence']
sample_idx = pc_info['sample_idx']
points = self.get_lidar(sequence_name, sample_idx)
if use_sequence_data:
points, num_points_all, sample_idx_pre_list = self.get_sequence_data(
info, points, sequence_name, sample_idx, self.dataset_cfg.SEQUENCE_CONFIG
)
annos = info['annos']
names = annos['name']
difficulty = annos['difficulty']
gt_boxes = annos['gt_boxes_lidar']
if info_idx % 4 != 0 and len(names) > 0:
mask = (names == 'Vehicle')
names = names[~mask]
difficulty = difficulty[~mask]
gt_boxes = gt_boxes[~mask]
if info_idx % 2 != 0 and len(names) > 0:
mask = (names == 'Pedestrian')
names = names[~mask]
difficulty = difficulty[~mask]
gt_boxes = gt_boxes[~mask]
num_obj = gt_boxes.shape[0]
if num_obj == 0:
return {}
if use_sequence_data and crop_gt_with_tail:
assert gt_boxes.shape[1] == 9
speed = gt_boxes[:, 7:9]
sequence_cfg = self.dataset_cfg.SEQUENCE_CONFIG
assert sequence_cfg.SAMPLE_OFFSET[1] == 0
assert sequence_cfg.SAMPLE_OFFSET[0] < 0
num_frames = sequence_cfg.SAMPLE_OFFSET[1] - sequence_cfg.SAMPLE_OFFSET[0] + 1
assert num_frames > 1
latest_center = gt_boxes[:, 0:2]
oldest_center = latest_center - speed * (num_frames - 1) * 0.1
new_center = (latest_center + oldest_center) * 0.5
new_length = gt_boxes[:, 3] + np.linalg.norm(latest_center - oldest_center, axis=-1)
gt_boxes_crop = gt_boxes.copy()
gt_boxes_crop[:, 0:2] = new_center
gt_boxes_crop[:, 3] = new_length
else:
gt_boxes_crop = gt_boxes
if use_cuda:
box_idxs_of_pts = roiaware_pool3d_utils.points_in_boxes_gpu(
torch.from_numpy(points[:, 0:3]).unsqueeze(dim=0).float().cuda(),
torch.from_numpy(gt_boxes_crop[:, 0:7]).unsqueeze(dim=0).float().cuda()
).long().squeeze(dim=0).cpu().numpy()
else:
box_point_mask = roiaware_pool3d_utils.points_in_boxes_cpu(
torch.from_numpy(points[:, 0:3]).float(),
torch.from_numpy(gt_boxes_crop[:, 0:7]).float()
).long().numpy() # (num_boxes, num_points)
for i in range(num_obj):
filename = '%s_%04d_%s_%d.bin' % (sequence_name, sample_idx, names[i], i)
filepath = database_save_path / filename
if use_cuda:
gt_points = points[box_idxs_of_pts == i]
else:
gt_points = points[box_point_mask[i] > 0]
gt_points[:, :3] -= gt_boxes[i, :3]
if (used_classes is None) or names[i] in used_classes:
gt_points = gt_points.astype(np.float32)
assert gt_points.dtype == np.float32
with open(filepath, 'w') as f:
gt_points.tofile(f)
db_path = str(filepath.relative_to(self.root_path)) # gt_database/xxxxx.bin
db_info = {'name': names[i], 'path': db_path, 'sequence_name': sequence_name,
'sample_idx': sample_idx, 'gt_idx': i, 'box3d_lidar': gt_boxes[i],
'num_points_in_gt': gt_points.shape[0], 'difficulty': difficulty[i],
'box3d_crop': gt_boxes_crop[i]}
if names[i] in all_db_infos:
all_db_infos[names[i]].append(db_info)
else:
all_db_infos[names[i]] = [db_info]
return all_db_infos
def create_groundtruth_database_parallel(self, info_path, save_path, used_classes=None, split='train', sampled_interval=10,
processed_data_tag=None, num_workers=16, crop_gt_with_tail=False):
use_sequence_data = self.dataset_cfg.get('SEQUENCE_CONFIG', None) is not None and self.dataset_cfg.SEQUENCE_CONFIG.ENABLED
if use_sequence_data:
st_frame, ed_frame = self.dataset_cfg.SEQUENCE_CONFIG.SAMPLE_OFFSET[0], self.dataset_cfg.SEQUENCE_CONFIG.SAMPLE_OFFSET[1]
self.dataset_cfg.SEQUENCE_CONFIG.SAMPLE_OFFSET[0] = min(-4, st_frame) # at least we use 5 frames for generating gt database to support various sequence configs (<= 5 frames)
st_frame = self.dataset_cfg.SEQUENCE_CONFIG.SAMPLE_OFFSET[0]
database_save_path = save_path / ('%s_gt_database_%s_sampled_%d_multiframe_%s_to_%s_%sparallel' % (processed_data_tag, split, sampled_interval, st_frame, ed_frame, 'tail_' if crop_gt_with_tail else ''))
db_info_save_path = save_path / ('%s_waymo_dbinfos_%s_sampled_%d_multiframe_%s_to_%s_%sparallel.pkl' % (processed_data_tag, split, sampled_interval, st_frame, ed_frame, 'tail_' if crop_gt_with_tail else ''))
else:
database_save_path = save_path / ('%s_gt_database_%s_sampled_%d_parallel' % (processed_data_tag, split, sampled_interval))
db_info_save_path = save_path / ('%s_waymo_dbinfos_%s_sampled_%d_parallel.pkl' % (processed_data_tag, split, sampled_interval))
database_save_path.mkdir(parents=True, exist_ok=True)
with open(info_path, 'rb') as f:
infos = pickle.load(f)
print(f'Number workers: {num_workers}')
create_gt_database_of_single_scene = partial(
self.create_gt_database_of_single_scene,
use_sequence_data=use_sequence_data, database_save_path=database_save_path,
used_classes=used_classes, total_samples=len(infos), use_cuda=False,
crop_gt_with_tail=crop_gt_with_tail
)
# create_gt_database_of_single_scene((infos[300], 0))
with multiprocessing.Pool(num_workers) as p:
all_db_infos_list = list(p.map(create_gt_database_of_single_scene, zip(infos, np.arange(len(infos)))))
all_db_infos = {}
for cur_db_infos in all_db_infos_list:
for key, val in cur_db_infos.items():
if key not in all_db_infos:
all_db_infos[key] = val
else:
all_db_infos[key].extend(val)
for k, v in all_db_infos.items():
print('Database %s: %d' % (k, len(v)))
with open(db_info_save_path, 'wb') as f:
pickle.dump(all_db_infos, f)
def create_waymo_infos(dataset_cfg, class_names, data_path, save_path, def create_waymo_infos(dataset_cfg, class_names, data_path, save_path,
raw_data_tag='raw_data', processed_data_tag='waymo_processed_data', raw_data_tag='raw_data', processed_data_tag='waymo_processed_data',
workers=min(16, multiprocessing.cpu_count())): workers=min(16, multiprocessing.cpu_count()), update_info_only=False):
dataset = WaymoDataset( dataset = WaymoDataset(
dataset_cfg=dataset_cfg, class_names=class_names, root_path=data_path, dataset_cfg=dataset_cfg, class_names=class_names, root_path=data_path,
training=False, logger=common_utils.create_logger() training=False, logger=common_utils.create_logger()
...@@ -372,7 +606,7 @@ def create_waymo_infos(dataset_cfg, class_names, data_path, save_path, ...@@ -372,7 +606,7 @@ def create_waymo_infos(dataset_cfg, class_names, data_path, save_path,
waymo_infos_train = dataset.get_infos( waymo_infos_train = dataset.get_infos(
raw_data_path=data_path / raw_data_tag, raw_data_path=data_path / raw_data_tag,
save_path=save_path / processed_data_tag, num_workers=workers, has_label=True, save_path=save_path / processed_data_tag, num_workers=workers, has_label=True,
sampled_interval=1 sampled_interval=1, update_info_only=update_info_only
) )
with open(train_filename, 'wb') as f: with open(train_filename, 'wb') as f:
pickle.dump(waymo_infos_train, f) pickle.dump(waymo_infos_train, f)
...@@ -382,12 +616,15 @@ def create_waymo_infos(dataset_cfg, class_names, data_path, save_path, ...@@ -382,12 +616,15 @@ def create_waymo_infos(dataset_cfg, class_names, data_path, save_path,
waymo_infos_val = dataset.get_infos( waymo_infos_val = dataset.get_infos(
raw_data_path=data_path / raw_data_tag, raw_data_path=data_path / raw_data_tag,
save_path=save_path / processed_data_tag, num_workers=workers, has_label=True, save_path=save_path / processed_data_tag, num_workers=workers, has_label=True,
sampled_interval=1 sampled_interval=1, update_info_only=update_info_only
) )
with open(val_filename, 'wb') as f: with open(val_filename, 'wb') as f:
pickle.dump(waymo_infos_val, f) pickle.dump(waymo_infos_val, f)
print('----------------Waymo info val file is saved to %s----------------' % val_filename) print('----------------Waymo info val file is saved to %s----------------' % val_filename)
if update_info_only:
return
print('---------------Start create groundtruth database for data augmentation---------------') print('---------------Start create groundtruth database for data augmentation---------------')
os.environ["CUDA_VISIBLE_DEVICES"] = "0" os.environ["CUDA_VISIBLE_DEVICES"] = "0"
dataset.set_split(train_split) dataset.set_split(train_split)
...@@ -398,24 +635,56 @@ def create_waymo_infos(dataset_cfg, class_names, data_path, save_path, ...@@ -398,24 +635,56 @@ def create_waymo_infos(dataset_cfg, class_names, data_path, save_path,
print('---------------Data preparation Done---------------') print('---------------Data preparation Done---------------')
def create_waymo_gt_database(
dataset_cfg, class_names, data_path, save_path, processed_data_tag='waymo_processed_data',
workers=min(16, multiprocessing.cpu_count()), use_parallel=False, crop_gt_with_tail=False):
dataset = WaymoDataset(
dataset_cfg=dataset_cfg, class_names=class_names, root_path=data_path,
training=False, logger=common_utils.create_logger()
)
train_split = 'train'
train_filename = save_path / ('%s_infos_%s.pkl' % (processed_data_tag, train_split))
print('---------------Start create groundtruth database for data augmentation---------------')
dataset.set_split(train_split)
if use_parallel:
dataset.create_groundtruth_database_parallel(
info_path=train_filename, save_path=save_path, split='train', sampled_interval=1,
used_classes=['Vehicle', 'Pedestrian', 'Cyclist'], processed_data_tag=processed_data_tag,
num_workers=workers, crop_gt_with_tail=crop_gt_with_tail
)
else:
dataset.create_groundtruth_database(
info_path=train_filename, save_path=save_path, split='train', sampled_interval=1,
used_classes=['Vehicle', 'Pedestrian', 'Cyclist'], processed_data_tag=processed_data_tag
)
print('---------------Data preparation Done---------------')
if __name__ == '__main__': if __name__ == '__main__':
import argparse import argparse
import yaml
from easydict import EasyDict
parser = argparse.ArgumentParser(description='arg parser') parser = argparse.ArgumentParser(description='arg parser')
parser.add_argument('--cfg_file', type=str, default=None, help='specify the config of dataset') parser.add_argument('--cfg_file', type=str, default=None, help='specify the config of dataset')
parser.add_argument('--func', type=str, default='create_waymo_infos', help='') parser.add_argument('--func', type=str, default='create_waymo_infos', help='')
parser.add_argument('--processed_data_tag', type=str, default='waymo_processed_data_v0_5_0', help='') parser.add_argument('--processed_data_tag', type=str, default='waymo_processed_data_v0_5_0', help='')
parser.add_argument('--update_info_only', action='store_true', default=False, help='')
parser.add_argument('--use_parallel', action='store_true', default=False, help='')
parser.add_argument('--wo_crop_gt_with_tail', action='store_true', default=False, help='')
args = parser.parse_args() args = parser.parse_args()
ROOT_DIR = (Path(__file__).resolve().parent / '../../../').resolve()
if args.func == 'create_waymo_infos': if args.func == 'create_waymo_infos':
import yaml
from easydict import EasyDict
try: try:
yaml_config = yaml.safe_load(open(args.cfg_file), Loader=yaml.FullLoader) yaml_config = yaml.safe_load(open(args.cfg_file), Loader=yaml.FullLoader)
except: except:
yaml_config = yaml.safe_load(open(args.cfg_file)) yaml_config = yaml.safe_load(open(args.cfg_file))
dataset_cfg = EasyDict(yaml_config) dataset_cfg = EasyDict(yaml_config)
ROOT_DIR = (Path(__file__).resolve().parent / '../../../').resolve()
dataset_cfg.PROCESSED_DATA_TAG = args.processed_data_tag dataset_cfg.PROCESSED_DATA_TAG = args.processed_data_tag
create_waymo_infos( create_waymo_infos(
dataset_cfg=dataset_cfg, dataset_cfg=dataset_cfg,
...@@ -423,5 +692,24 @@ if __name__ == '__main__': ...@@ -423,5 +692,24 @@ if __name__ == '__main__':
data_path=ROOT_DIR / 'data' / 'waymo', data_path=ROOT_DIR / 'data' / 'waymo',
save_path=ROOT_DIR / 'data' / 'waymo', save_path=ROOT_DIR / 'data' / 'waymo',
raw_data_tag='raw_data', raw_data_tag='raw_data',
processed_data_tag=args.processed_data_tag processed_data_tag=args.processed_data_tag,
update_info_only=args.update_info_only
)
elif args.func == 'create_waymo_gt_database':
try:
yaml_config = yaml.safe_load(open(args.cfg_file), Loader=yaml.FullLoader)
except:
yaml_config = yaml.safe_load(open(args.cfg_file))
dataset_cfg = EasyDict(yaml_config)
dataset_cfg.PROCESSED_DATA_TAG = args.processed_data_tag
create_waymo_gt_database(
dataset_cfg=dataset_cfg,
class_names=['Vehicle', 'Pedestrian', 'Cyclist'],
data_path=ROOT_DIR / 'data' / 'waymo',
save_path=ROOT_DIR / 'data' / 'waymo',
processed_data_tag=args.processed_data_tag,
use_parallel=args.use_parallel,
crop_gt_with_tail=not args.wo_crop_gt_with_tail
) )
else:
raise NotImplementedError
...@@ -60,6 +60,9 @@ class OpenPCDetWaymoDetectionMetricsEstimator(tf.test.TestCase): ...@@ -60,6 +60,9 @@ class OpenPCDetWaymoDetectionMetricsEstimator(tf.test.TestCase):
if fake_gt_infos: if fake_gt_infos:
info['gt_boxes_lidar'] = boxes3d_kitti_fakelidar_to_lidar(info['gt_boxes_lidar']) info['gt_boxes_lidar'] = boxes3d_kitti_fakelidar_to_lidar(info['gt_boxes_lidar'])
if info['gt_boxes_lidar'].shape[-1] == 9:
boxes3d.append(info['gt_boxes_lidar'][box_mask][:, 0:7])
else:
boxes3d.append(info['gt_boxes_lidar'][box_mask]) boxes3d.append(info['gt_boxes_lidar'][box_mask])
else: else:
num_boxes = len(info['boxes_lidar']) num_boxes = len(info['boxes_lidar'])
...@@ -67,6 +70,8 @@ class OpenPCDetWaymoDetectionMetricsEstimator(tf.test.TestCase): ...@@ -67,6 +70,8 @@ class OpenPCDetWaymoDetectionMetricsEstimator(tf.test.TestCase):
score.append(info['score']) score.append(info['score'])
boxes3d.append(np.array(info['boxes_lidar'])) boxes3d.append(np.array(info['boxes_lidar']))
box_name = info['name'] box_name = info['name']
if boxes3d[-1].shape[-1] == 9:
boxes3d[-1] = boxes3d[-1][:, 0:7]
obj_type += [self.WAYMO_CLASSES.index(name) for i, name in enumerate(box_name)] obj_type += [self.WAYMO_CLASSES.index(name) for i, name in enumerate(box_name)]
frame_id.append(np.array([frame_index] * num_boxes)) frame_id.append(np.array([frame_index] * num_boxes))
......
...@@ -20,11 +20,12 @@ except: ...@@ -20,11 +20,12 @@ except:
WAYMO_CLASSES = ['unknown', 'Vehicle', 'Pedestrian', 'Sign', 'Cyclist'] WAYMO_CLASSES = ['unknown', 'Vehicle', 'Pedestrian', 'Sign', 'Cyclist']
def generate_labels(frame): def generate_labels(frame, pose):
obj_name, difficulty, dimensions, locations, heading_angles = [], [], [], [], [] obj_name, difficulty, dimensions, locations, heading_angles = [], [], [], [], []
tracking_difficulty, speeds, accelerations, obj_ids = [], [], [], [] tracking_difficulty, speeds, accelerations, obj_ids = [], [], [], []
num_points_in_gt = [] num_points_in_gt = []
laser_labels = frame.laser_labels laser_labels = frame.laser_labels
for i in range(len(laser_labels)): for i in range(len(laser_labels)):
box = laser_labels[i].box box = laser_labels[i].box
class_ind = laser_labels[i].type class_ind = laser_labels[i].type
...@@ -37,6 +38,8 @@ def generate_labels(frame): ...@@ -37,6 +38,8 @@ def generate_labels(frame):
locations.append(loc) locations.append(loc)
obj_ids.append(laser_labels[i].id) obj_ids.append(laser_labels[i].id)
num_points_in_gt.append(laser_labels[i].num_lidar_points_in_box) num_points_in_gt.append(laser_labels[i].num_lidar_points_in_box)
speeds.append([laser_labels[i].metadata.speed_x, laser_labels[i].metadata.speed_y])
accelerations.append([laser_labels[i].metadata.accel_x, laser_labels[i].metadata.accel_y])
annotations = {} annotations = {}
annotations['name'] = np.array(obj_name) annotations['name'] = np.array(obj_name)
...@@ -48,15 +51,21 @@ def generate_labels(frame): ...@@ -48,15 +51,21 @@ def generate_labels(frame):
annotations['obj_ids'] = np.array(obj_ids) annotations['obj_ids'] = np.array(obj_ids)
annotations['tracking_difficulty'] = np.array(tracking_difficulty) annotations['tracking_difficulty'] = np.array(tracking_difficulty)
annotations['num_points_in_gt'] = np.array(num_points_in_gt) annotations['num_points_in_gt'] = np.array(num_points_in_gt)
annotations['speed_global'] = np.array(speeds)
annotations['accel_global'] = np.array(accelerations)
annotations = common_utils.drop_info_with_name(annotations, name='unknown') annotations = common_utils.drop_info_with_name(annotations, name='unknown')
if annotations['name'].__len__() > 0: if annotations['name'].__len__() > 0:
global_speed = np.pad(annotations['speed_global'], ((0, 0), (0, 1)), mode='constant', constant_values=0) # (N, 3)
speed = np.dot(global_speed, np.linalg.inv(pose[:3, :3].T))
speed = speed[:, :2]
gt_boxes_lidar = np.concatenate([ gt_boxes_lidar = np.concatenate([
annotations['location'], annotations['dimensions'], annotations['heading_angles'][..., np.newaxis]], annotations['location'], annotations['dimensions'], annotations['heading_angles'][..., np.newaxis], speed],
axis=1 axis=1
) )
else: else:
gt_boxes_lidar = np.zeros((0, 7)) gt_boxes_lidar = np.zeros((0, 9))
annotations['gt_boxes_lidar'] = gt_boxes_lidar annotations['gt_boxes_lidar'] = gt_boxes_lidar
return annotations return annotations
...@@ -158,8 +167,12 @@ def convert_range_image_to_point_cloud(frame, range_images, camera_projections, ...@@ -158,8 +167,12 @@ def convert_range_image_to_point_cloud(frame, range_images, camera_projections,
def save_lidar_points(frame, cur_save_path, use_two_returns=True): def save_lidar_points(frame, cur_save_path, use_two_returns=True):
range_images, camera_projections, range_image_top_pose = \ ret_outputs = frame_utils.parse_range_image_and_camera_projection(frame)
frame_utils.parse_range_image_and_camera_projection(frame) if len(ret_outputs) == 4:
range_images, camera_projections, seg_labels, range_image_top_pose = ret_outputs
else:
assert len(ret_outputs) == 3
range_images, camera_projections, range_image_top_pose = ret_outputs
points, cp_points, points_in_NLZ_flag, points_intensity, points_elongation = convert_range_image_to_point_cloud( points, cp_points, points_in_NLZ_flag, points_intensity, points_elongation = convert_range_image_to_point_cloud(
frame, range_images, camera_projections, range_image_top_pose, ri_index=(0, 1) if use_two_returns else (0,) frame, range_images, camera_projections, range_image_top_pose, ri_index=(0, 1) if use_two_returns else (0,)
...@@ -181,7 +194,7 @@ def save_lidar_points(frame, cur_save_path, use_two_returns=True): ...@@ -181,7 +194,7 @@ def save_lidar_points(frame, cur_save_path, use_two_returns=True):
return num_points_of_each_lidar return num_points_of_each_lidar
def process_single_sequence(sequence_file, save_path, sampled_interval, has_label=True, use_two_returns=True): def process_single_sequence(sequence_file, save_path, sampled_interval, has_label=True, use_two_returns=True, update_info_only=False):
sequence_name = os.path.splitext(os.path.basename(sequence_file))[0] sequence_name = os.path.splitext(os.path.basename(sequence_file))[0]
# print('Load record (sampled_interval=%d): %s' % (sampled_interval, sequence_name)) # print('Load record (sampled_interval=%d): %s' % (sampled_interval, sequence_name))
...@@ -197,8 +210,13 @@ def process_single_sequence(sequence_file, save_path, sampled_interval, has_labe ...@@ -197,8 +210,13 @@ def process_single_sequence(sequence_file, save_path, sampled_interval, has_labe
sequence_infos = [] sequence_infos = []
if pkl_file.exists(): if pkl_file.exists():
sequence_infos = pickle.load(open(pkl_file, 'rb')) sequence_infos = pickle.load(open(pkl_file, 'rb'))
sequence_infos_old = None
if not update_info_only:
print('Skip sequence since it has been processed before: %s' % pkl_file) print('Skip sequence since it has been processed before: %s' % pkl_file)
return sequence_infos return sequence_infos
else:
sequence_infos_old = sequence_infos
sequence_infos = []
for cnt, data in enumerate(dataset): for cnt, data in enumerate(dataset):
if cnt % sampled_interval != 0: if cnt % sampled_interval != 0:
...@@ -227,9 +245,13 @@ def process_single_sequence(sequence_file, save_path, sampled_interval, has_labe ...@@ -227,9 +245,13 @@ def process_single_sequence(sequence_file, save_path, sampled_interval, has_labe
info['pose'] = pose info['pose'] = pose
if has_label: if has_label:
annotations = generate_labels(frame) annotations = generate_labels(frame, pose=pose)
info['annos'] = annotations info['annos'] = annotations
if update_info_only and sequence_infos_old is not None:
assert info['frame_id'] == sequence_infos_old[cnt]['frame_id']
num_points_of_each_lidar = sequence_infos_old[cnt]['num_points_of_each_lidar']
else:
num_points_of_each_lidar = save_lidar_points( num_points_of_each_lidar = save_lidar_points(
frame, cur_save_dir / ('%04d.npy' % cnt), use_two_returns=use_two_returns frame, cur_save_dir / ('%04d.npy' % cnt), use_two_returns=use_two_returns
) )
......
...@@ -90,7 +90,7 @@ def corners_rect_to_camera(corners): ...@@ -90,7 +90,7 @@ def corners_rect_to_camera(corners):
return camera_rect return camera_rect
def mask_boxes_outside_range_numpy(boxes, limit_range, min_num_corners=1): def mask_boxes_outside_range_numpy(boxes, limit_range, min_num_corners=1, use_center_to_filter=True):
""" """
Args: Args:
boxes: (N, 7) [x, y, z, dx, dy, dz, heading, ...], (x, y, z) is the box center boxes: (N, 7) [x, y, z, dx, dy, dz, heading, ...], (x, y, z) is the box center
...@@ -102,8 +102,13 @@ def mask_boxes_outside_range_numpy(boxes, limit_range, min_num_corners=1): ...@@ -102,8 +102,13 @@ def mask_boxes_outside_range_numpy(boxes, limit_range, min_num_corners=1):
""" """
if boxes.shape[1] > 7: if boxes.shape[1] > 7:
boxes = boxes[:, 0:7] boxes = boxes[:, 0:7]
if use_center_to_filter:
box_centers = boxes[:, 0:3]
mask = ((box_centers >= limit_range[0:3]) & (box_centers <= limit_range[3:6])).all(axis=-1)
else:
corners = boxes_to_corners_3d(boxes) # (N, 8, 3) corners = boxes_to_corners_3d(boxes) # (N, 8, 3)
mask = ((corners >= limit_range[0:3]) & (corners <= limit_range[3:6])).all(axis=2) corners = corners[:, :, 0:2]
mask = ((corners >= limit_range[0:2]) & (corners <= limit_range[3:5])).all(axis=2)
mask = mask.sum(axis=1) >= min_num_corners # (N) mask = mask.sum(axis=1) >= min_num_corners # (N)
return mask return mask
......
...@@ -28,7 +28,7 @@ def write_version_to_file(version, target_file): ...@@ -28,7 +28,7 @@ def write_version_to_file(version, target_file):
if __name__ == '__main__': if __name__ == '__main__':
version = '0.5.2+%s' % get_git_commit_number() version = '0.6.0+%s' % get_git_commit_number()
write_version_to_file(version, 'pcdet/version.py') write_version_to_file(version, 'pcdet/version.py')
setup( setup(
......
...@@ -32,6 +32,12 @@ DATA_AUGMENTOR: ...@@ -32,6 +32,12 @@ DATA_AUGMENTOR:
DB_DATA_PATH: DB_DATA_PATH:
- waymo_processed_data_v0_5_0_gt_database_train_sampled_1_global.npy - waymo_processed_data_v0_5_0_gt_database_train_sampled_1_global.npy
BACKUP_DB_INFO:
# if the above DB_INFO cannot be found, will use this backup one
DB_INFO_PATH: waymo_processed_data_v0_5_0_waymo_dbinfos_train_sampled_1_multiframe_-4_to_0.pkl
DB_DATA_PATH: waymo_processed_data_v0_5_0_gt_database_train_sampled_1_multiframe_-4_to_0_global.npy
NUM_POINT_FEATURES: 6
PREPARE: { PREPARE: {
filter_by_min_points: ['Vehicle:5', 'Pedestrian:5', 'Cyclist:5'], filter_by_min_points: ['Vehicle:5', 'Pedestrian:5', 'Cyclist:5'],
filter_by_difficulty: [-1], filter_by_difficulty: [-1],
......
DATASET: 'WaymoDataset'
DATA_PATH: '../data/waymo'
PROCESSED_DATA_TAG: 'waymo_processed_data_v0_5_0'
POINT_CLOUD_RANGE: [-75.2, -75.2, -2, 75.2, 75.2, 4]
DATA_SPLIT: {
'train': train,
'test': val
}
SAMPLED_INTERVAL: {
'train': 5,
'test': 1
}
FILTER_EMPTY_BOXES_FOR_TRAIN: True
DISABLE_NLZ_FLAG_ON_POINTS: True
USE_SHARED_MEMORY: False # it will load the data to shared memory to speed up (DO NOT USE IT IF YOU DO NOT FULLY UNDERSTAND WHAT WILL HAPPEN)
SHARED_MEMORY_FILE_LIMIT: 35000 # set it based on the size of your shared memory
SEQUENCE_CONFIG:
ENABLED: True
SAMPLE_OFFSET: [-3, 0]
TRAIN_WITH_SPEED: True
DATA_AUGMENTOR:
DISABLE_AUG_LIST: ['placeholder']
AUG_CONFIG_LIST:
- NAME: gt_sampling
USE_ROAD_PLANE: False
DB_INFO_PATH:
- waymo_processed_data_v0_5_0_waymo_dbinfos_train_sampled_1_multiframe_-4_to_0.pkl
USE_SHARED_MEMORY: False # set it to True to speed up (it costs about 50GB? shared memory)
DB_DATA_PATH:
- waymo_processed_data_v0_5_0_gt_database_train_sampled_1_multiframe_-4_to_0_global.npy
PREPARE: {
filter_by_min_points: ['Vehicle:5', 'Pedestrian:5', 'Cyclist:5'],
filter_by_difficulty: [-1],
}
SAMPLE_GROUPS: ['Vehicle:15', 'Pedestrian:10', 'Cyclist:10']
NUM_POINT_FEATURES: 6
REMOVE_EXTRA_WIDTH: [0.0, 0.0, 0.0]
LIMIT_WHOLE_SCENE: True
FILTER_OBJ_POINTS_BY_TIMESTAMP: True
TIME_RANGE: [0.3, 0.0] # 0.3s-0.0s indicates 4 frames
- NAME: random_world_flip
ALONG_AXIS_LIST: ['x', 'y']
- NAME: random_world_rotation
WORLD_ROT_ANGLE: [-0.78539816, 0.78539816]
- NAME: random_world_scaling
WORLD_SCALE_RANGE: [0.95, 1.05]
POINT_FEATURE_ENCODING: {
encoding_type: absolute_coordinates_encoding,
used_feature_list: ['x', 'y', 'z', 'intensity', 'elongation', 'timestamp'],
src_feature_list: ['x', 'y', 'z', 'intensity', 'elongation', 'timestamp'],
}
DATA_PROCESSOR:
- NAME: mask_points_and_boxes_outside_range
REMOVE_OUTSIDE_BOXES: True
USE_CENTER_TO_FILTER: True
- NAME: shuffle_points
SHUFFLE_ENABLED: {
'train': True,
'test': True
}
- NAME: transform_points_to_voxels
VOXEL_SIZE: [0.1, 0.1, 0.15]
MAX_POINTS_PER_VOXEL: 5
MAX_NUMBER_OF_VOXELS: {
'train': 180000,
'test': 400000
}
...@@ -54,8 +54,7 @@ DATA_CONFIG: ...@@ -54,8 +54,7 @@ DATA_CONFIG:
WORLD_SCALE_RANGE: [0.95, 1.05] WORLD_SCALE_RANGE: [0.95, 1.05]
- NAME: random_world_translation - NAME: random_world_translation
WORLD_TRANSLATION_RANGE: [ -0.2, 0.2 ] NOISE_TRANSLATE_STD: [0.5, 0.5, 0.5]
ALONG_AXIS_LIST: ['x', 'y', 'z']
- NAME: random_local_translation - NAME: random_local_translation
LOCAL_TRANSLATION_RANGE: [0.95, 1.05] LOCAL_TRANSLATION_RANGE: [0.95, 1.05]
......
...@@ -38,8 +38,7 @@ DATA_CONFIG: ...@@ -38,8 +38,7 @@ DATA_CONFIG:
WORLD_SCALE_RANGE: [0.9, 1.1] WORLD_SCALE_RANGE: [0.9, 1.1]
- NAME: random_world_translation - NAME: random_world_translation
NOISE_TRANSLATE_STD: 0.5 NOISE_TRANSLATE_STD: [0.5, 0.5, 0.5]
ALONG_AXIS_LIST: ['x', 'y', 'z']
DATA_PROCESSOR: DATA_PROCESSOR:
......
CLASS_NAMES: ['Vehicle', 'Pedestrian', 'Cyclist']
DATA_CONFIG:
_BASE_CONFIG_: cfgs/dataset_configs/waymo_dataset_multiframe.yaml
MODEL:
NAME: CenterPoint
VFE:
NAME: MeanVFE
BACKBONE_3D:
NAME: VoxelResBackBone8x
MAP_TO_BEV:
NAME: HeightCompression
NUM_BEV_FEATURES: 256
BACKBONE_2D:
NAME: BaseBEVBackbone
LAYER_NUMS: [5, 5]
LAYER_STRIDES: [1, 2]
NUM_FILTERS: [128, 256]
UPSAMPLE_STRIDES: [1, 2]
NUM_UPSAMPLE_FILTERS: [256, 256]
DENSE_HEAD:
NAME: CenterHead
CLASS_AGNOSTIC: False
CLASS_NAMES_EACH_HEAD: [
['Vehicle', 'Pedestrian', 'Cyclist']
]
SHARED_CONV_CHANNEL: 64
USE_BIAS_BEFORE_NORM: True
NUM_HM_CONV: 2
SEPARATE_HEAD_CFG:
HEAD_ORDER: ['center', 'center_z', 'dim', 'rot', 'vel']
HEAD_DICT: {
'center': {'out_channels': 2, 'num_conv': 2},
'center_z': {'out_channels': 1, 'num_conv': 2},
'dim': {'out_channels': 3, 'num_conv': 2},
'rot': {'out_channels': 2, 'num_conv': 2},
'vel': {'out_channels': 2, 'num_conv': 2},
}
TARGET_ASSIGNER_CONFIG:
FEATURE_MAP_STRIDE: 8
NUM_MAX_OBJS: 500
GAUSSIAN_OVERLAP: 0.1
MIN_RADIUS: 2
LOSS_CONFIG:
LOSS_WEIGHTS: {
'cls_weight': 1.0,
'loc_weight': 2.0,
'code_weights': [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.2, 0.2]
}
POST_PROCESSING:
SCORE_THRESH: 0.1
POST_CENTER_LIMIT_RANGE: [-75.2, -75.2, -2, 75.2, 75.2, 4]
MAX_OBJ_PER_SAMPLE: 500
NMS_CONFIG:
NMS_TYPE: nms_gpu
NMS_THRESH: 0.7
NMS_PRE_MAXSIZE: 4096
NMS_POST_MAXSIZE: 500
POST_PROCESSING:
RECALL_THRESH_LIST: [0.3, 0.5, 0.7]
EVAL_METRIC: waymo
OPTIMIZATION:
BATCH_SIZE_PER_GPU: 4
NUM_EPOCHS: 30
OPTIMIZER: adam_onecycle
LR: 0.003
WEIGHT_DECAY: 0.01
MOMENTUM: 0.9
MOMS: [0.95, 0.85]
PCT_START: 0.4
DIV_FACTOR: 10
DECAY_STEP_LIST: [35, 45]
LR_DECAY: 0.1
LR_CLIP: 0.0000001
LR_WARMUP: False
WARMUP_EPOCH: 1
GRAD_NORM_CLIP: 10
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment