[Docs] Fix kitti_det.md, data_prepare.md and new_data_model.md. (#1777)

* fix some docs

* fix docs

* fix cfg

* fix docs

* fix comments

* fix comments

* fix comments:

* fix comments

* fix comments

configs/_base_/datasets/kitti-3d-3class.py

@@ -19,7 +19,11 @@ db_sampler = dict(
         type='LoadPointsFromFile', coord_type='LIDAR', load_dim=4, use_dim=4))
 
 train_pipeline = [
-    dict(type='LoadPointsFromFile', coord_type='LIDAR', load_dim=4, use_dim=4),
+    dict(
+        type='LoadPointsFromFile',
+        coord_type='LIDAR',
+        load_dim=4,  # x, y, z, intensity
+        use_dim=4),
     dict(type='LoadAnnotations3D', with_bbox_3d=True, with_label_3d=True),
     dict(type='ObjectSample', db_sampler=db_sampler),
     dict(

configs/_base_/datasets/kitti-3d-car.py

@@ -17,7 +17,11 @@ db_sampler = dict(
         type='LoadPointsFromFile', coord_type='LIDAR', load_dim=4, use_dim=4))
 
 train_pipeline = [
-    dict(type='LoadPointsFromFile', coord_type='LIDAR', load_dim=4, use_dim=4),
+    dict(
+        type='LoadPointsFromFile',
+        coord_type='LIDAR',
+        load_dim=4,  # x, y, z, intensity
+        use_dim=4),
     dict(type='LoadAnnotations3D', with_bbox_3d=True, with_label_3d=True),
     dict(type='ObjectSample', db_sampler=db_sampler),
     dict(

docs/en/advanced_guides/datasets/kitti_det.md

@@ -2,8 +2,6 @@
 
 This page provides specific tutorials about the usage of MMDetection3D for KITTI dataset.
 
-**Note**: Current tutorial is only for LiDAR-based and multi-modality 3D detection methods. Contents related to monocular methods will be supplemented afterwards.
-
 ## Prepare dataset
 
 You can download KITTI 3D detection data [HERE](http://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=3d) and unzip all zip files. Besides, the road planes could be downloaded from [HERE](https://download.openmmlab.com/mmdetection3d/data/train_planes.zip), which are optional for data augmentation during training for better performance. The road planes are generated by [AVOD](https://github.com/kujason/avod), you can see more details [HERE](https://github.com/kujason/avod/issues/19).
@@ -47,7 +45,6 @@ wget -c  https://raw.githubusercontent.com/traveller59/second.pytorch/master/sec
 
 
 python tools/create_data.py kitti --root-path ./data/kitti --out-dir ./data/kitti --extra-tag kitti --with-plane
-
 ```
 
 Note that if your local disk does not have enough space for saving converted data, you can change the `out-dir` to anywhere else, and you need to remove the `--with-plane` flag if `planes` are not prepared.
@@ -80,37 +77,42 @@ kitti
 ├── kitti_dbinfos_train.pkl
 ├── kitti_infos_test.pkl
 ├── kitti_infos_trainval.pkl
-├── kitti_infos_train_mono3d.coco.json
-├── kitti_infos_trainval_mono3d.coco.json
-├── kitti_infos_test_mono3d.coco.json
-├── kitti_infos_val_mono3d.coco.json
 ```
 
 - `kitti_gt_database/xxxxx.bin`: point cloud data included in each 3D bounding box of the training dataset
-- `kitti_infos_train.pkl`: training dataset infos, each frame info contains following details:
-  - info\['point_cloud'\]: {'num_features': 4, 'velodyne_path': velodyne_path}.
-  - info\['annos'\]: {
-    - location: x,y,z are bottom center in referenced camera coordinate system (in meters), an Nx3 array
-    - dimensions: height, width, length (in meters), an Nx3 array
-    - rotation_y: rotation ry around Y-axis in camera coordinates \[-pi..pi\], an N array
-    - name:  ground truth name array, an N array
-    - difficulty: kitti difficulty, Easy, Moderate, Hard
-    - group_ids: used for multi-part object
-      }
-  - (optional) info\['calib'\]: {
-    - P0: camera0 projection matrix after rectification, an 3x4 array
-    - P1: camera1 projection matrix after rectification, an 3x4 array
-    - P2: camera2 projection matrix after rectification, an 3x4 array
-    - P3: camera3 projection matrix after rectification, an 3x4 array
-    - R0_rect: rectifying rotation matrix, an 4x4 array
-    - Tr_velo_to_cam: transformation from Velodyne coordinate to camera coordinate, an 4x4 array
-    - Tr_imu_to_velo: transformation from IMU coordinate to Velodyne coordinate, an 4x4 array
-      }
-  - (optional) info\['image'\]:{'image_idx': idx, 'image_path': image_path, 'image_shape', image_shape}.
-
-**Note:** the info\['annos'\] is in the referenced camera coordinate system. More details please refer to [this](http://www.cvlibs.net/publications/Geiger2013IJRR.pdf)
-
-The core function to get kitti_infos_xxx.pkl and kitti_infos_xxx_mono3d.coco.json are [get_kitti_image_info](https://github.com/open-mmlab/mmdetection3d/blob/7873c8f62b99314f35079f369d1dab8d63f8a3ce/tools/dataset_converters/kitti_data_utils.py#L140) and [get_2d_boxes](https://github.com/open-mmlab/mmdetection3d/blob/7873c8f62b99314f35079f369d1dab8d63f8a3ce/tools/dataset_converters/kitti_converter.py#L378). Please refer to [kitti_converter.py](https://github.com/open-mmlab/mmdetection3d/blob/7873c8f62b99314f35079f369d1dab8d63f8a3ce/tools/dataset_converters/kitti_converter.py) for more details.
+- `kitti_infos_train.pkl`: training dataset info, each frame info has two keys: `metainfo` and `data_list`.
+  `metainfo` is a dict, it contains the essential information for the dataset, such as `CLASSES` and `version`.
+  `data_list` is a list, it has all the needed data information, and each item is detailed information dict for a single sample. Detailed information is as follows:
+  - info\['sample_idx'\]: The index of this sample in the whole dataset.
+  - info\['images'\]: Information of images captured by multiple cameras. A dict contains five keys including: `CAM0`, `CAM1`, `CAM2`, `CAM3`, `R0_rect`.
+    - info\['images'\]\['R0_rect'\]: Rectifying rotation matrix with shape (4, 4).
+    - info\['images'\]\['CAM2'\]: Include some information about the `CAM2` camera sensor.
+      - info\['images'\]\['CAM2'\]\['img_path'\]: The path to the image file.
+      - info\['images'\]\['CAM2'\]\['height'\]: The height of the image.
+      - info\['images'\]\['CAM2'\]\['width'\]: The width of the image.
+      - info\['images'\]\['CAM2'\]\['cam2img'\]: Transformation matrix from camera to image with shape (4, 4).
+      - info\['images'\]\['CAM2'\]\['lidar2cam'\]: Transformation matrix from lidar to camera with shape (4, 4).
+      - info\['images'\]\['CAM2'\]\['lidar2img'\]: Transformation matrix from lidar to image with shape (4, 4).
+  - info\['lidar_points'\]: Information of point cloud captured by Lidar. A dict contains information of LiDAR point cloud frame.
+    - info\['lidar_points'\]\['lidar_path'\]: The file path of the lidar point cloud data.
+    - info\['lidar_points'\]\['num_features'\]: Number of features for each point.
+    - info\['lidar_points'\]\['Tr_velo_to_cam'\]: Transformation from Velodyne coordinate to camera coordinate with shape (4, 4).
+    - info\['lidar_points'\]\['Tr_imu_to_velo'\]: Transformation from IMU coordinate to Velodyne coordinate with shape (4, 4).
+  - info\['instances'\]: Required by object detection task. A list contains some dict of instance infos. Each dict corresponds to annotations of one instance in this frame.
+    - info\['instances'\]\['bbox'\]: List of 4 numbers representing the 2D bounding box of the instance, in (x1, y1, x2, y2) order.
+    - info\['instances'\]\['bbox_3d'\]: List of 7 numbers representing the 3D bounding box of the instance, in (x, y, z, w, h, l, yaw) order.
+    - info\['instances'\]\['bbox_label'\]: An int indicate the 2D label of instance and the -1 indicating ignore.
+    - info\['instances'\]\['bbox_label_3d'\]: An int indicate the 3D label of instance and the -1 indicating ignore.
+    - info\['instances'\]\['depth'\]: Projected center depth of the 3D bounding box with respect to the image plane.
+    - info\['instances'\]\['num_lidar_pts'\]: The number of LiDAR points in the 3D bounding box.
+    - info\['instances'\]\['center_2d'\]: Projected 2D center of the 3D bounding box.
+    - info\['instances'\]\['difficulty'\]: Kitti difficulty, Easy, Moderate, Hard.
+    - info\['instances'\]\['truncated'\]: The instances bbox is truncated.
+    - info\['instances'\]\['occluded'\]: The instances bbox is semi occluded or fully occluded.
+    - info\['instances'\]\['group_ids'\]: Used for multi-part object.
+  - info\['plane'\](optional): Road level information.
+
+Please refer to [kitti_converter.py](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/tools/dataset_converters/kitti_converter.py) and [update_infos_to_v2.py ](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/tools/dataset_converters/update_infos_to_v2.py) for more details.
 
 ## Train pipeline
 
@@ -118,17 +120,11 @@ A typical train pipeline of 3D detection on KITTI is as below.
 
 ```python
 train_pipeline = [
-    dict(
-        type='LoadPointsFromFile',
+    dict(type='LoadPointsFromFile',
          coord_type='LIDAR',
          load_dim=4, # x, y, z, intensity
-        use_dim=4, # x, y, z, intensity
-        file_client_args=file_client_args),
-    dict(
-        type='LoadAnnotations3D',
-        with_bbox_3d=True,
-        with_label_3d=True,
-        file_client_args=file_client_args),
+         use_dim=4),
+    dict(type='LoadAnnotations3D', with_bbox_3d=True, with_label_3d=True),
     dict(type='ObjectSample', db_sampler=db_sampler),
     dict(
         type='ObjectNoise',
@@ -144,8 +140,9 @@ train_pipeline = [
     dict(type='PointsRangeFilter', point_cloud_range=point_cloud_range),
     dict(type='ObjectRangeFilter', point_cloud_range=point_cloud_range),
     dict(type='PointShuffle'),
-    dict(type='DefaultFormatBundle3D', class_names=class_names),
-    dict(type='Collect3D', keys=['points', 'gt_bboxes_3d', 'gt_labels_3d'])
+    dict(
+        type='Pack3DDetInputs',
+        keys=['points', 'gt_bboxes_3d', 'gt_labels_3d'])
 ]
 ```
 
@@ -159,7 +156,7 @@ train_pipeline = [
 An example to evaluate PointPillars with 8 GPUs with kitti metrics is as follows:
 
 ```shell
-bash tools/dist_test.sh configs/pointpillars/pointpillars_hv_secfpn_8xb6-160e_kitti-3d-3class.py work_dirs/hv_pointpillars_secfpn_6x8_160e_kitti-3d-3class/latest.pth 8 --eval bbox
+bash tools/dist_test.sh configs/pointpillars/pointpillars_hv_secfpn_8xb6-160e_kitti-3d-3class.py work_dirs/configs/pointpillars/pointpillars_hv_secfpn_8xb6-160e_kitti-3d-3class/latest.pth 8
 ```
 
 ## Metrics
@@ -183,12 +180,34 @@ aos  AP:97.70, 89.11, 87.38
 
 ## Testing and make a submission
 
-An example to test PointPillars on KITTI with 8 GPUs and generate a submission to the leaderboard is as follows:
+An example to test PointPillars on KITTI with 8 GPUs and generate a submission to the leaderboard. An example to test PointPillars on KITTI with 8 GPUs and generate a submission to the leaderboard is as follows:
+
+- First, you need to modify the `test_evaluator` dict in your config file to add `pklfile_prefix` and `submission_prefix`, just like:
+
+```python
+data_root = 'data/kitti/'
+test_evaluator = dict(
+    type='KittiMetric',
+    ann_file=data_root + 'kitti_infos_val.pkl',
+    metric='bbox',
+    pklfile_prefix='results/kitti-3class/kitti_results',
+    submission_prefix='results/kitti-3class/kitti_results')
+```
+
+- And then, you can run the test script.
+
+```shell
+mkdir -p results/kitti-3class
+
+./tools/dist_test.sh configs/pointpillars/configs/pointpillars/pointpillars_hv_secfpn_8xb6-160e_kitti-3d-3class.py work_dirs/configs/pointpillars/pointpillars_hv_secfpn_8xb6-160e_kitti-3d-3class/latest.pth 8
+```
+
+- Or you can use `--cfg-options "test_evaluator.jsonfile_prefix=work_dirs/pp-nus/results_eval.json)` after the test command, and run test script directly.
 
 ```shell
 mkdir -p results/kitti-3class
 
-./tools/dist_test.sh configs/pointpillars/pointpillars_hv_secfpn_8xb6-160e_kitti-3d-3class.py work_dirs/hv_pointpillars_secfpn_6x8_160e_kitti-3d-3class/latest.pth 8 --out results/kitti-3class/results_eval.pkl --format-only --eval-options 'pklfile_prefix=results/kitti-3class/kitti_results' 'submission_prefix=results/kitti-3class/kitti_results'
+./tools/dist_test.sh configs/pointpillars/configs/pointpillars/pointpillars_hv_secfpn_8xb6-160e_kitti-3d-3class.py work_dirs/configs/pointpillars/pointpillars_hv_secfpn_8xb6-160e_kitti-3d-3class/latest.pth 8 --cfg-options 'test_evaluator.pklfile_prefix=results/kitti-3class/kitti_results' 'test_evaluator.submission_prefix=results/kitti-3class/kitti_results'
 ```
 
 After generating `results/kitti-3class/kitti_results/xxxxx.txt` files, you can submit these files to KITTI benchmark. Please refer to the [KITTI official website](http://www.cvlibs.net/datasets/kitti/index.php) for more details.

docs/en/user_guides/2_new_data_model.md

+# 2: Train with customized datasets
+
+In this note, you will know how to train and test predefined models with customized datasets. We use the Waymo dataset as an example to describe the whole process.
+
+The basic steps are as below:
+
+1. Prepare the customized dataset
+2. Prepare a config
+3. Train, test, inference models on the customized dataset.
+
+## Prepare the customized dataset
+
+There are three ways to support a new dataset in MMDetection3D:
+
+1. reorganize the dataset into existing format.
+2. reorganize the dataset into a standard format.
+3. implement a new dataset.
+
+Usually we recommend to use the first two methods which are usually easier than the third.
+
+In this note, we give an example for converting the data into KITTI format, you can refer to this to reorganize your dataset into kitti format. About the standard format dataset, and you can refer to [customize_dataset.md](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/docs/en/advanced_guides/customize_dataset.md).
+
+**Note**: We take Waymo as the example here considering its format is totally different from other existing formats. For other datasets using similar methods to organize data, like Lyft compared to nuScenes, it would be easier to directly implement the new data converter (for the second approach above) instead of converting it to another format (for the first approach above).
+
+### KITTI dataset format
+
+Firstly, the raw data for 3D object detection from KITTI are typically organized as follows, where `ImageSets` contains split files indicating which files belong to training/validation/testing set, `calib` contains calibration information files, `image_2` and `velodyne` include image data and point cloud data, and `label_2` includes label files for 3D detection.
+
+```
+mmdetection3d
+├── mmdet3d
+├── tools
+├── configs
+├── data
+│   ├── kitti
+│   │   ├── ImageSets
+│   │   ├── testing
+│   │   │   ├── calib
+│   │   │   ├── image_2
+│   │   │   ├── velodyne
+│   │   ├── training
+│   │   │   ├── calib
+│   │   │   ├── image_2
+│   │   │   ├── label_2
+│   │   │   ├── velodyne
+```
+
+Specific annotation format is described in the official object development [kit](https://s3.eu-central-1.amazonaws.com/avg-kitti/devkit_object.zip). For example, it consists of the following labels:
+
+```
+#Values    Name      Description
+----------------------------------------------------------------------------
+   1    type         Describes the type of object: 'Car', 'Van', 'Truck',
+                     'Pedestrian', 'Person_sitting', 'Cyclist', 'Tram',
+                     'Misc' or 'DontCare'
+   1    truncated    Float from 0 (non-truncated) to 1 (truncated), where
+                     truncated refers to the object leaving image boundaries
+   1    occluded     Integer (0,1,2,3) indicating occlusion state:
+                     0 = fully visible, 1 = partly occluded
+                     2 = largely occluded, 3 = unknown
+   1    alpha        Observation angle of object, ranging [-pi..pi]
+   4    bbox         2D bounding box of object in the image (0-based index):
+                     contains left, top, right, bottom pixel coordinates
+   3    dimensions   3D object dimensions: height, width, length (in meters)
+   3    location     3D object location x,y,z in camera coordinates (in meters)
+   1    rotation_y   Rotation ry around Y-axis in camera coordinates [-pi..pi]
+   1    score        Only for results: Float, indicating confidence in
+                     detection, needed for p/r curves, higher is better.
+```
+
+Assume we use the Waymo dataset.
+
+After downloading the data, we need to implement a function to convert both the input data and annotation format into the KITTI style. Then we can implement `WaymoDataset` inherited from `KittiDataset` to load the data and perform training, and implement `WaymoMetric` inherited from `KittiMetric` for evaluation.
+
+Specifically, we implement a waymo [converter](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/tools/dataset_converters/waymo_converter.py) to convert Waymo data into KITTI format and a waymo dataset [class](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/mmdet3d/datasets/waymo_dataset.py) to process it, in addition need to add a waymo [metric](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/mmdet3d/evaluation/metrics/waymo_metric.py) to evaluate results. Because we preprocess the raw data and reorganize it like KITTI, the dataset class could be implemented more easily by inheriting from KittiDataset. Regarding the dataset evaluation metric, because Waymo has its own evaluation approach, we need further implement a new Waymo metric; more about the metric could refer to [metric_and_evaluator.md](https://github.com/open-mmlab/mmengine/blob/main/docs/en/tutorials/metric_and_evaluator.md). Afterward, users can successfully convert the data format and use `WaymoDataset` to train and evaluate the model by `WaymoMetric`.
+
+For more details about the intermediate results of preprocessing of Waymo dataset, please refer to its [waymo_det.md](https://mmdetection3d.readthedocs.io/en/latest/datasets/waymo_det.html).
+
+## Prepare a config
+
+The second step is to prepare configs such that the dataset could be successfully loaded. In addition, adjusting hyperparameters is usually necessary to obtain decent performance in 3D detection.
+
+Suppose we would like to train PointPillars on Waymo to achieve 3D detection for 3 classes, vehicle, cyclist and pedestrian, we need to prepare dataset config like [this](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/configs/_base_/datasets/waymoD5-3d-3class.py), model config like [this](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/configs/_base_/models/pointpillars_hv_secfpn_waymo.py) and combine them like [this](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/configs/pointpillars/pointpillars_hv_secfpn_sbn-all_16xb2-2x_waymoD5-3d-3class.py), compared to KITTI [dataset config](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/configs/_base_/datasets/kitti-3d-3class.py), [model config](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/configs/_base_/models/pointpillars_hv_secfpn_kitti.py) and [overall](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/configs/pointpillars/pointpillars_hv_secfpn_8xb6-160e_kitti-3d-3class.py).
+
+## Train a new model
+
+To train a model with the new config, you can simply run
+
+```shell
+python tools/train.py configs/pointpillars/pointpillars_hv_secfpn_sbn-all_16xb2-2x_waymoD5-3d-3class.py
+```
+
+For more detailed usages, please refer to the [Case 1](https://mmdetection3d.readthedocs.io/en/latest/1_exist_data_model.html).
+
+## Test and inference
+
+To test the trained model, you can simply run
+
+```shell
+python tools/test.py configs/pointpillars/pointpillars_hv_secfpn_sbn-all_16xb2-2x_waymoD5-3d-3class.py work_dirs/pointpillars_hv_secfpn_sbn-all_16xb2-2x_waymoD5-3d-3class/latest.pth
+```
+
+**Note**: To use Waymo evaluation protocol, you need to follow the [tutorial](https://mmdetection3d.readthedocs.io/en/latest/datasets/waymo_det.html) and prepare files related to metrics computation as official instructions.
+
+For more detailed usages for test and inference, please refer to the [Case 1](https://mmdetection3d.readthedocs.io/en/latest/1_exist_data_model.html).

docs/en/user_guides/dataset_prepare.md

@@ -133,12 +133,30 @@ Note that we follow the original folder names for clear organization. Please ren
 
 ### S3DIS, ScanNet and SUN RGB-D
 
-To prepare S3DIS data, please see its [README](https://github.com/open-mmlab/mmdetection3d/blob/master/data/s3dis/README.md/).
+To prepare S3DIS data, please see its [README](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/data/s3dis/README.md/).
 
-To prepare ScanNet data, please see its [README](https://github.com/open-mmlab/mmdetection3d/blob/master/data/scannet/README.md/).
+To prepare ScanNet data, please see its [README](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/data/scannet/README.md/).
 
-To prepare SUN RGB-D data, please see its [README](https://github.com/open-mmlab/mmdetection3d/blob/master/data/sunrgbd/README.md/).
+To prepare SUN RGB-D data, please see its [README](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/data/sunrgbd/README.md/).
 
 ### Customized Datasets
 
-For using custom datasets, please refer to [Tutorials 2: Customize Datasets](https://mmdetection3d.readthedocs.io/en/latest/tutorials/customize_dataset.html).
+For using custom datasets, please refer to [Customize Datasets](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/docs/en/advanced_guides/customize_dataset.md).
+
+### Update data infos
+
+If you have used v1.0.0rc1-v1.0.0rc4 mmdetection3d to create data infos before, and now you want to use the newest v1.1.0 mmdetection3d, you need to update the data infos file.
+
+```bash
+python tools/dataset_converters/update_infos_to_v2.py --dataset ${DATA_SET} --pkl ${PKL_PATH} --out-dir ${OUT_DIR}
+```
+
+- `dataset` : Name of dataset.
+- `pkl` : Specify the data infos pkl file path.
+- `out-dir` : Output direction of the data infos pkl file.
+
+Example
+
+```bash
+python tools/dataset_converters/update_infos_to_v2.py --dataset kitti --pkl ./data/kitti/kitti_infos_trainval.pkl --out-dir ./data/kitti
+```

mmdet3d/datasets/builder.py

@@ -18,7 +18,8 @@ PIPELINES = TRANSFORMS
 
 
 def build_dataset(cfg, default_args=None):
-    from mmengine import ClassBalancedDataset, ConcatDataset, RepeatDataset
+    from mmengine.dataset import (ClassBalancedDataset, ConcatDataset,
+                                  RepeatDataset)
 
     from mmdet3d.datasets.dataset_wrappers import CBGSDataset
     if isinstance(cfg, (list, tuple)):

tools/dataset_converters/update_infos_to_v2.py

@@ -390,7 +390,7 @@ def update_kitti_infos(pkl_path, out_dir):
         if 'plane' in ori_info_dict:
             temp_data_info['plane'] = ori_info_dict['plane']
 
-        temp_data_info['sample_id'] = ori_info_dict['image']['image_idx']
+        temp_data_info['sample_idx'] = ori_info_dict['image']['image_idx']
 
         temp_data_info['images']['CAM0']['cam2img'] = ori_info_dict['calib'][
             'P0'].tolist()