[Feature] Addd information of LiDAR segmentation in NuScenes annotation file (#2431)

* change md

* add pkl

docs/en/advanced_guides/datasets/nuscenes.md

@@ -4,7 +4,11 @@ This page provides specific tutorials about the usage of MMDetection3D for nuSce
 
 ## Before Preparation
 
-You can download nuScenes 3D detection data [HERE](https://www.nuscenes.org/download) and unzip all zip files.
+You can download nuScenes 3D detection `Full dataset (v1.0)` [HERE](https://www.nuscenes.org/download) and unzip all zip files.
+
+If you want to implement 3D semantic segmentation task, you need to additionally download the `nuScenes-lidarseg` data annotation and place the extracted files in the nuScenes corresponding folder.
+
+**Note**: `v1.0trainval(test)/categroy.json` in nuScenes-lidarseg will replace the original `v1.0trainval(test)/categroy.json` of the Full dataset (v1.0), but will not affect the 3D object detection task.
 
 Like the general way to prepare dataset, it is recommended to symlink the dataset root to `$MMDETECTION3D/data`.
 
@@ -20,6 +24,7 @@ mmdetection3d
 │   │   ├── maps
 │   │   ├── samples
 │   │   ├── sweeps
+│   │   ├── lidarseg (optional)
 │   │   ├── v1.0-test
 |   |   ├── v1.0-trainval
 ```
@@ -45,6 +50,7 @@ mmdetection3d
 │   │   ├── maps
 │   │   ├── samples
 │   │   ├── sweeps
+│   │   ├── lidarseg (optional)
 │   │   ├── v1.0-test
 |   |   ├── v1.0-trainval
 │   │   ├── nuscenes_database
@@ -60,11 +66,11 @@ mmdetection3d
   - info\['sample_idx'\]: The index of this sample in the whole dataset.
   - info\['token'\]: Sample data token.
   - info\['timestamp'\]: Timestamp of the sample data.
+  - info\['ego2global'\]: The transformation matrix from the ego vehicle to global coordinates. (4x4 list)
   - info\['lidar_points'\]: A dict containing all the information related to the lidar points.
     - info\['lidar_points'\]\['lidar_path'\]: The filename of the lidar point cloud data.
     - info\['lidar_points'\]\['num_pts_feats'\]: The feature dimension of point.
     - info\['lidar_points'\]\['lidar2ego'\]: The transformation matrix from this lidar sensor to ego vehicle. (4x4 list)
-    - info\['lidar_points'\]\['ego2global'\]: The transformation matrix from the ego vehicle to global coordinates. (4x4 list)
   - info\['lidar_sweeps'\]: A list contains sweeps information (The intermediate lidar frames without annotations)
     - info\['lidar_sweeps'\]\[i\]\['lidar_points'\]\['data_path'\]: The lidar data path of i-th sweep.
     - info\['lidar_sweeps'\]\[i\]\['lidar_points'\]\['lidar2ego'\]: The transformation matrix from this lidar sensor to ego vehicle. (4x4 list)
@@ -95,6 +101,7 @@ mmdetection3d
     - info\['cam_instances'\]\['CAM_XXX'\]\[i\]\['velocity'\]: Velocities of 3D bounding boxes (no vertical measurements due to inaccuracy), a list has shape (2,).
     - info\['cam_instances'\]\['CAM_XXX'\]\[i\]\['attr_label'\]: The attr label of instance. We maintain a default attribute collection and mapping for attribute classification.
     - info\['cam_instances'\]\['CAM_XXX'\]\[i\]\['bbox_3d'\]: List of 7 numbers representing the 3D bounding box of the instance, in (x, y, z, l, h, w, yaw) order.
+  - info\['pts_semantic_mask_path'\]：The filename of the lidar point cloud semantic segmentation annotation.
 
 Note:
 

docs/zh_cn/advanced_guides/datasets/nuscenes.md

@@ -4,7 +4,11 @@
 
 ## 准备之前
 
-您可以在[这里](https://www.nuscenes.org/download)下载 nuScenes 3D 检测数据并解压缩所有 zip 文件。
+您可以在[这里](https://www.nuscenes.org/download)下载 nuScenes 3D 检测数据 Full dataset (v1.0) 并解压缩所有 zip 文件。
+
+如果您想进行 3D 语义分割任务，需要额外下载 nuScenes-lidarseg 数据标注，并将解压的文件放入 nuScenes 对应的文件夹下。
+
+**注意**：nuScenes-lidarseg 中的 v1.0trainval(test)/categroy.json 会替换原先 Full dataset (v1.0) 原先的 v1.0trainval(test)/categroy.json，但是不会对 3D 目标检测任务造成影响。
 
 像准备数据集的一般方法一样，建议将数据集根目录链接到 `$MMDETECTION3D/data`。
 
@@ -20,6 +24,7 @@ mmdetection3d
 │   │   ├── maps
 │   │   ├── samples
 │   │   ├── sweeps
+│   │   ├── lidarseg (optional)
 │   │   ├── v1.0-test
 |   |   ├── v1.0-trainval
 ```
@@ -44,11 +49,11 @@ mmdetection3d
 │   │   ├── maps
 │   │   ├── samples
 │   │   ├── sweeps
+│   │   ├── lidarseg (optional)
 │   │   ├── v1.0-test
 |   |   ├── v1.0-trainval
 │   │   ├── nuscenes_database
 │   │   ├── nuscenes_infos_train.pkl
-│   │   ├── nuscenes_infos_trainval.pkl
 │   │   ├── nuscenes_infos_val.pkl
 │   │   ├── nuscenes_infos_test.pkl
 │   │   ├── nuscenes_dbinfos_train.pkl
@@ -59,11 +64,11 @@ mmdetection3d
   - info\['sample_idx'\]：样本在整个数据集的索引。
   - info\['token'\]：样本数据标记。
   - info\['timestamp'\]：样本数据时间戳。
+  - info\['ego2global'\]：自车到全局坐标的变换矩阵。（4x4 列表）
   - info\['lidar_points'\]：是一个字典，包含了所有与激光雷达点相关的信息。
     - info\['lidar_points'\]\['lidar_path'\]：激光雷达点云数据的文件名。
     - info\['lidar_points'\]\['num_pts_feats'\]：点的特征维度。
     - info\['lidar_points'\]\['lidar2ego'\]：该激光雷达传感器到自车的变换矩阵。（4x4 列表）
-    - info\['lidar_points'\]\['ego2global'\]：自车到全局坐标的变换矩阵。（4x4 列表）
   - info\['lidar_sweeps'\]：是一个列表，包含了扫描信息（没有标注的中间帧）。
     - info\['lidar_sweeps'\]\[i\]\['lidar_points'\]\['data_path'\]：第 i 次扫描的激光雷达数据的文件路径。
     - info\['lidar_sweeps'\]\[i\]\['lidar_points'\]\[lidar2ego''\]：当前激光雷达传感器到自车的变换矩阵。（4x4 列表）
@@ -94,6 +99,7 @@ mmdetection3d
     - info\['cam_instances'\]\['CAM_XXX'\]\[i\]\['velocity'\]：3D 边界框的速度（由于不正确，没有垂直测量），大小为 (2, ) 的列表。
     - info\['cam_instances'\]\['CAM_XXX'\]\[i\]\['attr_label'\]：实例的属性标签。我们为属性分类维护了一个属性集合和映射。
     - info\['cam_instances'\]\['CAM_XXX'\]\[i\]\['bbox_3d'\]：长度为 7 的列表，以 (x, y, z, l, h, w, yaw) 的顺序表示实例的 3D 边界框。
+  - info\['pts_semantic_mask_path'\]：激光雷达语义分割标注的文件名。
 
 注意：
 

tools/dataset_converters/nuscenes_converter.py

@@ -267,6 +267,11 @@ def _fill_trainval_infos(nusc,
                 [a['num_radar_pts'] for a in annotations])
             info['valid_flag'] = valid_flag
 
+            if 'lidarseg' in nusc.table_names:
+                info['pts_semantic_mask_path'] = osp.join(
+                    nusc.dataroot,
+                    nusc.get('lidarseg', lidar_token)['filename'])
+
         if sample['scene_token'] in train_scenes:
             train_nusc_infos.append(info)
         else:

tools/dataset_converters/update_infos_to_v2.py

@@ -365,6 +365,9 @@ def update_nuscenes_infos(pkl_path, out_dir):
             temp_data_info[
                 'cam_instances'] = generate_nuscenes_camera_instances(
                     ori_info_dict, nusc)
+        if 'pts_semantic_mask_path' in ori_info_dict:
+            temp_data_info['pts_semantic_mask_path'] = Path(
+                ori_info_dict['pts_semantic_mask_path']).name
         temp_data_info, _ = clear_data_info_unused_keys(temp_data_info)
         converted_list.append(temp_data_info)
     pkl_name = Path(pkl_path).name