[Fix] Fix part of documentations (#1758)

* fix liar_det3d * fix ch liar_det3d * fix en and ch liar_seg and add formart results * resolve the comments * fix the nuscence * fix scannet * fix waymo det * fix docs * fix the doc * fix the command line * fix the command line * fix nuscence * fix nuscence * resolve the comments * resp;ve * Minor fix * Minor fix Co-authored-by: Tai-Wang <tab_wang@outlook.com>

[Fix] Fix part of documentations (#1758)
* fix liar_det3d * fix ch liar_det3d * fix en and ch liar_seg and add formart results * resolve the comments * fix the nuscence * fix scannet * fix waymo det * fix docs * fix the doc * fix the command line * fix the command line * fix nuscence * fix nuscence * resolve the comments * resp;ve * Minor fix * Minor fix Co-authored-by: Tai-Wang <tab_wang@outlook.com>
326bbf03 · Shilong Zhang · GitHub · b297f21d · 326bbf03 · 326bbf03
Unverified Commit 326bbf03 authored Sep 01, 2022 by Shilong Zhang Committed by GitHub Sep 01, 2022
15 changed files
--- a/docs/en/advanced_guides/datasets/nuscenes_det.md
+++ b/docs/en/advanced_guides/datasets/nuscenes_det.md
@@ -52,78 +52,58 @@ mmdetection3d
 │   │   ├── nuscenes_infos_val.pkl
 │   │   ├── nuscenes_infos_test.pkl
 │   │   ├── nuscenes_dbinfos_train.pkl
-│   │   ├── nuscenes_infos_train_mono3d.coco.json
-│   │   ├── nuscenes_infos_val_mono3d.coco.json
-│   │   ├── nuscenes_infos_test_mono3d.coco.json
 ```
-Here, .pkl files are generally used for methods involving point clouds and coco-style .json files are more suitable for image-based methods, such as image-based 2D and 3D detection.
-Next, we will elaborate on the details recorded in these info files.
 - `nuscenes_database/xxxxx.bin`: point cloud data included in each 3D bounding box of the training dataset
- `nuscenes_infos_train.pkl`: training dataset info, each frame info has two keys: `metadata` and `infos`.
+- `nuscenes_infos_train.pkl`: training dataset, a dict contains two keys: `metainfo` and `data_list`.
-  `metadata` contains the basic information for the dataset itself, such as `{'version': 'v1.0-trainval'}`, while `infos` contains the detailed information as follows:
+  `metadata` contains the basic information for the dataset itself, such as `CLASSES` and `version`, while `data_list` is a list of dict, each dict ( hereinafter referred to as`info`) contains all the detailed information of single sample as follows:
-  - info\['lidar_path'\]: The file path of the lidar point cloud data.
+  - info\['sample_idx'\]: The index of this sample in the whole dataset.
  - info\['token'\]: Sample data token.
-  - info\['sweeps'\]: Sweeps information (`sweeps` in the nuScenes refer to the intermediate frames without annotations, while `samples` refer to those key frames with annotations).
-    - info\['sweeps'\]\[i\]\['data_path'\]: The data path of i-th sweep.
-    - info\['sweeps'\]\[i\]\['type'\]: The sweep data type, e.g., `'lidar'`.
-    - info\['sweeps'\]\[i\]\['sample_data_token'\]: The sweep sample data token.
-    - info\['sweeps'\]\[i\]\['sensor2ego_translation'\]: The translation from the current sensor (for collecting the sweep data) to ego vehicle. (1x3 list)
-    - info\['sweeps'\]\[i\]\['sensor2ego_rotation'\]: The rotation from the current sensor (for collecting the sweep data) to ego vehicle. (1x4 list in the quaternion format)
-    - info\['sweeps'\]\[i\]\['ego2global_translation'\]: The translation from the ego vehicle to global coordinates. (1x3 list)
-    - info\['sweeps'\]\[i\]\['ego2global_rotation'\]: The rotation from the ego vehicle to global coordinates. (1x4 list in the quaternion format)
-    - info\['sweeps'\]\[i\]\['timestamp'\]: Timestamp of the sweep data.
-    - info\['sweeps'\]\[i\]\['sensor2lidar_translation'\]: The translation from the current sensor (for collecting the sweep data) to lidar. (1x3 list)
-    - info\['sweeps'\]\[i\]\['sensor2lidar_rotation'\]: The rotation from the current sensor (for collecting the sweep data) to lidar. (1x4 list in the quaternion format)
-  - info\['cams'\]: Cameras calibration information. It contains six keys corresponding to each camera: `'CAM_FRONT'`, `'CAM_FRONT_RIGHT'`, `'CAM_FRONT_LEFT'`, `'CAM_BACK'`, `'CAM_BACK_LEFT'`, `'CAM_BACK_RIGHT'`.
-    Each dictionary contains detailed information following the above way for each sweep data (has the same keys for each information as above). In addition, each camera has a key `'cam_intrinsic'` for recording the intrinsic parameters when projecting 3D points to each image plane.
-  - info\['lidar2ego_translation'\]: The translation from lidar to ego vehicle. (1x3 list)
-  - info\['lidar2ego_rotation'\]: The rotation from lidar to ego vehicle. (1x4 list in the quaternion format)
-  - info\['ego2global_translation'\]: The translation from the ego vehicle to global coordinates. (1x3 list)
-  - info\['ego2global_rotation'\]: The rotation from the ego vehicle to global coordinates. (1x4 list in the quaternion format)
  - info\['timestamp'\]: Timestamp of the sample data.
-  - info\['gt_boxes'\]: 7-DoF annotations of 3D bounding boxes, an Nx7 array.
+  - info\['lidar_points'\]: A dict contains all the information related to the lidar points.
-  - info\['gt_names'\]: Categories of 3D bounding boxes, an 1xN array.
+    - info\['lidar_points'\]\['lidar_path'\]: The filename of the lidar point cloud data.
-  - info\['gt_velocity'\]: Velocities of 3D bounding boxes (no vertical measurements due to inaccuracy), an Nx2 array.
+    - info\['lidar_points'\]\['lidar2ego'\]: The transformation matrix from this lidar sensor to ego vehicle. (4x4 list)
-  - info\['num_lidar_pts'\]: Number of lidar points included in each 3D bounding box.
+    - info\['lidar_points'\]\['ego2global'\]: The transformation matrix from the ego vehicle to global coordinates. (4x4 list)
-  - info\['num_radar_pts'\]: Number of radar points included in each 3D bounding box.
+  - info\['lidar_sweeps'\]: A list contains sweeps information (The intermediate lidar frames without annotations)
-  - info\['valid_flag'\]: Whether each bounding box is valid. In general, we only take the 3D boxes that include at least one lidar or radar point as valid boxes.
+    - info\['lidar_sweeps'\]\[i\]\['lidar_points'\]\['data_path'\]: The lidar data path of i-th sweep.
- `nuscenes_infos_train_mono3d.coco.json`: training dataset coco-style info. This file organizes image-based data into three categories (keys): `'categories'`, `'images'`, `'annotations'`.
+    - info\['lidar_sweeps'\]\[i\]\['lidar_points'\]\['lidar2ego'\]: The transformation matrix from this lidar sensor to ego vehicle. (4x4 list)
-  - info\['categories'\]: A list containing all the category names. Each element follows the dictionary format and consists of two keys: `'id'` and `'name'`.
+    - info\['lidar_sweeps'\]\[i\]\['lidar_points'\]\['ego2global'\]: The transformation matrix from the ego vehicle to global coordinates. (4x4 list)
-  - info\['images'\]: A list containing all the image info.
+    - info\['lidar_sweeps'\]\[i\]\['lidar2sensor'\]: The transformation matrix from the main lidar sensor to the current sensor (for collecting the sweep data) to lidar. (4x4 list)
-    - info\['images'\]\[i\]\['file_name'\]: The file name of the i-th image.
+    - info\['lidar_sweeps'\]\[i\]\['timestamp'\]: Timestamp of the sweep data.
-    - info\['images'\]\[i\]\['id'\]: Sample data token of the i-th image.
+    - info\['lidar_sweeps'\]\[i\]\['sample_data_token'\]: The sweep sample data token.
-    - info\['images'\]\[i\]\['token'\]: Sample token corresponding to this frame.
+  - info\['images'\]: A dict contains six keys corresponding to each camera: `'CAM_FRONT'`, `'CAM_FRONT_RIGHT'`, `'CAM_FRONT_LEFT'`, `'CAM_BACK'`, `'CAM_BACK_LEFT'`, `'CAM_BACK_RIGHT'`. Each dict contains all data information related to  corresponding camera.
-    - info\['images'\]\[i\]\['cam2ego_rotation'\]: The rotation from the camera to ego vehicle. (1x4 list in the quaternion format)
+    - info\['images'\]\['CAM_XXX'\]\['img_path'\]: Filename of image.
-    - info\['images'\]\[i\]\['cam2ego_translation'\]: The translation from the camera to ego vehicle. (1x3 list)
+    - info\['images'\]\['CAM_XXX'\]\['cam2img'\]: The transformation matrix recording the intrinsic parameters when projecting 3D points to each image plane. (3x3 list)
-    - info\['images'\]\[i\]\['ego2global_rotation''\]: The rotation from the ego vehicle to global coordinates. (1x4 list in the quaternion format)
+    - info\['images'\]\['CAM_XXX'\]\['sample_data_token'\]: Sample data token of image.
-    - info\['images'\]\[i\]\['ego2global_translation'\]: The translation from the ego vehicle to global coordinates. (1x3 list)
+    - info\['images'\]\['CAM_XXX'\]\['timestamp'\]: Timestamp of the image.
-    - info\['images'\]\[i\]\['cam_intrinsic'\]: Camera intrinsic matrix. (3x3 list)
+    - info\['images'\]\['CAM_XXX'\]\['cam2ego'\]: The transformation matrix from this camera sensor to ego vehicle. (4x4 list)
-    - info\['images'\]\[i\]\['width'\]: Image width, 1600 by default in nuScenes.
+    - info\['images'\]\['CAM_XXX'\]\['lidar2cam'\]: The transformation matrix from lidar sensor to this camera. (4x4 list)
-    - info\['images'\]\[i\]\['height'\]: Image height, 900 by default in nuScenes.
+  - info\['instances'\]: It is a list of dict. Each dict contains all annotation information of single instance.
-  - info\['annotations'\]: A list containing all the annotation info.
+    - info\['instances'\]\['bbox_3d'\]: List of 7 numbers representing the 3D bounding box of the instance, in (x, y, z, l, w, h, yaw) order.
-    - info\['annotations'\]\[i\]\['file_name'\]: The file name of the corresponding image.
+    - info\['instances'\]\['bbox_label_3d'\]: A int indicate the label of instance and the -1 indicate ignore.
-    - info\['annotations'\]\[i\]\['image_id'\]: The image id (token) of the corresponding image.
+    - info\['instances'\]\['velocity'\]: Velocities of 3D bounding boxes (no vertical measurements due to inaccuracy), a list has shape (2.).
-    - info\['annotations'\]\[i\]\['area'\]: Area of the 2D bounding box.
+    - info\['instances'\]\['num_lidar_pts'\]: Number of lidar points included in each 3D bounding box.
-    - info\['annotations'\]\[i\]\['category_name'\]: Category name.
+    - info\['instances'\]\['num_radar_pts'\]: Number of radar points included in each 3D bounding box.
-    - info\['annotations'\]\[i\]\['category_id'\]: Category id.
+    - info\['instances'\]\['bbox_3d_isvalid'\]: Whether each bounding box is valid. In general, we only take the 3D boxes that include at least one lidar or radar point as valid boxes.
-    - info\['annotations'\]\[i\]\['bbox'\]: 2D bounding box annotation (exterior rectangle of the projected 3D box), 1x4 list following \[x1, y1, x2-x1, y2-y1\].
+  - info\['cam_instances'\]: It is a dict contains keys `'CAM_FRONT'`, `'CAM_FRONT_RIGHT'`, `'CAM_FRONT_LEFT'`, `'CAM_BACK'`, `'CAM_BACK_LEFT'`, `'CAM_BACK_RIGHT'`. For vision-based 3D object detection task, we split 3D annotations of the whole scenes according to the camera they belong to.
-      x1/y1 are minimum coordinates along horizontal/vertical direction of the image.
+    - info\['cam_instances'\]\['CAM_XXX'\]\['bbox_label'\]: Label of instance.
-    - info\['annotations'\]\[i\]\['iscrowd'\]: Whether the region is crowded. Defaults to 0.
+    - info\['cam_instances'\]\['CAM_XXX'\]\['bbox_label_3d'\]: Label of instance.
-    - info\['annotations'\]\[i\]\['bbox_cam3d'\]: 3D bounding box (gravity) center location (3), size (3), (global) yaw angle (1), 1x7 list.
+    - info\['cam_instances'\]\['CAM_XXX'\]\['bbox'\]: 2D bounding box annotation (exterior rectangle of the projected 3D box), a list arrange as \[x1, y1, x2, y2\].
-    - info\['annotations'\]\[i\]\['velo_cam3d'\]: Velocities of 3D bounding boxes (no vertical measurements due to inaccuracy), an Nx2 array.
+    - info\['cam_instances'\]\['CAM_XXX'\]\['center_2d'\]: Projected center location on the image, a list has shape (2,), .
-    - info\['annotations'\]\[i\]\['center2d'\]: Projected 3D-center containing 2.5D information: projected center location on the image (2) and depth (1), 1x3 list.
+    - info\['cam_instances'\]\['CAM_XXX'\]\['depth'\]: The depth of projected center.
-    - info\['annotations'\]\[i\]\['attribute_name'\]: Attribute name.
+    - info\['cam_instances'\]\['CAM_XXX'\]\['velocity'\]: Velocities of 3D bounding boxes (no vertical measurements due to inaccuracy), a list has shape (2,).
-    - info\['annotations'\]\[i\]\['attribute_id'\]: Attribute id.
+    - info\['cam_instances'\]\['CAM_XXX'\]\['attr_label'\]: The attr label of instance. We maintain a default attribute collection and mapping for attribute classification.
-      We maintain a default attribute collection and mapping for attribute classification.
+    - info\['cam_instances'\]\['CAM_XXX'\]\['bbox_3d'\]: List of 7 numbers representing the 3D bounding box of the instance, in (x, y, z, l, h, w, yaw) order.
-      Please refer to [here](https://github.com/open-mmlab/mmdetection3d/blob/master/mmdet3d/datasets/nuscenes_mono_dataset.py#L53) for more details.
-    - info\['annotations'\]\[i\]\['id'\]: Annotation id. Defaults to `i`.
+Note:
-Here we only explain the data recorded in the training info files. The same applies to validation and testing set.
+1. The differences between  `bbox_3d` in `instances`  and that in `cam_instances`.
+   Both `bbox_3d`  have been converted to MMDet3D coordinate system, but `bboxes_3d` in `instances` is in LiDAR coordinate format and `bboxes_3d` in `cam_instances` is in Camera coordinate format. Mind the difference between them in 3D Box representation ('l, w, h' and 'l, h, w').
-The core function to get `nuscenes_infos_xxx.pkl` and `nuscenes_infos_xxx_mono3d.coco.json` are [\_fill_trainval_infos](https://github.com/open-mmlab/mmdetection3d/blob/master/tools/dataset_converters/nuscenes_converter.py#L143) and [get_2d_boxes](https://github.com/open-mmlab/mmdetection3d/blob/master/tools/dataset_converters/nuscenes_converter.py#L397), respectively.
-Please refer to [nuscenes_converter.py](https://github.com/open-mmlab/mmdetection3d/blob/master/tools/dataset_converters/nuscenes_converter.py) for more details.
+2. Here we only explain the data recorded in the training info files. The same applies to validation and testing set (the pkl of test set does not contains `instances` and `cam_instances`).
+The core function to get `nuscenes_infos_xxx.pkl` are  [\_fill_trainval_infos](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/tools/dataset_converters/nuscenes_converter.py#L146) and [get_2d_boxes](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/tools/dataset_converters/nuscenes_converter.py#L397), respectively.
+Please refer to [nuscenes_converter.py](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/tools/dataset_converters/nuscenes_converter.py) for more details.
 ## Training pipeline
@@ -138,11 +118,11 @@ train_pipeline = [
        coord_type='LIDAR',
        load_dim=5,
        use_dim=5,
-        file_client_args=file_client_args),
+        ),
    dict(
        type='LoadPointsFromMultiSweeps',
        sweeps_num=10,
-        file_client_args=file_client_args),
+        ),
    dict(type='LoadAnnotations3D', with_bbox_3d=True, with_label_3d=True),
    dict(
        type='GlobalRotScaleTrans',
@@ -154,8 +134,9 @@ train_pipeline = [
    dict(type='ObjectRangeFilter', point_cloud_range=point_cloud_range),
    dict(type='ObjectNameFilter', classes=class_names),
    dict(type='PointShuffle'),
-    dict(type='DefaultFormatBundle3D', class_names=class_names),
+    dict(
-    dict(type='Collect3D', keys=['points', 'gt_bboxes_3d', 'gt_labels_3d'])
+        type='Pack3DDetInputs',
+        keys=['points', 'gt_bboxes_3d', 'gt_labels_3d'])
 ]
 ```
@@ -179,16 +160,13 @@ train_pipeline = [
        with_bbox_3d=True,
        with_label_3d=True,
        with_bbox_depth=True),
-    dict(type='Resize', img_scale=(1600, 900), keep_ratio=True),
+    dict(type='mmdet.Resize', scale=(1600, 900), keep_ratio=True),
    dict(type='RandomFlip3D', flip_ratio_bev_horizontal=0.5),
-    dict(type='Normalize', **img_norm_cfg),
-    dict(type='Pad', size_divisor=32),
-    dict(type='DefaultFormatBundle3D', class_names=class_names),
    dict(
-        type='Collect3D',
+        type='Pack3DDetInputs',
        keys=[
-            'img', 'gt_bboxes', 'gt_labels', 'attr_labels', 'gt_bboxes_3d',
+            'img', 'gt_bboxes', 'gt_bboxes_labels', 'attr_labels', 'gt_bboxes_3d',
-            'gt_labels_3d', 'centers2d', 'depths'
+            'gt_labels_3d', 'centers_2d', 'depths'
        ]),
 ]
 ```
@@ -205,7 +183,7 @@ It follows the general pipeline of 2D detection while differs in some details:
 An example to evaluate PointPillars with 8 GPUs with nuScenes metrics is as follows.
 ```shell
-bash ./tools/dist_test.sh configs/pointpillars/pointpillars_hv_fpn_sbn-all_8xb4-2x_nus-3d.py checkpoints/hv_pointpillars_fpn_sbn-all_4x8_2x_nus-3d_20200620_230405-2fa62f3d.pth 8 --eval bbox
+bash ./tools/dist_test.sh configs/pointpillars/pointpillars_hv_fpn_sbn-all_8xb4-2x_nus-3d.py checkpoints/hv_pointpillars_fpn_sbn-all_4x8_2x_nus-3d_20200620_230405-2fa62f3d.pth 8
 ```
 ## Metrics
@@ -244,11 +222,13 @@ barrier 0.466   0.581   0.269   0.169   nan     nan
 An example to test PointPillars on nuScenes with 8 GPUs and generate a submission to the leaderboard is as follows.
+You should modify the `jsonfile_prefix` in the `test_evaluator` of corresponding configuration. For example, adding `test_evaluator = dict(type='NuScenesMetric', jsonfile_prefix='work_dirs/pp-nus/results_eval.json')` or using `--cfg-options "test_evaluator.jsonfile_prefix=work_dirs/pp-nus/results_eval.json)` after the test command.
 ```shell
-./tools/dist_test.sh configs/pointpillars/pointpillars_hv_fpn_sbn-all_8xb4-2x_nus-3d.py work_dirs/pp-nus/latest.pth 8 --out work_dirs/pp-nus/results_eval.pkl --format-only --eval-options 'jsonfile_prefix=work_dirs/pp-nus/results_eval'
+./tools/dist_test.sh configs/pointpillars/pointpillars_hv_fpn_sbn-all_8xb4-2x_nus-3d.py work_dirs/pp-nus/latest.pth 8 --cfg-options 'test_evaluator.jsonfile_prefix=work_dirs/pp-nus/results_eval'
 ```
-Note that the testing info should be changed to that for testing set instead of validation set [here](https://github.com/open-mmlab/mmdetection3d/blob/master/configs/_base_/datasets/nus-3d.py#L132).
+Note that the testing info should be changed to that for testing set instead of validation set [here](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/configs/_base_/datasets/nus-3d.py#L132).
 After generating the `work_dirs/pp-nus/results_eval.json`, you can compress it and submit it to nuScenes benchmark. Please refer to the [nuScenes official website](https://www.nuscenes.org/object-detection?externalData=all&mapData=all&modalities=Any) for more information.

--- a/docs/en/advanced_guides/datasets/scannet_det.md
+++ b/docs/en/advanced_guides/datasets/scannet_det.md
@@ -2,7 +2,7 @@
 ## Dataset preparation
-For the overall process, please refer to the [README](https://github.com/open-mmlab/mmdetection3d/blob/master/data/scannet/README.md/) page for ScanNet.
+For the overall process, please refer to the [README](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/data/scannet/README.md/) page for ScanNet.
 ### Export ScanNet point cloud data
@@ -226,21 +226,14 @@ scannet
 - `semantic_mask/xxxxx.bin`: The semantic label for each point, value range: \[1, 40\], i.e. `nyu40id` standard. Note: the `nyu40id` ID will be mapped to train ID in train pipeline `PointSegClassMapping`.
 - `posed_images/scenexxxx_xx`: The set of `.jpg` images with `.txt` 4x4 poses and the single `.txt` file with camera intrinsic matrix.
 - `scannet_infos_train.pkl`: The train data infos, the detailed info of each scan is as follows:
-  - info\['point_cloud'\]: {'num_features': 6, 'lidar_idx': sample_idx}.
+  - info\['lidar_points'\]\['lidar_path'\]: The filename of `xxx.bin` of lidar points.
-  - info\['pts_path'\]: The path of `points/xxxxx.bin`.
+  - info\['lidar_points'\]\['num_pts_feats'\]: The feature dimension of point.
-  - info\['pts_instance_mask_path'\]: The path of `instance_mask/xxxxx.bin`.
+  - info\['lidar_points'\]\['pts_semantic_mask_path'\]: The filename of `xxx.bin` contains semantic mask annotation.
-  - info\['pts_semantic_mask_path'\]: The path of `semantic_mask/xxxxx.bin`.
+  - info\['lidar_points'\]\['pts_instance_mask_path'\]: The filename of `xxx.bin` contains semantic mask annotation.
-  - info\['annos'\]: The annotations of each scan.
+  - info\['lidar_points'\]\['axis_align_matrix'\]: The transformation matrix to align the axis.
-    - annotations\['gt_num'\]: The number of ground truths.
+  - info\['instances'\]: A list of dict contains all annotations, each dict contains all annotation information of single instance.
-    - annotations\['name'\]： The semantic name of all ground truths, e.g. `chair`.
+    - info\['instances'\]\[i\]\['bbox_3d'\]: List of 6 numbers representing the axis-aligned 3D bounding box of the instance in depth coordinate system, in (x, y, z, l, w, h) order.
-    - annotations\['location'\]: The gravity center of the axis-aligned 3D bounding boxes in depth coordinate system. Shape: \[K, 3\], K is the number of ground truths.
+    - info\['instances'\]\[i\]\['bbox_label_3d'\]: The label of each 3d bounding boxes.
-    - annotations\['dimensions'\]: The dimensions of the axis-aligned 3D bounding boxes in depth coordinate system, i.e. (x_size, y_size, z_size), shape: \[K, 3\].
-    - annotations\['gt_boxes_upright_depth'\]: The axis-aligned 3D bounding boxes in depth coordinate system, each bounding box is (x, y, z, x_size, y_size, z_size), shape: \[K, 6\].
-    - annotations\['unaligned_location'\]: The gravity center of the axis-unaligned 3D bounding boxes in depth coordinate system.
-    - annotations\['unaligned_dimensions'\]: The dimensions of the axis-unaligned 3D bounding boxes in depth coordinate system.
-    - annotations\['unaligned_gt_boxes_upright_depth'\]: The axis-unaligned 3D bounding boxes in depth coordinate system.
-    - annotations\['index'\]: The index of all ground truths, i.e. \[0, K).
-    - annotations\['class'\]: The train class ID of the bounding boxes, value range: \[0, 18), shape: \[K, \].
 - `scannet_infos_val.pkl`: The val data infos, which shares the same format as `scannet_infos_train.pkl`.
 - `scannet_infos_test.pkl`: The test data infos, which almost shares the same format as `scannet_infos_train.pkl` except for the lack of annotation.
@@ -264,10 +257,7 @@ train_pipeline = [
        with_seg_3d=True),
    dict(type='GlobalAlignment', rotation_axis=2),
    dict(
-        type='PointSegClassMapping',
+        type='PointSegClassMapping'),
-        valid_cat_ids=(3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 16, 24, 28, 33, 34,
-                       36, 39),
-        max_cat_id=40),
    dict(type='PointSample', num_points=40000),
    dict(
        type='RandomFlip3D',
@@ -279,9 +269,8 @@ train_pipeline = [
        rot_range=[-0.087266, 0.087266],
        scale_ratio_range=[1.0, 1.0],
        shift_height=True),
-    dict(type='DefaultFormatBundle3D', class_names=class_names),
    dict(
-        type='Collect3D',
+        type='Pack3DDetInputs',
        keys=[
            'points', 'gt_bboxes_3d', 'gt_labels_3d', 'pts_semantic_mask',
            'pts_instance_mask'
@@ -298,6 +287,6 @@ train_pipeline = [
 ## Metrics
-Typically mean Average Precision (mAP) is used for evaluation on ScanNet, e.g. `mAP@0.25` and `mAP@0.5`. In detail, a generic function to compute precision and recall for 3D object detection for multiple classes is called, please refer to [indoor_eval](https://github.com/open-mmlab/mmdetection3d/blob/master/mmdet3D/core/evaluation/indoor_eval.py).
+Typically mean Average Precision (mAP) is used for evaluation on ScanNet, e.g. `mAP@0.25` and `mAP@0.5`. In detail, a generic function to compute precision and recall for 3D object detection for multiple classes is called, please refer to [indoor_eval](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/mmdet3d/evaluation/functional/indoor_eval.py).
 As introduced in section `Export ScanNet data`, all ground truth 3D bounding box are axis-aligned, i.e. the yaw is zero. So the yaw target of network predicted 3D bounding box is also zero and axis-aligned 3D Non-Maximum Suppression (NMS), which is regardless of rotation, is adopted during post-processing .
--- a/docs/en/advanced_guides/datasets/scannet_sem_seg.md
+++ b/docs/en/advanced_guides/datasets/scannet_sem_seg.md
@@ -90,10 +90,7 @@ train_pipeline = [
        with_mask_3d=False,
        with_seg_3d=True),
    dict(
-        type='PointSegClassMapping',
+        type='PointSegClassMapping'),
-        valid_cat_ids=(1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 16, 24, 28,
-                       33, 34, 36, 39),
-        max_cat_id=40),
    dict(
        type='IndoorPatchPointSample',
        num_points=num_points,

--- a/docs/en/advanced_guides/datasets/waymo_det.md
+++ b/docs/en/advanced_guides/datasets/waymo_det.md
@@ -122,19 +122,16 @@ sudo apt install build-essential
 bazel clean
 bazel build waymo_open_dataset/metrics/tools/compute_detection_metrics_main
-cp bazel-bin/waymo_open_dataset/metrics/tools/compute_detection_metrics_main ../mmdetection3d/mmdet3d/core/evaluation/waymo_utils/
+cp bazel-bin/waymo_open_dataset/metrics/tools/compute_detection_metrics_main ../mmdetection3d/mmdet3d/evaluation/functional/waymo_utils/
 ```
 Then you can evaluate your models on Waymo. An example to evaluate PointPillars on Waymo with 8 GPUs with Waymo metrics is as follows.
 ```shell
-./tools/slurm_test.sh ${PARTITION} ${JOB_NAME} configs/pointpillars/hv_pointpillars_secfpn_sbn-2x16_2x_waymo-3d-car.py \
+./tools/dist_test.sh configs/pointpillars/pointpillars_hv_secfpn_sbn-all_16xb2-2x_waymo-3d-car.py checkpoints/hv_pointpillars_secfpn_sbn-2x16_2x_waymo-3d-car_latest.pth
-    checkpoints/hv_pointpillars_secfpn_sbn-2x16_2x_waymo-3d-car_latest.pth --out results/waymo-car/results_eval.pkl \
-    --eval waymo --eval-options 'pklfile_prefix=results/waymo-car/kitti_results' \
-    'submission_prefix=results/waymo-car/kitti_results'
 ```
-`pklfile_prefix` should be given in the `--eval-options` if the bin file is needed to be generated. For metrics, `waymo` is the recommended official evaluation prototype. Currently, evaluating with choice `kitti` is adapted from KITTI and the results for each difficulty are not exactly the same as the definition of KITTI. Instead, most of objects are marked with difficulty 0 currently, which will be fixed in the future. The reasons of its instability include the large computation for evaluation, the lack of occlusion and truncation in the converted data, different definitions of difficulty and different methods of computing Average Precision.
+`pklfile_prefix` should be set in `test_evaluator` of configuration if the bin file is needed to be generated, so you can add `--cfg-options "test_evaluator.pklfile_prefix=xxxx"` in the end of command if you want do it.
 **Notice**:
@@ -148,12 +145,7 @@ Then you can evaluate your models on Waymo. An example to evaluate PointPillars
 An example to test PointPillars on Waymo with 8 GPUs, generate the bin files and make a submission to the leaderboard.
-```shell
+`submission_prefix` should be set in `test_evaluator` of configuration before you run the test command  if you want to generate the bin files and make a submission to the leaderboard..
-./tools/slurm_test.sh ${PARTITION} ${JOB_NAME} configs/pointpillars/hv_pointpillars_secfpn_sbn-2x16_2x_waymo-3d-car.py \
-    checkpoints/hv_pointpillars_secfpn_sbn-2x16_2x_waymo-3d-car_latest.pth --out results/waymo-car/results_eval.pkl \
-    --format-only --eval-options 'pklfile_prefix=results/waymo-car/kitti_results' \
-    'submission_prefix=results/waymo-car/kitti_results'
-```
 After generating the bin file, you can simply build the binary file `create_submission` and use them to create a submission file by following the [instruction](https://github.com/waymo-research/waymo-open-dataset/blob/master/docs/quick_start.md/). Basically, here are some example commands.
@@ -162,11 +154,11 @@ cd ../waymo-od/
 bazel build waymo_open_dataset/metrics/tools/create_submission
 cp bazel-bin/waymo_open_dataset/metrics/tools/create_submission ../mmdetection3d/mmdet3d/core/evaluation/waymo_utils/
 vim waymo_open_dataset/metrics/tools/submission.txtpb  # set the metadata information
-cp waymo_open_dataset/metrics/tools/submission.txtpb ../mmdetection3d/mmdet3d/core/evaluation/waymo_utils/
+cp waymo_open_dataset/metrics/tools/submission.txtpb ../mmdetection3d/mmdet3d/evaluation/functional/waymo_utils/
 cd ../mmdetection3d
 # suppose the result bin is in `results/waymo-car/submission`
-mmdet3d/core/evaluation/waymo_utils/create_submission  --input_filenames='results/waymo-car/kitti_results_test.bin' --output_filename='results/waymo-car/submission/model' --submission_filename='mmdet3d/core/evaluation/waymo_utils/submission.txtpb'
+mmdet3d/core/evaluation/waymo_utils/create_submission  --input_filenames='results/waymo-car/kitti_results_test.bin' --output_filename='results/waymo-car/submission/model' --submission_filename='mmdet3d/evaluation/functional/waymo_utils/submission.txtpb'
 tar cvf results/waymo-car/submission/my_model.tar results/waymo-car/submission/my_model/
 gzip results/waymo-car/submission/my_model.tar

--- a/docs/en/advanced_guides/supported_tasks/lidar_det3d.md
+++ b/docs/en/advanced_guides/supported_tasks/lidar_det3d.md
@@ -31,11 +31,13 @@ mmdetection3d
 │   │   │   ├── calib
 │   │   │   ├── image_2
 │   │   │   ├── velodyne
+│   │   │   ├── velodyne_reduced
 │   │   ├── training
 │   │   │   ├── calib
 │   │   │   ├── image_2
 │   │   │   ├── label_2
 │   │   │   ├── velodyne
+│   │   │   ├── velodyne_reduced
 │   │   ├── kitti_gt_database
 │   │   ├── kitti_infos_train.pkl
 │   │   ├── kitti_infos_trainval.pkl
@@ -47,20 +49,21 @@ mmdetection3d
 ## Training
 Then let us train a model with provided configs for PointPillars.
-You can basically follow this [tutorial](https://mmdetection3d.readthedocs.io/en/latest/1_exist_data_model.html#inference-with-existing-models) for sample scripts when training with different GPU settings.
+You can basically follow this [tutorial](https://mmdetection3d.readthedocs.io/en/dev-1.x/user_guides/1_exist_data_model.html) for sample scripts when training with different GPU settings.
 Suppose we use 8 GPUs on a single machine with distributed training:
 ```
-./tools/dist_train.sh configs/pointpillars/hv_pointpillars_secfpn_6x8_160e_kitti-3d-3class.py 8
+./tools/dist_train.sh configs/pointpillars/pointpillars_hv-secfpn_8xb6-160e_kitti-3d-3class.py 8
 ```
-Note that `6x8` in the config name refers to the training is completed with 8 GPUs and 6 samples on each GPU.
+Note that `8xb6` in the config name refers to the training is completed with 8 GPUs and 6 samples on each GPU.
 If your customized setting is different from this, sometimes you need to adjust the learning rate accordingly.
-A basic rule can be referred to [here](https://arxiv.org/abs/1706.02677).
+A basic rule can be referred to [here](https://arxiv.org/abs/1706.02677). We have supported `--auto-scale-lr` to
+enable automatically scaling LR
 ## Quantitative Evaluation
-During training, the model checkpoints will be evaluated regularly according to the setting of `evaluation = dict(interval=xxx)` in the config.
+During training, the model checkpoints will be evaluated regularly according to the setting of `train_cfg = dict(val_interval=xxx)` in the config.
 We support official evaluation protocols for different datasets.
 For KITTI, the model will be evaluated with mean average precision (mAP) with Intersection over Union (IoU) thresholds 0.5/0.7 for 3 categories respectively.
 The evaluation results will be printed in the command like:
@@ -81,22 +84,22 @@ aos AP:97.70, 88.73, 87.34
 In addition, you can also evaluate a specific model checkpoint after training is finished. Simply run scripts like the following:
 ```
-./tools/dist_test.sh configs/pointpillars/hv_pointpillars_secfpn_6x8_160e_kitti-3d-3class.py \
+./tools/dist_test.sh configs/pointpillars/pointpillars_hv-secfpn_8xb6-160e_kitti-3d-3class.py work_dirs/pointpillars/latest.pth 8
-    work_dirs/pointpillars/latest.pth --eval mAP
 ```
 ## Testing and Making a Submission
 If you would like to only conduct inference or test the model performance on the online benchmark,
-you just need to replace the `--eval mAP` with `--format-only` in the previous evaluation script and specify the `pklfile_prefix` and `submission_prefix` if necessary,
+you just need to specify the `submission_prefix` for corresponding evaluator,
-e.g., adding an option `--eval-options submission_prefix=work_dirs/pointpillars/test_submission`.
+e.g., add `test_evaluator = dict(type='KittiMetric', submission_prefix=work_dirs/pointpillars/test_submission)` in the configuration then you can get the results file or you can just add
-Please guarantee the [info for testing](https://github.com/open-mmlab/mmdetection3d/blob/master/configs/_base_/datasets/kitti-3d-3class.py#L131) in the config corresponds to the test set instead of validation set.
+`--cfg-options "test_evaluator.submission_prefix=work_dirs/pointpillars/test_submission` in the end of test command.
+Please guarantee the `data_prefix` and `ann_file` in [info for testing](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/configs/_base_/datasets/kitti-3d-3class.py#L113) in the config corresponds to the test set instead of validation set.
 After generating the results, you can basically compress the folder and upload to the KITTI evaluation server.
 ## Qualitative Validation
 MMDetection3D also provides versatile tools for visualization such that we can have an intuitive feeling of the detection results predicted by our trained models.
-You can either set the `--eval-options 'show=True' 'out_dir=${SHOW_DIR}'` option to visualize the detection results online during evaluation,
+You can either set the `--show'` option to visualize the detection results online during evaluation,
 or using `tools/misc/visualize_results.py` for offline visualization.
 Besides, we also provide scripts `tools/misc/browse_dataset.py` to visualize the dataset without inference.
-Please refer more details in the [doc for visualization](https://mmdetection3d.readthedocs.io/en/latest/useful_tools.html#visualization).
+Please refer more details in the [doc for visualization](https://mmdetection3d.readthedocs.io/en/dev-1.x/useful_tools.html#visualization).
--- a/docs/en/advanced_guides/supported_tasks/lidar_sem_seg3d.md
+++ b/docs/en/advanced_guides/supported_tasks/lidar_sem_seg3d.md
@@ -49,16 +49,16 @@ You can basically follow this [tutorial](https://mmdetection3d.readthedocs.io/en
 Suppose we use 2 GPUs on a single machine with distributed training:
 ```
-./tools/dist_train.sh configs/pointnet2/pointnet2_ssg_16x2_cosine_200e_scannet_seg-3d-20class.py 2
+./tools/dist_train.sh configs/pointnet2/pointnet2_ssg_2xb16-cosine-200e_scannet-seg.py 2
 ```
-Note that `16x2` in the config name refers to the training is completed with 2 GPUs and 16 samples on each GPU.
+Note that `2xb16` in the config name refers to the training is completed with 2 GPUs and 16 samples on each GPU.
 If your customized setting is different from this, sometimes you need to adjust the learning rate accordingly.
 A basic rule can be referred to [here](https://arxiv.org/abs/1706.02677).
 ## Quantitative Evaluation
-During training, the model checkpoints will be evaluated regularly according to the setting of `evaluation = dict(interval=xxx)` in the config.
+During training, the model checkpoints will be evaluated regularly according to the setting of `train_cfg = dict(val_interval=xxx)` in the config.
 We support official evaluation protocols for different datasets.
 For ScanNet, the model will be evaluated with mean Intersection over Union (mIoU) over all 20 categories.
 The evaluation results will be printed in the command like:
@@ -74,21 +74,20 @@ The evaluation results will be printed in the command like:
 In addition, you can also evaluate a specific model checkpoint after training is finished. Simply run scripts like the following:
 ```
-./tools/dist_test.sh configs/pointnet2/pointnet2_ssg_16x2_cosine_200e_scannet_seg-3d-20class.py \
+./tools/dist_test.sh configs/pointnet2/pointnet2_ssg_16x2_cosine_200e_scannet_seg-3d-20class.py work_dirs/pointnet2_ssg/latest.pth 8
-    work_dirs/pointnet2_ssg/latest.pth --eval mIoU
 ```
 ## Testing and Making a Submission
 If you would like to only conduct inference or test the model performance on the online benchmark,
-you need to replace the `--eval mIoU` with `--format-only` in the previous evaluation script and change `ann_file=data_root + 'scannet_infos_val.pkl'` to `ann_file=data_root + 'scannet_infos_test.pkl'` in the ScanNet dataset's [config](https://github.com/open-mmlab/mmdetection3d/blob/master/configs/_base_/datasets/scannet_seg-3d-20class.py#L126). Remember to specify the `txt_prefix` as the directory to save the testing results,
+you should change `ann_file='scannet_infos_val.pkl'` to `ann_file='scannet_infos_test.pkl'` in the
-e.g., adding an option `--eval-options txt_prefix=work_dirs/pointnet2_ssg/test_submission`.
+ScanNet dataset's [config](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/configs/_base_/datasets/scannet_seg-3d-20class.py#L129). Remember to
+specify the `submission_prefix` in the `test_evaluator`,
+e.g., adding `test_evaluator = dict(type='SegMetric', submission_prefix=work_dirs/pointnet2_ssg/test_submission`) or just add `--cfg-options test_evaluator.submission_prefix=work_dirs/pointnet2_ssg/test_submission` in the end of command.
 After generating the results, you can basically compress the folder and upload to the [ScanNet evaluation server](http://kaldir.vc.in.tum.de/scannet_benchmark/semantic_label_3d).
 ## Qualitative Validation
-MMDetection3D also provides versatile tools for visualization such that we can have an intuitive feeling of the segmentation results predicted by our trained models.
+MMDetection3D also provides versatile tools for visualization such that you can use `tools/misc/visualize_results.py` with results pkl file for offline visualization of add `--show` in the end of test command to do the online visualization.
-You can either set the `--eval-options 'show=True' 'out_dir=${SHOW_DIR}'` option to visualize the segmentation results online during evaluation,
-or using `tools/misc/visualize_results.py` for offline visualization.
 Besides, we also provide scripts `tools/misc/browse_dataset.py` to visualize the dataset without inference.
 Please refer more details in the [doc for visualization](https://mmdetection3d.readthedocs.io/en/latest/useful_tools.html#visualization).
--- a/docs/en/user_guides/1_exist_data_model.md
+++ b/docs/en/user_guides/1_exist_data_model.md
-# Inference and train
+# Inference and train with existing models and standard datasets
 ## Inference with existing models
 Here we provide testing scripts to evaluate a whole dataset (SUNRGBD, ScanNet, KITTI, etc.).
-For high-level apis easier to integrated into other projects and basic demos, please refer to Verification/Demo under [Get Started](https://mmdetection3d.readthedocs.io/en/latest/getting_started.html).
+For high-level apis easier to integrated into other projects and basic demos, please refer to Verification/Demo under [Get Started](https://mmdetection3d.readthedocs.io/en/dev-1.x/inference.html).
 ### Test existing models on standard datasets
@@ -17,14 +17,14 @@ You can use the following commands to test a dataset.
 ```shell
 # single-gpu testing
-python tools/test.py ${CONFIG_FILE} ${CHECKPOINT_FILE} [--out ${RESULT_FILE}] [--eval ${EVAL_METRICS}] [--show] [--show-dir ${SHOW_DIR}]
+python tools/test.py ${CONFIG_FILE} ${CHECKPOINT_FILE} [--cfg-options test_evaluator.pklfile_prefix=${RESULT_FILE}]  [--show] [--show-dir ${SHOW_DIR}]
 # CPU: disable GPUs and run single-gpu testing script (experimental)
 export CUDA_VISIBLE_DEVICES=-1
-python tools/test.py ${CONFIG_FILE} ${CHECKPOINT_FILE} [--out ${RESULT_FILE}] [--eval ${EVAL_METRICS}] [--show] [--show-dir ${SHOW_DIR}]
+python tools/test.py ${CONFIG_FILE} ${CHECKPOINT_FILE} [--cfg-options test_evaluator.pklfile_prefix=${RESULT_FILE}]  [--show] [--show-dir ${SHOW_DIR}]
 # multi-gpu testing
-./tools/dist_test.sh ${CONFIG_FILE} ${CHECKPOINT_FILE} ${GPU_NUM} [--out ${RESULT_FILE}] [--eval ${EVAL_METRICS}]
+./tools/dist_test.sh ${CONFIG_FILE} ${CHECKPOINT_FILE} ${GPU_NUM} [--cfg-options test_evaluator.pklfile_prefix=${RESULT_FILE}]  [--show] [--show-dir ${SHOW_DIR}]
 ```
 **Note**:
@@ -33,11 +33,19 @@ For now, CPU testing is only supported for SMOKE.
 Optional arguments:
- `RESULT_FILE`: Filename of the output results in pickle format. If not specified, the results will not be saved to a file.
- `EVAL_METRICS`: Items to be evaluated on the results. Allowed values depend on the dataset. Typically we default to use official metrics for evaluation on different datasets, so it can be simply set to `mAP` as a placeholder for detection tasks, which applies to nuScenes, Lyft, ScanNet and SUNRGBD. For KITTI, if we only want to evaluate the 2D detection performance, we can simply set the metric to `img_bbox` (unstable, stay tuned). For Waymo, we provide both KITTI-style evaluation (unstable) and Waymo-style official protocol, corresponding to metric `kitti` and `waymo` respectively. We recommend to use the default official metric for stable performance and fair comparison with other methods. Similarly, the metric can be set to `mIoU` for segmentation tasks, which applies to S3DIS and ScanNet.
 - `--show`: If specified, detection results will be plotted in the silient mode. It is only applicable to single GPU testing and used for debugging and visualization. This should be used with `--show-dir`.
 - `--show-dir`: If specified, detection results will be plotted on the `***_points.obj` and `***_pred.obj` files in the specified directory. It is only applicable to single GPU testing and used for debugging and visualization. You do NOT need a GUI available in your environment for using this option.
+All evaluation related arguments are set in the `test_evaluator` in corresponding dataset configuration. such as
+`test_evaluator = dict(type='KittiMetric', ann_file=data_root + 'kitti_infos_val.pkl', pklfile_prefix=None, submission_prefix=None)`
+The arguments:
+- `type`: The name of the corresponding metric, usually associated with the dataset.
+- `ann_file`: The path of annotation file.
+- `pklfile_prefix`: An optional argument. The filename of the output results in pickle format. If not specified, the results will not be saved to a file.
+- `submission_prefix`: An optional argument. The the results will not be saved to a file then you can upload it to do the official evaluation.
 Examples:
 Assume that you have already downloaded the checkpoints to the directory `checkpoints/`.
@@ -55,8 +63,7 @@ Assume that you have already downloaded the checkpoints to the directory `checkp
   ```shell
   python tools/test.py configs/votenet/votenet_8xb8_scannet-3d.py \
       checkpoints/votenet_8x8_scannet-3d-18class_20200620_230238-2cea9c3a.pth \
-       --eval mAP
+       --show --show-dir ./data/scannet/show_results
-       --eval-options 'show=True' 'out_dir=./data/scannet/show_results'
   ```
 3. Test VoteNet on ScanNet (without saving the test results) and evaluate the mAP.
@@ -64,23 +71,21 @@ Assume that you have already downloaded the checkpoints to the directory `checkp
   ```shell
   python tools/test.py configs/votenet/votenet_8xb8_scannet-3d.py \
       checkpoints/votenet_8x8_scannet-3d-18class_20200620_230238-2cea9c3a.pth \
-       --eval mAP
   ```
 4. Test SECOND on KITTI with 8 GPUs, and evaluate the mAP.
   ```shell
-   ./tools/slurm_test.sh ${PARTITION} ${JOB_NAME} configs/second/hv_second_secfpn_6x8_80e_kitti-3d-3class.py \
+   ./tools/slurm_test.sh ${PARTITION} ${JOB_NAME} configs/second/second_hv_secfpn_8xb6-80e_kitti-3d-3class.py \
-       checkpoints/hv_second_secfpn_6x8_80e_kitti-3d-3class_20200620_230238-9208083a.pth \
+       checkpoints/hv_second_secfpn_6x8_80e_kitti-3d-3class_20200620_230238-9208083a.pth
-       --out results.pkl --eval mAP
   ```
 5. Test PointPillars on nuScenes with 8 GPUs, and generate the json file to be submit to the official evaluation server.
   ```shell
-   ./tools/slurm_test.sh ${PARTITION} ${JOB_NAME} configs/pointpillars/pointpillars_hv_fpn_sbn-all_8xb4-2x_nus-3d.py \
+   ./tools/slurm_test.sh ${PARTITION} ${JOB_NAME} configs/pointpillars/pointpillars_hv_secfpn_sbn-all_8xb4-2x_nus-3d.py \
       checkpoints/hv_pointpillars_fpn_sbn-all_4x8_2x_nus-3d_20200620_230405-2fa62f3d.pth \
-       --format-only --eval-options 'jsonfile_prefix=./pointpillars_nuscenes_results'
+      --cfg-options 'test_evaluator.jsonfile_prefix=./pointpillars_nuscenes_results'
   ```
   The generated results be under `./pointpillars_nuscenes_results` directory.
@@ -88,9 +93,9 @@ Assume that you have already downloaded the checkpoints to the directory `checkp
 6. Test SECOND on KITTI with 8 GPUs, and generate the pkl files and submission data to be submit to the official evaluation server.
   ```shell
-   ./tools/slurm_test.sh ${PARTITION} ${JOB_NAME} configs/second/hv_second_secfpn_6x8_80e_kitti-3d-3class.py \
+   ./tools/slurm_test.sh ${PARTITION} ${JOB_NAME} configs/second/second_hv_secfpn_8xb6-80e_kitti-3d-3class.py \
       checkpoints/hv_second_secfpn_6x8_80e_kitti-3d-3class_20200620_230238-9208083a.pth \
-       --format-only --eval-options 'pklfile_prefix=./second_kitti_results' 'submission_prefix=./second_kitti_results'
+       --cfg-options 'test_evaluator.pklfile_prefix=./second_kitti_results' 'submission_prefix=./second_kitti_results'
   ```
   The generated results be under `./second_kitti_results` directory.
@@ -99,22 +104,23 @@ Assume that you have already downloaded the checkpoints to the directory `checkp
   ```shell
   ./tools/slurm_test.sh ${PARTITION} ${JOB_NAME} configs/pointpillars/hv_pointpillars_fpn_sbn-2x8_2x_lyft-3d.py \
-       checkpoints/hv_pointpillars_fpn_sbn-2x8_2x_lyft-3d_latest.pth --out results/pp_lyft/results_challenge.pkl \
+       checkpoints/hv_pointpillars_fpn_sbn-2x8_2x_lyft-3d_latest.pth \
-       --format-only --eval-options 'jsonfile_prefix=results/pp_lyft/results_challenge' \
+       --cfg-options 'test_evaluator.jsonfile_prefix=results/pp_lyft/results_challenge' \
-       'csv_savepath=results/pp_lyft/results_challenge.csv'
+       'test_evaluator.csv_savepath=results/pp_lyft/results_challenge.csv' \
+       'test_evaluator.pklfile_prefix=results/pp_lyft/results_challenge.pkl'
   ```
-   **Notice**: To generate submissions on Lyft, `csv_savepath` must be given in the `--eval-options`. After generating the csv file, you can make a submission with kaggle commands given on the [website](https://www.kaggle.com/c/3d-object-detection-for-autonomous-vehicles/submit).
+   **Notice**: To generate submissions on Lyft, `csv_savepath` must be given in the `--cfg-options`. After generating the csv file, you can make a submission with kaggle commands given on the [website](https://www.kaggle.com/c/3d-object-detection-for-autonomous-vehicles/submit).
-   Note that in the [config of Lyft dataset](../../configs/_base_/datasets/lyft-3d.py), the value of `ann_file` keyword in `test` is `data_root + 'lyft_infos_test.pkl'`, which is the official test set of Lyft without annotation. To test on the validation set, please change this to `data_root + 'lyft_infos_val.pkl'`.
+   Note that in the [config of Lyft dataset](../../configs/_base_/datasets/lyft-3d.py), the value of `ann_file` keyword in `test` is `'lyft_infos_test.pkl'`, which is the official test set of Lyft without annotation. To test on the validation set, please change this to `'lyft_infos_val.pkl'`.
 8. Test PointPillars on waymo with 8 GPUs, and evaluate the mAP with waymo metrics.
   ```shell
-   ./tools/slurm_test.sh ${PARTITION} ${JOB_NAME} configs/pointpillars/hv_pointpillars_secfpn_sbn-2x16_2x_waymo-3d-car.py \
+   ./tools/slurm_test.sh ${PARTITION} ${JOB_NAME} configs/pointpillars/pointpillars_hv_secfpn_sbn-all_16xb2-2x_waymo-3d-car.py  \
-       checkpoints/hv_pointpillars_secfpn_sbn-2x16_2x_waymo-3d-car_latest.pth --out results/waymo-car/results_eval.pkl \
+       checkpoints/hv_pointpillars_secfpn_sbn-2x16_2x_waymo-3d-car_latest.pth \
-       --eval waymo --eval-options 'pklfile_prefix=results/waymo-car/kitti_results' \
+       --cfg-options 'test_evaluator.pklfile_prefix=results/waymo-car/kitti_results' \
-       'submission_prefix=results/waymo-car/kitti_results'
+       'test_evaluator.submission_prefix=results/waymo-car/kitti_results'
   ```
   **Notice**: For evaluation on waymo, please follow the [instruction](https://github.com/waymo-research/waymo-open-dataset/blob/master/docs/quick_start.md/) to build the binary file `compute_detection_metrics_main` for metrics computation and put it into `mmdet3d/core/evaluation/waymo_utils/`.(Sometimes when using bazel to build `compute_detection_metrics_main`, an error `'round' is not a member of 'std'` may appear. We just need to remove the `std::` before `round` in that file.) `pklfile_prefix` should be given in the `--eval-options` for the bin file generation. For metrics, `waymo` is the recommended official evaluation prototype. Currently, evaluating with choice `kitti` is adapted from KITTI and the results for each difficulty are not exactly the same as the definition of KITTI. Instead, most of objects are marked with difficulty 0 currently, which will be fixed in the future. The reasons of its instability include the large computation for evaluation, the lack of occlusion and truncation in the converted data, different definition of difficulty and different methods of computing average precision.
@@ -122,10 +128,10 @@ Assume that you have already downloaded the checkpoints to the directory `checkp
 9. Test PointPillars on waymo with 8 GPUs, generate the bin files and make a submission to the leaderboard.
   ```shell
-   ./tools/slurm_test.sh ${PARTITION} ${JOB_NAME} configs/pointpillars/hv_pointpillars_secfpn_sbn-2x16_2x_waymo-3d-car.py \
+   ./tools/slurm_test.sh ${PARTITION} ${JOB_NAME} configs/pointpillars/pointpillars_hv_secfpn_sbn-all_16xb2-2x_waymo-3d-car.py  \
-       checkpoints/hv_pointpillars_secfpn_sbn-2x16_2x_waymo-3d-car_latest.pth --out results/waymo-car/results_eval.pkl \
+       checkpoints/hv_pointpillars_secfpn_sbn-2x16_2x_waymo-3d-car_latest.pth \
-       --format-only --eval-options 'pklfile_prefix=results/waymo-car/kitti_results' \
+       --cfg-options 'test_evaluator.pklfile_prefix=results/waymo-car/kitti_results' \
-       'submission_prefix=results/waymo-car/kitti_results'
+       'test_evaluator.submission_prefix=results/waymo-car/kitti_results'
   ```
   **Notice**: After generating the bin file, you can simply build the binary file `create_submission` and use them to create a submission file by following the [instruction](https://github.com/waymo-research/waymo-open-dataset/blob/master/docs/quick_start.md/). For evaluation on the validation set with the eval server, you can also use the same way to generate a submission.
@@ -141,10 +147,10 @@ which is specified by `work_dir` in the config file.
 By default we evaluate the model on the validation set after each epoch, you can change the evaluation interval by adding the interval argument in the training config.
 ```python
-evaluation = dict(interval=12)  # This evaluate the model per 12 epoch.
+train_cfg = dict(type='EpochBasedTrainLoop', val_interval=1)  # This evaluate the model per 12 epoch.
 ```
-**Important**: The default learning rate in config files is for 8 GPUs and the exact batch size is marked by the config's file name, e.g. '2x8' means 2 samples per GPU using 8 GPUs.
+**Important**: The default learning rate in config files is for 8 GPUs and the exact batch size is marked by the config's file name, e.g. '2xb8' means 2 samples per GPU using 8 GPUs.
 According to the [Linear Scaling Rule](https://arxiv.org/abs/1706.02677), you need to set the learning rate proportional to the batch size if you use different GPUs or images per GPU, e.g., lr=0.01 for 4 GPUs * 2 img/gpu and lr=0.08 for 16 GPUs * 4 img/gpu. However, since most of the models in this repo use ADAM rather than SGD for optimization, the rule may not hold and users need to tune the learning rate by themselves.
 ### Train with a single GPU
@@ -177,15 +183,7 @@ For now, most of the point cloud related algorithms rely on 3D CUDA op, which ca
 Optional arguments are:
- `--no-validate` (**not suggested**): By default, the codebase will perform evaluation at every k (default value is 1, which can be modified like [this](https://github.com/open-mmlab/mmdetection3d/blob/master/configs/fcos3d/fcos3d_r101_caffe_fpn_gn-head_dcn_2x8_1x_nus-mono3d.py#L75)) epochs during the training. To disable this behavior, use `--no-validate`.
+- `--cfg-options 'Key=value'`: Override some settings in the used config.
- `--work-dir ${WORK_DIR}`: Override the working directory specified in the config file.
- `--resume-from ${CHECKPOINT_FILE}`: Resume from a previous checkpoint file.
- `--options 'Key=value'`: Override some settings in the used config.
-Difference between `resume-from` and `load-from`:
- `resume-from` loads both the model weights and optimizer status, and the epoch is also inherited from the specified checkpoint. It is usually used for resuming the training process that is interrupted accidentally.
- `load-from` only loads the model weights and the training epoch starts from 0. It is usually used for finetuning.
 ### Train with multiple machines
@@ -198,7 +196,7 @@ If you run MMDetection3D on a cluster managed with [slurm](https://slurm.schedmd
 Here is an example of using 16 GPUs to train Mask R-CNN on the dev partition.
 ```shell
-GPUS=16 ./tools/slurm_train.sh dev pp_kitti_3class pointpillars_hv_secfpn_8xb6-160e_kitti-3d-3class.py /nfs/xxxx/pp_kitti_3class
+GPUS=16 ./tools/slurm_train.sh dev pp_kitti_3class configs/pointpillars/pointpillars_hv_secfpn_8xb6-160e_kitti-3d-3class.py /nfs/xxxx/pp_kitti_3class
 ```
 You can check [slurm_train.sh](https://github.com/open-mmlab/mmdetection/blob/master/tools/slurm_train.sh) for full arguments and environment variables.
@@ -233,11 +231,11 @@ CUDA_VISIBLE_DEVICES=4,5,6,7 PORT=29501 ./tools/dist_train.sh ${CONFIG_FILE} 4
 If you use launch training jobs with Slurm, there are two ways to specify the ports.
-1. Set the port through `--options`. This is more recommended since it does not change the original configs.
+1. Set the port through `--cfg-options`. This is more recommended since it does not change the original configs.
   ```shell
-   CUDA_VISIBLE_DEVICES=0,1,2,3 GPUS=4 ./tools/slurm_train.sh ${PARTITION} ${JOB_NAME} config1.py ${WORK_DIR} --options 'dist_params.port=29500'
+   CUDA_VISIBLE_DEVICES=0,1,2,3 GPUS=4 ./tools/slurm_train.sh ${PARTITION} ${JOB_NAME} config1.py ${WORK_DIR} --cfg-options 'env_cfg.dist_cfg.port=29500'
-   CUDA_VISIBLE_DEVICES=4,5,6,7 GPUS=4 ./tools/slurm_train.sh ${PARTITION} ${JOB_NAME} config2.py ${WORK_DIR} --options 'dist_params.port=29501'
+   CUDA_VISIBLE_DEVICES=4,5,6,7 GPUS=4 ./tools/slurm_train.sh ${PARTITION} ${JOB_NAME} config2.py ${WORK_DIR} --cfg-options 'env_cfg.dist_cfg.port=29501'
   ```
 2. Modify the config files (usually the 6th line from the bottom in config files) to set different communication ports.
@@ -245,13 +243,17 @@ If you use launch training jobs with Slurm, there are two ways to specify the po
   In `config1.py`,
   ```python
-   dist_params = dict(backend='nccl', port=29500)
+    env_cfg = dict(
+        dist_cfg=dict(backend='nccl', port=29500)
+    )
   ```
   In `config2.py`,
   ```python
-   dist_params = dict(backend='nccl', port=29501)
+    env_cfg = dict(
+        dist_cfg=dict(backend='nccl', port=29500)
+    )
   ```
   Then you can launch two jobs with `config1.py` and `config2.py`.

--- a/docs/zh_cn/advanced_guides/datasets/scannet_det.md
+++ b/docs/zh_cn/advanced_guides/datasets/scannet_det.md
@@ -265,10 +265,7 @@ train_pipeline = [
        with_seg_3d=True),
    dict(type='GlobalAlignment', rotation_axis=2),
    dict(
-        type='PointSegClassMapping',
+        type='PointSegClassMapping'),
-        valid_cat_ids=(3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 16, 24, 28, 33, 34,
-                       36, 39),
-        max_cat_id=40),
    dict(type='PointSample', num_points=40000),
    dict(
        type='RandomFlip3D',

--- a/docs/zh_cn/advanced_guides/datasets/scannet_sem_seg.md
+++ b/docs/zh_cn/advanced_guides/datasets/scannet_sem_seg.md
@@ -92,10 +92,7 @@ train_pipeline = [
        with_mask_3d=False,
        with_seg_3d=True),
    dict(
-        type='PointSegClassMapping',
+        type='PointSegClassMapping'),
-        valid_cat_ids=(1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 16, 24, 28,
-                       33, 34, 36, 39),
-        max_cat_id=40),
    dict(
        type='IndoorPatchPointSample',
        num_points=num_points,

--- a/docs/zh_cn/advanced_guides/supported_tasks/lidar_det3d.md
+++ b/docs/zh_cn/advanced_guides/supported_tasks/lidar_det3d.md
@@ -26,11 +26,13 @@ mmdetection3d
 │   │   │   ├── calib
 │   │   │   ├── image_2
 │   │   │   ├── velodyne
+│   │   │   ├── velodyne_reduced
 │   │   ├── training
 │   │   │   ├── calib
 │   │   │   ├── image_2
 │   │   │   ├── label_2
 │   │   │   ├── velodyne
+│   │   │   ├── velodyne_reduced
 │   │   ├── kitti_gt_database
 │   │   ├── kitti_infos_train.pkl
 │   │   ├── kitti_infos_trainval.pkl
@@ -44,7 +46,7 @@ mmdetection3d
 接着，我们将使用提供的配置文件训练 PointPillars。当你使用不同的 GPU 设置进行训练时，你基本上可以按照这个[教程](https://mmdetection3d.readthedocs.io/zh_CN/latest/1_exist_data_model.html)的示例脚本进行训练。假设我们在一台具有 8 块 GPU 的机器上进行分布式训练：
 ```
-./tools/dist_train.sh configs/pointpillars/hv_pointpillars_secfpn_6x8_160e_kitti-3d-3class.py 8
+./tools/dist_train.sh configs/pointpillars/pointpillars_hv-secfpn_8xb6-160e_kitti-3d-3class.py 8
 ```
 注意到，配置文件名字中的 `6x8` 是指训练时是用了 8 块 GPU，每块 GPU 上有 6 个样本。如果你有不同的自定义的设置，那么有时你可能需要调整学习率。可以参考这篇[文献](https://arxiv.org/abs/1706.02677)。
@@ -69,14 +71,15 @@ aos AP:97.70, 88.73, 87.34
 评估某个特定的模型权重文件。你可以简单地执行下列的脚本：
 ```
-./tools/dist_test.sh configs/pointpillars/hv_pointpillars_secfpn_6x8_160e_kitti-3d-3class.py \
+./tools/dist_test.sh configs/pointpillars/pointpillars_hv-secfpn_8xb6-160e_kitti-3d-3class.py work_dirs/pointpillars/latest.pth 8
-    work_dirs/pointpillars/latest.pth --eval mAP
 ```
 ## 测试与提交
-如果你只想在线上基准上进行推理或者测试模型的表现，你只需要把上面评估脚本中的 `--eval mAP` 替换为 `--format-only`。如果需要的话，还可以指定 `pklfile_prefix` 和 `submission_prefix`，如，添加命令行选项 `--eval-options submission_prefix=work_dirs/pointpillars/test_submission`。请确保配置文件中的[测试信息](https://github.com/open-mmlab/mmdetection3d/blob/master/configs/_base_/datasets/kitti-3d-3class.py#L131)与测试集对应，而不是验证集。在生成结果后，你可以压缩文件夹，并上传到 KITTI 的评估服务器上。
+如果你只想在线上基准上进行推理或者测试模型的表现，你只需要把修改配置文件中的 evaluator 部分。 例如， 在你的配置文件中修改 `test_evaluator = dict(type='KittiMetric', submission_prefix=work_dirs/pointpillars/test_submission)`，就可以在推理结束后得到结果文件。
+。请确保配置文件中的 `data_prefix` 与 `ann_file` [测试信息](https://github.com/open-mmlab/mmdetection3d/blob/master/configs/_base_/datasets/kitti-3d-3class.py#L113)与测试集对应，而不是验证集。在生成结果后，你可以压缩文件夹，并上传到 KITTI 的评估服务器上。
 ## 定性验证
-MMDetection3D 还提供了通用的可视化工具，以便于我们可以对训练好的模型的预测结果有一个直观的感受。你可以在命令行中添加 `--eval-options 'show=True' 'out_dir=${SHOW_DIR}'` 选项，在评估过程中在线地可视化检测结果；你也可以使用 `tools/misc/visualize_results.py`, 离线地进行可视化。另外，我们还提供了脚本 `tools/misc/browse_dataset.py`， 可视化数据集而不做推理。更多的细节请参考[可视化文档](https://mmdetection3d.readthedocs.io/zh_CN/latest/useful_tools.html#id2)
+MMDetection3D 还提供了通用的可视化工具，以便于我们可以对训练好的模型的预测结果有一个直观的感受。你可以使用 `tools/misc/visualize_results.py`, 离线地进行可视化存储的 pkl 结果文件。另外，我们还提供了脚本 `tools/misc/browse_dataset.py`， 可视化数据集而不做推理。更多的细节请参考[可视化文档](https://mmdetection3d.readthedocs.io/zh_CN/latest/useful_tools.html#id2)
--- a/docs/zh_cn/advanced_guides/supported_tasks/lidar_sem_seg3d.md
+++ b/docs/zh_cn/advanced_guides/supported_tasks/lidar_sem_seg3d.md
@@ -45,14 +45,14 @@ mmdetection3d
 接着，我们将使用提供的配置文件训练 PointNet++ (SSG) 模型。当你使用不同的 GPU 设置进行训练时，你基本上可以按照这个[教程](https://mmdetection3d.readthedocs.io/zh_CN/latest/1_exist_data_model.html#inference-with-existing-models)的示例脚本。假设我们在一台具有 2 块 GPU 的机器上使用分布式训练：
 ```
-./tools/dist_train.sh configs/pointnet2/pointnet2_ssg_16x2_cosine_200e_scannet_seg-3d-20class.py 2
+./tools/dist_train.sh configs/pointnet2/pointnet2_ssg_2xb16-cosine-200e_scannet-seg.py 2
 ```
 注意，配置文件名中的 `16x2` 是指训练时用了 2 块 GPU，每块 GPU 上有 16 个样本。如果你的自定义设置不同于此，那么有时候你需要相应的调整学习率。基本规则可以参考[此处](https://arxiv.org/abs/1706.02677)。
 ## 定量评估
-在训练期间，模型权重将会根据配置文件中的 `evaluation = dict(interval=xxx)` 设置被周期性地评估。我们支持不同数据集的官方评估方案。对于 ScanNet，将使用 20 个类别的平均交并比 (mIoU) 对模型进行评估。评估结果将会被打印到终端中，如下所示：
+在训练期间，模型权重将会根据配置文件中的 `train_cfg = dict(val_interval=xxx)` 设置被周期性地评估。我们支持不同数据集的官方评估方案。对于 ScanNet，将使用 20 个类别的平均交并比 (mIoU) 对模型进行评估。评估结果将会被打印到终端中，如下所示：
 ```
 +---------+--------+--------+---------+--------+--------+--------+--------+--------+--------+-----------+---------+---------+--------+---------+--------------+----------------+--------+--------+---------+----------------+--------+--------+---------+
@@ -65,13 +65,13 @@ mmdetection3d
 此外，在训练完成后你也可以评估特定的模型权重文件。你可以简单地执行以下脚本：
 ```
-./tools/dist_test.sh configs/pointnet2/pointnet2_ssg_16x2_cosine_200e_scannet_seg-3d-20class.py \
+./tools/dist_test.sh configs/pointnet2/pointnet2_ssg_16x2_cosine_200e_scannet_seg-3d-20class.py work_dirs/pointnet2_ssg/latest.pth 8
-    work_dirs/pointnet2_ssg/latest.pth --eval mIoU
 ```
 ## 测试与提交
-如果你只想在在线基准上进行推理或测试模型性能，你需要将之前评估脚本中的 `--eval mIoU` 替换成 `--format-only`，并将 ScanNet 数据集[配置文件](https://github.com/open-mmlab/mmdetection3d/blob/master/configs/_base_/datasets/scannet_seg-3d-20class.py#L126)中的 `ann_file=data_root + 'scannet_infos_val.pkl'` 变成 `ann_file=data_root + 'scannet_infos_test.pkl'`。记住将 `txt_prefix` 指定为保存测试结果的目录，例如，添加选项 `--eval-options txt_prefix=work_dirs/pointnet2_ssg/test_submission`。在生成结果后，你可以压缩文件夹并上传至 [ScanNet 评估服务器](http://kaldir.vc.in.tum.de/scannet_benchmark/semantic_label_3d)上。
+如果你只想在在线基准上进行推理或测试模型性能，你需要在配置文件中的 `test_evalutor` 字段增加 `submission_prefix`， 例如配置文件增加 `test_evaluator = dict(type='SegMetric',submission_prefix=work_dirs/pointnet2_ssg/test_submission`)。
+并将 ScanNet 数据集[配置文件](https://github.com/open-mmlab/mmdetection3d/blob/master/configs/_base_/datasets/scannet_seg-3d-20class.py#L129)中的 `ann_file=scannet_infos_val.pkl` 变成 `ann_file=scannet_infos_test.pkl`。在生成结果后，你可以压缩文件夹并上传至 [ScanNet 评估服务器](http://kaldir.vc.in.tum.de/scannet_benchmark/semantic_label_3d)上。
 ## 定性评估

--- a/docs/zh_cn/user_guides/1_exist_data_model.md
+++ b/docs/zh_cn/user_guides/1_exist_data_model.md
@@ -106,7 +106,7 @@ python tools/test.py ${CONFIG_FILE} ${CHECKPOINT_FILE} [--out ${RESULT_FILE}] [-
   **注意**：为了生成 Lyft 数据集的提交结果，`--eval-options` 必须指定 `csv_savepath`。生成 csv 文件后，你可以使用[网站](https://www.kaggle.com/c/3d-object-detection-for-autonomous-vehicles/submit)上给出的 kaggle 命令提交结果。
-   注意在 [Lyft 数据集的配置文件](../../configs/_base_/datasets/lyft-3d.py)，`test` 中的 `ann_file` 值为 `data_root + 'lyft_infos_test.pkl'`，是没有标注的 Lyft 官方测试集。要在验证数据集上测试，请把它改为 `data_root + 'lyft_infos_val.pkl'`。
+   注意在 [Lyft 数据集的配置文件](../../configs/_base_/datasets/lyft-3d.py)，`test` 中的 `ann_file` 值为 `lyft_infos_test.pkl`，是没有标注的 Lyft 官方测试集。要在验证数据集上测试，请把它改为 `lyft_infos_val.pkl`。
 8. 使用8块显卡在 waymo 数据集上测试 PointPillars，使用 waymo 度量方法计算 mAP

--- a/docs/zh_cn/user_guides/config.md
+++ b/docs/zh_cn/user_guides/config.md
@@ -283,10 +283,7 @@ data = dict(
                    with_mask_3d=True,
                    with_seg_3d=True),
                dict(
-                    type='PointSegClassMapping',
+                    type='PointSegClassMapping'),
-                    valid_cat_ids=(3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 14, 16, 24,
-                                   28, 33, 34, 36, 39),
-                    max_cat_id=40),
                dict(type='PointSample', num_points=40000),
                dict(
                    type='IndoorFlipData',

--- a/mmdet3d/datasets/transforms/loading.py
+++ b/mmdet3d/datasets/transforms/loading.py
@@ -255,9 +255,10 @@ class LoadPointsFromMultiSweeps(BaseTransform):
                    points_sweep = self._remove_close(points_sweep)
                # bc-breaking: Timestamp has divided 1e6 in pkl infos.
                sweep_ts = sweep['timestamp']
-                lidar2cam = np.array(sweep['lidar_points']['lidar2sensor'])
+                lidar2sensor = np.array(sweep['lidar_points']['lidar2sensor'])
-                points_sweep[:, :3] = points_sweep[:, :3] @ lidar2cam[:3, :3]
+                points_sweep[:, :
-                points_sweep[:, :3] -= lidar2cam[:3, 3]
+                             3] = points_sweep[:, :3] @ lidar2sensor[:3, :3]
+                points_sweep[:, :3] -= lidar2sensor[:3, 3]
                points_sweep[:, 4] = ts - sweep_ts
                points_sweep = points.new_point(points_sweep)
                sweep_points_list.append(points_sweep)

--- a/mmdet3d/evaluation/metrics/seg_metric.py
+++ b/mmdet3d/evaluation/metrics/seg_metric.py
 # Copyright (c) OpenMMLab. All rights reserved.
+import os.path as osp
+import tempfile
 from typing import Dict, Optional, Sequence
+import mmcv
+import numpy as np
 from mmengine.evaluator import BaseMetric
 from mmengine.logging import MMLogger
@@ -19,13 +23,23 @@ class SegMetric(BaseMetric):
        prefix (str): The prefix that will be added in the metric
            names to disambiguate homonymous metrics of different evaluators.
            If prefix is not provided in the argument, self.default_prefix
-            will be used instead. Default: None
+            will be used instead. Default: None.
+        pklfile_prefix (str, optional): The prefix of pkl files, including
+            the file path and the prefix of filename, e.g., "a/b/prefix".
+            If not specified, a temp file will be created. Default: None.
+        submission_prefix (str, optional): The prefix of submission data.
+            If not specified, the submission data will not be generated.
+            Default: None.
    """
    def __init__(self,
                 collect_device: str = 'cpu',
                 prefix: Optional[str] = None,
+                 pklfile_prefix: str = None,
+                 submission_prefix: str = None,
                 **kwargs):
+        self.pklfile_prefix = pklfile_prefix
+        self.submission_prefix = submission_prefix
        super(SegMetric, self).__init__(
            prefix=prefix, collect_device=collect_device)
@@ -52,6 +66,40 @@ class SegMetric(BaseMetric):
                    cpu_pred_3d[k] = v
            self.results.append((eval_ann_info, cpu_pred_3d))
+    def format_results(self, results):
+        r"""Format the results to txt file. Refer to `ScanNet documentation
+        <http://kaldir.vc.in.tum.de/scannet_benchmark/documentation>`_.
+        Args:
+            outputs (list[dict]): Testing results of the dataset.
+        Returns:
+            tuple: (outputs, tmp_dir), outputs is the detection results,
+                tmp_dir is the temporal directory created for saving submission
+                files when ``submission_prefix`` is not specified.
+        """
+        submission_prefix = self.submission_prefix
+        if submission_prefix is None:
+            tmp_dir = tempfile.TemporaryDirectory()
+            submission_prefix = osp.join(tmp_dir.name, 'results')
+        mmcv.mkdir_or_exist(submission_prefix)
+        ignore_index = self.dataset_meta['ignore_index']
+        # need to map network output to original label idx
+        cat2label = np.zeros(len(self.dataset_meta['label2cat'])).astype(
+            np.int)
+        for original_label, output_idx in self.dataset_meta['label2cat'].items(
+        ):
+            if output_idx != ignore_index:
+                cat2label[output_idx] = original_label
+        for i, (eval_ann, result) in enumerate(results):
+            sample_idx = eval_ann['point_cloud']['lidar_idx']
+            pred_sem_mask = result['semantic_mask'].numpy().astype(np.int)
+            pred_label = cat2label[pred_sem_mask]
+            curr_file = f'{submission_prefix}/{sample_idx}.txt'
+            np.savetxt(curr_file, pred_label, fmt='%d')
    def compute_metrics(self, results: list) -> Dict[str, float]:
        """Compute the metrics from processed results.
@@ -64,6 +112,10 @@ class SegMetric(BaseMetric):
        """
        logger: MMLogger = MMLogger.get_current_instance()
+        if self.submission_prefix:
+            self.format_results(results)
+            return None
        label2cat = self.dataset_meta['label2cat']
        ignore_index = self.dataset_meta['ignore_index']