[Enhance] Benchmark PAConv on S3DIS dataset (#809)

* add cfg

* add eps format

* update README

* update model zoo

* add paconv readme

* add metafile

* hack drop_ratio

* use base schedule cfg

* remove paconv cuda

* monor fix

* remove paconv cuda config related

* update paconv benchmark

* update model link

.dev_scripts/gather_models.py

@@ -24,6 +24,7 @@ SCHEDULES_LUT = {
     '_6x_': 73,
     '_50e_': 50,
     '_80e_': 80,
+    '_150e_': 150,
     '_200e_': 200,
     '_250e_': 250,
     '_400e_': 400

README.md

@@ -92,6 +92,8 @@ Support methods
 - [x] [FCOS3D (Arxiv'2021)](configs/fcos3d/README.md)
 - [x] [PointNet++ (NeurIPS'2017)](configs/pointnet2/README.md)
 - [x] [Group-Free-3D (Arxiv'2021)](configs/groupfree3d/README.md)
+- [x] [ImVoxelNet (Arxiv'2021)](configs/imvoxelnet/README.md)
+- [x] [PAConv (CVPR'2021)](configs/paconv/README.md)
 
 |                    | ResNet   | ResNeXt  | SENet    |PointNet++ | HRNet | RegNetX | Res2Net |
 |--------------------|:--------:|:--------:|:--------:|:---------:|:-----:|:--------:|:-----:|
@@ -109,6 +111,8 @@ Support methods
 | FCOS3D               | ✓        | ☐        | ☐        | ✗         | ☐     | ☐        | ☐     |
 | PointNet++           | ✗        | ✗        | ✗        | ✓         | ✗     | ✗        | ✗     |
 | Group-Free-3D        | ✗        | ✗        | ✗        | ✓         | ✗     | ✗        | ✗     |
+| ImVoxelNet           | ✓         | ✗        | ✗        | ✗        | ✗     | ✗        | ✗     |
+| PAConv               | ✗        | ✗        | ✗        | ✓         | ✗     | ✗        | ✗     |
 
 Other features
 - [x] [Dynamic Voxelization](configs/dynamic_voxelization/README.md)

README_zh-CN.md

@@ -90,6 +90,9 @@ MMDetection3D 是一个基于 PyTorch 的目标检测开源工具箱, 下一代
 - [x] [ImVoteNet (CVPR'2020)](configs/imvotenet/README.md)
 - [x] [FCOS3D (Arxiv'2021)](configs/fcos3d/README.md)
 - [x] [PointNet++ (NeurIPS'2017)](configs/pointnet2/README.md)
+- [x] [Group-Free-3D (Arxiv'2021)](configs/groupfree3d/README.md)
+- [x] [ImVoxelNet (Arxiv'2021)](configs/imvoxelnet/README.md)
+- [x] [PAConv (CVPR'2021)](configs/paconv/README.md)
 
 |                    | ResNet   | ResNeXt  | SENet    |PointNet++ | HRNet | RegNetX | Res2Net |
 |--------------------|:--------:|:--------:|:--------:|:---------:|:-----:|:--------:|:-----:|
@@ -106,6 +109,9 @@ MMDetection3D 是一个基于 PyTorch 的目标检测开源工具箱, 下一代
 | ImVoteNet            | ✗        | ✗        | ✗        | ✓         | ✗     | ✗        | ✗     |
 | FCOS3D               | ✓        | ☐        | ☐        | ✗         | ☐     | ☐        | ☐     |
 | PointNet++           | ✗        | ✗        | ✗        | ✓         | ✗     | ✗        | ✗     |
+| Group-Free-3D        | ✗        | ✗        | ✗        | ✓         | ✗     | ✗        | ✗     |
+| ImVoxelNet           | ✓         | ✗        | ✗        | ✗        | ✗     | ✗        | ✗     |
+| PAConv               | ✗        | ✗        | ✗        | ✓         | ✗     | ✗        | ✗     |
 
 其他特性
 - [x] [Dynamic Voxelization](configs/dynamic_voxelization/README.md)

configs/_base_/schedules/seg_cosine_150e.py

+# optimizer
+# This schedule is mainly used on S3DIS dataset in segmentation task
+optimizer = dict(type='SGD', lr=0.2, weight_decay=0.0001, momentum=0.9)
+optimizer_config = dict(grad_clip=None)
+lr_config = dict(policy='CosineAnnealing', warmup=None, min_lr=0.002)
+momentum_config = None
+
+# runtime settings
+runner = dict(type='EpochBasedRunner', max_epochs=150)

configs/paconv/README.md

+# PAConv: Position Adaptive Convolution with Dynamic Kernel Assembling on Point Clouds
+
+## Introduction
+
+<!-- [ALGORITHM] -->
+
+We implement PAConv and provide the result and checkpoints on S3DIS dataset.
+
+```
+@inproceedings{xu2021paconv,
+  title={PAConv: Position Adaptive Convolution with Dynamic Kernel Assembling on Point Clouds},
+  author={Xu, Mutian and Ding, Runyu and Zhao, Hengshuang and Qi, Xiaojuan},
+  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
+  pages={3173--3182},
+  year={2021}
+}
+```
+
+**Notice**: The original PAConv paper used step learning rate schedule. We discovered that cosine schedule achieves slightly better results and adopt it in our implementations.
+
+## Results
+
+### S3DIS
+
+|                                   Method                                    | Split  |   Lr schd   | Mem (GB) | Inf time (fps) | mIoU (Val set) |         Download         |
+| :-------------------------------------------------------------------------: | :----: | :---------: | :------: | :------------: | :------------: | :----------------------: |
+|    [PAConv (SSG)](./paconv_ssg_8x8_cosine_150e_s3dis_seg-3d-13class.py)     | Area_5 | cosine 150e |   5.8    |                |       66.65        | [model](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/paconv/paconv_ssg_8x8_cosine_150e_s3dis_seg-3d-13class/paconv_ssg_8x8_cosine_150e_s3dis_seg-3d-13class_20210729_200615-2147b2d1.pth) &#124; [log](https://download.openmmlab.com/mmdetection3d/v0.1.0_models/paconv/paconv_ssg_8x8_cosine_150e_s3dis_seg-3d-13class/paconv_ssg_8x8_cosine_150e_s3dis_seg-3d-13class_20210729_200615.log.json) |
+
+**Notes:**
+
+- We use XYZ+Color+Normalized_XYZ as input in all the experiments on S3DIS datasets.
+- `Area_5` Split means training the model on Area_1, 2, 3, 4, 6 and testing on Area_5.
+
+## Indeterminism
+
+Since PAConv testing adopts sliding patch inference which involves random point sampling, and the test script uses fixed random seeds while the random seeds of validation in training are not fixed, the test results may be slightly different from the results reported above.

configs/paconv/metafile.yml

+Collections:
+  - Name: PAConv
+    Metadata:
+      Training Techniques:
+        - SGD
+      Training Resources: 8x Titan XP GPUs
+      Architecture:
+        - PAConv
+    Paper: https://arxiv.org/abs/2103.14635
+    README: configs/paconv/README.md
+
+Models:
+  - Name: paconv_ssg_8x8_cosine_150e_s3dis_seg-3d-13class.py
+    In Collection: PAConv
+    Config: configs/paconv/paconv_ssg_8x8_cosine_150e_s3dis_seg-3d-13class.py
+    Metadata:
+      Training Data: S3DIS
+      Training Memory (GB): 5.8
+    Results:
+      - Task: 3D Semantic Segmentation
+        Dataset: S3DIS
+        Metrics:
+          mIoU: 66.65
+    Weights: https://download.openmmlab.com/mmdetection3d/v0.1.0_models/paconv/paconv_ssg_8x8_cosine_150e_s3dis_seg-3d-13class/paconv_ssg_8x8_cosine_150e_s3dis_seg-3d-13class_20210729_200615-2147b2d1.pth

configs/paconv/paconv_ssg_8x8_cosine_150e_s3dis_seg-3d-13class.py

+_base_ = [
+    '../_base_/datasets/s3dis_seg-3d-13class.py',
+    '../_base_/models/paconv_ssg.py', '../_base_/schedules/seg_cosine_150e.py',
+    '../_base_/default_runtime.py'
+]
+
+# data settings
+class_names = ('ceiling', 'floor', 'wall', 'beam', 'column', 'window', 'door',
+               'table', 'chair', 'sofa', 'bookcase', 'board', 'clutter')
+num_points = 4096
+train_pipeline = [
+    dict(
+        type='LoadPointsFromFile',
+        coord_type='DEPTH',
+        shift_height=False,
+        use_color=True,
+        load_dim=6,
+        use_dim=[0, 1, 2, 3, 4, 5]),
+    dict(
+        type='LoadAnnotations3D',
+        with_bbox_3d=False,
+        with_label_3d=False,
+        with_mask_3d=False,
+        with_seg_3d=True),
+    dict(
+        type='PointSegClassMapping',
+        valid_cat_ids=tuple(range(len(class_names))),
+        max_cat_id=13),
+    dict(
+        type='IndoorPatchPointSample',
+        num_points=num_points,
+        block_size=1.0,
+        use_normalized_coord=True,
+        num_try=10000,
+        enlarge_size=None,
+        min_unique_num=num_points // 4,
+        eps=0.0),
+    dict(type='NormalizePointsColor', color_mean=None),
+    dict(
+        type='GlobalRotScaleTrans',
+        rot_range=[0.0, 6.283185307179586],  # [0, 2 * pi]
+        scale_ratio_range=[0.8, 1.2],
+        translation_std=[0, 0, 0]),
+    dict(
+        type='RandomJitterPoints',
+        jitter_std=[0.01, 0.01, 0.01],
+        clip_range=[-0.05, 0.05]),
+    dict(type='RandomDropPointsColor', drop_ratio=0.2),
+    dict(type='DefaultFormatBundle3D', class_names=class_names),
+    dict(type='Collect3D', keys=['points', 'pts_semantic_mask'])
+]
+
+data = dict(samples_per_gpu=8, train=dict(pipeline=train_pipeline))
+evaluation = dict(interval=1)
+
+# model settings
+model = dict(
+    decode_head=dict(
+        num_classes=13, ignore_index=13,
+        loss_decode=dict(class_weight=None)),  # S3DIS doesn't use class_weight
+    test_cfg=dict(
+        num_points=4096,
+        block_size=1.0,
+        sample_rate=0.5,
+        use_normalized_coord=True,
+        batch_size=12))

configs/pointnet2/README.md

@@ -50,8 +50,8 @@ We implement PointNet++ and provide the result and checkpoints on ScanNet and S3
 
 **Notes:**
 
--   We use XYZ+Color+Normalized_XYZ as input in all the experiments on S3DIS datasets.
--   `Area_5` Split means training the model on Area_1, 2, 3, 4, 6 and testing on Area_5.
+- We use XYZ+Color+Normalized_XYZ as input in all the experiments on S3DIS datasets.
+- `Area_5` Split means training the model on Area_1, 2, 3, 4, 6 and testing on Area_5.
 
 ## Indeterminism
 

docs/model_zoo.md

@@ -72,4 +72,8 @@ Please refer to [Group-Free-3D](https://github.com/open-mmlab/mmdetection3d/blob
 
 ### ImVoxelNet
 
-Please refer to [ImVoxelNet](https://github.com/open-mmlab/mmdetection3d/blob/master/configs/imvoxelnet) for details. We provide  ImVoxelNet baselines on KITTI dataset.
+Please refer to [ImVoxelNet](https://github.com/open-mmlab/mmdetection3d/blob/master/configs/imvoxelnet) for details. We provide ImVoxelNet baselines on KITTI dataset.
+
+### PAConv
+
+Please refer to [PAConv](https://github.com/open-mmlab/mmdetection3d/blob/master/configs/paconv) for details. We provide PAConv baselines on S3DIS dataset.

mmdet3d/datasets/pipelines/transforms_3d.py

@@ -45,7 +45,13 @@ class RandomDropPointsColor(object):
             'color' in points.attribute_dims, \
             'Expect points have color attribute'
 
-        if np.random.rand() < self.drop_ratio:
+        # this if-expression is a bit strange
+        # `RandomDropPointsColor` is used in training 3D segmentor PAConv
+        # we discovered in our experiments that, using
+        # `if np.random.rand() > 1.0 - self.drop_ratio` consistently leads to
+        # better results than using `if np.random.rand() < self.drop_ratio`
+        # so we keep this hack in our codebase
+        if np.random.rand() > 1.0 - self.drop_ratio:
             points.color = points.color * 0.0
         return input_dict
 
@@ -1151,7 +1157,8 @@ class IndoorPatchPointSample(object):
         repr_str += f' use_normalized_coord={self.use_normalized_coord},'
         repr_str += f' num_try={self.num_try},'
         repr_str += f' enlarge_size={self.enlarge_size},'
-        repr_str += f' min_unique_num={self.min_unique_num})'
+        repr_str += f' min_unique_num={self.min_unique_num},'
+        repr_str += f' eps={self.eps})'
         return repr_str
 
 

tests/test_data/test_pipelines/test_indoor_sample.py

@@ -114,7 +114,8 @@ def test_indoor_seg_sample():
                         'use_normalized_coord=True, ' \
                         'num_try=10, ' \
                         'enlarge_size=0.2, ' \
-                        'min_unique_num=None)'
+                        'min_unique_num=None, ' \
+                        'eps=0.01)'
     assert repr_str == expected_repr_str
 
     # when enlarge_size and min_unique_num are set

tests/test_models/test_segmentors.py

@@ -167,7 +167,7 @@ def test_paconv_ssg():
 
     set_random_seed(0, True)
     paconv_ssg_cfg = _get_segmentor_cfg(
-        'paconv/paconv_ssg_8x2_step_100e_s3dis_seg-3d-13class.py')
+        'paconv/paconv_ssg_8x8_cosine_150e_s3dis_seg-3d-13class.py')
     # for GPU memory consideration
     paconv_ssg_cfg.backbone.num_points = (256, 64, 16, 4)
     paconv_ssg_cfg.test_cfg.num_points = 32
@@ -231,75 +231,3 @@ def test_paconv_ssg():
         results = self.forward(return_loss=False, **data_dict)
         assert results[0]['semantic_mask'].shape == torch.Size([200])
         assert results[1]['semantic_mask'].shape == torch.Size([100])
-
-
-def test_paconv_cuda_ssg():
-    if not torch.cuda.is_available():
-        pytest.skip('test requires GPU and torch+cuda')
-
-    set_random_seed(0, True)
-    paconv_cuda_ssg_cfg = _get_segmentor_cfg(
-        'paconv/paconv_ssg_8x2_step_100e_s3dis_seg-3d-13class.py')
-    # for GPU memory consideration
-    paconv_cuda_ssg_cfg.backbone.num_points = (256, 64, 16, 4)
-    paconv_cuda_ssg_cfg.test_cfg.num_points = 32
-    self = build_segmentor(paconv_cuda_ssg_cfg).cuda()
-    points = [torch.rand(1024, 9).float().cuda() for _ in range(2)]
-    img_metas = [dict(), dict()]
-    gt_masks = [torch.randint(0, 13, (1024, )).long().cuda() for _ in range(2)]
-
-    # test forward_train
-    losses = self.forward_train(points, img_metas, gt_masks)
-    assert losses['decode.loss_sem_seg'].item() >= 0
-    assert losses['regularize.loss_regularize'].item() >= 0
-
-    # test forward function
-    set_random_seed(0, True)
-    data_dict = dict(
-        points=points, img_metas=img_metas, pts_semantic_mask=gt_masks)
-    forward_losses = self.forward(return_loss=True, **data_dict)
-    assert np.allclose(losses['decode.loss_sem_seg'].item(),
-                       forward_losses['decode.loss_sem_seg'].item())
-    assert np.allclose(losses['regularize.loss_regularize'].item(),
-                       forward_losses['regularize.loss_regularize'].item())
-
-    # test loss with ignore_index
-    ignore_masks = [torch.ones_like(gt_masks[0]) * 13 for _ in range(2)]
-    losses = self.forward_train(points, img_metas, ignore_masks)
-    assert losses['decode.loss_sem_seg'].item() == 0
-
-    # test simple_test
-    self.eval()
-    with torch.no_grad():
-        scene_points = [
-            torch.randn(200, 6).float().cuda() * 3.0,
-            torch.randn(100, 6).float().cuda() * 2.5
-        ]
-        results = self.simple_test(scene_points, img_metas)
-        assert results[0]['semantic_mask'].shape == torch.Size([200])
-        assert results[1]['semantic_mask'].shape == torch.Size([100])
-
-    # test forward function calling simple_test
-    with torch.no_grad():
-        data_dict = dict(points=[scene_points], img_metas=[img_metas])
-        results = self.forward(return_loss=False, **data_dict)
-        assert results[0]['semantic_mask'].shape == torch.Size([200])
-        assert results[1]['semantic_mask'].shape == torch.Size([100])
-
-    # test aug_test
-    with torch.no_grad():
-        scene_points = [
-            torch.randn(2, 200, 6).float().cuda() * 3.0,
-            torch.randn(2, 100, 6).float().cuda() * 2.5
-        ]
-        img_metas = [[dict(), dict()], [dict(), dict()]]
-        results = self.aug_test(scene_points, img_metas)
-        assert results[0]['semantic_mask'].shape == torch.Size([200])
-        assert results[1]['semantic_mask'].shape == torch.Size([100])
-
-    # test forward function calling aug_test
-    with torch.no_grad():
-        data_dict = dict(points=scene_points, img_metas=img_metas)
-        results = self.forward(return_loss=False, **data_dict)
-        assert results[0]['semantic_mask'].shape == torch.Size([200])
-        assert results[1]['semantic_mask'].shape == torch.Size([100])