Skip to content

GitLab

Unverified Commit 328590d2 authored by VVsssssk's avatar VVsssssk Committed by GitHub
Browse files

[Docs]Fix docs about custom runtime and custom model (#1770) 

* fix custom runtime

* fix custom_model

* fix cfg

* fix comments

* fix comments

* fix comments
parent 8de2a289
Show whitespace changes
Inline Side-by-side
Showing with 409 additions and 262 deletions
configs/_base_/schedules/cyclic-20e.py
View file @ 328590d2
...@@ -12,6 +12,10 @@ optim_wrapper = dict( ...@@ -12,6 +12,10 @@ optim_wrapper = dict(
clip_grad=dict(max_norm=35, norm_type=2)) clip_grad=dict(max_norm=35, norm_type=2))
# learning rate # learning rate
param_scheduler = [ param_scheduler = [
# learning rate scheduler
# During the first 8 epochs, learning rate increases from 0 to lr * 10
# during the next 12 epochs, learning rate decreases from lr * 10 to
# lr * 1e-4
dict( dict(
type='CosineAnnealingLR', type='CosineAnnealingLR',
T_max=8, T_max=8,
...@@ -28,6 +32,9 @@ param_scheduler = [ ...@@ -28,6 +32,9 @@ param_scheduler = [
end=20, end=20,
by_epoch=True, by_epoch=True,
convert_to_iter_based=True), convert_to_iter_based=True),
# momentum scheduler
# During the first 8 epochs, momentum increases from 0 to 0.85 / 0.95
# during the next 12 epochs, momentum increases from 0.85 / 0.95 to 1
dict( dict(
type='CosineAnnealingMomentum', type='CosineAnnealingMomentum',
T_max=8, T_max=8,
......
configs/_base_/schedules/cyclic-40e.py
View file @ 328590d2
...@@ -11,6 +11,10 @@ optim_wrapper = dict( ...@@ -11,6 +11,10 @@ optim_wrapper = dict(
clip_grad=dict(max_norm=10, norm_type=2)) clip_grad=dict(max_norm=10, norm_type=2))
# learning rate # learning rate
param_scheduler = [ param_scheduler = [
# learning rate scheduler
# During the first 16 epochs, learning rate increases from 0 to lr * 10
# during the next 24 epochs, learning rate decreases from lr * 10 to
# lr * 1e-4
dict( dict(
type='CosineAnnealingLR', type='CosineAnnealingLR',
T_max=16, T_max=16,
...@@ -27,6 +31,9 @@ param_scheduler = [ ...@@ -27,6 +31,9 @@ param_scheduler = [
end=40, end=40,
by_epoch=True, by_epoch=True,
convert_to_iter_based=True), convert_to_iter_based=True),
# momentum scheduler
# During the first 16 epochs, momentum increases from 0 to 0.85 / 0.95
# during the next 24 epochs, momentum increases from 0.85 / 0.95 to 1
dict( dict(
type='CosineAnnealingMomentum', type='CosineAnnealingMomentum',
T_max=16, T_max=16,
......
docs/en/advanced_guides/customize_models.md
View file @ 328590d2
...@@ -22,10 +22,10 @@ Create a new file `mmdet3d/models/voxel_encoders/voxel_encoder.py`. ...@@ -22,10 +22,10 @@ Create a new file `mmdet3d/models/voxel_encoders/voxel_encoder.py`.
```python ```python
import torch.nn as nn import torch.nn as nn
from ..builder import VOXEL_ENCODERS from mmdet3d.registry import MODELS
@VOXEL_ENCODERS.register_module() @MODELS.register_module()
class HardVFE(nn.Module): class HardVFE(nn.Module):
def __init__(self, arg1, arg2): def __init__(self, arg1, arg2):
...@@ -76,10 +76,10 @@ Create a new file `mmdet3d/models/backbones/second.py`. ...@@ -76,10 +76,10 @@ Create a new file `mmdet3d/models/backbones/second.py`.
```python ```python
import torch.nn as nn import torch.nn as nn
from ..builder import BACKBONES from mmdet3d.registry import MODELS
@BACKBONES.register_module() @MODELS.register_module()
class SECOND(BaseModule): class SECOND(BaseModule):
def __init__(self, arg1, arg2): def __init__(self, arg1, arg2):
...@@ -126,9 +126,9 @@ model = dict( ...@@ -126,9 +126,9 @@ model = dict(
Create a new file `mmdet3d/models/necks/second_fpn.py`. Create a new file `mmdet3d/models/necks/second_fpn.py`.
```python ```python
from ..builder import NECKS from mmdet3d.registry import MODELS
@NECKS.register @MODELS.register
class SECONDFPN(BaseModule): class SECONDFPN(BaseModule):
def __init__(self, def __init__(self,
...@@ -186,13 +186,13 @@ Here we show how to develop a new head with the example of [PartA2 Head](https:/ ...@@ -186,13 +186,13 @@ Here we show how to develop a new head with the example of [PartA2 Head](https:/
First, add a new bbox head in `mmdet3d/models/roi_heads/bbox_heads/parta2_bbox_head.py`. First, add a new bbox head in `mmdet3d/models/roi_heads/bbox_heads/parta2_bbox_head.py`.
PartA2 RoI Head implements a new bbox head for object detection. PartA2 RoI Head implements a new bbox head for object detection.
To implement a bbox head, basically we need to implement three functions of the new module as the following. Sometimes other related functions like `loss` and `get_targets` are also required. To implement a bbox head, basically we need to implement two functions of the new module as the following. Sometimes other related functions like `loss` and `get_targets` are also required.
```python ```python
from mmdet.models.builder import HEADS from mmdet3d.registry import MODELS
from .bbox_head import BBoxHead from mmengine.model import BaseModule
@HEADS.register_module() @MODELS.register_module()
class PartA2BboxHead(BaseModule): class PartA2BboxHead(BaseModule):
"""PartA2 RoI head.""" """PartA2 RoI head."""
...@@ -224,71 +224,63 @@ class PartA2BboxHead(BaseModule): ...@@ -224,71 +224,63 @@ class PartA2BboxHead(BaseModule):
super(PartA2BboxHead, self).__init__(init_cfg=init_cfg) super(PartA2BboxHead, self).__init__(init_cfg=init_cfg)
def forward(self, seg_feats, part_feats): def forward(self, seg_feats, part_feats):
``` ```
Second, implement a new RoI Head if it is necessary. We plan to inherit the new `PartAggregationROIHead` from `Base3DRoIHead`. We can find that a `Base3DRoIHead` already implements the following functions. Second, implement a new RoI Head if it is necessary. We plan to inherit the new `PartAggregationROIHead` from `Base3DRoIHead`. We can find that a `Base3DRoIHead` already implements the following functions.
```python ```python
from abc import ABCMeta, abstractmethod from mmdet3d.registry import MODELS, TASK_UTILS
from torch import nn as nn from mmdet.models.roi_heads import BaseRoIHead
@HEADS.register_module() class Base3DRoIHead(BaseRoIHead):
class Base3DRoIHead(BaseModule, metaclass=ABCMeta):
"""Base class for 3d RoIHeads.""" """Base class for 3d RoIHeads."""
def __init__(self, def __init__(self,
bbox_head=None, bbox_head=None,
mask_roi_extractor=None, bbox_roi_extractor=None,
mask_head=None, mask_head=None,
mask_roi_extractor=None,
train_cfg=None, train_cfg=None,
test_cfg=None, test_cfg=None,
init_cfg=None): init_cfg=None):
super(Base3DRoIHead, self).__init__(
bbox_head=bbox_head,
bbox_roi_extractor=bbox_roi_extractor,
mask_head=mask_head,
mask_roi_extractor=mask_roi_extractor,
train_cfg=train_cfg,
test_cfg=test_cfg,
init_cfg=init_cfg)
@property def init_bbox_head(self, bbox_roi_extractor: dict,
def with_bbox(self): bbox_head: dict) -> None:
"""Initialize box head and box roi extractor.
@property
def with_mask(self):
@abstractmethod
def init_weights(self, pretrained):
@abstractmethod
def init_bbox_head(self):
@abstractmethod Args:
def init_mask_head(self): bbox_roi_extractor (dict or ConfigDict): Config of box
roi extractor.
bbox_head (dict or ConfigDict): Config of box in box head.
"""
self.bbox_roi_extractor = MODELS.build(bbox_roi_extractor)
self.bbox_head = MODELS.build(bbox_head)
@abstractmethod
def init_assigner_sampler(self): def init_assigner_sampler(self):
"""Initialize assigner and sampler."""
self.bbox_assigner = None
self.bbox_sampler = None
if self.train_cfg:
if isinstance(self.train_cfg.assigner, dict):
self.bbox_assigner = TASK_UTILS.build(self.train_cfg.assigner)
elif isinstance(self.train_cfg.assigner, list):
self.bbox_assigner = [
TASK_UTILS.build(res) for res in self.train_cfg.assigner
]
self.bbox_sampler = TASK_UTILS.build(self.train_cfg.sampler)
@abstractmethod def init_mask_head(self):
def forward_train(self, """Initialize mask head, skip since ``PartAggregationROIHead`` does not
x, have one."""
img_metas,
proposal_list,
gt_bboxes,
gt_labels,
gt_bboxes_ignore=None,
**kwargs):
def simple_test(self,
x,
proposal_list,
img_metas,
proposals=None,
rescale=False,
**kwargs):
"""Test without augmentation."""
pass
def aug_test(self, x, proposal_list, img_metas, rescale=False, **kwargs):
"""Test with augmentations.
If rescale is False, then returned bboxes and masks will fit the scale
of imgs[0].
"""
pass pass
``` ```
...@@ -297,85 +289,192 @@ Double Head's modification is mainly in the bbox_forward logic, and it inherits ...@@ -297,85 +289,192 @@ Double Head's modification is mainly in the bbox_forward logic, and it inherits
In the `mmdet3d/models/roi_heads/part_aggregation_roi_head.py`, we implement the new RoI Head as the following: In the `mmdet3d/models/roi_heads/part_aggregation_roi_head.py`, we implement the new RoI Head as the following:
```python ```python
from typing import Dict, List, Tuple
from mmcv import ConfigDict
from torch import Tensor
from torch.nn import functional as F from torch.nn import functional as F
from mmdet3d.core import AssignResult from mmdet3d.registry import MODELS
from mmdet3d.core.bbox import bbox3d2result, bbox3d2roi from mmdet3d.structures import bbox3d2roi
from mmdet.core import build_assigner, build_sampler from mmdet3d.utils import InstanceList
from mmdet.models import HEADS from mmdet.models.task_modules import AssignResult, SamplingResult
from ..builder import build_head, build_roi_extractor from ...structures.det3d_data_sample import SampleList
from .base_3droi_head import Base3DRoIHead from .base_3droi_head import Base3DRoIHead
@HEADS.register_module() @MODELS.register_module()
class PartAggregationROIHead(Base3DRoIHead): class PartAggregationROIHead(Base3DRoIHead):
"""Part aggregation roi head for PartA2. """Part aggregation roi head for PartA2.
Args: Args:
semantic_head (ConfigDict): Config of semantic head. semantic_head (ConfigDict): Config of semantic head.
num_classes (int): The number of classes. num_classes (int): The number of classes.
seg_roi_extractor (ConfigDict): Config of seg_roi_extractor. seg_roi_extractor (ConfigDict): Config of seg_roi_extractor.
part_roi_extractor (ConfigDict): Config of part_roi_extractor. bbox_roi_extractor (ConfigDict): Config of part_roi_extractor.
bbox_head (ConfigDict): Config of bbox_head. bbox_head (ConfigDict): Config of bbox_head.
train_cfg (ConfigDict): Training config. train_cfg (ConfigDict): Training config.
test_cfg (ConfigDict): Testing config. test_cfg (ConfigDict): Testing config.
""" """
def __init__(self, def __init__(self,
semantic_head, semantic_head: dict,
num_classes=3, num_classes: int = 3,
seg_roi_extractor=None, seg_roi_extractor: dict = None,
part_roi_extractor=None, bbox_head: dict = None,
bbox_head=None, bbox_roi_extractor: dict = None,
train_cfg=None, train_cfg: dict = None,
test_cfg=None, test_cfg: dict = None,
init_cfg=None): init_cfg: dict = None) -> None:
super(PartAggregationROIHead, self).__init__( super(PartAggregationROIHead, self).__init__(
bbox_head=bbox_head, bbox_head=bbox_head,
bbox_roi_extractor=bbox_roi_extractor,
train_cfg=train_cfg, train_cfg=train_cfg,
test_cfg=test_cfg, test_cfg=test_cfg,
init_cfg=init_cfg) init_cfg=init_cfg)
self.num_classes = num_classes self.num_classes = num_classes
assert semantic_head is not None assert semantic_head is not None
self.semantic_head = build_head(semantic_head) self.init_seg_head(seg_roi_extractor, semantic_head)
def init_seg_head(self, seg_roi_extractor: dict,
semantic_head: dict) -> None:
"""Initialize semantic head and seg roi extractor.
Args:
seg_roi_extractor (dict): Config of seg
roi extractor.
semantic_head (dict): Config of semantic head.
"""
self.semantic_head = MODELS.build(semantic_head)
self.seg_roi_extractor = MODELS.build(seg_roi_extractor)
@property
def with_semantic(self):
"""bool: whether the head has semantic branch"""
return hasattr(self,
'semantic_head') and self.semantic_head is not None
def predict(self,
feats_dict: Dict,
rpn_results_list: InstanceList,
batch_data_samples: SampleList,
rescale: bool = False,
**kwargs) -> InstanceList:
"""Perform forward propagation of the roi head and predict detection
results on the features of the upstream network.
Args:
feats_dict (dict): Contains features from the first stage.
rpn_results_list (List[:obj:`InstancesData`]): Detection results
of rpn head.
batch_data_samples (List[:obj:`Det3DDataSample`]): The Data
samples. It usually includes information such as
`gt_instance_3d`, `gt_panoptic_seg_3d` and `gt_sem_seg_3d`.
rescale (bool): If True, return boxes in original image space.
Defaults to False.
if seg_roi_extractor is not None: Returns:
self.seg_roi_extractor = build_roi_extractor(seg_roi_extractor) list[:obj:`InstanceData`]: Detection results of each sample
if part_roi_extractor is not None: after the post process.
self.part_roi_extractor = build_roi_extractor(part_roi_extractor) Each item usually contains following keys.
- scores_3d (Tensor): Classification scores, has a shape
(num_instances, )
- labels_3d (Tensor): Labels of bboxes, has a shape
(num_instances, ).
- bboxes_3d (BaseInstance3DBoxes): Prediction of bboxes,
contains a tensor with shape (num_instances, C), where
C >= 7.
"""
assert self.with_bbox, 'Bbox head must be implemented in PartA2.'
assert self.with_semantic, 'Semantic head must be implemented' \
' in PartA2.'
batch_input_metas = [
data_samples.metainfo for data_samples in batch_data_samples
]
voxels_dict = feats_dict.pop('voxels_dict')
# TODO: Split predict semantic and bbox
results_list = self.predict_bbox(feats_dict, voxels_dict,
batch_input_metas, rpn_results_list,
self.test_cfg)
return results_list
def predict_bbox(self, feats_dict: Dict, voxel_dict: Dict,
batch_input_metas: List[dict],
rpn_results_list: InstanceList,
test_cfg: ConfigDict) -> InstanceList:
"""Perform forward propagation of the bbox head and predict detection
results on the features of the upstream network.
self.init_assigner_sampler() Args:
feats_dict (dict): Contains features from the first stage.
voxel_dict (dict): Contains information of voxels.
batch_input_metas (list[dict], Optional): Batch image meta info.
Defaults to None.
rpn_results_list (List[:obj:`InstancesData`]): Detection results
of rpn head.
test_cfg (Config): Test config.
Returns:
list[:obj:`InstanceData`]: Detection results of each sample
after the post process.
Each item usually contains following keys.
- scores_3d (Tensor): Classification scores, has a shape
(num_instances, )
- labels_3d (Tensor): Labels of bboxes, has a shape
(num_instances, ).
- bboxes_3d (BaseInstance3DBoxes): Prediction of bboxes,
contains a tensor with shape (num_instances, C), where
C >= 7.
"""
...
def loss(self, feats_dict: Dict, rpn_results_list: InstanceList,
batch_data_samples: SampleList, **kwargs) -> dict:
"""Perform forward propagation and loss calculation of the detection
roi on the features of the upstream network.
def _bbox_forward(self, seg_feats, part_feats, voxels_dict, rois):
"""Forward function of roi_extractor and bbox_head used in both
training and testing.
Args: Args:
seg_feats (torch.Tensor): Point-wise semantic features. feats_dict (dict): Contains features from the first stage.
part_feats (torch.Tensor): Point-wise part prediction features. rpn_results_list (List[:obj:`InstancesData`]): Detection results
voxels_dict (dict): Contains information of voxels. of rpn head.
rois (Tensor): Roi boxes. batch_data_samples (List[:obj:`Det3DDataSample`]): The Data
samples. It usually includes information such as
`gt_instance_3d`, `gt_panoptic_seg_3d` and `gt_sem_seg_3d`.
Returns: Returns:
dict: Contains predictions of bbox_head and dict[str, Tensor]: A dictionary of loss components
features of roi_extractor.
""" """
pooled_seg_feats = self.seg_roi_extractor(seg_feats, assert len(rpn_results_list) == len(batch_data_samples)
voxels_dict['voxel_centers'], losses = dict()
voxels_dict['coors'][..., 0], batch_gt_instances_3d = []
rois) batch_gt_instances_ignore = []
pooled_part_feats = self.part_roi_extractor( voxels_dict = feats_dict.pop('voxels_dict')
part_feats, voxels_dict['voxel_centers'], for data_sample in batch_data_samples:
voxels_dict['coors'][..., 0], rois) batch_gt_instances_3d.append(data_sample.gt_instances_3d)
cls_score, bbox_pred = self.bbox_head(pooled_seg_feats, if 'ignored_instances' in data_sample:
pooled_part_feats) batch_gt_instances_ignore.append(data_sample.ignored_instances)
else:
bbox_results = dict( batch_gt_instances_ignore.append(None)
cls_score=cls_score, if self.with_semantic:
bbox_pred=bbox_pred, semantic_results = self._semantic_forward_train(
pooled_seg_feats=pooled_seg_feats, feats_dict, voxels_dict, batch_gt_instances_3d)
pooled_part_feats=pooled_part_feats) losses.update(semantic_results.pop('loss_semantic'))
return bbox_results
sample_results = self._assign_and_sample(rpn_results_list,
batch_gt_instances_3d)
if self.with_bbox:
feats_dict.update(semantic_results)
bbox_results = self._bbox_forward_train(feats_dict, voxels_dict,
sample_results)
losses.update(bbox_results['loss_bbox'])
return losses
``` ```
Here we omit more details related to other functions. Please see the [code](https://github.com/open-mmlab/mmdetection3d/blob/master/mmdet3d/models/roi_heads/part_aggregation_roi_head.py) for more details. Here we omit more details related to other functions. Please see the [code](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/mmdet3d/models/roi_heads/part_aggregation_roi_head.py) for more details.
Last, the users need to add the module in Last, the users need to add the module in
`mmdet3d/models/bbox_heads/__init__.py` and `mmdet3d/models/roi_heads/__init__.py` thus the corresponding registry could find and load them. `mmdet3d/models/bbox_heads/__init__.py` and `mmdet3d/models/roi_heads/__init__.py` thus the corresponding registry could find and load them.
...@@ -404,14 +503,16 @@ model = dict( ...@@ -404,14 +503,16 @@ model = dict(
seg_score_thr=0.3, seg_score_thr=0.3,
num_classes=3, num_classes=3,
loss_seg=dict( loss_seg=dict(
type='FocalLoss', type='mmdet.FocalLoss',
use_sigmoid=True, use_sigmoid=True,
reduction='sum', reduction='sum',
gamma=2.0, gamma=2.0,
alpha=0.25, alpha=0.25,
loss_weight=1.0), loss_weight=1.0),
loss_part=dict( loss_part=dict(
type='CrossEntropyLoss', use_sigmoid=True, loss_weight=1.0)), type='mmdet.CrossEntropyLoss',
use_sigmoid=True,
loss_weight=1.0)),
seg_roi_extractor=dict( seg_roi_extractor=dict(
type='Single3DRoIAwareExtractor', type='Single3DRoIAwareExtractor',
roi_layer=dict( roi_layer=dict(
...@@ -419,7 +520,7 @@ model = dict( ...@@ -419,7 +520,7 @@ model = dict(
out_size=14, out_size=14,
max_pts_per_voxel=128, max_pts_per_voxel=128,
mode='max')), mode='max')),
part_roi_extractor=dict( bbox_roi_extractor=dict(
type='Single3DRoIAwareExtractor', type='Single3DRoIAwareExtractor',
roi_layer=dict( roi_layer=dict(
type='RoIAwarePool3d', type='RoIAwarePool3d',
...@@ -443,15 +544,15 @@ model = dict( ...@@ -443,15 +544,15 @@ model = dict(
roi_feat_size=14, roi_feat_size=14,
with_corner_loss=True, with_corner_loss=True,
loss_bbox=dict( loss_bbox=dict(
type='SmoothL1Loss', type='mmdet.SmoothL1Loss',
beta=1.0 / 9.0, beta=1.0 / 9.0,
reduction='sum', reduction='sum',
loss_weight=1.0), loss_weight=1.0),
loss_cls=dict( loss_cls=dict(
type='CrossEntropyLoss', type='mmdet.CrossEntropyLoss',
use_sigmoid=True, use_sigmoid=True,
reduction='sum', reduction='sum',
loss_weight=1.0))) loss_weight=1.0))),
... ...
) )
``` ```
......
docs/en/advanced_guides/customize_runtime.md
View file @ 328590d2
...@@ -2,16 +2,41 @@ ...@@ -2,16 +2,41 @@
## Customize optimization settings ## Customize optimization settings
### Customize optimizer supported by PyTorch Optimization related configuration is now all managed by `optim_wrapper` which usually has three fields: `optimizer`, `paramwise_cfg`, `clip_grad`. Please refer to [OptimWrapper](https://mmengine.readthedocs.io/en/latest/tutorials/optim_wrapper.md) for more detail. See the example below, where `Adamw` is used as an `optimizer`, the learning rate of the backbone is reduced by a factor of 10, and gradient clipping is added.
We already support to use all the optimizers implemented by PyTorch, and the only modification is to change the `optimizer` field of config files. ```python
For example, if you want to use `ADAM` (note that the performance could drop a lot), the modification could be as the following. optim_wrapper = dict(
type='OptimWrapper',
# optimizer
optimizer=dict(
type='AdamW',
lr=0.0001,
weight_decay=0.05,
eps=1e-8,
betas=(0.9, 0.999)),
# Parameter-level learning rate and weight decay settings
paramwise_cfg=dict(
custom_keys={
'backbone': dict(lr_mult=0.1, decay_mult=1.0),
},
norm_decay_mult=0.0),
# gradient clipping
clip_grad=dict(max_norm=0.01, norm_type=2))
```
### Customize optimizer supported by Pytorch
We already support to use all the optimizers implemented by PyTorch, and the only modification is to change the `optimizer` field in `optim_wrapper` field of config files. For example, if you want to use `ADAM` (note that the performance could drop a lot), the modification could be as the following.
```python ```python
optimizer = dict(type='Adam', lr=0.0003, weight_decay=0.0001) optim_wrapper = dict(
type='OptimWrapper',
optimizer=dict(type='Adam', lr=0.0003, weight_decay=0.0001))
``` ```
To modify the learning rate of the model, the users only need to modify the `lr` in the config of optimizer. The users can directly set arguments following the [API doc](https://pytorch.org/docs/stable/optim.html?highlight=optim#module-torch.optim) of PyTorch. To modify the learning rate of the model, the users only need to modify the `lr` in `optimizer`. The users can directly set arguments following the [API doc](https://pytorch.org/docs/stable/optim.html?highlight=optim#module-torch.optim) of PyTorch.
### Customize self-implemented optimizer ### Customize self-implemented optimizer
...@@ -20,18 +45,17 @@ To modify the learning rate of the model, the users only need to modify the `lr` ...@@ -20,18 +45,17 @@ To modify the learning rate of the model, the users only need to modify the `lr`
A customized optimizer could be defined as following. A customized optimizer could be defined as following.
Assume you want to add a optimizer named `MyOptimizer`, which has arguments `a`, `b`, and `c`. Assume you want to add a optimizer named `MyOptimizer`, which has arguments `a`, `b`, and `c`.
You need to create a new directory named `mmdet3d/core/optimizer`. You need to create a new directory named `mmdet3d/engine/optimizers`, and then implement the new optimizer in a file, e.g., in `mmdet3d/engine/optimizers/my_optimizer.py`:
And then implement the new optimizer in a file, e.g., in `mmdet3d/core/optimizer/my_optimizer.py`:
```python ```python
from mmcv.runner.optimizer import OPTIMIZERS from mmdet3d.registry import OPTIMIZERS
from torch.optim import Optimizer from torch.optim import Optimizer
@OPTIMIZERS.register_module() @OPTIMIZERS.register_module()
class MyOptimizer(Optimizer): class MyOptimizer(Optimizer):
def __init__(self, a, b, c) def __init__(self, a, b, c):
``` ```
...@@ -39,221 +63,250 @@ class MyOptimizer(Optimizer): ...@@ -39,221 +63,250 @@ class MyOptimizer(Optimizer):
To find the above module defined above, this module should be imported into the main namespace at first. There are two options to achieve it. To find the above module defined above, this module should be imported into the main namespace at first. There are two options to achieve it.
- Add `mmdet3d/core/optimizer/__init__.py` to import it. - Modify `mmdet3d/engine/optimizers/__init__.py` to import it.
The newly defined module should be imported in `mmdet3d/core/optimizer/__init__.py` so that the registry will The newly defined module should be imported in `mmdet3d/engine/optimizers/__init__.py` so that the registry will find the new module and add it:
find the new module and add it:
```python ```python
from .my_optimizer import MyOptimizer from .my_optimizer import MyOptimizer
__all__ = ['MyOptimizer']
```
You also need to import `optimizer` in `mmdet3d/core/__init__.py` by adding:
```python
from .optimizer import *
``` ```
Or use `custom_imports` in the config to manually import it - Use `custom_imports` in the config to manually import it
```python ```python
custom_imports = dict(imports=['mmdet3d.core.optimizer.my_optimizer'], allow_failed_imports=False) custom_imports = dict(imports=['mmdet3d.engine.optimizers.my_optimizer'], allow_failed_imports=False)
``` ```
The module `mmdet3d.core.optimizer.my_optimizer` will be imported at the beginning of the program and the class `MyOptimizer` is then automatically registered. The module `mmdet3d.engine.optimizers.my_optimizer` will be imported at the beginning of the program and the class `MyOptimizer` is then automatically registered.
Note that only the package containing the class `MyOptimizer` should be imported. Note that only the package containing the class `MyOptimizer` should be imported.
`mmdet3d.core.optimizer.my_optimizer.MyOptimizer` **cannot** be imported directly. `mmdet3d.engine.optimizers.my_optimizer.MyOptimizer` **cannot** be imported directly.
Actually users can use a totally different file directory structure in this importing method, as long as the module root can be located in `PYTHONPATH`. Actually users can use a totally different file directory structure with this importing method, as long as the module root is located in `PYTHONPATH`.
#### 3. Specify the optimizer in the config file #### 3. Specify the optimizer in the config file
Then you can use `MyOptimizer` in `optimizer` field of config files. Then you can use `MyOptimizer` in `optimizer` field in `optim_wrapper` field of config files. In the configs, the optimizers are defined by the field `optimizer` like the following:
In the configs, the optimizers are defined by the field `optimizer` like the following:
```python ```python
optimizer = dict(type='SGD', lr=0.02, momentum=0.9, weight_decay=0.0001) optim_wrapper = dict(
type='OptimWrapper',
optimizer=dict(type='SGD', lr=0.02, momentum=0.9, weight_decay=0.0001))
``` ```
To use your own optimizer, the field can be changed to To use your own optimizer, the field can be changed to
```python ```python
optimizer = dict(type='MyOptimizer', a=a_value, b=b_value, c=c_value) optim_wrapper = dict(
type='OptimWrapper',
optimizer=dict(type='MyOptimizer', a=a_value, b=b_value, c=c_value))
``` ```
### Customize optimizer constructor ### Customize optimizer wrapper constructor
Some models may have some parameter-specific settings for optimization, e.g. weight decay for BatchNorm layers. Some models may have some parameter-specific settings for optimization, e.g. weight decay for BatchNorm layers.
The users can tune those fine-grained parameters through customizing optimizer constructor. The users can do those fine-grained parameter tuning through customizing optimizer wrapper constructor.
```python ```python
from mmcv.utils import build_from_cfg from mmengine.optim import DefaultOptiWrapperConstructor
from mmcv.runner.optimizer import OPTIMIZER_BUILDERS, OPTIMIZERS from mmdet3d.registry import OPTIM_WRAPPER_CONSTRUCTORS
from mmengine.logging import MMLogger
from .my_optimizer import MyOptimizer from .my_optimizer import MyOptimizer
@OPTIMIZER_BUILDERS.register_module() @OPTIM_WRAPPER_CONSTRUCTORS.register_module()
class MyOptimizerConstructor(object): class MyOptimizerWrapperConstructor(DefaultOptimWrapperConstructor):
def __init__(self, optimizer_cfg, paramwise_cfg=None): def __init__(self,
optim_wrapper_cfg: dict,
paramwise_cfg: Optional[dict] = None):
def __call__(self, model): def __call__(self, model: nn.Module) -> OptimWrapper:
return my_optimizer return optim_wrapper
``` ```
The default optimizer constructor is implemented [here](https://github.com/open-mmlab/mmcv/blob/v1.3.7/mmcv/runner/optimizer/default_constructor.py#L11), which could also serve as a template for new optimizer constructor. The default optimizer wrapper constructor is implemented [here](https://github.com/open-mmlab/mmengine/blob/main/mmengine/optim/optimizer/default_constructor.py#L18), which could also serve as a template for the new optimizer wrapper constructor.
### Additional settings ### Additional settings
Tricks not implemented by the optimizer should be implemented through optimizer constructor (e.g., set parameter-wise learning rates) or hooks. We list some common settings that could stabilize the training or accelerate the training. Feel free to create PR, issue for more settings. Tricks not implemented by the optimizer should be implemented through optimizer wrapper constructor (e.g., set parameter-wise learning rates) or hooks. We list some common settings that could stabilize the training or accelerate the training. Feel free to create PR, issue for more settings.
- __Use gradient clip to stabilize training__: - __Use gradient clip to stabilize training__:
Some models need gradient clip to clip the gradients to stabilize the training process. An example is as below: Some models need gradient clip to clip the gradients to stabilize the training process. An example is as below:
```python ```python
optimizer_config = dict( optim_wrapper = dict(
_delete_=True, grad_clip=dict(max_norm=35, norm_type=2)) _delete_=True, clip_grad=dict(max_norm=35, norm_type=2))
``` ```
If your config inherits the base config which already sets the `optimizer_config`, you might need `_delete_=True` to override the unnecessary settings in the base config. See the [config documentation](https://mmdetection.readthedocs.io/en/latest/tutorials/config.html) for more details. If your config inherits the base config which already sets the `optim_wrapper`, you might need `_delete_=True` to override the unnecessary settings. See the [config documentation](https://mmdetection3d.readthedocs.io/en/latest/tutorials/config.html) for more details.
- __Use momentum schedule to accelerate model convergence__: - __Use momentum schedule to accelerate model convergence__:
We support momentum scheduler to modify model's momentum according to learning rate, which could make the model converge in a faster way. We support momentum scheduler to modify model's momentum according to learning rate, which could make the model converge in a faster way.
Momentum scheduler is usually used with LR scheduler, for example, the following config is used in 3D detection to accelerate convergence. Momentum scheduler is usually used with LR scheduler, for example, the following config is used in [3D detection](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/configs/_base_/schedules/cyclic_20e.py) to accelerate convergence.
For more details, please refer to the implementation of [CyclicLrUpdater](https://github.com/open-mmlab/mmcv/blob/v1.3.7/mmcv/runner/hooks/lr_updater.py#L358) and [CyclicMomentumUpdater](https://github.com/open-mmlab/mmcv/blob/v1.3.7/mmcv/runner/hooks/momentum_updater.py#L225). For more details, please refer to the implementation of [CosineAnnealingLR](https://github.com/open-mmlab/mmengine/blob/main/mmengine/optim/scheduler/lr_scheduler.py#L43) and [CosineAnnealingMomentum](https://github.com/open-mmlab/mmengine/blob/main/mmengine/optim/scheduler/momentum_scheduler.py#L71).
```python ```python
lr_config = dict( param_scheduler = [
policy='cyclic', # learning rate scheduler
target_ratio=(10, 1e-4), # During the first 8 epochs, learning rate increases from 0 to lr * 10
cyclic_times=1, # during the next 12 epochs, learning rate decreases from lr * 10 to lr * 1e-4
step_ratio_up=0.4, dict(
) type='CosineAnnealingLR',
momentum_config = dict( T_max=8,
policy='cyclic', eta_min=lr * 10,
target_ratio=(0.85 / 0.95, 1), begin=0,
cyclic_times=1, end=8,
step_ratio_up=0.4, by_epoch=True,
) convert_to_iter_based=True),
dict(
type='CosineAnnealingLR',
T_max=12,
eta_min=lr * 1e-4,
begin=8,
end=20,
by_epoch=True,
convert_to_iter_based=True),
# momentum scheduler
# During the first 8 epochs, momentum increases from 0 to 0.85 / 0.95
# during the next 12 epochs, momentum increases from 0.85 / 0.95 to 1
dict(
type='CosineAnnealingMomentum',
T_max=8,
eta_min=0.85 / 0.95,
begin=0,
end=8,
by_epoch=True,
convert_to_iter_based=True),
dict(
type='CosineAnnealingMomentum',
T_max=12,
eta_min=1,
begin=8,
end=20,
by_epoch=True,
convert_to_iter_based=True)
]
``` ```
## Customize training schedules ## Customize training schedules
By default we use step learning rate with 1x schedule, this calls [`StepLRHook`](https://github.com/open-mmlab/mmcv/blob/v1.3.7/mmcv/runner/hooks/lr_updater.py#L167) in MMCV. By default we use step learning rate with 1x schedule, this calls [MultiStepLR](https://github.com/open-mmlab/mmengine/blob/main/mmengine/optim/scheduler/lr_scheduler.py#L139) in MMEngine.
We support many other learning rate schedule [here](https://github.com/open-mmlab/mmcv/blob/v1.3.7/mmcv/runner/hooks/lr_updater.py), such as `CosineAnnealing` and `Poly` schedule. Here are some examples We support many other learning rate schedule [here](https://github.com/open-mmlab/mmengine/blob/main/mmengine/optim/scheduler/lr_scheduler.py), such as `CosineAnnealingLR` and `PolyLR` schedules. Here are some examples
- Poly schedule: - Poly schedule:
```python ```python
lr_config = dict(policy='poly', power=0.9, min_lr=1e-4, by_epoch=False) param_scheduler = [
dict(
type='PolyLR',
power=0.9,
eta_min=1e-4,
begin=0,
end=8,
by_epoch=True)]
``` ```
- ConsineAnnealing schedule: - ConsineAnnealing schedule:
```python ```python
lr_config = dict( param_scheduler = [
policy='CosineAnnealing', dict(
warmup='linear', type='CosineAnnealingLR',
warmup_iters=1000, T_max=8,
warmup_ratio=1.0 / 10, eta_min=lr * 1e-5,
min_lr_ratio=1e-5) begin=0,
end=8,
by_epoch=True)]
``` ```
## Customize workflow ## Customize train loop
Workflow is a list of (phase, epochs) to specify the running order and epochs. By default, `EpochBasedTrainLoop` is used in `train_cfg` and validation is done after every train epoch, as follows.
By default it is set to be
```python ```python
workflow = [('train', 1)] train_cfg = dict(type='EpochBasedTrainLoop', max_epochs=12, val_begin=1, val_interval=1)
``` ```
which means running 1 epoch for training. Actually, both [`IterBasedTrainLoop`](https://github.com/open-mmlab/mmengine/blob/main/mmengine/runner/loops.py#L183%5D) and [`EpochBasedTrainLoop`](https://github.com/open-mmlab/mmengine/blob/main/mmengine/runner/loops.py#L18) support dynamical interval, see the following example.
Sometimes user may want to check some metrics (e.g. loss, accuracy) about the model on the validate set.
In such case, we can set the workflow as
```python ```python
[('train', 1), ('val', 1)] # Before 365001th iteration, we do evaluation every 5000 iterations.
# After 365000th iteration, we do evaluation every 368750 iteraions,
# which means that we do evaluation at the end of training.
interval = 5000
max_iters = 368750
dynamic_intervals = [(max_iters // interval * interval + 1, max_iters)]
train_cfg = dict(
type='IterBasedTrainLoop',
max_iters=max_iters,
val_interval=interval,
dynamic_intervals=dynamic_intervals)
``` ```
so that 1 epoch for training and 1 epoch for validation will be run iteratively.
**Note**:
1. The parameters of model will not be updated during val epoch.
2. Keyword `max_epochs` in `runner` in the config only controls the number of training epochs and will not affect the validation workflow.
3. Workflows `[('train', 1), ('val', 1)]` and `[('train', 1)]` will not change the behavior of `EvalHook` because `EvalHook` is called by `after_train_epoch` and validation workflow only affect hooks that are called through `after_val_epoch`. Therefore, the only difference between `[('train', 1), ('val', 1)]` and `[('train', 1)]` is that the runner will calculate losses on validation set after each training epoch.
## Customize hooks ## Customize hooks
### Customize self-implemented hooks ### Customize self-implemented hooks
#### 1. Implement a new hook #### 1. Implement a new hook
There are some occasions when the users might need to implement a new hook. MMDetection supports customized hooks in training (#3395) since v2.3.0. Thus the users could implement a hook directly in mmdet or their mmdet-based codebases and use the hook by only modifying the config in training. MMEngine provides many useful [hooks](https://github.com/open-mmlab/mmengine/blob/main/docs/en/tutorials/hook.md), but there are some occasions when the users might need to implement a new hook. MMDetection3D supports customized hooks in training based on MMEngine after v1.1.0rc0. Thus the users could implement a hook directly in mmdet3d or their mmdet3d-based codebases and use the hook by only modifying the config in training.
Before v2.3.0, the users need to modify the code to get the hook registered before training starts.
Here we give an example of creating a new hook in mmdet3d and using it in training. Here we give an example of creating a new hook in mmdet3d and using it in training.
```python ```python
from mmcv.runner import HOOKS, Hook from mmengine.hooks import HOOKS, Hook
@HOOKS.register_module() @HOOKS.register_module()
class MyHook(Hook): class MyHook(Hook):
def __init__(self, a, b): def __init__(self, a, b):
pass
def before_run(self, runner): def before_run(self, runner) -> None:
pass
def after_run(self, runner): def after_run(self, runner) -> None:
pass
def before_epoch(self, runner): def before_train(self, runner) -> None:
pass
def after_epoch(self, runner): def after_train(self, runner) -> None:
pass
def before_iter(self, runner): def before_train_epoch(self, runner) -> None:
pass
def after_iter(self, runner): def after_train_epoch(self, runner) -> None:
pass
def before_train_iter(self,
runner,
batch_idx: int,
data_batch: DATA_BATCH = None) -> None:
def after_train_iter(self,
runner,
batch_idx: int,
data_batch: DATA_BATCH = None,
outputs: Optional[dict] = None) -> None:
``` ```
Depending on the functionality of the hook, the users need to specify what the hook will do at each stage of the training in `before_run`, `after_run`, `before_epoch`, `after_epoch`, `before_iter`, and `after_iter`. Depending on the functionality of the hook, users need to specify what the hook will do at each stage of the training in `before_run`, `after_run`, `before_train`, `after_train` , `before_train_epoch`, `after_train_epoch`, `before_train_iter`, and `after_train_iter`. There are more points where hooks can be inserted, refer to [base hook class](https://github.com/open-mmlab/mmengine/blob/main/mmengine/hooks/hook.py#L9) for more detail.
#### 2. Register the new hook #### 2. Register the new hook
Then we need to make `MyHook` imported. Assuming the hook is in `mmdet3d/core/utils/my_hook.py` there are two ways to do that: Then we need to make `MyHook` imported. Assuming the file is in `mmdet3d/engine/hooks/my_hook.py` there are two ways to do that:
- Modify `mmdet3d/core/utils/__init__.py` to import it. - Modify `mmdet3d/engine/hooks/__init__.py` to import it.
The newly defined module should be imported in `mmdet3d/core/utils/__init__.py` so that the registry will The newly defined module should be imported in `mmdet3d/engine/hooks/__init__.py` so that the registry will find the new module and add it:
find the new module and add it:
```python ```python
from .my_hook import MyHook from .my_hook import MyHook
__all__ = [..., 'MyHook']
``` ```
Or use `custom_imports` in the config to manually import it - Use `custom_imports` in the config to manually import it
```python ```python
custom_imports = dict(imports=['mmdet3d.core.utils.my_hook'], allow_failed_imports=False) custom_imports = dict(imports=['mmdet3d.engine.hooks.my_hook'], allow_failed_imports=False)
``` ```
#### 3. Modify the config #### 3. Modify the config
...@@ -264,7 +317,7 @@ custom_hooks = [ ...@@ -264,7 +317,7 @@ custom_hooks = [
] ]
``` ```
You can also set the priority of the hook by setting key `priority` to `'NORMAL'` or `'HIGHEST'` as below You can also set the priority of the hook by adding key `priority` to `'NORMAL'` or `'HIGHEST'` as below
```python ```python
custom_hooks = [ custom_hooks = [
...@@ -274,60 +327,39 @@ custom_hooks = [ ...@@ -274,60 +327,39 @@ custom_hooks = [
By default the hook's priority is set as `NORMAL` during registration. By default the hook's priority is set as `NORMAL` during registration.
### Use hooks implemented in MMCV ### Use hooks implemented in MMEngine
If the hook is already implemented in MMCV, you can directly modify the config to use the hook as below If the hook is already implemented in MMEngine, you can directly modify the config to use the hook as below
```python
custom_hooks = [
dict(type='MyHook', a=a_value, b=b_value, priority='NORMAL')
]
```
### Modify default runtime hooks ### Modify default runtime hooks
There are some common hooks that are not registered through `custom_hooks`, they are There are some common hooks that are registered through `default_hooks`, they are
- log_config
- checkpoint_config
- evaluation
- lr_config
- optimizer_config
- momentum_config
In those hooks, only the logger hook has the `VERY_LOW` priority, others' priority are `NORMAL`.
The above-mentioned tutorials already covers how to modify `optimizer_config`, `momentum_config`, and `lr_config`.
Here we reveal what we can do with `log_config`, `checkpoint_config`, and `evaluation`.
#### Checkpoint config - `IterTimerHook`: A hook that logs 'data_time' for loading data and 'time' for a model train step.
- `LoggerHook`: A hook that Collect logs from different components of `Runner` and write them to terminal, JSON file, tensorboard and wandb .etc.
The MMCV runner will use `checkpoint_config` to initialize [`CheckpointHook`](https://github.com/open-mmlab/mmcv/blob/v1.3.7/mmcv/runner/hooks/checkpoint.py#L9). - `ParamSchedulerHook`: A hook to update some hyper-parameters in optimizer, e.g., learning rate and momentum.
- `CheckpointHook`: A hook that saves checkpoints periodically.
```python - `DistSamplerSeedHook`: A hook that sets the seed for sampler and batch_sampler.
checkpoint_config = dict(interval=1)
```
The users could set `max_keep_ckpts` to save only small number of checkpoints or decide whether to store state dict of optimizer by `save_optimizer`. More details of the arguments are [here](https://mmcv.readthedocs.io/en/latest/api.html#mmcv.runner.CheckpointHook). `IterTimerHook`, `ParamSchedulerHook` and `DistSamplerSeedHook` are simple and no need to be modified usually, so here we reveals how what we can do with `LoggerHook`, `CheckpointHook` and `DetVisualizationHook`.
#### Log config #### CheckpointHook
The `log_config` wraps multiple logger hooks and enables to set intervals. Now MMCV supports `WandbLoggerHook`, `MlflowLoggerHook`, and `TensorboardLoggerHook`. Except saving checkpoints periodically, [`CheckpointHook`](https://github.com/open-mmlab/mmengine/blob/main/mmengine/hooks/checkpoint_hook.py#L19) provides other options such as `max_keep_ckpts`, `save_optimizer` and etc. The users could set `max_keep_ckpts` to only save small number of checkpoints or decide whether to store state dict of optimizer by `save_optimizer`. More details of the arguments are [here](https://github.com/open-mmlab/mmengine/blob/main/mmengine/hooks/checkpoint_hook.py#L19)
The detailed usages can be found in the [docs](https://mmcv.readthedocs.io/en/latest/api.html#mmcv.runner.LoggerHook).
```python ```python
log_config = dict( default_hooks = dict(
interval=50, checkpoint=dict(
hooks=[ type='CheckpointHook',
dict(type='TextLoggerHook'), interval=1,
dict(type='TensorboardLoggerHook') max_keep_ckpts=3,
]) save_optimizer=True))
``` ```
#### Evaluation config #### LoggerHook
The config of `evaluation` will be used to initialize the [`EvalHook`](https://github.com/open-mmlab/mmdetection/blob/v2.13.0/mmdet/core/evaluation/eval_hooks.py#L9). The `LoggerHook` enables setting intervals. Detailed instructions can be found in the [docstring](https://github.com/open-mmlab/mmengine/blob/main/mmengine/hooks/logger_hook.py#L18).
Except the key `interval`, other arguments such as `metric` will be passed to the `dataset.evaluate()`.
```python ```python
evaluation = dict(interval=1, metric='bbox') default_hooks = dict(logger=dict(type='LoggerHook', interval=50))
``` ```
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment