Skip to content

GitLab

"format/byte_test.go" did not exist on "f00d359a6729ab52a300b8ef8a738f3faa1006af"
Unverified Commit 72ac2a5e authored by Xiang Xu's avatar Xiang Xu Committed by GitHub
Browse files

[Docs] Update `customize_models` and `customize_runtime` (#2194) 

* Update customize_models.md

* Update customize_runtime.md

* Update customize_models.md

* Update customize_runtime.md

* fix typo

* fix typo

* fix
parent e1f695d3
Show whitespace changes
Inline Side-by-side
Showing with 256 additions and 190 deletions
docs/en/advanced_guides/customize_models.md
View file @ 72ac2a5e
# Customize Models
We basically categorize model components into 6 types.
We basically categorize model components into 6 types:
- encoder: including voxel layer, voxel encoder and middle encoder used in voxel-based methods before backbone, e.g., HardVFE and PointPillarsScatter.
- backbone: usually an FCN network to extract feature maps, e.g., ResNet, SECOND.
- neck: the component between backbones and heads, e.g., FPN, SECONDFPN.
- head: the component for specific tasks, e.g., bbox prediction and mask prediction.
- RoI extractor: the part for extracting RoI features from feature maps, e.g., H3DRoIHead and PartAggregationROIHead.
- loss: the component in heads for calculating losses, e.g., FocalLoss, L1Loss, and GHMLoss.
- encoder: Including voxel encoder and middle encoder used in voxel-based methods before backbone, e.g., `HardVFE` and `PointPillarsScatter`.
- backbone: Usually an FCN network to extract feature maps, e.g., `ResNet`, `SECOND`.
- neck: The component between backbones and heads, e.g., `FPN`, `SECONDFPN`.
- head: The component for specific tasks, e.g., `bbox prediction` and `mask prediction`.
- RoI extractor: The part for extracting RoI features from feature maps, e.g., `H3DRoIHead` and `PartAggregationROIHead`.
- loss: The component in heads for calculating losses, e.g., `FocalLoss`, `L1Loss`, and `GHMLoss`.
## Develop new components
......@@ -15,7 +15,7 @@ We basically categorize model components into 6 types.
Here we show how to develop new components with an example of HardVFE.
#### 1. Define a new voxel encoder (e.g. HardVFE: Voxel feature encoder used in DV-SECOND)
#### 1. Define a new voxel encoder (e.g. HardVFE: Voxel feature encoder used in HV-SECOND)
Create a new file `mmdet3d/models/voxel_encoders/voxel_encoder.py`.
......@@ -37,7 +37,7 @@ class HardVFE(nn.Module):
#### 2. Import the module
You can either add the following line to `mmdet3d/models/voxel_encoders/__init__.py`
You can either add the following line to `mmdet3d/models/voxel_encoders/__init__.py`:
```python
from .voxel_encoder import HardVFE
......@@ -47,7 +47,7 @@ or alternatively add
```python
custom_imports = dict(
imports=['mmdet3d.models.voxel_encoders.HardVFE'],
imports=['mmdet3d.models.voxel_encoders.voxel_encoder'],
allow_failed_imports=False)
```
......@@ -61,8 +61,9 @@ model = dict(
voxel_encoder=dict(
type='HardVFE',
arg1=xxx,
arg2=xxx),
arg2=yyy),
...
)
```
### Add a new backbone
......@@ -74,7 +75,7 @@ Here we show how to develop new components with an example of [SECOND](https://w
Create a new file `mmdet3d/models/backbones/second.py`.
```python
import torch.nn as nn
from mmengine.model import BaseModule
from mmdet3d.registry import MODELS
......@@ -91,7 +92,7 @@ class SECOND(BaseModule):
#### 2. Import the module
You can either add the following line to `mmdet3d/models/backbones/__init__.py`
You can either add the following line to `mmdet3d/models/backbones/__init__.py`:
```python
from .second import SECOND
......@@ -115,19 +116,23 @@ model = dict(
backbone=dict(
type='SECOND',
arg1=xxx,
arg2=xxx),
arg2=yyy),
...
)
```
### Add new necks
### Add a new neck
#### 1. Define a neck (e.g. SECONDFPN)
#### 1. Define a new neck (e.g. SECONDFPN)
Create a new file `mmdet3d/models/necks/second_fpn.py`.
```python
from mmengine.model import BaseModule
from mmdet3d.registry import MODELS
@MODELS.register_module()
class SECONDFPN(BaseModule):
......@@ -142,14 +147,14 @@ class SECONDFPN(BaseModule):
init_cfg=None):
pass
def forward(self, X):
def forward(self, x):
# implementation is ignored
pass
```
#### 2. Import the module
You can either add the following line to `mmdet3D/models/necks/__init__.py`,
You can either add the following line to `mmdet3d/models/necks/__init__.py`:
```python
from .second_fpn import SECONDFPN
......@@ -163,7 +168,7 @@ custom_imports = dict(
allow_failed_imports=False)
```
to the config file and avoid modifying the original code.
to the config file to avoid modifying the original code.
#### 3. Use the neck in your config file
......@@ -176,22 +181,25 @@ model = dict(
upsample_strides=[1, 2, 4],
out_channels=[128, 128, 128]),
...
)
```
### Add new heads
### Add a new head
Here we show how to develop a new head with the example of [PartA2 Head](https://arxiv.org/abs/1907.03670) as the following.
**Note**: Here the example of PartA2 RoI Head is used in the second stage. For one-stage heads, please refer to examples in `mmdet3d/models/dense_heads/`. They are more commonly used in 3D detection for autonomous driving due to its simplicity and high efficiency.
**Note**: Here the example of `PartA2 RoI Head` is used in the second stage. For one-stage heads, please refer to examples in `mmdet3d/models/dense_heads/`. They are more commonly used in 3D detection for autonomous driving due to its simplicity and high efficiency.
First, add a new bbox head in `mmdet3d/models/roi_heads/bbox_heads/parta2_bbox_head.py`.
PartA2 RoI Head implements a new bbox head for object detection.
`PartA2 RoI Head` implements a new bbox head for object detection.
To implement a bbox head, basically we need to implement two functions of the new module as the following. Sometimes other related functions like `loss` and `get_targets` are also required.
```python
from mmdet3d.registry import MODELS
from mmengine.model import BaseModule
from mmdet3d.registry import MODELS
@MODELS.register_module()
class PartA2BboxHead(BaseModule):
"""PartA2 RoI head."""
......@@ -224,14 +232,16 @@ class PartA2BboxHead(BaseModule):
super(PartA2BboxHead, self).__init__(init_cfg=init_cfg)
def forward(self, seg_feats, part_feats):
pass
```
Second, implement a new RoI Head if it is necessary. We plan to inherit the new `PartAggregationROIHead` from `Base3DRoIHead`. We can find that a `Base3DRoIHead` already implements the following functions.
```python
from mmdet3d.registry import MODELS, TASK_UTILS
from mmdet.models.roi_heads import BaseRoIHead
from mmdet3d.registry import MODELS, TASK_UTILS
class Base3DRoIHead(BaseRoIHead):
"""Base class for 3d RoIHeads."""
......@@ -282,7 +292,6 @@ class Base3DRoIHead(BaseRoIHead):
"""Initialize mask head, skip since ``PartAggregationROIHead`` does not
have one."""
pass
```
Double Head's modification is mainly in the bbox_forward logic, and it inherits other logics from the `Base3DRoIHead`.
......@@ -291,14 +300,14 @@ In the `mmdet3d/models/roi_heads/part_aggregation_roi_head.py`, we implement the
```python
from typing import Dict, List, Tuple
from mmcv import ConfigDict
from mmdet.models.task_modules import AssignResult, SamplingResult
from mmengine import ConfigDict
from torch import Tensor
from torch.nn import functional as F
from mmdet3d.registry import MODELS
from mmdet3d.structures import bbox3d2roi
from mmdet3d.utils import InstanceList
from mmdet.models.task_modules import AssignResult, SamplingResult
from ...structures.det3d_data_sample import SampleList
from .base_3droi_head import Base3DRoIHead
......@@ -477,18 +486,19 @@ class PartAggregationROIHead(Base3DRoIHead):
Here we omit more details related to other functions. Please see the [code](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/mmdet3d/models/roi_heads/part_aggregation_roi_head.py) for more details.
Last, the users need to add the module in
`mmdet3d/models/bbox_heads/__init__.py` and `mmdet3d/models/roi_heads/__init__.py` thus the corresponding registry could find and load them.
`mmdet3d/models/roi_heads/bbox_heads/__init__.py` and `mmdet3d/models/roi_heads/__init__.py` thus the corresponding registry could find and load them.
Alternatively, the users can add
```python
custom_imports=dict(
imports=['mmdet3d.models.roi_heads.part_aggregation_roi_head', 'mmdet3d.models.roi_heads.bbox_heads.parta2_bbox_head'])
imports=['mmdet3d.models.roi_heads.part_aggregation_roi_head', 'mmdet3d.models.roi_heads.bbox_heads.parta2_bbox_head'],
allow_failed_imports=False)
```
to the config file and achieve the same goal.
The config file of PartAggregationROIHead is as the following
The config file of `PartAggregationROIHead` is as the following:
```python
model = dict(
......@@ -554,25 +564,26 @@ model = dict(
reduction='sum',
loss_weight=1.0))),
...
)
)
```
Since MMDetection 2.0, the config system supports to inherit configs such that the users can focus on the modification.
The second stage of PartA2 Head mainly uses a new `PartAggregationROIHead` and a new
`PartA2BboxHead`, the arguments are set according to the `__init__` function of each module.
### Add new loss
### Add a new loss
Assume you want to add a new loss as `MyLoss`, for bounding box regression.
To add a new loss function, the users need implement it in `mmdet3d/models/losses/my_loss.py`.
The decorator `weighted_loss` enable the loss to be weighted for each element.
Assume you want to add a new loss as `MyLoss` for bounding box regression.
To add a new loss function, the users need to implement it in `mmdet3d/models/losses/my_loss.py`.
The decorator `weighted_loss` enables the loss to be weighted for each element.
```python
import torch
import torch.nn as nn
from mmdet.models.losses.utils import weighted_loss
from mmdet3d.registry import MODELS
from mmdet.models.losses.utils import weighted_loss
@weighted_loss
def my_loss(pred, target):
......@@ -580,6 +591,7 @@ def my_loss(pred, target):
loss = torch.abs(pred - target)
return loss
@MODELS.register_module()
class MyLoss(nn.Module):
......@@ -606,21 +618,21 @@ Then the users need to add it in the `mmdet3d/models/losses/__init__.py`.
```python
from .my_loss import MyLoss, my_loss
```
Alternatively, you can add
```python
custom_imports=dict(
imports=['mmdet3d.models.losses.my_loss'])
imports=['mmdet3d.models.losses.my_loss'],
allow_failed_imports=False)
```
to the config file and achieve the same goal.
To use it, modify the `loss_xxx` field.
Since MyLoss is for regression, you need to modify the `loss_bbox` field in the head.
To use it, users should modify the `loss_xxx` field.
Since `MyLoss` is for regression, you need to modify the `loss_bbox` field in the head.
```python
loss_bbox=dict(type='MyLoss', loss_weight=1.0))
loss_bbox=dict(type='MyLoss', loss_weight=1.0)
```
docs/en/advanced_guides/customize_runtime.md
View file @ 72ac2a5e
......@@ -2,7 +2,7 @@
## Customize optimization settings
Optimization related configuration is now all managed by `optim_wrapper` which usually has three fields: `optimizer`, `paramwise_cfg`, `clip_grad`. Please refer to [OptimWrapper](https://mmengine.readthedocs.io/en/latest/tutorials/optim_wrapper.md) for more detail. See the example below, where `Adamw` is used as an `optimizer`, the learning rate of the backbone is reduced by a factor of 10, and gradient clipping is added.
Optimization related configuration is now all managed by `optim_wrapper` which usually has three fields: `optimizer`, `paramwise_cfg`, `clip_grad`. Please refer to [OptimWrapper](https://mmengine.readthedocs.io/en/latest/tutorials/optim_wrapper.html) for more details. See the example below, where `AdamW` is used as an `optimizer`, the learning rate of the backbone is reduced by a factor of 10, and gradient clipping is added.
```python
optim_wrapper = dict(
......@@ -28,7 +28,7 @@ optim_wrapper = dict(
### Customize optimizer supported by PyTorch
We already support to use all the optimizers implemented by PyTorch, and the only modification is to change the `optimizer` field in `optim_wrapper` field of config files. For example, if you want to use `ADAM` (note that the performance could drop a lot), the modification could be as the following.
We already support to use all the optimizers implemented by PyTorch, and the only modification is to change the `optimizer` field in `optim_wrapper` field of config files. For example, if you want to use `Adam` (note that the performance could drop a lot), the modification could be as the following:
```python
optim_wrapper = dict(
......@@ -42,21 +42,22 @@ To modify the learning rate of the model, the users only need to modify the `lr`
#### 1. Define a new optimizer
A customized optimizer could be defined as following.
A customized optimizer could be defined as following:
Assume you want to add a optimizer named `MyOptimizer`, which has arguments `a`, `b`, and `c`.
You need to create a new directory named `mmdet3d/engine/optimizers`, and then implement the new optimizer in a file, e.g., in `mmdet3d/engine/optimizers/my_optimizer.py`:
```python
from mmdet3d.registry import OPTIMIZERS
from torch.optim import Optimizer
from mmdet3d.registry import OPTIMIZERS
@OPTIMIZERS.register_module()
class MyOptimizer(Optimizer):
def __init__(self, a, b, c):
pass
```
#### 2. Add the optimizer to registry
......@@ -67,21 +68,21 @@ To find the above module defined above, this module should be imported into the
The newly defined module should be imported in `mmdet3d/engine/optimizers/__init__.py` so that the registry will find the new module and add it:
```python
from .my_optimizer import MyOptimizer
```
```python
from .my_optimizer import MyOptimizer
```
- Use `custom_imports` in the config to manually import it
- Use `custom_imports` in the config to manually import it.
```python
custom_imports = dict(imports=['mmdet3d.engine.optimizers.my_optimizer'], allow_failed_imports=False)
```
```python
custom_imports = dict(imports=['mmdet3d.engine.optimizers.my_optimizer'], allow_failed_imports=False)
```
The module `mmdet3d.engine.optimizers.my_optimizer` will be imported at the beginning of the program and the class `MyOptimizer` is then automatically registered.
Note that only the package containing the class `MyOptimizer` should be imported.
`mmdet3d.engine.optimizers.my_optimizer.MyOptimizer` **cannot** be imported directly.
The module `mmdet3d.engine.optimizers.my_optimizer` will be imported at the beginning of the program and the class `MyOptimizer` is then automatically registered.
Note that only the package containing the class `MyOptimizer` should be imported.
`mmdet3d.engine.optimizers.my_optimizer.MyOptimizer` **cannot** be imported directly.
Actually users can use a totally different file directory structure with this importing method, as long as the module root is located in `PYTHONPATH`.
Actually users can use a totally different file directory structure with this importing method, as long as the module root is located in `PYTHONPATH`.
#### 3. Specify the optimizer in the config file
......@@ -93,7 +94,7 @@ optim_wrapper = dict(
optimizer=dict(type='SGD', lr=0.02, momentum=0.9, weight_decay=0.0001))
```
To use your own optimizer, the field can be changed to
To use your own optimizer, the field can be changed to:
```python
optim_wrapper = dict(
......@@ -107,7 +108,7 @@ Some models may have some parameter-specific settings for optimization, e.g. wei
The users can do those fine-grained parameter tuning through customizing optimizer wrapper constructor.
```python
from mmengine.optim import DefaultOptiWrapperConstructor
from mmengine.optim import DefaultOptimWrapperConstructor
from mmdet3d.registry import OPTIM_WRAPPER_CONSTRUCTORS
from .my_optimizer import MyOptimizer
......@@ -119,11 +120,11 @@ class MyOptimizerWrapperConstructor(DefaultOptimWrapperConstructor):
def __init__(self,
optim_wrapper_cfg: dict,
paramwise_cfg: Optional[dict] = None):
pass
def __call__(self, model: nn.Module) -> OptimWrapper:
return optim_wrapper
```
The default optimizer wrapper constructor is implemented [here](https://github.com/open-mmlab/mmengine/blob/main/mmengine/optim/optimizer/default_constructor.py#L18), which could also serve as a template for the new optimizer wrapper constructor.
......@@ -140,11 +141,11 @@ Tricks not implemented by the optimizer should be implemented through optimizer
_delete_=True, clip_grad=dict(max_norm=35, norm_type=2))
```
If your config inherits the base config which already sets the `optim_wrapper`, you might need `_delete_=True` to override the unnecessary settings. See the [config documentation](https://mmdetection3d.readthedocs.io/en/latest/tutorials/config.html) for more details.
If your config inherits the base config which already sets the `optim_wrapper`, you might need `_delete_=True` to override the unnecessary settings. See the [config documentation](https://mmdetection3d.readthedocs.io/en/dev-1.x/user_guides/config.html) for more details.
- __Use momentum schedule to accelerate model convergence__:
We support momentum scheduler to modify model's momentum according to learning rate, which could make the model converge in a faster way.
Momentum scheduler is usually used with LR scheduler, for example, the following config is used in [3D detection](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/configs/_base_/schedules/cyclic_20e.py) to accelerate convergence.
Momentum scheduler is usually used with LR scheduler, for example, the following config is used in [3D detection](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/configs/_base_/schedules/cyclic-20e.py) to accelerate convergence.
For more details, please refer to the implementation of [CosineAnnealingLR](https://github.com/open-mmlab/mmengine/blob/main/mmengine/optim/scheduler/lr_scheduler.py#L43) and [CosineAnnealingMomentum](https://github.com/open-mmlab/mmengine/blob/main/mmengine/optim/scheduler/momentum_scheduler.py#L71).
```python
......@@ -192,8 +193,8 @@ Tricks not implemented by the optimizer should be implemented through optimizer
## Customize training schedules
By default we use step learning rate with 1x schedule, this calls [`MultiStepLR`](https://github.com/open-mmlab/mmengine/blob/main/mmengine/optim/scheduler/lr_scheduler.py#L139) in MMEngine.
We support many other learning rate schedule [here](https://github.com/open-mmlab/mmengine/blob/main/mmengine/optim/scheduler/lr_scheduler.py), such as `CosineAnnealingLR` and `PolyLR` schedules. Here are some examples
By default we use step learning rate with 1x schedule, this calls [`MultiStepLR`](https://github.com/open-mmlab/mmengine/blob/main/mmengine/optim/scheduler/lr_scheduler.py#L144) in MMEngine.
We support many other learning rate schedule [here](https://github.com/open-mmlab/mmengine/blob/main/mmengine/optim/scheduler/lr_scheduler.py), such as `CosineAnnealingLR` and `PolyLR` schedules. Here are some examples:
- Poly schedule:
......@@ -208,7 +209,7 @@ We support many other learning rate schedule [here](https://github.com/open-mmla
by_epoch=True)]
```
- ConsineAnnealing schedule:
- CosineAnnealing schedule:
```python
param_scheduler = [
......@@ -223,13 +224,13 @@ We support many other learning rate schedule [here](https://github.com/open-mmla
## Customize train loop
By default, `EpochBasedTrainLoop` is used in `train_cfg` and validation is done after every train epoch, as follows.
By default, `EpochBasedTrainLoop` is used in `train_cfg` and validation is done after every train epoch, as follows:
```python
train_cfg = dict(type='EpochBasedTrainLoop', max_epochs=12, val_begin=1, val_interval=1)
```
Actually, both [`IterBasedTrainLoop`](https://github.com/open-mmlab/mmengine/blob/main/mmengine/runner/loops.py#L183%5D) and [`EpochBasedTrainLoop`](https://github.com/open-mmlab/mmengine/blob/main/mmengine/runner/loops.py#L18) support dynamical interval, see the following example.
Actually, both [`IterBasedTrainLoop`](https://github.com/open-mmlab/mmengine/blob/main/mmengine/runner/loops.py#L185) and [`EpochBasedTrainLoop`](https://github.com/open-mmlab/mmengine/blob/main/mmengine/runner/loops.py#L18) support dynamic interval, see the following example:
```python
# Before 365001th iteration, we do evaluation every 5000 iterations.
......@@ -252,7 +253,7 @@ train_cfg = dict(
#### 1. Implement a new hook
MMEngine provides many useful [hooks](https://github.com/open-mmlab/mmengine/blob/main/docs/en/tutorials/hook.md), but there are some occasions when the users might need to implement a new hook. MMDetection3D supports customized hooks in training based on MMEngine after v1.1.0rc0. Thus the users could implement a hook directly in mmdet3d or their mmdet3d-based codebases and use the hook by only modifying the config in training.
MMEngine provides many useful [hooks](https://mmengine.readthedocs.io/en/latest/tutorials/hook.html), but there are some occasions when the users might need to implement a new hook. MMDetection3D supports customized hooks in training based on MMEngine after v1.1.0rc0. Thus the users could implement a hook directly in mmdet3d or their mmdet3d-based codebases and use the hook by only modifying the config in training.
Here we give an example of creating a new hook in mmdet3d and using it in training.
```python
......@@ -290,25 +291,25 @@ class MyHook(Hook):
outputs: Optional[dict] = None) -> None:
```
Depending on the functionality of the hook, users need to specify what the hook will do at each stage of the training in `before_run`, `after_run`, `before_train`, `after_train` , `before_train_epoch`, `after_train_epoch`, `before_train_iter`, and `after_train_iter`. There are more points where hooks can be inserted, refer to [base hook class](https://github.com/open-mmlab/mmengine/blob/main/mmengine/hooks/hook.py#L9) for more detail.
Depending on the functionality of the hook, users need to specify what the hook will do at each stage of the training in `before_run`, `after_run`, `before_train`, `after_train`, `before_train_epoch`, `after_train_epoch`, `before_train_iter`, and `after_train_iter`. There are more points where hooks can be inserted, refer to [base hook class](https://github.com/open-mmlab/mmengine/blob/main/mmengine/hooks/hook.py#L9) for more details.
#### 2. Register the new hook
Then we need to make `MyHook` imported. Assuming the file is in `mmdet3d/engine/hooks/my_hook.py` there are two ways to do that:
Then we need to make `MyHook` imported. Assuming the file is in `mmdet3d/engine/hooks/my_hook.py`, there are two ways to do that:
- Modify `mmdet3d/engine/hooks/__init__.py` to import it.
The newly defined module should be imported in `mmdet3d/engine/hooks/__init__.py` so that the registry will find the new module and add it:
```python
from .my_hook import MyHook
```
```python
from .my_hook import MyHook
```
- Use `custom_imports` in the config to manually import it
- Use `custom_imports` in the config to manually import it.
```python
custom_imports = dict(imports=['mmdet3d.engine.hooks.my_hook'], allow_failed_imports=False)
```
```python
custom_imports = dict(imports=['mmdet3d.engine.hooks.my_hook'], allow_failed_imports=False)
```
#### 3. Modify the config
......@@ -318,7 +319,7 @@ custom_hooks = [
]
```
You can also set the priority of the hook by adding key `priority` to `'NORMAL'` or `'HIGHEST'` as below
You can also set the priority of the hook by adding key `priority` to `'NORMAL'` or `'HIGHEST'` as below:
```python
custom_hooks = [
......@@ -328,25 +329,36 @@ custom_hooks = [
By default the hook's priority is set as `NORMAL` during registration.
### Use hooks implemented in MMEngine
### Use hooks implemented in MMDetection3D
If the hook is already implemented in MMDetection3D, you can directly modify the config to use the hook as below.
If the hook is already implemented in MMEngine, you can directly modify the config to use the hook as below
#### Example: `DisableObjectSampleHook`
We implement a customized hook named [DisableObjectSampleHook](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/mmdet3d/engine/hooks/disable_object_sample_hook.py) to disable `ObjectSample` augmentation during training after specified epoch.
We can set it in the config file if needed:
```python
custom_hooks = [dict(type='DisableObjectSampleHook', disable_after_epoch=15)]
```
### Modify default runtime hooks
There are some common hooks that are registered through `default_hooks`, they are
- `IterTimerHook`: A hook that logs 'data_time' for loading data and 'time' for a model train step.
- `LoggerHook`: A hook that Collect logs from different components of `Runner` and write them to terminal, JSON file, tensorboard and wandb .etc.
- `ParamSchedulerHook`: A hook to update some hyper-parameters in optimizer, e.g., learning rate and momentum.
- `IterTimerHook`: A hook that logs 'data_time' for loading data and 'time' for a model training step.
- `LoggerHook`: A hook that collects logs from different components of `Runner` and writes them to terminal, json file, tensorboard and wandb etc.
- `ParamSchedulerHook`: A hook that updates some hyper-parameters in optimizer, e.g., learning rate and momentum.
- `CheckpointHook`: A hook that saves checkpoints periodically.
- `DistSamplerSeedHook`: A hook that sets the seed for sampler and batch_sampler.
- `Det3DVisualizationHook`: A hook used to visualize validation and testing process prediction results.
`IterTimerHook`, `ParamSchedulerHook` and `DistSamplerSeedHook` are simple and no need to be modified usually, so here we reveals how what we can do with `LoggerHook`, `CheckpointHook` and `Det3DVisualizationHook`.
`IterTimerHook`, `ParamSchedulerHook` and `DistSamplerSeedHook` are simple and no need to be modified usually, so here we reveal what we can do with `LoggerHook`, `CheckpointHook` and `Det3DVisualizationHook`.
#### CheckpointHook
Except saving checkpoints periodically, [`CheckpointHook`](https://github.com/open-mmlab/mmengine/blob/main/mmengine/hooks/checkpoint_hook.py#L19) provides other options such as `max_keep_ckpts`, `save_optimizer` and etc. The users could set `max_keep_ckpts` to only save small number of checkpoints or decide whether to store state dict of optimizer by `save_optimizer`. More details of the arguments are [here](https://github.com/open-mmlab/mmengine/blob/main/mmengine/hooks/checkpoint_hook.py#L19)
Except saving checkpoints periodically, [`CheckpointHook`](https://github.com/open-mmlab/mmengine/blob/main/mmengine/hooks/checkpoint_hook.py#L18) provides other options such as `max_keep_ckpts`, `save_optimizer` and etc. The users could set `max_keep_ckpts` to only save small number of checkpoints or decide whether to store state dict of optimizer by `save_optimizer`. More details of the arguments are [here](https://github.com/open-mmlab/mmengine/blob/main/mmengine/hooks/checkpoint_hook.py#L18).
```python
default_hooks = dict(
......@@ -359,8 +371,22 @@ default_hooks = dict(
#### LoggerHook
The `LoggerHook` enables setting intervals. Detailed instructions can be found in the [docstring](https://github.com/open-mmlab/mmengine/blob/main/mmengine/hooks/logger_hook.py#L18).
The `LoggerHook` enables setting intervals. Detailed instructions can be found in the [docstring](https://github.com/open-mmlab/mmengine/blob/main/mmengine/hooks/logger_hook.py#L19).
```python
default_hooks = dict(logger=dict(type='LoggerHook', interval=50))
```
#### Det3DVisualizationHook
`Det3DVisualizationHook` use `DetLocalVisualizer` to visualize prediction results, and `Det3DLocalVisualizer` current supports different backends, e.g., `TensorboardVisBackend` and `WandbVisBackend` (see [docstring](https://github.com/open-mmlab/mmengine/blob/main/mmengine/visualization/vis_backend.py) for more details). The users could add multi backends to do visualization as follows.
```python
default_hooks = dict(
visualization=dict(type='Det3DVisualizationHook', draw=True))
vis_backends = [dict(type='LocalVisBackend'),
dict(type='TensorboardVisBackend')]
visualizer = dict(
type='Det3DLocalVisualizer', vis_backends=vis_backends, name='visualizer')
```
docs/zh_cn/advanced_guides/customize_models.md
View file @ 72ac2a5e
......@@ -2,22 +2,22 @@
我们通常把模型的各个组成成分分成 6 种类型：
- 编码器（encoder）：包括 voxel layer、voxel encoder 和 middle encoder 等进入 backbone 前所使用的基于 voxel 的方法，如 HardVFE 和 PointPillarsScatter。
- 骨干网络（backbone）：通常采用 FCN 网络来提取特征图，如 ResNet 和 SECOND。
- 颈部网络（neck）：位于 backbones 和 heads 之间的组成模块，如 FPN 和 SECONDFPN。
- 检测头（head）：用于特定任务的组成模块，如检测框的预测和掩码的预测。
- RoI 提取器（RoI extractor）：用于从特征图中提取 RoI 特征的组成模块，如 H3DRoIHead 和 PartAggregationROIHead。
- 损失函数（loss）：heads 中用于计算损失函数的组成模块，如 FocalLoss、L1Loss 和 GHMLoss。
- 编码器（encoder）：包括 voxel encoder 和 middle encoder 等进入 backbone 前所使用的基于体素的方法，如 `HardVFE` 和 `PointPillarsScatter`。
- 骨干网络（backbone）：通常采用 FCN 网络来提取特征图，如 `ResNet` 和 `SECOND`。
- 颈部网络（neck）：位于 backbones 和 heads 之间的组成模块，如 `FPN` 和 `SECONDFPN`。
- 检测头（head）：用于特定任务的组成模块，如`检测框的预测`和`掩码的预测`。
- RoI 提取器（RoI extractor）：用于从特征图中提取 RoI 特征的组成模块，如 `H3DRoIHead` 和 `PartAggregationROIHead`。
- 损失函数（loss）：heads 中用于计算损失函数的组成模块，如 `FocalLoss`、`L1Loss` 和 `GHMLoss`。
## 开发新的组成模块
### 添加新建 encoder
### 添加新的编码器
接下来我们以 HardVFE 为例展示如何开发新的组成模块。
#### 1. 定义一个新的 voxel encoder（如 HardVFE：即 DV-SECOND 中所提出的 Voxel 特征提取器）
#### 1. 定义一个新的体素编码器（如 HardVFE：即 HV-SECOND 中使用的体素特征编码器）
创建一个新文件 `mmdet3d/models/voxel_encoders/voxel_encoder.py` ：
创建一个新文件 `mmdet3d/models/voxel_encoders/voxel_encoder.py`。
```python
import torch.nn as nn
......@@ -31,27 +31,27 @@ class HardVFE(nn.Module):
def __init__(self, arg1, arg2):
pass
def forward(self, x): # should return a tuple
def forward(self, x): # 需要返回一个元组
pass
```
#### 2. 导入新建模块
#### 2. 导入该模块
用户可以通过添加下面这行代码到 `mmdet3d/models/voxel_encoders/__init__.py` 中
您可以在 `mmdet3d/models/voxel_encoders/__init__.py` 中添加以下代码：
```python
from .voxel_encoder import HardVFE
```
或者添加以下的代码到配置文件中，从而能够在避免修改源码的情况下导入新建模块。
或者在配置文件中添加以下代码，从而避免修改源码：
```python
custom_imports = dict(
imports=['mmdet3d.models.voxel_encoders.HardVFE'],
imports=['mmdet3d.models.voxel_encoders.voxel_encoder'],
allow_failed_imports=False)
```
#### 3. 在配置文件中使用 voxel encoder
#### 3. 在配置文件中使用体素编码器
```python
model = dict(
......@@ -59,20 +59,21 @@ model = dict(
voxel_encoder=dict(
type='HardVFE',
arg1=xxx,
arg2=xxx),
arg2=yyy),
...
)
```
### 添加新建 backbone
### 添加新的骨干网络
接下来我们以 [SECOND](https://www.mdpi.com/1424-8220/18/10/3337)（Sparsely Embedded Convolutional Detection） 为例展示如何开发新的组成模块。
接下来我们以 [SECOND](https://www.mdpi.com/1424-8220/18/10/3337)（Sparsely Embedded Convolutional Detection）为例展示如何开发新的组成模块。
#### 1. 定义一个新的 backbone（如 SECOND）
#### 1. 定义一个新的骨干网络（如 SECOND）
创建一个新文件 `mmdet3d/models/backbones/second.py` ：
创建一个新文件 `mmdet3d/models/backbones/second.py`。
```python
import torch.nn as nn
from mmengine.model import BaseModule
from mmdet3d.registry import MODELS
......@@ -83,19 +84,19 @@ class SECOND(BaseModule):
def __init__(self, arg1, arg2):
pass
def forward(self, x): # should return a tuple
def forward(self, x): # 需要返回一个元组
pass
```
#### 2. 导入新建模块
#### 2. 导入该模块
用户可以通过添加下面这行代码到 `mmdet3d/models/backbones/__init__.py` 中
您可以在 `mmdet3d/models/backbones/__init__.py` 中添加以下代码：
```python
from .second import SECOND
```
或者添加以下的代码到配置文件中，从而能够在避免修改源码的情况下导入新建模块。
或者在配置文件中添加以下代码，从而避免修改源码：
```python
custom_imports = dict(
......@@ -103,7 +104,7 @@ custom_imports = dict(
allow_failed_imports=False)
```
#### 3. 在配置文件中使用 backbone
#### 3. 在配置文件中使用骨干网络
```python
model = dict(
......@@ -111,19 +112,23 @@ model = dict(
backbone=dict(
type='SECOND',
arg1=xxx,
arg2=xxx),
arg2=yyy),
...
)
```
### 添加新建 necks
### 添加新的颈部网络
#### 1. 定义一个新的 neck（如 SECONDFPN）
#### 1. 定义一个新的颈部网络（如 SECONDFPN）
创建一个新文件 `mmdet3d/models/necks/second_fpn.py` ：
创建一个新文件 `mmdet3d/models/necks/second_fpn.py`。
```python
from mmengine.model import BaseModule
from mmdet3d.registry import MODELS
@MODELS.register_module()
class SECONDFPN(BaseModule):
......@@ -138,20 +143,20 @@ class SECONDFPN(BaseModule):
init_cfg=None):
pass
def forward(self, X):
# implementation is ignored
def forward(self, x):
# 具体实现忽略
pass
```
#### 2. 导入新建模块
#### 2. 导入该模块
用户可以通过添加下面这行代码到 `mmdet3D/models/necks/__init__.py` 中
您可以在 `mmdet3d/models/necks/__init__.py` 中添加以下代码：
```python
from .second_fpn import SECONDFPN
```
或者添加以下的代码到配置文件中，从而能够在避免修改源码的情况下导入新建模块。
或者在配置文件中添加以下代码，从而避免修改源码：
```python
custom_imports = dict(
......@@ -159,7 +164,7 @@ custom_imports = dict(
allow_failed_imports=False)
```
#### 3. 在配置文件中使用 neck
#### 3. 在配置文件中使用颈部网络
```python
model = dict(
......@@ -170,22 +175,23 @@ model = dict(
upsample_strides=[1, 2, 4],
out_channels=[128, 128, 128]),
...
)
```
### 添加新建 heads
### 添加新的检测头
接下来我们以 [PartA2 Head](https://arxiv.org/abs/1907.03670) 为例展示如何开发新的组成模块。
接下来我们以 [PartA2 Head](https://arxiv.org/abs/1907.03670) 为例展示如何开发新的检测头。
**注意**：此处展示的 PartA2 RoI Head 将应用于双阶段检测器中，对于单阶段检测器，请参考 `mmdet3d/models/dense_heads/` 中所展示的例子。由于这些 heads 简单高效，因此这些 heads 普遍应用在自动驾驶场景下的 3D 检测任务中。
**注意**：此处展示的 `PartA2 RoI Head` 将用于检测器的第二阶段。对于单阶段的检测头，请参考 `mmdet3d/models/dense_heads/` 中的例子。由于其简单高效，它们更常用于自动驾驶场景下的 3D 检测中。
首先，在 `mmdet3d/models/roi_heads/bbox_heads/parta2_bbox_head.py` 中创建一个新的 bbox head。
PartA2 RoI Head 实现一个新的 bbox head ，并用于目标检测的任务中。
为了实现一个新的 bbox head，通常需要在其中实现三个功能，如下所示，有时该模块还需要实现其他相关的功能，如 `loss` 和 `get_targets`。
首先，在 `mmdet3d/models/roi_heads/bbox_heads/parta2_bbox_head.py` 中添加新的 bbox head。`PartA2 RoI Head` 为目标检测实现了一个新的 bbox head。为了实现一个 bbox head，我们通常需要在新模块中实现如下两个函数。有时还需要实现其他相关函数，如 `loss` 和 `get_targets`。
```python
from mmdet3d.registry import MODELS
from mmengine.model import BaseModule
from mmdet3d.registry import MODELS
@MODELS.register_module()
class PartA2BboxHead(BaseModule):
"""PartA2 RoI head."""
......@@ -218,14 +224,16 @@ class PartA2BboxHead(BaseModule):
super(PartA2BboxHead, self).__init__(init_cfg=init_cfg)
def forward(self, seg_feats, part_feats):
pass
```
其次，如果有必要的话，用户还需要实现一个新的 RoI Head，此处我们从 `Base3DRoIHead` 中继承得到一个新类 `PartAggregationROIHead`，此时我们就能发现 `Base3DRoIHead` 已经实现了下面的功能：
其次，如果有必要的话需要实现一个新的 RoI Head。我们从 `Base3DRoIHead` 中继承得到新的 `PartAggregationROIHead`。我们可以发现 `Base3DRoIHead` 已经实现了如下函数。
```python
from mmdet3d.registry import MODELS, TASK_UTILS
from mmdet.models.roi_heads import BaseRoIHead
from mmdet3d.registry import MODELS, TASK_UTILS
class Base3DRoIHead(BaseRoIHead):
"""Base class for 3d RoIHeads."""
......@@ -276,22 +284,21 @@ class Base3DRoIHead(BaseRoIHead):
"""Initialize mask head, skip since ``PartAggregationROIHead`` does not
have one."""
pass
```
接着将会对 bbox_forward 的逻辑进行修改，同时，bbox_forward 还会继承来自 `Base3DRoIHead` 的其他逻辑，在 `mmdet3d/models/roi_heads/part_aggregation_roi_head.py` 中，我们实现了新的 RoI Head，如下所示：
接下来主要对 bbox_forward 的逻辑进行修改，同时其继承了来自 `Base3DRoIHead` 的其它逻辑。在 `mmdet3d/models/roi_heads/part_aggregation_roi_head.py` 中，我们实现了新的 RoI Head，如下所示：
```python
from typing import Dict, List, Tuple
from mmcv import ConfigDict
from mmdet.models.task_modules import AssignResult, SamplingResult
from mmengine import ConfigDict
from torch import Tensor
from torch.nn import functional as F
from mmdet3d.registry import MODELS
from mmdet3d.structures import bbox3d2roi
from mmdet3d.utils import InstanceList
from mmdet.models.task_modules import AssignResult, SamplingResult
from ...structures.det3d_data_sample import SampleList
from .base_3droi_head import Base3DRoIHead
......@@ -358,7 +365,7 @@ class PartAggregationROIHead(Base3DRoIHead):
Args:
feats_dict (dict): Contains features from the first stage.
rpn_results_list (List[:obj:`InstancesData`]): Detection results
rpn_results_list (List[:obj:`InstanceData`]): Detection results
of rpn head.
batch_data_samples (List[:obj:`Det3DDataSample`]): The Data
samples. It usually includes information such as
......@@ -405,7 +412,7 @@ class PartAggregationROIHead(Base3DRoIHead):
voxel_dict (dict): Contains information of voxels.
batch_input_metas (list[dict], Optional): Batch image meta info.
Defaults to None.
rpn_results_list (List[:obj:`InstancesData`]): Detection results
rpn_results_list (List[:obj:`InstanceData`]): Detection results
of rpn head.
test_cfg (Config): Test config.
......@@ -431,7 +438,7 @@ class PartAggregationROIHead(Base3DRoIHead):
Args:
feats_dict (dict): Contains features from the first stage.
rpn_results_list (List[:obj:`InstancesData`]): Detection results
rpn_results_list (List[:obj:`InstanceData`]): Detection results
of rpn head.
batch_data_samples (List[:obj:`Det3DDataSample`]): The Data
samples. It usually includes information such as
......@@ -467,18 +474,19 @@ class PartAggregationROIHead(Base3DRoIHead):
return losses
```
此处我们省略相关函数的更多细节。请参考[代码](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/mmdet3d/models/roi_heads/part_aggregation_roi_head.py)了解更多细节。
此处我们省略了相关函数的更多细节。更多细节请参考[代码](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/mmdet3d/models/roi_heads/part_aggregation_roi_head.py)。
最后，用户需要在 `mmdet3d/models/bbox_heads/__init__.py` 和 `mmdet3d/models/roi_heads/__init__.py` 中添加模块以确保相应的注册器能够找到并加载它们。
最后，用户需要在 `mmdet3d/models/roi_heads/bbox_heads/__init__.py` 和 `mmdet3d/models/roi_heads/__init__.py` 添加模块，从而能被相应的注册器找到并加载。
此外，用户也可以添加以下的代码到配置文件中，从而实现相同的目标。
此外，用户也可以在配置文件中添加以下代码以达到相同的目的。
```python
custom_imports=dict(
imports=['mmdet3d.models.roi_heads.part_aggregation_roi_head', 'mmdet3d.models.roi_heads.bbox_heads.parta2_bbox_head'])
imports=['mmdet3d.models.roi_heads.part_aggregation_roi_head', 'mmdet3d.models.roi_heads.bbox_heads.parta2_bbox_head'],
allow_failed_imports=False)
```
PartAggregationROIHead 的配置文件如下所示。
`PartAggregationROIHead` 的配置文件如下所示：
```python
model = dict(
......@@ -544,24 +552,22 @@ model = dict(
reduction='sum',
loss_weight=1.0))),
...
)
)
```
MMDetection 2.0 支持配置文件之间的继承，使得用户能够更加关注自己的配置文件的修改。
PartA2 Head 的第二阶段主要使用新建的 `PartAggregationROIHead` 和 `PartA2BboxHead`，需要根据对应模块的 `__init__` 参数来设置对应的参数。
MMDetection 2.0 开始支持配置文件之间的继承，因此用户可以关注配置文件的修改。PartA2 Head 的第二阶段主要使用了新的 `PartAggregationROIHead` 和 `PartA2BboxHead`，需要根据对应模块的 `__init__` 函数来设置参数。
### 添加新建 loss
### 添加新的损失函数
假定用户想要新添一个用于检测框回归的 loss，并命名为 `MyLoss`。
为了添加一个新的 loss ，用于需要在 `mmdet3d/models/losses/my_loss.py` 中实现对应的逻辑。
装饰器 `weighted_loss` 能够保证对 batch 中每个样本的 loss 进行加权平均。
假设您想要为检测框的回归添加一个新的损失函数 `MyLoss`。为了添加一个新的损失函数，用户需要在 `mmdet3d/models/losses/my_loss.py` 中实现该函数。装饰器 `weighted_loss` 能够保证对每个元素的损失进行加权平均。
```python
import torch
import torch.nn as nn
from mmdet.models.losses.utils import weighted_loss
from mmdet3d.registry import MODELS
from mmdet.models.losses.utils import weighted_loss
@weighted_loss
def my_loss(pred, target):
......@@ -569,6 +575,7 @@ def my_loss(pred, target):
loss = torch.abs(pred - target)
return loss
@MODELS.register_module()
class MyLoss(nn.Module):
......@@ -591,23 +598,22 @@ class MyLoss(nn.Module):
return loss_bbox
```
接着，用户需要将 loss 添加到 `mmdet3d/models/losses/__init__.py`：
接下来，用户需要在 `mmdet3d/models/losses/__init__.py` 添加该函数。
```python
from .my_loss import MyLoss, my_loss
```
此外，用户也可以添加以下的代码到配置文件中，从而实现相同的目标。
或者在配置文件中添加以下代码以达到相同的目的。
```python
custom_imports=dict(
imports=['mmdet3d.models.losses.my_loss'])
imports=['mmdet3d.models.losses.my_loss'],
allow_failed_imports=False)
```
为了使用该 loss，需要对 `loss_xxx` 域进行修改。
因为 MyLoss 主要用于检测框的回归，因此需要在对应的 head 中修改 `loss_bbox` 域的值。
为了使用该函数，用户需要修改 `loss_xxx` 域。由于 `MyLoss` 是用于回归的，您需要修改 head 中的 `loss_bbox` 域。
```python
loss_bbox=dict(type='MyLoss', loss_weight=1.0))
loss_bbox=dict(type='MyLoss', loss_weight=1.0)
```
docs/zh_cn/advanced_guides/customize_runtime.md
View file @ 72ac2a5e
......@@ -28,7 +28,7 @@ optim_wrapper = dict(
### 自定义 PyTorch 支持的优化器
我们已经支持使用所有 PyTorch 实现的优化器，且唯一需要修改的地方就是改变配置文件中的 `optim_wrapper` 字段中的 `optimizer` 字段。例如，如果您想使用 `ADAM`（注意到这样可能会使性能大幅下降），您可以这样修改：
我们已经支持使用所有 PyTorch 实现的优化器，且唯一需要修改的地方就是改变配置文件中的 `optim_wrapper` 字段中的 `optimizer` 字段。例如，如果您想使用 `Adam`（注意这样可能会使性能大幅下降），您可以这样修改：
```python
optim_wrapper = dict(
......@@ -47,15 +47,16 @@ optim_wrapper = dict(
假设您想要添加一个叫 `MyOptimizer` 的，拥有参数 `a`，`b` 和 `c` 的优化器，您需要创建一个叫做 `mmdet3d/engine/optimizers` 的目录。接下来，应该在目录下某个文件中实现新的优化器，比如 `mmdet3d/engine/optimizers/my_optimizer.py`：
```python
from mmdet3d.registry import OPTIMIZERS
from torch.optim import Optimizer
from mmdet3d.registry import OPTIMIZERS
@OPTIMIZERS.register_module()
class MyOptimizer(Optimizer):
def __init__(self, a, b, c):
pass
```
#### 2. 将优化器添加到注册器
......@@ -70,15 +71,15 @@ class MyOptimizer(Optimizer):
from .my_optimizer import MyOptimizer
```
- 在配置中使用 `custom_imports` 来人工导入新优化器：
- 在配置中使用 `custom_imports` 来人工导入新优化器。
```python
custom_imports = dict(imports=['mmdet3d.engine.optimizers.my_optimizer'], allow_failed_imports=False)
```
模块 `mmdet3d.engine.optimizers.my_optimizer` 会在程序开始被导入，且 `MyOptimizer` 类在那时会自动被注册。注意到应该只有包含 `MyOptimizer` 类的包被导入。`mmdet3d.engine.optimizers.my_optimizer.MyOptimizer`**不能**被直接导入。
模块 `mmdet3d.engine.optimizers.my_optimizer` 会在程序开始被导入，且 `MyOptimizer` 类在那时会自动被注册。注意到应该只有包含 `MyOptimizer` 类的包被导入。`mmdet3d.engine.optimizers.my_optimizer.MyOptimizer`**不能**被直接导入。
事实上，用户可以在这种导入的方法中使用完全不同的文件目录结构，只要保证根目录能在 `PYTHONPATH` 中被定位。
事实上，用户可以在这种导入的方法中使用完全不同的文件目录结构，只要保证根目录能在 `PYTHONPATH` 中被定位。
#### 3. 在配置文件中指定优化器
......@@ -103,7 +104,7 @@ optim_wrapper = dict(
部分模型可能会拥有一些参数专属的优化器设置，比如 BatchNorm 层的权重衰减 (weight decay)。用户可以通过自定义优化器封装构造器来对那些细粒度的参数进行调优。
```python
from mmengine.optim import DefaultOptiWrapperConstructor
from mmengine.optim import DefaultOptimWrapperConstructor
from mmdet3d.registry import OPTIM_WRAPPER_CONSTRUCTORS
from .my_optimizer import MyOptimizer
......@@ -115,33 +116,29 @@ class MyOptimizerWrapperConstructor(DefaultOptimWrapperConstructor):
def __init__(self,
optim_wrapper_cfg: dict,
paramwise_cfg: Optional[dict] = None):
pass
def __call__(self, model: nn.Module) -> OptimWrapper:
return optim_wrapper
```
默认优化器封装构造器在[这里](https://github.com/open-mmlab/mmengine/blob/main/mmengine/optim/optimizer/default_constructor.py#L18)实现。这部分代码也可以用作新优化器封装构造器的模版。
默认优化器封装构造器在[这里](https://github.com/open-mmlab/mmengine/blob/main/mmengine/optim/optimizer/default_constructor.py#L18)实现。这部分代码也可以用作新优化器封装构造器的模板。
### 额外的设置
没有在优化器部分实现的技巧应该通过优化器封装构造器或者钩子来实现（比如逐参数的学习率设置）。我们列举了一些常用的可以稳定训练过程或者加速训练的设置。我们欢迎提供更多类似设置的 PR 和 issue。
- __使用梯度裁剪 (gradient clip) 来稳定训练过程__：
一些模型依赖梯度裁剪技术来裁剪训练中的梯度，以稳定训练过程。举例如下：
- __使用梯度裁剪 (gradient clip) 来稳定训练过程__：一些模型依赖梯度裁剪技术来裁剪训练中的梯度，以稳定训练过程。举例如下：
```python
optim_wrapper = dict(
_delete_=True, clip_grad=dict(max_norm=35, norm_type=2))
```
如果您的配置继承了一个已经设置了 `optim_wrapper` 的基础配置，那么您可能需要 `_delete_=True` 字段来覆盖基础配置中无用的设置。更多细节请参考[说明文档](https://mmdetection3d.readthedocs.io/en/latest/tutorials/config.html)。
- __使用动量调度器 (momentum scheduler) 来加速模型收敛__：
如果您的配置继承了一个已经设置了 `optim_wrapper` 的基础配置，那么您可能需要 `_delete_=True` 字段来覆盖基础配置中无用的设置。更多细节请参考[配置文档](https://mmdetection3d.readthedocs.io/zh_CN/dev-1.x/user_guides/config.html)。
我们支持用动量调度器来根据学习率更改模型的动量，这样可以使模型更快地收敛。动量调度器通常和学习率调度器一起使用，例如，如下配置文件在 [3D 检测](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/configs/_base_/schedules/cyclic_20e.py)中被用于加速模型收敛。更多细节请参考 [CosineAnnealingLR](https://github.com/open-mmlab/mmengine/blob/main/mmengine/optim/scheduler/lr_scheduler.py#L43) 和 [CosineAnnealingMomentum](https://github.com/open-mmlab/mmengine/blob/main/mmengine/optim/scheduler/momentum_scheduler.py#L71) 的实现方法。
- __使用动量调度器 (momentum scheduler) 来加速模型收敛__：我们支持用动量调度器来根据学习率更改模型的动量，这样可以使模型更快地收敛。动量调度器通常和学习率调度器一起使用，例如，如下配置文件在 [3D 检测](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/configs/_base_/schedules/cyclic-20e.py)中被用于加速模型收敛。更多细节请参考 [CosineAnnealingLR](https://github.com/open-mmlab/mmengine/blob/main/mmengine/optim/scheduler/lr_scheduler.py#L43) 和 [CosineAnnealingMomentum](https://github.com/open-mmlab/mmengine/blob/main/mmengine/optim/scheduler/momentum_scheduler.py#L71) 的实现方法。
```python
param_scheduler = [
......@@ -188,9 +185,9 @@ class MyOptimizerWrapperConstructor(DefaultOptimWrapperConstructor):
## 自定义训练调度
默认情况下我们使用阶梯式学习率衰减的 1 倍训练调度。这会调用 MMEngine 中的 [`MultiStepLR`](https://github.com/open-mmlab/mmengine/blob/main/mmengine/optim/scheduler/lr_scheduler.py#L139)。我们在[这里](https://github.com/open-mmlab/mmengine/blob/main/mmengine/optim/scheduler/lr_scheduler.py)支持了很多其他学习率调度，比如`余弦退火`和`多项式衰减`调度。下面是一些样例：
默认情况下我们使用阶梯式学习率衰减的 1 倍训练调度，这会调用 MMEngine 中的 [`MultiStepLR`](https://github.com/open-mmlab/mmengine/blob/main/mmengine/optim/scheduler/lr_scheduler.py#L144)。我们在[这里](https://github.com/open-mmlab/mmengine/blob/main/mmengine/optim/scheduler/lr_scheduler.py)支持了很多其他学习率调度，比如`余弦退火`和`多项式衰减`调度。下面是一些样例：
- 多项式衰减调度:
- 多项式衰减调度：
```python
param_scheduler = [
......@@ -203,7 +200,7 @@ class MyOptimizerWrapperConstructor(DefaultOptimWrapperConstructor):
by_epoch=True)]
```
- 余弦退火调度:
- 余弦退火调度：
```python
param_scheduler = [
......@@ -224,11 +221,11 @@ class MyOptimizerWrapperConstructor(DefaultOptimWrapperConstructor):
train_cfg = dict(type='EpochBasedTrainLoop', max_epochs=12, val_begin=1, val_interval=1)
```
事实上，[`IterBasedTrainLoop`](https://github.com/open-mmlab/mmengine/blob/main/mmengine/runner/loops.py#L183%5D) 和 [`EpochBasedTrainLoop`](https://github.com/open-mmlab/mmengine/blob/main/mmengine/runner/loops.py#L18) 都支持动态间隔验证，如下所示：
事实上，[`IterBasedTrainLoop`](https://github.com/open-mmlab/mmengine/blob/main/mmengine/runner/loops.py#L185) 和 [`EpochBasedTrainLoop`](https://github.com/open-mmlab/mmengine/blob/main/mmengine/runner/loops.py#L18) 都支持动态间隔验证，如下所示：
```python
# 在第 365001 次迭代之前，我们每隔 5000 次迭代验证一次
# 在第 365000 次迭代之后，我们每隔 368750 次迭代验证一次
# 在第 365001 次迭代之前，我们每隔 5000 次迭代验证一次。
# 在第 365000 次迭代之后，我们每隔 368750 次迭代验证一次，
# 这意味着我们在训练结束后进行验证。
interval = 5000
......@@ -247,7 +244,7 @@ train_cfg = dict(
#### 1. 实现一个新钩子
MMEngine 提供了一些实用的[钩子](https://github.com/open-mmlab/mmengine/blob/main/docs/en/tutorials/hook.md)，但有些场合用户可能需要实现一个新的钩子。在 v1.1.0rc0 之后，MMDetection3D 在训练时支持基于 MMEngine 自定义钩子。因此用户可以直接在 mmdet3d 或者基于 mmdet3d 的代码库中实现钩子并通过更改训练配置来使用钩子。这里我们给出一个在 mmdet3d 中创建并使用新钩子的例子。
MMEngine 提供了一些实用的[钩子](https://mmengine.readthedocs.io/zh_CN/latest/tutorials/hook.html)，但有些场合用户可能需要实现一个新的钩子。在 v1.1.0rc0 之后，MMDetection3D 在训练时支持基于 MMEngine 自定义钩子。因此用户可以直接在 mmdet3d 或者基于 mmdet3d 的代码库中实现钩子并通过更改训练配置来使用钩子。这里我们给出一个在 mmdet3d 中创建并使用新钩子的例子。
```python
from mmengine.hooks import Hook
......@@ -290,7 +287,7 @@ class MyHook(Hook):
接下来我们需要导入 `MyHook`。假设新钩子位于文件 `mmdet3d/engine/hooks/my_hook.py` 中，有两种实现方法：
- 修改 `mmdet3d/engine/hooks/__init__.py` 导入该模块：
- 修改 `mmdet3d/engine/hooks/__init__.py` 导入该模块。
新定义的模块应该在 `mmdet3d/engine/hooks/__init__.py` 中被导入，从而被找到并且被添加到注册器中：
......@@ -298,7 +295,7 @@ class MyHook(Hook):
from .my_hook import MyHook
```
- 在配置中使用 `custom_imports` 来人为地导入新钩子：
- 在配置中使用 `custom_imports` 来人为地导入新钩子。
```python
custom_imports = dict(imports=['mmdet3d.engine.hooks.my_hook'], allow_failed_imports=False)
......@@ -320,27 +317,38 @@ custom_hooks = [
]
```
默认情况下，注册阶段钩子的优先级为 `NORMAL`。
默认情况下，注册阶段钩子的优先级为 `'NORMAL'`。
### 使用 MMDetection3D 中实现的钩子
如果 MMDetection3D 中已经实现了该钩子，您可以直接通过更改配置文件来使用该钩子。
#### 例子：`DisableObjectSampleHook`
### 使用 MMEngine 中实现的钩子
我们实现了一个名为 [DisableObjectSampleHook](https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/mmdet3d/engine/hooks/disable_object_sample_hook.py) 的自定义钩子在训练阶段达到指定 epoch 后禁用 `ObjectSample` 增强策略。
如果钩子已经在 MMEngine 中被实现了，您可以直接通过更改配置文件来使用该钩子：
如果有需要的话我们可以在配置文件中设置它：
```python
custom_hooks = [dict(type='DisableObjectSampleHook', disable_after_epoch=15)]
```
### 更改默认的运行时钩子
有一些常用的钩子通过 `default_hooks` 注册，它们是：
- `IterTimerHook`：该钩子用来记录加载数据的时间 'data_time' 和模型训练一步的时间 'time' 。
- `LoggerHook`：该钩子用来从`执行器（Runner）`的不同组件收集日志并将其写入终端，Json 文件，tensorboard 和 wandb 等。
- `IterTimerHook`：该钩子用来记录加载数据的时间 'data_time' 和模型训练一步的时间 'time'。
- `LoggerHook`：该钩子用来从`执行器（Runner）`的不同组件收集日志并将其写入终端，json 文件，tensorboard 和 wandb 等。
- `ParamSchedulerHook`：该钩子用来更新优化器中的一些超参数，例如学习率和动量。
- `CheckpointHook`：该钩子用来定期地保存检查点。
- `DistSamplerSeedHook`：该钩子用来设置采样和批采样的种子。
- `Det3DVisualizationHook`：该钩子用来可视化验证和测试过程的预测结果。
`IterTimerHook`，`ParamSchedulerHook` 和 `DistSamplerSeedHook` 都很简单，通常不需要修改，因此此处我们将介绍如何使用 `LoggerHook`，`CheckpointHook` 和 `Det3DVisualizationHook`。
#### CheckpointHook
除了定期地保存检查点，[`CheckpointHook`](https://github.com/open-mmlab/mmengine/blob/main/mmengine/hooks/checkpoint_hook.py#L19) 提供了其它的可选项例如 `max_keep_ckpts`，`save_optimizer` 等。用户可以设置 `max_keep_ckpts` 只保存少量的检查点或者通过 `save_optimizer` 决定是否保存优化器的状态。参数的更多细节参考[此处](https://github.com/open-mmlab/mmengine/blob/main/mmengine/hooks/checkpoint_hook.py#L19)。
除了定期地保存检查点，[`CheckpointHook`](https://github.com/open-mmlab/mmengine/blob/main/mmengine/hooks/checkpoint_hook.py#L18) 提供了其它的可选项例如 `max_keep_ckpts`，`save_optimizer` 等。用户可以设置 `max_keep_ckpts` 只保存少量的检查点或者通过 `save_optimizer` 决定是否保存优化器的状态。参数的更多细节请参考[此处](https://github.com/open-mmlab/mmengine/blob/main/mmengine/hooks/checkpoint_hook.py#L18)。
```python
default_hooks = dict(
......@@ -353,8 +361,22 @@ default_hooks = dict(
#### LoggerHook
`LoggerHook` 允许设置日志记录间隔。详细介绍可参考[文档](https://github.com/open-mmlab/mmengine/blob/main/mmengine/hooks/logger_hook.py#L18)。
`LoggerHook` 允许设置日志记录间隔。详细介绍可参考[文档](https://github.com/open-mmlab/mmengine/blob/main/mmengine/hooks/logger_hook.py#L19)。
```python
default_hooks = dict(logger=dict(type='LoggerHook', interval=50))
```
#### Det3DVisualizationHook
`Det3DVisualizationHook` 使用 `DetLocalVisualizer` 来可视化预测结果，`Det3DLocalVisualizer` 支持不同的后端，例如 `TensorboardVisBackend` 和 `WandbVisBackend`（更多细节请参考[文档](https://github.com/open-mmlab/mmengine/blob/main/mmengine/visualization/vis_backend.py)）。用户可以添加多个后端来进行可视化，如下所示。
```python
default_hooks = dict(
visualization=dict(type='Det3DVisualizationHook', draw=True))
vis_backends = [dict(type='LocalVisBackend'),
dict(type='TensorboardVisBackend')]
visualizer = dict(
type='Det3DLocalVisualizer', vis_backends=vis_backends, name='visualizer')
```
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment