Initial add code.

docs_zh_CN/tutorials/6_export_model.md

+# 教程 6：如何导出模型为 onnx 格式
+
+开放式神经网络交换格式（Open Neural Network Exchange，即 [ONNX](https://onnx.ai/)）是一个开放的生态系统，使 AI 开发人员能够随着项目的发展选择正确的工具。
+
+<!-- TOC -->
+
+- [支持的模型](#%E6%94%AF%E6%8C%81%E7%9A%84%E6%A8%A1%E5%9E%8B)
+- [如何使用](#%E5%A6%82%E4%BD%95%E4%BD%BF%E7%94%A8)
+  - [准备工作](#%E5%87%86%E5%A4%87%E5%B7%A5%E4%BD%9C)
+  - [行为识别器](#%E8%A1%8C%E4%B8%BA%E8%AF%86%E5%88%AB%E5%99%A8)
+  - [时序动作检测器](#%E6%97%B6%E5%BA%8F%E5%8A%A8%E4%BD%9C%E6%A3%80%E6%B5%8B%E5%99%A8)
+
+<!-- TOC -->
+
+## 支持的模型
+
+到目前为止，MMAction2 支持将训练的 pytorch 模型中进行 onnx 导出。支持的模型有：
+
+- I3D
+- TSN
+- TIN
+- TSM
+- R(2+1)D
+- SLOWFAST
+- SLOWONLY
+- BMN
+- BSN(tem, pem)
+
+## 如何使用
+
+对于简单的模型导出，用户可以使用这里的 [脚本](/tools/deployment/pytorch2onnx.py)。
+注意，需要安装 `onnx` 和 `onnxruntime` 包以进行导出后的验证。
+
+### 准备工作
+
+首先，安装 onnx
+
+```shell
+pip install onnx onnxruntime
+```
+
+MMAction2 提供了一个 python 脚本，用于将 MMAction2 训练的 pytorch 模型导出到 ONNX。
+
+```shell
+python tools/deployment/pytorch2onnx.py ${CONFIG_FILE} ${CHECKPOINT_FILE} [--shape ${SHAPE}] \
+    [--verify] [--show] [--output-file ${OUTPUT_FILE}]  [--is-localizer] [--opset-version ${VERSION}]
+```
+
+可选参数：
+
+- `--shape`: 模型输入张量的形状。对于 2D 模型（如 TSN），输入形状应当为 `$batch $clip $channel $height $width` (例如，`1 1 3 224 224`)；对于 3D 模型（如 I3D），输入形状应当为 `$batch $clip $channel $time $height $width` (如，`1 1 3 32 224 224`)；对于时序检测器如 BSN，每个模块的数据都不相同，请查看对应的 `forward` 函数。如果没有被指定，它将被置为 `1 1 3 224 224`。
+- `--verify`: 决定是否对导出模型进行验证，验证项包括是否可运行，数值是否正确等。如果没有被指定，它将被置为 `False`。
+- `--show`: 决定是否打印导出模型的结构。如果没有被指定，它将被置为 `False`。
+- `--output-file`: 导出的 onnx 模型名。如果没有被指定，它将被置为 `tmp.onnx`。
+- `--is-localizer`：决定导出的模型是否为时序检测器。如果没有被指定，它将被置为 `False`。
+- `--opset-version`：决定 onnx 的执行版本，MMAction2 推荐用户使用高版本（例如 11 版本）的 onnx 以确保稳定性。如果没有被指定，它将被置为 `11`。
+- `--softmax`: 是否在行为识别器末尾添加 Softmax。如果没有指定，将被置为 `False`。目前仅支持行为识别器，不支持时序动作检测器。
+
+### 行为识别器
+
+对于行为识别器，可运行：
+
+```shell
+python tools/deployment/pytorch2onnx.py $CONFIG_PATH $CHECKPOINT_PATH --shape $SHAPE --verify
+```
+
+### 时序动作检测器
+
+对于时序动作检测器，可运行：
+
+```shell
+python tools/deployment/pytorch2onnx.py $CONFIG_PATH $CHECKPOINT_PATH --is-localizer --shape $SHAPE --verify
+```
+
+如果发现提供的模型权重文件没有被成功导出，或者存在精度损失，可以在本 repo 下提出问题（issue）。

docs_zh_CN/tutorials/7_customize_runtime.md

+# 教程 7：如何自定义模型运行参数
+
+在本教程中，我们将介绍如何在运行自定义模型时，进行自定义参数优化方法，学习率调整策略，工作流和钩子的方法。
+
+<!-- TOC -->
+
+- [定制优化方法](#%E5%AE%9A%E5%88%B6%E4%BC%98%E5%8C%96%E6%96%B9%E6%B3%95)
+  - [使用 PyTorch 内置的优化器](#%E4%BD%BF%E7%94%A8-PyTorch-%E5%86%85%E7%BD%AE%E7%9A%84%E4%BC%98%E5%8C%96%E5%99%A8)
+  - [定制用户自定义的优化器](#%E5%AE%9A%E5%88%B6%E7%94%A8%E6%88%B7%E8%87%AA%E5%AE%9A%E4%B9%89%E7%9A%84%E4%BC%98%E5%8C%96%E5%99%A8)
+    - [1. 定义一个新的优化器](#1-%E5%AE%9A%E4%B9%89%E4%B8%80%E4%B8%AA%E6%96%B0%E7%9A%84%E4%BC%98%E5%8C%96%E5%99%A8)
+    - [2. 注册优化器](#2-%E6%B3%A8%E5%86%8C%E4%BC%98%E5%8C%96%E5%99%A8)
+    - [3. 在配置文件中指定优化器](#3-%E5%9C%A8%E9%85%8D%E7%BD%AE%E6%96%87%E4%BB%B6%E4%B8%AD%E6%8C%87%E5%AE%9A%E4%BC%98%E5%8C%96%E5%99%A8)
+  - [定制优化器构造器](#%E5%AE%9A%E5%88%B6%E4%BC%98%E5%8C%96%E5%99%A8%E6%9E%84%E9%80%A0%E5%99%A8)
+  - [额外设定](#%E9%A2%9D%E5%A4%96%E8%AE%BE%E5%AE%9A)
+- [定制学习率调整策略](#%E5%AE%9A%E5%88%B6%E5%AD%A6%E4%B9%A0%E7%8E%87%E8%B0%83%E6%95%B4%E7%AD%96%E7%95%A5)
+- [定制工作流](#%E5%AE%9A%E5%88%B6%E5%B7%A5%E4%BD%9C%E6%B5%81)
+- [定制钩子](#%E5%AE%9A%E5%88%B6%E9%92%A9%E5%AD%90)
+  - [定制用户自定义钩子](#%E5%AE%9A%E5%88%B6%E7%94%A8%E6%88%B7%E8%87%AA%E5%AE%9A%E4%B9%89%E9%92%A9%E5%AD%90)
+    - [1. 创建一个新钩子](#1-%E5%88%9B%E5%BB%BA%E4%B8%80%E4%B8%AA%E6%96%B0%E9%92%A9%E5%AD%90)
+    - [2. 注册新钩子](#2-%E6%B3%A8%E5%86%8C%E6%96%B0%E9%92%A9%E5%AD%90)
+    - [3. 修改配置](#3-%E4%BF%AE%E6%94%B9%E9%85%8D%E7%BD%AE)
+  - [使用 MMCV 内置钩子](#%E4%BD%BF%E7%94%A8-MMCV-%E5%86%85%E7%BD%AE%E9%92%A9%E5%AD%90)
+  - [修改默认运行的钩子](#%E4%BF%AE%E6%94%B9%E9%BB%98%E8%AE%A4%E8%BF%90%E8%A1%8C%E7%9A%84%E9%92%A9%E5%AD%90)
+    - [模型权重文件配置](#%E6%A8%A1%E5%9E%8B%E6%9D%83%E9%87%8D%E6%96%87%E4%BB%B6%E9%85%8D%E7%BD%AE)
+    - [日志配置](#%E6%97%A5%E5%BF%97%E9%85%8D%E7%BD%AE)
+    - [验证配置](#%E9%AA%8C%E8%AF%81%E9%85%8D%E7%BD%AE)
+
+<!-- TOC -->
+
+## 定制优化方法
+
+### 使用 PyTorch 内置的优化器
+
+MMAction2 支持 PyTorch 实现的所有优化器，仅需在配置文件中，指定 “optimizer” 字段
+例如，如果要使用 “Adam”，则修改如下。
+
+```python
+optimizer = dict(type='Adam', lr=0.0003, weight_decay=0.0001)
+```
+
+要修改模型的学习率，用户只需要在优化程序的配置中修改 “lr” 即可。
+用户可根据 [PyTorch API 文档](https://pytorch.org/docs/stable/optim.html?highlight=optim#module-torch.optim) 进行参数设置
+
+例如，如果想使用 `Adam` 并设置参数为 `torch.optim.Adam(params, lr=0.001, betas=(0.9, 0.999), eps=1e-08, weight_decay=0, amsgrad=False)`，
+则需要进行如下修改
+
+```python
+optimizer = dict(type='Adam', lr=0.001, betas=(0.9, 0.999), eps=1e-08, weight_decay=0, amsgrad=False)
+```
+
+### 定制用户自定义的优化器
+
+#### 1. 定义一个新的优化器
+
+一个自定义的优化器可根据如下规则进行定制
+
+假设用户想添加一个名为 `MyOptimzer` 的优化器，其拥有参数 `a`, `b` 和 `c`，
+可以创建一个名为 `mmaction/core/optimizer` 的文件夹，并在目录下的文件进行构建，如 `mmaction/core/optimizer/my_optimizer.py`：
+
+```python
+from mmcv.runner import OPTIMIZERS
+from torch.optim import Optimizer
+
+
+@OPTIMIZERS.register_module()
+class MyOptimizer(Optimizer):
+
+    def __init__(self, a, b, c):
+
+```
+
+#### 2. 注册优化器
+
+要找到上面定义的上述模块，首先应将此模块导入到主命名空间中。有两种方法可以实现它。
+
+- 修改 `mmaction/core/optimizer/__init__.py` 来进行调用
+
+  新定义的模块应导入到 `mmaction/core/optimizer/__init__.py` 中，以便注册器能找到新模块并将其添加：
+
+```python
+from .my_optimizer import MyOptimizer
+```
+
+- 在配置中使用 `custom_imports` 手动导入
+
+```python
+custom_imports = dict(imports=['mmaction.core.optimizer.my_optimizer'], allow_failed_imports=False)
+```
+
+`mmaction.core.optimizer.my_optimizer` 模块将会在程序开始阶段被导入，`MyOptimizer` 类会随之自动被注册。
+注意，只有包含 `MyOptmizer` 类的包会被导入。`mmaction.core.optimizer.my_optimizer.MyOptimizer` **不会** 被直接导入。
+
+#### 3. 在配置文件中指定优化器
+
+之后，用户便可在配置文件的 `optimizer` 域中使用 `MyOptimizer`。
+在配置中，优化器由 “optimizer” 字段定义，如下所示：
+
+```python
+optimizer = dict(type='SGD', lr=0.02, momentum=0.9, weight_decay=0.0001)
+```
+
+要使用自定义的优化器，可以将该字段更改为
+
+```python
+optimizer = dict(type='MyOptimizer', a=a_value, b=b_value, c=c_value)
+```
+
+### 定制优化器构造器
+
+某些模型可能具有一些特定于参数的设置以进行优化，例如 BatchNorm 层的权重衰减。
+用户可以通过自定义优化器构造函数来进行那些细粒度的参数调整。
+
+```python
+from mmcv.runner.optimizer import OPTIMIZER_BUILDERS
+
+
+@OPTIMIZER_BUILDERS.register_module()
+class MyOptimizerConstructor:
+
+    def __init__(self, optimizer_cfg, paramwise_cfg=None):
+        pass
+
+    def __call__(self, model):
+
+        return my_optimizer
+```
+
+默认的优化器构造器被创建于[此](https://github.com/open-mmlab/mmcv/blob/9ecd6b0d5ff9d2172c49a182eaa669e9f27bb8e7/mmcv/runner/optimizer/default_constructor.py#L11)，
+可被视为新优化器构造器的模板。
+
+### 额外设定
+
+优化器没有实现的优化技巧（trick）可通过优化器构造函数（例如，设置按参数的学习率）或钩子来实现。
+下面列出了一些可以稳定训练或加快训练速度的常用设置。用户亦可通过为 MMAction2 创建 PR，发布更多设置。
+
+- __使用梯度裁剪来稳定训练__
+  一些模型需要使用梯度裁剪来剪辑渐变以稳定训练过程。 一个例子如下：
+
+  ```python
+  optimizer_config = dict(grad_clip=dict(max_norm=35, norm_type=2))
+  ```
+
+- __使用动量调整来加速模型收敛__
+  MMAction2 支持动量调整器根据学习率修改模型的动量，从而使模型收敛更快。
+  动量调整程序通常与学习率调整器一起使用，例如，以下配置用于3D检测以加速收敛。
+  更多细节可参考 [CyclicLrUpdater](https://github.com/open-mmlab/mmcv/blob/f48241a65aebfe07db122e9db320c31b685dc674/mmcv/runner/hooks/lr_updater.py#L327)
+  和 [CyclicMomentumUpdater](https://github.com/open-mmlab/mmcv/blob/f48241a65aebfe07db122e9db320c31b685dc674/mmcv/runner/hooks/momentum_updater.py#L130)。
+
+  ```python
+  lr_config = dict(
+      policy='cyclic',
+      target_ratio=(10, 1e-4),
+      cyclic_times=1,
+      step_ratio_up=0.4,
+  )
+  momentum_config = dict(
+      policy='cyclic',
+      target_ratio=(0.85 / 0.95, 1),
+      cyclic_times=1,
+      step_ratio_up=0.4,
+  )
+  ```
+
+## 定制学习率调整策略
+
+在配置文件中使用默认值的逐步学习率调整，它调用 MMCV 中的 [`StepLRHook`](https://github.com/open-mmlab/mmcv/blob/f48241a65aebfe07db122e9db320c31b685dc674/mmcv/runner/hooks/lr_updater.py#L153)。
+此外，也支持其他学习率调整方法，如 `CosineAnnealing` 和 `Poly`。 详情可见 [这里](https://github.com/open-mmlab/mmcv/blob/master/mmcv/runner/hooks/lr_updater.py)
+
+- Poly:
+
+  ```python
+  lr_config = dict(policy='poly', power=0.9, min_lr=1e-4, by_epoch=False)
+  ```
+
+- ConsineAnnealing:
+
+  ```python
+  lr_config = dict(
+      policy='CosineAnnealing',
+      warmup='linear',
+      warmup_iters=1000,
+      warmup_ratio=1.0 / 10,
+      min_lr_ratio=1e-5)
+  ```
+
+## 定制工作流
+
+默认情况下，MMAction2 推荐用户在训练周期中使用 “EvalHook” 进行模型验证，也可以选择 “val” 工作流模型进行模型验证。
+
+工作流是一个形如 (工作流名, 周期数) 的列表，用于指定运行顺序和周期。其默认设置为：
+
+```python
+workflow = [('train', 1)]
+```
+
+其代表要进行一轮周期的训练。
+有时，用户可能希望检查有关验证集中模型的某些指标（例如，损失，准确性）。
+在这种情况下，可以将工作流程设置为
+
+```python
+[('train', 1), ('val', 1)]
+```
+
+从而将迭代运行1个训练时间和1个验证时间。
+
+**值得注意的是**：
+
+1. 在验证周期时不会更新模型参数。
+2. 配置文件内的关键词 `total_epochs` 控制训练时期数，并且不会影响验证工作流程。
+3. 工作流 `[('train', 1), ('val', 1)]` 和 `[('train', 1)]` 不会改变 `EvalHook` 的行为。
+   因为 `EvalHook` 由 `after_train_epoch` 调用，而验证工作流只会影响 `after_val_epoch` 调用的钩子。
+   因此，`[('train', 1), ('val', 1)]` 和 `[('train', 1)]` 的区别在于，runner 在完成每一轮训练后，会计算验证集上的损失。
+
+## 定制钩子
+
+### 定制用户自定义钩子
+
+#### 1. 创建一个新钩子
+
+这里举一个在 MMAction2 中创建一个新钩子，并在训练中使用它的示例：
+
+```python
+from mmcv.runner import HOOKS, Hook
+
+
+@HOOKS.register_module()
+class MyHook(Hook):
+
+    def __init__(self, a, b):
+        pass
+
+    def before_run(self, runner):
+        pass
+
+    def after_run(self, runner):
+        pass
+
+    def before_epoch(self, runner):
+        pass
+
+    def after_epoch(self, runner):
+        pass
+
+    def before_iter(self, runner):
+        pass
+
+    def after_iter(self, runner):
+        pass
+```
+
+根据钩子的功能，用户需要指定钩子在训练的每个阶段将要执行的操作，比如 `before_run`，`after_run`，`before_epoch`，`after_epoch`，`before_iter` 和 `after_iter`。
+
+#### 2. 注册新钩子
+
+之后，需要导入 `MyHook`。假设该文件在 `mmaction/core/utils/my_hook.py`，有两种办法导入它：
+
+- 修改 `mmaction/core/utils/__init__.py` 进行导入
+
+  新定义的模块应导入到 `mmaction/core/utils/__init__py` 中，以便注册表能找到并添加新模块：
+
+```python
+from .my_hook import MyHook
+```
+
+- 使用配置文件中的 `custom_imports` 变量手动导入
+
+```python
+custom_imports = dict(imports=['mmaction.core.utils.my_hook'], allow_failed_imports=False)
+```
+
+#### 3. 修改配置
+
+```python
+custom_hooks = [
+    dict(type='MyHook', a=a_value, b=b_value)
+]
+```
+
+还可通过 `priority` 参数（可选参数值包括 `'NORMAL'` 或 `'HIGHEST'`）设置钩子优先级，如下所示：
+
+```python
+custom_hooks = [
+    dict(type='MyHook', a=a_value, b=b_value, priority='NORMAL')
+]
+```
+
+默认情况下，在注册过程中，钩子的优先级设置为 “NORMAL”。
+
+### 使用 MMCV 内置钩子
+
+如果该钩子已在 MMCV 中实现，则可以直接修改配置以使用该钩子，如下所示
+
+```python
+mmcv_hooks = [
+    dict(type='MMCVHook', a=a_value, b=b_value, priority='NORMAL')
+]
+```
+
+### 修改默认运行的钩子
+
+有一些常见的钩子未通过 `custom_hooks` 注册，但在导入 MMCV 时已默认注册，它们是：
+
+- log_config
+- checkpoint_config
+- evaluation
+- lr_config
+- optimizer_config
+- momentum_config
+
+在这些钩子中，只有 log_config 具有 “VERY_LOW” 优先级，其他钩子具有 “NORMAL” 优先级。
+上述教程已经介绍了如何修改 “optimizer_config”，“momentum_config” 和 “lr_config”。
+下面介绍如何使用 log_config，checkpoint_config，以及 evaluation 能做什么。
+
+#### 模型权重文件配置
+
+MMCV 的 runner 使用 `checkpoint_config` 来初始化 [`CheckpointHook`](https://github.com/open-mmlab/mmcv/blob/9ecd6b0d5ff9d2172c49a182eaa669e9f27bb8e7/mmcv/runner/hooks/checkpoint.py#L9)。
+
+```python
+checkpoint_config = dict(interval=1)
+```
+
+用户可以设置 “max_keep_ckpts” 来仅保存少量模型权重文件，或者通过 “save_optimizer” 决定是否存储优化器的状态字典。
+更多细节可参考 [这里](https://mmcv.readthedocs.io/en/latest/api.html#mmcv.runner.CheckpointHook)。
+
+#### 日志配置
+
+`log_config` 包装了多个记录器钩子，并可以设置间隔。
+目前，MMCV 支持 `WandbLoggerHook`，`MlflowLoggerHook` 和 `TensorboardLoggerHook`。
+更多细节可参考[这里](https://mmcv.readthedocs.io/en/latest/api.html#mmcv.runner.LoggerHook)。
+
+```python
+log_config = dict(
+    interval=50,
+    hooks=[
+        dict(type='TextLoggerHook'),
+        dict(type='TensorboardLoggerHook')
+    ])
+```
+
+#### 验证配置
+
+评估的配置将用于初始化 [`EvalHook`](https://github.com/open-mmlab/mmaction2/blob/master/mmaction/core/evaluation/eval_hooks.py#L12)。
+除了键 `interval` 外，其他参数，如 “metrics” 也将传递给 `dataset.evaluate()`。
+
+```python
+evaluation = dict(interval=1, metrics='bbox')
+```

docs_zh_CN/useful_tools.md

+除了训练/测试脚本外，MMAction2 还在 `tools/` 目录下提供了许多有用的工具。
+
+## 目录
+
+<!-- TOC -->
+
+- [日志分析](#%E6%97%A5%E5%BF%97%E5%88%86%E6%9E%90)
+- [模型复杂度分析](#%E6%A8%A1%E5%9E%8B%E5%A4%8D%E6%9D%82%E5%BA%A6%E5%88%86%E6%9E%90)
+- [模型转换](#%E6%A8%A1%E5%9E%8B%E8%BD%AC%E6%8D%A2)
+  - [导出 MMAction2 模型为 ONNX 格式（实验特性）](#%E5%AF%BC%E5%87%BA-MMAction2-%E6%A8%A1%E5%9E%8B%E4%B8%BA-ONNX-%E6%A0%BC%E5%BC%8F%EF%BC%88%E5%AE%9E%E9%AA%8C%E7%89%B9%E6%80%A7%EF%BC%89)
+  - [发布模型](#%E5%8F%91%E5%B8%83%E6%A8%A1%E5%9E%8B)
+- [其他脚本](#%E5%85%B6%E4%BB%96%E8%84%9A%E6%9C%AC)
+  - [指标评价](#%E6%8C%87%E6%A0%87%E8%AF%84%E4%BB%B7)
+  - [打印完整配置](#%E6%89%93%E5%8D%B0%E5%AE%8C%E6%95%B4%E9%85%8D%E7%BD%AE)
+
+<!-- TOC -->
+
+## 日志分析
+
+输入变量指定一个训练日志文件，可通过 `tools/analysis/analyze_logs.py` 脚本绘制 loss/top-k 曲线。本功能依赖于 `seaborn`，使用前请先通过 `pip install seaborn` 安装依赖包。
+
+![准确度曲线图](https://github.com/open-mmlab/mmaction2/raw/master/resources/acc_curve.png)
+
+```shell
+python tools/analysis/analyze_logs.py plot_curve ${JSON_LOGS} [--keys ${KEYS}] [--title ${TITLE}] [--legend ${LEGEND}] [--backend ${BACKEND}] [--style ${STYLE}] [--out ${OUT_FILE}]
+```
+
+例如:
+
+- 绘制某日志文件对应的分类损失曲线图。
+
+  ```shell
+  python tools/analysis/analyze_logs.py plot_curve log.json --keys loss_cls --legend loss_cls
+  ```
+
+- 绘制某日志文件对应的 top-1 和 top-5 准确率曲线图，并将曲线图导出为 PDF 文件。
+
+  ```shell
+  python tools/analysis/analyze_logs.py plot_curve log.json --keys top1_acc top5_acc --out results.pdf
+  ```
+
+- 在同一图像内绘制两份日志文件对应的 top-1 准确率曲线图。
+
+  ```shell
+  python tools/analysis/analyze_logs.py plot_curve log1.json log2.json --keys top1_acc --legend run1 run2
+  ```
+
+  用户还可以通过本工具计算平均训练速度。
+
+  ```shell
+  python tools/analysis/analyze_logs.py cal_train_time ${JSON_LOGS} [--include-outliers]
+  ```
+
+- 计算某日志文件对应的平均训练速度。
+
+  ```shell
+  python tools/analysis/analyze_logs.py cal_train_time work_dirs/some_exp/20200422_153324.log.json
+  ```
+
+  预计输出结果如下所示：
+
+  ```text
+  -----Analyze train time of work_dirs/some_exp/20200422_153324.log.json-----
+  slowest epoch 60, average time is 0.9736
+  fastest epoch 18, average time is 0.9001
+  time std over epochs is 0.0177
+  average iter time: 0.9330 s/iter
+  ```
+
+## 模型复杂度分析
+
+`/tools/analysis/get_flops.py` 是根据 [flops-counter.pytorch](https://github.com/sovrasov/flops-counter.pytorch) 库改编的脚本，用于计算输入变量指定模型的 FLOPs 和参数量。
+
+```shell
+python tools/analysis/get_flops.py ${CONFIG_FILE} [--shape ${INPUT_SHAPE}]
+```
+
+预计输出结果如下所示：
+
+```text
+==============================
+Input shape: (1, 3, 32, 340, 256)
+Flops: 37.1 GMac
+Params: 28.04 M
+==============================
+```
+
+**注意**：该工具仍处于试验阶段，不保证该数字绝对正确。
+用户可以将结果用于简单比较，但若要在技术报告或论文中采用该结果，请仔细检查。
+
+(1) FLOPs 与输入变量形状有关，但是模型的参数量与输入变量形状无关。2D 行为识别器的默认形状为 (1, 3, 340, 256)，3D 行为识别器的默认形状为 (1, 3, 32, 340, 256)。
+(2) 部分算子不参与 FLOPs 以及参数量的计算，如 GN 和一些自定义算子。更多详细信息请参考 [`mmcv.cnn.get_model_complexity_info()`](https://github.com/open-mmlab/mmcv/blob/master/mmcv/cnn/utils/flops_counter.py)
+
+## 模型转换
+
+### 导出 MMAction2 模型为 ONNX 格式（实验特性）
+
+`/tools/deployment/pytorch2onnx.py` 脚本用于将模型转换为 [ONNX](https://github.com/onnx/onnx) 格式。
+同时，该脚本支持比较 PyTorch 模型和 ONNX 模型的输出结果，验证输出结果是否相同。
+本功能依赖于 `onnx` 以及 `onnxruntime`，使用前请先通过 `pip install onnx onnxruntime` 安装依赖包。
+请注意，可通过 `--softmax` 选项在行为识别器末尾添加 Softmax 层，从而获取 `[0, 1]` 范围内的预测结果。
+
+- 对于行为识别模型，请运行：
+
+  ```shell
+  python tools/deployment/pytorch2onnx.py $CONFIG_PATH $CHECKPOINT_PATH --shape $SHAPE --verify
+  ```
+
+- 对于时序动作检测模型，请运行：
+
+  ```shell
+  python tools/deployment/pytorch2onnx.py $CONFIG_PATH $CHECKPOINT_PATH --is-localizer --shape $SHAPE --verify
+  ```
+
+### 发布模型
+
+`tools/deployment/publish_model.py` 脚本用于进行模型发布前的准备工作，主要包括：
+
+(1) 将模型的权重张量转化为 CPU 张量。
+(2) 删除优化器状态信息。
+(3) 计算模型权重文件的哈希值，并将哈希值添加到文件名后。
+
+```shell
+python tools/deployment/publish_model.py ${INPUT_FILENAME} ${OUTPUT_FILENAME}
+```
+
+例如,
+
+```shell
+python tools/deployment/publish_model.py work_dirs/tsn_r50_1x1x3_100e_kinetics400_rgb/latest.pth tsn_r50_1x1x3_100e_kinetics400_rgb.pth
+```
+
+最终，输出文件名为 `tsn_r50_1x1x3_100e_kinetics400_rgb-{hash id}.pth`。
+
+## 其他脚本
+
+### 指标评价
+
+`tools/analysis/eval_metric.py` 脚本通过输入变量指定配置文件，以及对应的结果存储文件，计算某一评价指标。
+
+结果存储文件通过 `tools/test.py` 脚本（通过参数 `--out ${RESULT_FILE}` 指定）生成，保存了指定模型在指定数据集中的预测结果。
+
+```shell
+python tools/analysis/eval_metric.py ${CONFIG_FILE} ${RESULT_FILE} [--eval ${EVAL_METRICS}] [--cfg-options ${CFG_OPTIONS}] [--eval-options ${EVAL_OPTIONS}]
+```
+
+### 打印完整配置
+
+`tools/analysis/print_config.py` 脚本会解析所有输入变量，并打印完整配置信息。
+
+```shell
+python tools/print_config.py ${CONFIG} [-h] [--options ${OPTIONS [OPTIONS...]}]
+```
+
+### 检查视频
+
+`tools/analysis/check_videos.py` 脚本利用指定视频编码器，遍历指定配置文件视频数据集中所有样本，寻找无效视频文件（文件破损或者文件不存在），并将无效文件路径保存到输出文件中。请注意，删除无效视频文件后，需要重新生成视频文件列表。
+
+```shell
+python tools/analysis/check_videos.py ${CONFIG} [-h] [--options OPTIONS [OPTIONS ...]] [--cfg-options CFG_OPTIONS [CFG_OPTIONS ...]] [--output-file OUTPUT_FILE] [--split SPLIT] [--decoder DECODER] [--num-processes NUM_PROCESSES] [--remove-corrupted-videos]
+```

mmaction/__init__.py

+# Copyright (c) OpenMMLab. All rights reserved.
+import mmcv
+from mmcv import digit_version
+
+from .version import __version__
+
+mmcv_minimum_version = '1.3.6'
+mmcv_maximum_version = '1.7.0'
+mmcv_version = digit_version(mmcv.__version__)
+
+assert (digit_version(mmcv_minimum_version) <= mmcv_version
+        <= digit_version(mmcv_maximum_version)), \
+    f'MMCV=={mmcv.__version__} is used but incompatible. ' \
+    f'Please install mmcv>={mmcv_minimum_version}, <={mmcv_maximum_version}.'
+
+__all__ = ['__version__']

mmaction/apis/__init__.py

+# Copyright (c) OpenMMLab. All rights reserved.
+from .inference import inference_recognizer, init_recognizer
+from .test import multi_gpu_test, single_gpu_test
+from .train import init_random_seed, train_model
+
+__all__ = [
+    'train_model', 'init_recognizer', 'inference_recognizer', 'multi_gpu_test',
+    'single_gpu_test', 'init_random_seed'
+]

mmaction/apis/inference.py

+# Copyright (c) OpenMMLab. All rights reserved.
+import os
+import os.path as osp
+import re
+import warnings
+from operator import itemgetter
+
+import mmcv
+import numpy as np
+import torch
+from mmcv.parallel import collate, scatter
+from mmcv.runner import load_checkpoint
+
+from mmaction.core import OutputHook
+from mmaction.datasets.pipelines import Compose
+from mmaction.models import build_recognizer
+
+
+def init_recognizer(config, checkpoint=None, device='cuda:0', **kwargs):
+    """Initialize a recognizer from config file.
+
+    Args:
+        config (str | :obj:`mmcv.Config`): Config file path or the config
+            object.
+        checkpoint (str | None, optional): Checkpoint path/url. If set to None,
+            the model will not load any weights. Default: None.
+        device (str | :obj:`torch.device`): The desired device of returned
+            tensor. Default: 'cuda:0'.
+
+    Returns:
+        nn.Module: The constructed recognizer.
+    """
+    if 'use_frames' in kwargs:
+        warnings.warn('The argument `use_frames` is deprecated PR #1191. '
+                      'Now you can use models trained with frames or videos '
+                      'arbitrarily. ')
+
+    if isinstance(config, str):
+        config = mmcv.Config.fromfile(config)
+    elif not isinstance(config, mmcv.Config):
+        raise TypeError('config must be a filename or Config object, '
+                        f'but got {type(config)}')
+
+    # pretrained model is unnecessary since we directly load checkpoint later
+    config.model.backbone.pretrained = None
+    model = build_recognizer(config.model, test_cfg=config.get('test_cfg'))
+
+    if checkpoint is not None:
+        load_checkpoint(model, checkpoint, map_location='cpu')
+    model.cfg = config
+    model.to(device)
+    model.eval()
+    return model
+
+
+def inference_recognizer(model, video, outputs=None, as_tensor=True, **kwargs):
+    """Inference a video with the recognizer.
+
+    Args:
+        model (nn.Module): The loaded recognizer.
+        video (str | dict | ndarray): The video file path / url or the
+            rawframes directory path / results dictionary (the input of
+            pipeline) / a 4D array T x H x W x 3 (The input video).
+        outputs (list(str) | tuple(str) | str | None) : Names of layers whose
+            outputs need to be returned, default: None.
+        as_tensor (bool): Same as that in ``OutputHook``. Default: True.
+
+    Returns:
+        dict[tuple(str, float)]: Top-5 recognition result dict.
+        dict[torch.tensor | np.ndarray]:
+            Output feature maps from layers specified in `outputs`.
+    """
+    if 'use_frames' in kwargs:
+        warnings.warn('The argument `use_frames` is deprecated PR #1191. '
+                      'Now you can use models trained with frames or videos '
+                      'arbitrarily. ')
+    if 'label_path' in kwargs:
+        warnings.warn('The argument `use_frames` is deprecated PR #1191. '
+                      'Now the label file is not needed in '
+                      'inference_recognizer. ')
+
+    input_flag = None
+    if isinstance(video, dict):
+        input_flag = 'dict'
+    elif isinstance(video, np.ndarray):
+        assert len(video.shape) == 4, 'The shape should be T x H x W x C'
+        input_flag = 'array'
+    elif isinstance(video, str) and video.startswith('http'):
+        input_flag = 'video'
+    elif isinstance(video, str) and osp.exists(video):
+        if osp.isfile(video):
+            if video.endswith('.npy'):
+                input_flag = 'audio'
+            else:
+                input_flag = 'video'
+        if osp.isdir(video):
+            input_flag = 'rawframes'
+    else:
+        raise RuntimeError('The type of argument video is not supported: '
+                           f'{type(video)}')
+
+    if isinstance(outputs, str):
+        outputs = (outputs, )
+    assert outputs is None or isinstance(outputs, (tuple, list))
+
+    cfg = model.cfg
+    device = next(model.parameters()).device  # model device
+    # build the data pipeline
+    test_pipeline = cfg.data.test.pipeline
+    # Alter data pipelines & prepare inputs
+    if input_flag == 'dict':
+        data = video
+    if input_flag == 'array':
+        modality_map = {2: 'Flow', 3: 'RGB'}
+        modality = modality_map.get(video.shape[-1])
+        data = dict(
+            total_frames=video.shape[0],
+            label=-1,
+            start_index=0,
+            array=video,
+            modality=modality)
+        for i in range(len(test_pipeline)):
+            if 'Decode' in test_pipeline[i]['type']:
+                test_pipeline[i] = dict(type='ArrayDecode')
+        test_pipeline = [x for x in test_pipeline if 'Init' not in x['type']]
+    if input_flag == 'video':
+        data = dict(filename=video, label=-1, start_index=0, modality='RGB')
+        if 'Init' not in test_pipeline[0]['type']:
+            test_pipeline = [dict(type='OpenCVInit')] + test_pipeline
+        else:
+            test_pipeline[0] = dict(type='OpenCVInit')
+        for i in range(len(test_pipeline)):
+            if 'Decode' in test_pipeline[i]['type']:
+                test_pipeline[i] = dict(type='OpenCVDecode')
+    if input_flag == 'rawframes':
+        filename_tmpl = cfg.data.test.get('filename_tmpl', 'img_{:05}.jpg')
+        modality = cfg.data.test.get('modality', 'RGB')
+        start_index = cfg.data.test.get('start_index', 1)
+
+        # count the number of frames that match the format of `filename_tmpl`
+        # RGB pattern example: img_{:05}.jpg -> ^img_\d+.jpg$
+        # Flow patteren example: {}_{:05d}.jpg -> ^x_\d+.jpg$
+        pattern = f'^{filename_tmpl}$'
+        if modality == 'Flow':
+            pattern = pattern.replace('{}', 'x')
+        pattern = pattern.replace(
+            pattern[pattern.find('{'):pattern.find('}') + 1], '\\d+')
+        total_frames = len(
+            list(
+                filter(lambda x: re.match(pattern, x) is not None,
+                       os.listdir(video))))
+        data = dict(
+            frame_dir=video,
+            total_frames=total_frames,
+            label=-1,
+            start_index=start_index,
+            filename_tmpl=filename_tmpl,
+            modality=modality)
+        if 'Init' in test_pipeline[0]['type']:
+            test_pipeline = test_pipeline[1:]
+        for i in range(len(test_pipeline)):
+            if 'Decode' in test_pipeline[i]['type']:
+                test_pipeline[i] = dict(type='RawFrameDecode')
+    if input_flag == 'audio':
+        data = dict(
+            audio_path=video,
+            total_frames=len(np.load(video)),
+            start_index=cfg.data.test.get('start_index', 1),
+            label=-1)
+
+    test_pipeline = Compose(test_pipeline)
+    data = test_pipeline(data)
+    data = collate([data], samples_per_gpu=1)
+
+    if next(model.parameters()).is_cuda:
+        # scatter to specified GPU
+        data = scatter(data, [device])[0]
+
+    # forward the model
+    with OutputHook(model, outputs=outputs, as_tensor=as_tensor) as h:
+        with torch.no_grad():
+            scores = model(return_loss=False, **data)[0]
+        returned_features = h.layer_outputs if outputs else None
+
+    num_classes = scores.shape[-1]
+    score_tuples = tuple(zip(range(num_classes), scores))
+    score_sorted = sorted(score_tuples, key=itemgetter(1), reverse=True)
+
+    top5_label = score_sorted[:5]
+    if outputs:
+        return top5_label, returned_features
+    return top5_label

mmaction/apis/test.py

+# Copyright (c) OpenMMLab. All rights reserved.
+import os.path as osp
+import pickle
+import shutil
+import tempfile
+# TODO import test functions from mmcv and delete them from mmaction2
+import warnings
+
+import mmcv
+import torch
+import torch.distributed as dist
+from mmcv.runner import get_dist_info
+
+try:
+    from mmcv.engine import (collect_results_cpu, collect_results_gpu,
+                             multi_gpu_test, single_gpu_test)
+    from_mmcv = True
+except (ImportError, ModuleNotFoundError):
+    warnings.warn(
+        'DeprecationWarning: single_gpu_test, multi_gpu_test, '
+        'collect_results_cpu, collect_results_gpu from mmaction2 will be '
+        'deprecated. Please install mmcv through master branch.')
+    from_mmcv = False
+
+if not from_mmcv:
+
+    def single_gpu_test(model, data_loader):  # noqa: F811
+        """Test model with a single gpu.
+
+        This method tests model with a single gpu and
+        displays test progress bar.
+
+        Args:
+            model (nn.Module): Model to be tested.
+            data_loader (nn.Dataloader): Pytorch data loader.
+
+        Returns:
+            list: The prediction results.
+        """
+        model.eval()
+        results = []
+        dataset = data_loader.dataset
+        prog_bar = mmcv.ProgressBar(len(dataset))
+        for data in data_loader:
+            with torch.no_grad():
+                result = model(return_loss=False, **data)
+            results.extend(result)
+
+            # use the first key as main key to calculate the batch size
+            batch_size = len(next(iter(data.values())))
+            for _ in range(batch_size):
+                prog_bar.update()
+        return results
+
+    def multi_gpu_test(  # noqa: F811
+            model, data_loader, tmpdir=None, gpu_collect=True):
+        """Test model with multiple gpus.
+
+        This method tests model with multiple gpus and collects the results
+        under two different modes: gpu and cpu modes. By setting
+        'gpu_collect=True' it encodes results to gpu tensors and use gpu
+        communication for results collection. On cpu mode it saves the results
+        on different gpus to 'tmpdir' and collects them by the rank 0 worker.
+
+        Args:
+            model (nn.Module): Model to be tested.
+            data_loader (nn.Dataloader): Pytorch data loader.
+            tmpdir (str): Path of directory to save the temporary results from
+                different gpus under cpu mode. Default: None
+            gpu_collect (bool): Option to use either gpu or cpu to collect
+                results. Default: True
+
+        Returns:
+            list: The prediction results.
+        """
+        model.eval()
+        results = []
+        dataset = data_loader.dataset
+        rank, world_size = get_dist_info()
+        if rank == 0:
+            prog_bar = mmcv.ProgressBar(len(dataset))
+        for data in data_loader:
+            with torch.no_grad():
+                result = model(return_loss=False, **data)
+            results.extend(result)
+
+            if rank == 0:
+                # use the first key as main key to calculate the batch size
+                batch_size = len(next(iter(data.values())))
+                for _ in range(batch_size * world_size):
+                    prog_bar.update()
+
+        # collect results from all ranks
+        if gpu_collect:
+            results = collect_results_gpu(results, len(dataset))
+        else:
+            results = collect_results_cpu(results, len(dataset), tmpdir)
+        return results
+
+    def collect_results_cpu(result_part, size, tmpdir=None):  # noqa: F811
+        """Collect results in cpu mode.
+
+        It saves the results on different gpus to 'tmpdir' and collects
+        them by the rank 0 worker.
+
+        Args:
+            result_part (list): Results to be collected
+            size (int): Result size.
+            tmpdir (str): Path of directory to save the temporary results from
+                different gpus under cpu mode. Default: None
+
+        Returns:
+            list: Ordered results.
+        """
+        rank, world_size = get_dist_info()
+        # create a tmp dir if it is not specified
+        if tmpdir is None:
+            MAX_LEN = 512
+            # 32 is whitespace
+            dir_tensor = torch.full((MAX_LEN, ),
+                                    32,
+                                    dtype=torch.uint8,
+                                    device='cuda')
+            if rank == 0:
+                mmcv.mkdir_or_exist('.dist_test')
+                tmpdir = tempfile.mkdtemp(dir='.dist_test')
+                tmpdir = torch.tensor(
+                    bytearray(tmpdir.encode()),
+                    dtype=torch.uint8,
+                    device='cuda')
+                dir_tensor[:len(tmpdir)] = tmpdir
+            dist.broadcast(dir_tensor, 0)
+            tmpdir = dir_tensor.cpu().numpy().tobytes().decode().rstrip()
+        else:
+            tmpdir = osp.join(tmpdir, '.dist_test')
+            mmcv.mkdir_or_exist(tmpdir)
+        # synchronizes all processes to make sure tmpdir exist
+        dist.barrier()
+        # dump the part result to the dir
+        mmcv.dump(result_part, osp.join(tmpdir, f'part_{rank}.pkl'))
+        # synchronizes all processes for loading pickle file
+        dist.barrier()
+        # collect all parts
+        if rank != 0:
+            return None
+        # load results of all parts from tmp dir
+        part_list = []
+        for i in range(world_size):
+            part_file = osp.join(tmpdir, f'part_{i}.pkl')
+            part_list.append(mmcv.load(part_file))
+        # sort the results
+        ordered_results = []
+        for res in zip(*part_list):
+            ordered_results.extend(list(res))
+        # the dataloader may pad some samples
+        ordered_results = ordered_results[:size]
+        # remove tmp dir
+        shutil.rmtree(tmpdir)
+        return ordered_results
+
+    def collect_results_gpu(result_part, size):  # noqa: F811
+        """Collect results in gpu mode.
+
+        It encodes results to gpu tensors and use gpu communication for results
+        collection.
+
+        Args:
+            result_part (list): Results to be collected
+            size (int): Result size.
+
+        Returns:
+            list: Ordered results.
+        """
+        rank, world_size = get_dist_info()
+        # dump result part to tensor with pickle
+        part_tensor = torch.tensor(
+            bytearray(pickle.dumps(result_part)),
+            dtype=torch.uint8,
+            device='cuda')
+        # gather all result part tensor shape
+        shape_tensor = torch.tensor(part_tensor.shape, device='cuda')
+        shape_list = [shape_tensor.clone() for _ in range(world_size)]
+        dist.all_gather(shape_list, shape_tensor)
+        # padding result part tensor to max length
+        shape_max = torch.tensor(shape_list).max()
+        part_send = torch.zeros(shape_max, dtype=torch.uint8, device='cuda')
+        part_send[:shape_tensor[0]] = part_tensor
+        part_recv_list = [
+            part_tensor.new_zeros(shape_max) for _ in range(world_size)
+        ]
+        # gather all result part
+        dist.all_gather(part_recv_list, part_send)
+
+        if rank == 0:
+            part_list = []
+            for recv, shape in zip(part_recv_list, shape_list):
+                part_list.append(
+                    pickle.loads(recv[:shape[0]].cpu().numpy().tobytes()))
+            # sort the results
+            ordered_results = []
+            for res in zip(*part_list):
+                ordered_results.extend(list(res))
+            # the dataloader may pad some samples
+            ordered_results = ordered_results[:size]
+            return ordered_results
+        return None

mmaction/apis/train.py

+# Copyright (c) OpenMMLab. All rights reserved.
+import copy as cp
+import os
+import os.path as osp
+import time
+
+import numpy as np
+import torch
+import torch.distributed as dist
+from mmcv.runner import (DistSamplerSeedHook, EpochBasedRunner, OptimizerHook,
+                         build_optimizer, get_dist_info)
+from mmcv.runner.hooks import Fp16OptimizerHook
+
+from ..core import (DistEvalHook, EvalHook, OmniSourceDistSamplerSeedHook,
+                    OmniSourceRunner)
+from ..datasets import build_dataloader, build_dataset
+from ..utils import (PreciseBNHook, build_ddp, build_dp, default_device,
+                     get_root_logger)
+from .test import multi_gpu_test
+
+
+def init_random_seed(seed=None, device=default_device, distributed=True):
+    """Initialize random seed.
+
+    If the seed is not set, the seed will be automatically randomized,
+    and then broadcast to all processes to prevent some potential bugs.
+    Args:
+        seed (int, Optional): The seed. Default to None.
+        device (str): The device where the seed will be put on.
+            Default to 'cuda'.
+        distributed (bool): Whether to use distributed training.
+            Default: True.
+    Returns:
+        int: Seed to be used.
+    """
+    if seed is not None:
+        return seed
+
+    # Make sure all ranks share the same random seed to prevent
+    # some potential bugs. Please refer to
+    # https://github.com/open-mmlab/mmdetection/issues/6339
+    rank, world_size = get_dist_info()
+    seed = np.random.randint(2**31)
+
+    if world_size == 1:
+        return seed
+
+    if rank == 0:
+        random_num = torch.tensor(seed, dtype=torch.int32, device=device)
+    else:
+        random_num = torch.tensor(0, dtype=torch.int32, device=device)
+
+    if distributed:
+        dist.broadcast(random_num, src=0)
+    return random_num.item()
+
+
+def train_model(model,
+                dataset,
+                cfg,
+                distributed=False,
+                validate=False,
+                test=dict(test_best=False, test_last=False),
+                timestamp=None,
+                meta=None):
+    """Train model entry function.
+
+    Args:
+        model (nn.Module): The model to be trained.
+        dataset (:obj:`Dataset`): Train dataset.
+        cfg (dict): The config dict for training.
+        distributed (bool): Whether to use distributed training.
+            Default: False.
+        validate (bool): Whether to do evaluation. Default: False.
+        test (dict): The testing option, with two keys: test_last & test_best.
+            The value is True or False, indicating whether to test the
+            corresponding checkpoint.
+            Default: dict(test_best=False, test_last=False).
+        timestamp (str | None): Local time for runner. Default: None.
+        meta (dict | None): Meta dict to record some important information.
+            Default: None
+    """
+    logger = get_root_logger(log_level=cfg.log_level)
+
+    # prepare data loaders
+    dataset = dataset if isinstance(dataset, (list, tuple)) else [dataset]
+
+    dataloader_setting = dict(
+        videos_per_gpu=cfg.data.get('videos_per_gpu', 1),
+        workers_per_gpu=cfg.data.get('workers_per_gpu', 1),
+        persistent_workers=cfg.data.get('persistent_workers', False),
+        num_gpus=len(cfg.gpu_ids),
+        dist=distributed,
+        seed=cfg.seed)
+    dataloader_setting = dict(dataloader_setting,
+                              **cfg.data.get('train_dataloader', {}))
+
+    if cfg.omnisource:
+        # The option can override videos_per_gpu
+        train_ratio = cfg.data.get('train_ratio', [1] * len(dataset))
+        omni_videos_per_gpu = cfg.data.get('omni_videos_per_gpu', None)
+        if omni_videos_per_gpu is None:
+            dataloader_settings = [dataloader_setting] * len(dataset)
+        else:
+            dataloader_settings = []
+            for videos_per_gpu in omni_videos_per_gpu:
+                this_setting = cp.deepcopy(dataloader_setting)
+                this_setting['videos_per_gpu'] = videos_per_gpu
+                dataloader_settings.append(this_setting)
+        data_loaders = [
+            build_dataloader(ds, **setting)
+            for ds, setting in zip(dataset, dataloader_settings)
+        ]
+
+    else:
+        data_loaders = [
+            build_dataloader(ds, **dataloader_setting) for ds in dataset
+        ]
+
+    # put model on gpus
+    if distributed:
+        find_unused_parameters = cfg.get('find_unused_parameters', False)
+        # Sets the `find_unused_parameters` parameter in
+        # torch.nn.parallel.DistributedDataParallel
+
+        model = build_ddp(
+            model,
+            default_device,
+            default_args=dict(
+                device_ids=[int(os.environ['LOCAL_RANK'])],
+                broadcast_buffers=False,
+                find_unused_parameters=find_unused_parameters))
+    else:
+        model = build_dp(
+            model, default_device, default_args=dict(device_ids=cfg.gpu_ids))
+
+    # build runner
+    optimizer = build_optimizer(model, cfg.optimizer)
+
+    Runner = OmniSourceRunner if cfg.omnisource else EpochBasedRunner
+    runner = Runner(
+        model,
+        optimizer=optimizer,
+        work_dir=cfg.work_dir,
+        logger=logger,
+        meta=meta)
+    # an ugly workaround to make .log and .log.json filenames the same
+    runner.timestamp = timestamp
+
+    # fp16 setting
+    fp16_cfg = cfg.get('fp16', None)
+    if fp16_cfg is not None:
+        optimizer_config = Fp16OptimizerHook(
+            **cfg.optimizer_config, **fp16_cfg, distributed=distributed)
+    elif distributed and 'type' not in cfg.optimizer_config:
+        optimizer_config = OptimizerHook(**cfg.optimizer_config)
+    else:
+        optimizer_config = cfg.optimizer_config
+
+    # register hooks
+    runner.register_training_hooks(
+        cfg.lr_config,
+        optimizer_config,
+        cfg.checkpoint_config,
+        cfg.log_config,
+        cfg.get('momentum_config', None),
+        custom_hooks_config=cfg.get('custom_hooks', None))
+
+    # multigrid setting
+    multigrid_cfg = cfg.get('multigrid', None)
+    if multigrid_cfg is not None:
+        from mmaction.utils.multigrid import LongShortCycleHook
+        multigrid_scheduler = LongShortCycleHook(cfg)
+        runner.register_hook(multigrid_scheduler)
+        logger.info('Finish register multigrid hook')
+
+        # subbn3d aggregation is HIGH, as it should be done before
+        # saving and evaluation
+        from mmaction.utils.multigrid import SubBatchNorm3dAggregationHook
+        subbn3d_aggre_hook = SubBatchNorm3dAggregationHook()
+        runner.register_hook(subbn3d_aggre_hook, priority='VERY_HIGH')
+        logger.info('Finish register subbn3daggre hook')
+
+    # precise bn setting
+    if cfg.get('precise_bn', False):
+        precise_bn_dataset = build_dataset(cfg.data.train)
+        dataloader_setting = dict(
+            videos_per_gpu=cfg.data.get('videos_per_gpu', 1),
+            workers_per_gpu=1,  # save memory and time
+            persistent_workers=cfg.data.get('persistent_workers', False),
+            num_gpus=len(cfg.gpu_ids),
+            dist=distributed,
+            seed=cfg.seed)
+        data_loader_precise_bn = build_dataloader(precise_bn_dataset,
+                                                  **dataloader_setting)
+        precise_bn_hook = PreciseBNHook(data_loader_precise_bn,
+                                        **cfg.get('precise_bn'))
+        runner.register_hook(precise_bn_hook, priority='HIGHEST')
+        logger.info('Finish register precisebn hook')
+
+    if distributed:
+        if cfg.omnisource:
+            runner.register_hook(OmniSourceDistSamplerSeedHook())
+        else:
+            runner.register_hook(DistSamplerSeedHook())
+
+    if validate:
+        eval_cfg = cfg.get('evaluation', {})
+        val_dataset = build_dataset(cfg.data.val, dict(test_mode=True))
+        dataloader_setting = dict(
+            videos_per_gpu=cfg.data.get('videos_per_gpu', 1),
+            workers_per_gpu=cfg.data.get('workers_per_gpu', 1),
+            persistent_workers=cfg.data.get('persistent_workers', False),
+            # cfg.gpus will be ignored if distributed
+            num_gpus=len(cfg.gpu_ids),
+            dist=distributed,
+            shuffle=False)
+        dataloader_setting = dict(dataloader_setting,
+                                  **cfg.data.get('val_dataloader', {}))
+        val_dataloader = build_dataloader(val_dataset, **dataloader_setting)
+        eval_hook = DistEvalHook(val_dataloader, **eval_cfg) if distributed \
+            else EvalHook(val_dataloader, **eval_cfg)
+        runner.register_hook(eval_hook, priority='LOW')
+
+    if cfg.resume_from:
+        runner.resume(cfg.resume_from)
+    elif cfg.load_from:
+        runner.load_checkpoint(cfg.load_from)
+    runner_kwargs = dict()
+    if cfg.omnisource:
+        runner_kwargs = dict(train_ratio=train_ratio)
+    runner.run(data_loaders, cfg.workflow, cfg.total_epochs, **runner_kwargs)
+
+    if distributed:
+        dist.barrier()
+    time.sleep(5)
+
+    if test['test_last'] or test['test_best']:
+        best_ckpt_path = None
+        if test['test_best']:
+            ckpt_paths = [x for x in os.listdir(cfg.work_dir) if 'best' in x]
+            ckpt_paths = [x for x in ckpt_paths if x.endswith('.pth')]
+            if len(ckpt_paths) == 0:
+                runner.logger.info('Warning: test_best set, but no ckpt found')
+                test['test_best'] = False
+                if not test['test_last']:
+                    return
+            elif len(ckpt_paths) > 1:
+                epoch_ids = [
+                    int(x.split('epoch_')[-1][:-4]) for x in ckpt_paths
+                ]
+                best_ckpt_path = ckpt_paths[np.argmax(epoch_ids)]
+            else:
+                best_ckpt_path = ckpt_paths[0]
+            if best_ckpt_path:
+                best_ckpt_path = osp.join(cfg.work_dir, best_ckpt_path)
+
+        test_dataset = build_dataset(cfg.data.test, dict(test_mode=True))
+        gpu_collect = cfg.get('evaluation', {}).get('gpu_collect', False)
+        tmpdir = cfg.get('evaluation', {}).get('tmpdir',
+                                               osp.join(cfg.work_dir, 'tmp'))
+        dataloader_setting = dict(
+            videos_per_gpu=cfg.data.get('videos_per_gpu', 1),
+            workers_per_gpu=cfg.data.get('workers_per_gpu', 1),
+            persistent_workers=cfg.data.get('persistent_workers', False),
+            num_gpus=len(cfg.gpu_ids),
+            dist=distributed,
+            shuffle=False)
+        dataloader_setting = dict(dataloader_setting,
+                                  **cfg.data.get('test_dataloader', {}))
+
+        test_dataloader = build_dataloader(test_dataset, **dataloader_setting)
+
+        names, ckpts = [], []
+
+        if test['test_last']:
+            names.append('last')
+            ckpts.append(None)
+        if test['test_best'] and best_ckpt_path is not None:
+            names.append('best')
+            ckpts.append(best_ckpt_path)
+
+        for name, ckpt in zip(names, ckpts):
+            if ckpt is not None:
+                runner.load_checkpoint(ckpt)
+
+            outputs = multi_gpu_test(runner.model, test_dataloader, tmpdir,
+                                     gpu_collect)
+            rank, _ = get_dist_info()
+            if rank == 0:
+                out = osp.join(cfg.work_dir, f'{name}_pred.pkl')
+                test_dataset.dump_results(outputs, out)
+
+                eval_cfg = cfg.get('evaluation', {})
+                for key in [
+                        'interval', 'tmpdir', 'start', 'gpu_collect',
+                        'save_best', 'rule', 'by_epoch', 'broadcast_bn_buffers'
+                ]:
+                    eval_cfg.pop(key, None)
+
+                eval_res = test_dataset.evaluate(outputs, **eval_cfg)
+                runner.logger.info(f'Testing results of the {name} checkpoint')
+                for metric_name, val in eval_res.items():
+                    runner.logger.info(f'{metric_name}: {val:.04f}')

mmaction/core/__init__.py

+# Copyright (c) OpenMMLab. All rights reserved.
+from .bbox import *  # noqa: F401, F403
+from .dist_utils import *  # noqa: F401, F403
+from .evaluation import *  # noqa: F401, F403
+from .hooks import *  # noqa: F401, F403
+from .lr import *  # noqa: F401, F403
+from .optimizer import *  # noqa: F401, F403
+from .runner import *  # noqa: F401, F403
+from .scheduler import *  # noqa: F401, F403

mmaction/core/bbox/__init__.py

+# Copyright (c) OpenMMLab. All rights reserved.
+from .assigners import MaxIoUAssignerAVA
+from .bbox_target import bbox_target
+from .transforms import bbox2result
+
+__all__ = ['MaxIoUAssignerAVA', 'bbox_target', 'bbox2result']

mmaction/core/bbox/assigners/__init__.py

+# Copyright (c) OpenMMLab. All rights reserved.
+from .max_iou_assigner_ava import MaxIoUAssignerAVA
+
+__all__ = ['MaxIoUAssignerAVA']

mmaction/core/bbox/assigners/max_iou_assigner_ava.py

+# Copyright (c) OpenMMLab. All rights reserved.
+import torch
+
+try:
+    from mmdet.core.bbox import AssignResult, MaxIoUAssigner
+    from mmdet.core.bbox.builder import BBOX_ASSIGNERS
+    mmdet_imported = True
+except (ImportError, ModuleNotFoundError):
+    mmdet_imported = False
+
+if mmdet_imported:
+
+    @BBOX_ASSIGNERS.register_module()
+    class MaxIoUAssignerAVA(MaxIoUAssigner):
+        """Assign a corresponding gt bbox or background to each bbox.
+
+        Each proposals will be assigned with `-1`, `0`, or a positive integer
+        indicating the ground truth index.
+
+        - -1: don't care
+        - 0: negative sample, no assigned gt
+        - positive integer: positive sample, index (1-based) of assigned gt
+
+        Args:
+            pos_iou_thr (float): IoU threshold for positive bboxes.
+            neg_iou_thr (float | tuple): IoU threshold for negative bboxes.
+            min_pos_iou (float): Minimum iou for a bbox to be considered as a
+                positive bbox. Positive samples can have smaller IoU than
+                pos_iou_thr due to the 4th step (assign max IoU sample to each
+                gt). Default: 0.
+            gt_max_assign_all (bool): Whether to assign all bboxes with the
+                same highest overlap with some gt to that gt. Default: True.
+        """
+
+        # The function is overridden, to handle the case that gt_label is not
+        # int
+        def assign_wrt_overlaps(self, overlaps, gt_labels=None):
+            """Assign w.r.t. the overlaps of bboxes with gts.
+
+            Args:
+                overlaps (Tensor): Overlaps between k gt_bboxes and n bboxes,
+                    shape(k, n).
+                gt_labels (Tensor, optional): Labels of k gt_bboxes, shape
+                    (k, ).
+
+            Returns:
+                :obj:`AssignResult`: The assign result.
+            """
+            num_gts, num_bboxes = overlaps.size(0), overlaps.size(1)
+
+            # 1. assign -1 by default
+            assigned_gt_inds = overlaps.new_full((num_bboxes, ),
+                                                 -1,
+                                                 dtype=torch.long)
+
+            if num_gts == 0 or num_bboxes == 0:
+                # No ground truth or boxes, return empty assignment
+                max_overlaps = overlaps.new_zeros((num_bboxes, ))
+                if num_gts == 0:
+                    # No truth, assign everything to background
+                    assigned_gt_inds[:] = 0
+                if gt_labels is None:
+                    assigned_labels = None
+                else:
+                    assigned_labels = overlaps.new_full((num_bboxes, ),
+                                                        -1,
+                                                        dtype=torch.long)
+                return AssignResult(
+                    num_gts,
+                    assigned_gt_inds,
+                    max_overlaps,
+                    labels=assigned_labels)
+
+            # for each anchor, which gt best overlaps with it
+            # for each anchor, the max iou of all gts
+            max_overlaps, argmax_overlaps = overlaps.max(dim=0)
+            # for each gt, which anchor best overlaps with it
+            # for each gt, the max iou of all proposals
+            gt_max_overlaps, gt_argmax_overlaps = overlaps.max(dim=1)
+
+            # 2. assign negative: below
+            # the negative inds are set to be 0
+            if isinstance(self.neg_iou_thr, float):
+                assigned_gt_inds[(max_overlaps >= 0)
+                                 & (max_overlaps < self.neg_iou_thr)] = 0
+            elif isinstance(self.neg_iou_thr, tuple):
+                assert len(self.neg_iou_thr) == 2
+                assigned_gt_inds[(max_overlaps >= self.neg_iou_thr[0])
+                                 & (max_overlaps < self.neg_iou_thr[1])] = 0
+
+            # 3. assign positive: above positive IoU threshold
+            pos_inds = max_overlaps >= self.pos_iou_thr
+            assigned_gt_inds[pos_inds] = argmax_overlaps[pos_inds] + 1
+
+            if self.match_low_quality:
+                # Low-quality matching will overwrite the assigned_gt_inds
+                # assigned in Step 3. Thus, the assigned gt might not be the
+                # best one for prediction.
+                # For example, if bbox A has 0.9 and 0.8 iou with GT bbox
+                # 1 & 2, bbox 1 will be assigned as the best target for bbox A
+                # in step 3. However, if GT bbox 2's gt_argmax_overlaps = A,
+                # bbox A's assigned_gt_inds will be overwritten to be bbox B.
+                # This might be the reason that it is not used in ROI Heads.
+                for i in range(num_gts):
+                    if gt_max_overlaps[i] >= self.min_pos_iou:
+                        if self.gt_max_assign_all:
+                            max_iou_inds = overlaps[i, :] == gt_max_overlaps[i]
+                            assigned_gt_inds[max_iou_inds] = i + 1
+                        else:
+                            assigned_gt_inds[gt_argmax_overlaps[i]] = i + 1
+
+            if gt_labels is not None:
+                # consider multi-class case (AVA)
+                assert len(gt_labels[0]) > 1
+                assigned_labels = assigned_gt_inds.new_zeros(
+                    (num_bboxes, len(gt_labels[0])), dtype=torch.float32)
+
+                # If not assigned, labels will be all 0
+                pos_inds = torch.nonzero(
+                    assigned_gt_inds > 0, as_tuple=False).squeeze()
+                if pos_inds.numel() > 0:
+                    assigned_labels[pos_inds] = gt_labels[
+                        assigned_gt_inds[pos_inds] - 1]
+            else:
+                assigned_labels = None
+
+            return AssignResult(
+                num_gts,
+                assigned_gt_inds,
+                max_overlaps,
+                labels=assigned_labels)
+
+else:
+    # define an empty class, so that can be imported
+    class MaxIoUAssignerAVA:
+
+        def __init__(self, *args, **kwargs):
+            raise ImportError(
+                'Failed to import `AssignResult`, `MaxIoUAssigner` from '
+                '`mmdet.core.bbox` or failed to import `BBOX_ASSIGNERS` from '
+                '`mmdet.core.bbox.builder`. The class `MaxIoUAssignerAVA` is '
+                'invalid. ')

mmaction/core/bbox/bbox_target.py

+# Copyright (c) OpenMMLab. All rights reserved.
+import torch
+import torch.nn.functional as F
+
+
+def bbox_target(pos_bboxes_list, neg_bboxes_list, gt_labels, cfg):
+    """Generate classification targets for bboxes.
+
+    Args:
+        pos_bboxes_list (list[Tensor]): Positive bboxes list.
+        neg_bboxes_list (list[Tensor]): Negative bboxes list.
+        gt_labels (list[Tensor]): Groundtruth classification label list.
+        cfg (Config): RCNN config.
+
+    Returns:
+        (Tensor, Tensor): Label and label_weight for bboxes.
+    """
+    labels, label_weights = [], []
+    pos_weight = 1.0 if cfg.pos_weight <= 0 else cfg.pos_weight
+
+    assert len(pos_bboxes_list) == len(neg_bboxes_list) == len(gt_labels)
+    length = len(pos_bboxes_list)
+
+    for i in range(length):
+        pos_bboxes = pos_bboxes_list[i]
+        neg_bboxes = neg_bboxes_list[i]
+        gt_label = gt_labels[i]
+
+        num_pos = pos_bboxes.size(0)
+        num_neg = neg_bboxes.size(0)
+        num_samples = num_pos + num_neg
+        label = F.pad(gt_label, (0, 0, 0, num_neg))
+        label_weight = pos_bboxes.new_zeros(num_samples)
+        label_weight[:num_pos] = pos_weight
+        label_weight[-num_neg:] = 1.
+
+        labels.append(label)
+        label_weights.append(label_weight)
+
+    labels = torch.cat(labels, 0)
+    label_weights = torch.cat(label_weights, 0)
+    return labels, label_weights

mmaction/core/bbox/transforms.py

+# Copyright (c) OpenMMLab. All rights reserved.
+import numpy as np
+
+
+def bbox2result(bboxes, labels, num_classes, thr=0.01):
+    """Convert detection results to a list of numpy arrays.
+
+    This identifies single-label classification (as opposed to multi-label)
+    through the thr parameter which is set to a negative value.
+
+    Currently, the way to set this is to set
+       `test_cfg.rcnn.action_thr=-1.0`
+    ToDo: The ideal way would be for this to be automatically set when the
+    model cfg uses multilabel=False, however this could be a breaking change
+    and is left as a future exercise.
+    NB - this should not interfere with the evaluation in any case.
+
+    Args:
+        bboxes (Tensor): shape (n, 4)
+        labels (Tensor): shape (n, #num_classes)
+        num_classes (int): class number, including background class
+        thr (float): The score threshold used when converting predictions to
+            detection results. If a single negative value, uses single-label
+            classification
+    Returns:
+        list(ndarray): bbox results of each class
+    """
+    if bboxes.shape[0] == 0:
+        return list(np.zeros((num_classes - 1, 0, 5), dtype=np.float32))
+
+    bboxes = bboxes.cpu().numpy()
+    scores = labels.cpu().numpy()  # rename for clarification
+
+    # Although we can handle single-label classification, we still want scores
+    assert scores.shape[-1] > 1
+
+    # Robustly check for multi/single-label:
+    if not hasattr(thr, '__len__'):
+        multilabel = thr >= 0
+        thr = (thr, ) * num_classes
+    else:
+        multilabel = True
+
+    # Check Shape
+    assert scores.shape[1] == num_classes
+    assert len(thr) == num_classes
+
+    result = []
+    for i in range(num_classes - 1):
+        if multilabel:
+            where = (scores[:, i + 1] > thr[i + 1])
+        else:
+            where = (scores[:, 1:].argmax(axis=1) == i)
+        result.append(
+            np.concatenate((bboxes[where, :4], scores[where, i + 1:i + 2]),
+                           axis=1))
+    return result

mmaction/core/dist_utils.py

+# Copyright (c) OpenMMLab. All rights reserved.
+import numpy as np
+import torch
+import torch.distributed as dist
+from mmcv.runner import get_dist_info
+
+from ..utils import default_device
+
+
+def sync_random_seed(seed=None, device=default_device):
+    """Make sure different ranks share the same seed. All workers must call
+    this function, otherwise it will deadlock. This method is generally used in
+    `DistributedSampler`, because the seed should be identical across all
+    processes in the distributed group.
+
+    In distributed sampling, different ranks should sample non-overlapped
+    data in the dataset. Therefore, this function is used to make sure that
+    each rank shuffles the data indices in the same order based
+    on the same seed. Then different ranks could use different indices
+    to select non-overlapped data from the same data list.
+
+    Args:
+        seed (int, Optional): The seed. Default to None.
+        device (str): The device where the seed will be put on.
+            Default to 'cuda'.
+    Returns:
+        int: Seed to be used.
+    """
+    if seed is None:
+        seed = np.random.randint(2**31)
+    assert isinstance(seed, int)
+
+    rank, world_size = get_dist_info()
+
+    if world_size == 1:
+        return seed
+
+    if rank == 0:
+        random_num = torch.tensor(seed, dtype=torch.int32, device=device)
+    else:
+        random_num = torch.tensor(0, dtype=torch.int32, device=device)
+    dist.broadcast(random_num, src=0)
+    return random_num.item()

mmaction/core/evaluation/__init__.py

+# Copyright (c) OpenMMLab. All rights reserved.
+from .accuracy import (average_precision_at_temporal_iou,
+                       average_recall_at_avg_proposals, confusion_matrix,
+                       get_weighted_score, interpolated_precision_recall,
+                       mean_average_precision, mean_class_accuracy,
+                       mmit_mean_average_precision, pairwise_temporal_iou,
+                       softmax, top_k_accuracy, top_k_classes)
+from .eval_detection import ActivityNetLocalization
+from .eval_hooks import DistEvalHook, EvalHook
+
+__all__ = [
+    'DistEvalHook', 'EvalHook', 'top_k_accuracy', 'mean_class_accuracy',
+    'confusion_matrix', 'mean_average_precision', 'get_weighted_score',
+    'average_recall_at_avg_proposals', 'pairwise_temporal_iou',
+    'average_precision_at_temporal_iou', 'ActivityNetLocalization', 'softmax',
+    'interpolated_precision_recall', 'mmit_mean_average_precision',
+    'top_k_classes'
+]