GitLab

work around hipify not finding headers

add distributed fused lamb

Make torch version check numeric

IFU-2020-03-04

Revert "pass all TensorListMetadata as pointer to pinned host memory (#13)

This reverts commit bdd481d1.

* copy-paste friendly

* fix import container_abcs issue

Nightly PyTorch has removed `container_abcs` from `torch._six`.  https://github.com/pytorch/pytorch/commit/58eb23378f2a376565a66ac32c93a316c45b6131#diff-b3c160475f0fbe8ad50310f92d3534172ba98203387a962b7dc8f4a23b15cf4dL35

* fix import container_abcs issue

Nightly PyTorch has removed `container_abcs` from `torch._six`.
https://github.com/pytorch/pytorch/commit/58eb23378f2a376565a66ac32c93a316c45b6131#diff-b3c160475f0fbe8ad50310f92d3534172ba98203387a962b7dc8f4a23b15cf4dL35

* keep existing for pytorch1.7 and earlier

- incorrect use of __shfl_down
- fix warp size assumptions
- update unit tests to exit on failure

use __launch_bounds__(1024) for multi_tensor_apply, re-enable skipped tests

Co-authored-by: Sukru Eryilmaz <seryilmaz@computelab-dgx1v-32.nvidia.com>

IFU-2021-01-18

Mostly whitespace or formatting issues addressed.
Diff with upstream is reduced; ROCm changes are more clear.

Conflicts:
csrc/multi_tensor_apply.cuh
setup.py
tests/L0/run_optimizers/test_adagrad.py
tests/L0/run_optimizers/test_fused_optimizer.py
tests/L0/run_optimizers/test_lamb.py

Fix reduce_block_into_lanes for multi_tensor_l2norm for ROCm

Skip the unit tests

Update ASP README to highlight default recipe

The Recipe was presented after some non-standard API calls, so moving the suggested usage up, giving it its own section, and reinforcing the suggested usage in the non-standard section.

Hipify revamp changes for apex extensions on ROCm.

* add flag for DistributedAdam: step_support_amp_scaling
Co-authored-by: Kexin Yu <kexiny@nvidia.com>
Co-authored-by: Kexin Yu <kexinznzn›@gmail.com>

* fuse dropout into softmax in fprop for additive mask case