Commits · d99142a0177a2462cbda07a31aaa8e68b4e85461 · OpenDAS / TransformerEngine

23 Jan, 2024 1 commit

[PyTorch] Fix for deferred init bug causing NeMo MLPerf LLM crash (#619) · 3c04c417

Alp Dener authored Jan 23, 2024



* added missing parameter materialization on real device for LayerNorm and RMSNorm
Signed-off-by: Alp Dener <adener@nvidia.com>

* added new unittest for deferred initialization and modified parameter materialization to support standalone execution outside of FSDP
Signed-off-by: Alp Dener <adener@nvidia.com>

* restored tensor parallel attributes that were being wiped out by the parameter reset
Signed-off-by: Alp Dener <adener@nvidia.com>

* fixed incorrect order of fp8 metadata initialization
Signed-off-by: Alp Dener <adener@nvidia.com>

* added deferred init unittest to the QA script
Signed-off-by: Alp Dener <adener@nvidia.com>

---------
Signed-off-by: Alp Dener <adener@nvidia.com>

3c04c417