HF <-> megatron checkpoint reshaping and conversion for GPT (#19317)

* HF <-> megatron checkpoint conversion handling reshaping from different tensor and parallel sizes * Apply suggestions from code review Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com> * addressing comments * add doc strings and 🐛 fixes Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

HF <-> megatron checkpoint reshaping and conversion for GPT (#19317)
* HF <-> megatron checkpoint conversion handling reshaping from different tensor and parallel sizes * Apply suggestions from code review Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com> * addressing comments * add doc strings and 🐛 fixes Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
56af8df3 · Sourab Mangrulkar · GitHub · 41ec5d0c · 56af8df3
Unverified Commit 56af8df3 authored Oct 07, 2022 by Sourab Mangrulkar Committed by GitHub Oct 07, 2022
Expand all Hide whitespace changes
Inline Side-by-side

Showing with 900 additions and 0 deletions

src/transformers/models/megatron_gpt2/checkpoint_reshaping_and_interoperability.py ...egatron_gpt2/checkpoint_reshaping_and_interoperability.py +900 -0

No files found.
--- a/src/transformers/models/megatron_gpt2/checkpoint_reshaping_and_interoperability.py
+++ b/src/transformers/models/megatron_gpt2/checkpoint_reshaping_and_interoperability.py