transformer_engine/pytorch/attention.py · c0caadbe1cdba42efb09aba205f4de0905739f26 · OpenDAS / TransformerEngine

"docs/conf.py" did not exist on "8c2527766639f8cd7a86400a154e0a2fe44c90e6"

Expose `rotary_base` as an arg instead of hardcoding (#944) · c0caadbe

Sudhakar Singh authored Sep 18, 2024



* make rotary_base arg
Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>

* rotary base can be a float
Co-authored-by: Tim Moon <4406448+timmoon10@users.noreply.github.com>
Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>

---------
Signed-off-by: Sudhakar Singh <sudhakars@nvidia.com>
Co-authored-by: Tim Moon <4406448+timmoon10@users.noreply.github.com>

c0caadbe

attention.py 373 KB

Replace attention.py