Skip to content

GitLab

"src/git@developer.sourcefind.cn:renzhc/diffusers_dcu.git" did not exist on "5ffb73d4aeac9eaef8366d7b21872d64009bd1c7"
Commit 1362b21b authored by Myle Ott's avatar Myle Ott Committed by Facebook Github Bot
Browse files

Correctly zero padding index in TransformerSentenceEncoder 

Summary: Pull Request resolved: https://github.com/pytorch/fairseq/pull/912

Differential Revision: D16536561

Pulled By: myleott

fbshipit-source-id: 54c5c20a826a14f4e690770e027bcb282acdf911
parent 8207f263
Hide whitespace changes
Inline Side-by-side
Showing with 1 addition and 0 deletions
fairseq/modules/transformer_sentence_encoder.py
View file @ 1362b21b
...@@ -38,6 +38,7 @@ def init_bert_params(module): ...@@ -38,6 +38,7 @@ def init_bert_params(module):
module.bias.data.zero_() module.bias.data.zero_()
if isinstance(module, nn.Embedding): if isinstance(module, nn.Embedding):
module.weight.data.normal_(mean=0.0, std=0.02) module.weight.data.normal_(mean=0.0, std=0.02)
module.weight.data[module.padding_idx].zero_()
if isinstance(module, MultiheadAttention): if isinstance(module, MultiheadAttention):
module.in_proj_weight.data.normal_(mean=0.0, std=0.02) module.in_proj_weight.data.normal_(mean=0.0, std=0.02)
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment