"git@developer.sourcefind.cn:OpenDAS/vision.git" did not exist on "face20bdbd29b78976ed4d01524b2de4b7d77a2f"
Commit 6be5f07c authored by Matt Le's avatar Matt Le Committed by Facebook Github Bot
Browse files

Use bert init for xlm_base

Summary:
Use bert init for xlm_base.  This seems to be much closer to what is done in the [XLM](https://github.com/facebookresearch/XLM/blob/master/src/model/transformer.py#L44) repo.

At update 10 with BERT init (f121471600), loss starts at 14.234

At update 10 without BERT init (f121471612), loss starts at 154.423

Reviewed By: liezl200, pipibjc

Differential Revision: D15874836

fbshipit-source-id: f81bf83a078992d7476ba7fdf263b731a9f5b66d
parent 881381cf
...@@ -352,4 +352,5 @@ def xlm_architecture(args): ...@@ -352,4 +352,5 @@ def xlm_architecture(args):
args.activation_fn = getattr(args, 'activation_fn', 'gelu') args.activation_fn = getattr(args, 'activation_fn', 'gelu')
args.encoder_normalize_before = getattr(args, 'encoder_normalize_before', False) args.encoder_normalize_before = getattr(args, 'encoder_normalize_before', False)
args.pooler_activation_fn = getattr(args, 'pooler_activation_fn', 'tanh') args.pooler_activation_fn = getattr(args, 'pooler_activation_fn', 'tanh')
args.apply_bert_init = getattr(args, 'apply_bert_init', True)
base_architecture(args) base_architecture(args)
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment