GitLab

Summary:
Pull Request resolved: https://github.com/pytorch/fairseq/pull/710

Previously there was a bug in how we dealt with padding when computing the input representation from the segment and position embedding. D15144912 fixed this by adding an offset based on the padding id. However this makes assumptions about the padding id which may not hold true for vocabularies built outside of pyText and fairseq. Based on a discussion with barlaso, this diff 0's out all the embeddings associated with the padding.

Reviewed By: borguz

Differential Revision: D15209395

fbshipit-source-id: 5573020e610f5466e673fe3845c3ed34ebb5c44d

Summary:
Co-authored-by: myleott <myleott@fb.com>

Changing `data` to be `str` with colon separated list for loading sharded datasets. This change is useful for loading large datasets that cannot fit into, memory. The large dataset can be sharded and then each shard is loaded in one epoch in roudrobin manner.

For example, if there are `5` shards of data and `10` epochs then the shards will be iterated upon `[0, 1, 2, 3, 4, 0, 1, 2, 3, 4]`.

myleott We need to look into `translation.py` as it currently already expects a list and then concats the datasets.
Pull Request resolved: https://github.com/pytorch/fairseq/pull/696

Differential Revision: D15214049

fbshipit-source-id: 03e43a7b69c7aefada2ca668abf1eac1969fe013

Summary: Pull Request resolved: https://github.com/pytorch/fairseq/pull/707

Differential Revision: D15219014

Pulled By: myleott

fbshipit-source-id: f38f2cf817d05e0871ff9084a810d109848e827c

Summary:
Co-authored-by: jingfeidu <jingfeidu@fb.com>

1) Adding `masked_lm` task for BERT like training. Code mostly taken from jingfeidu 's implementation.

2) Added `has_eos` option to `block_pair_dataset` for working with dataset that has been preprocessed with having `eos`.

Depends on: https://github.com/pytorch/fairseq/pull/696
Pull Request resolved: https://github.com/pytorch/fairseq/pull/697

Differential Revision: D15214050

fbshipit-source-id: c179ce2d70e59d2ddc941b13ceda99d929878931

Summary:
Pass required "sample_key" argument to forward-backward call in semi-supervised task.
Pull Request resolved: https://github.com/pytorch/fairseq/pull/706

Differential Revision: D15217957

Pulled By: pipibjc

fbshipit-source-id: bf943d566c5caa67682dfb16ff8b7c432323cdba

Summary:
Pull Request resolved: https://github.com/pytorch/fairseq/pull/705

This adds functionality in fairseq to load a pretrained encoder or decoder from another pretrained model into the current model.

Reviewed By: jmp84

Differential Revision: D15207084

fbshipit-source-id: 32a710ff77389928e20793c71d312863df9dd8ae

Reviewed By: chenyangyu1988

Differential Revision: D14784219

fbshipit-source-id: 273888d6e3d22a01d5e7edfbc786195e7b78efef

Summary:
Pull Request resolved: https://github.com/pytorch/fairseq/pull/703

It's better to write one checkpoint and copy it, rather than repeatedly pickling the model via torch.save.

Differential Revision: D15213778

fbshipit-source-id: 27dad39853b09dab7f0e11c030313019f035dbb0

Summary: Pull Request resolved: https://github.com/pytorch/fairseq/pull/695

Differential Revision: D15182613

Pulled By: myleott

fbshipit-source-id: 4196346517d8e75ed9e903e9e01ab943d086f6f1

Summary:
Pull Request resolved: https://github.com/pytorch/translate/pull/508

The previous version applied the temperature after the softmax. Fix that, and
also generalize so it works with other search approaches.
Pull Request resolved: https://github.com/pytorch/fairseq/pull/694

Differential Revision: D15175160

Pulled By: myleott

fbshipit-source-id: cc87ff0e97a8a1dd37f9983163f58a8641155ab0

Summary:
It was tedious defining these, let's try just taking the first batch lazily instead.
Pull Request resolved: https://github.com/pytorch/fairseq/pull/699

Differential Revision: D15188266

Pulled By: myleott

fbshipit-source-id: a4c9f7ee3111278faaffa8a22ba91ed5f50e143d

Summary:
We can later get rid off `BertLayerNorm` also, as I think the implementation of that is exactly same as `LayerNorm`. (will confirm with jingfeidu on that).
But this should be drop and replace.
Pull Request resolved: https://github.com/pytorch/fairseq/pull/702

Differential Revision: D15213116

Pulled By: myleott

fbshipit-source-id: ba5c00e1129a4443ef5d3d8bebd0bb6c6ee3b188

Summary: See comment

Reviewed By: jay-mahadeokar

Differential Revision: D15070187

fbshipit-source-id: ffefca0effb2cc866ce6fa22a59d5419b592fb7b

Summary:
Pull Request resolved: https://github.com/fairinternal/fairspeq/pull/2

Pull Request resolved: https://github.com/pytorch/fairseq/pull/689

We found not raising OOM during trainer.train_step causes various
issue, including NCCL hangs / gloo sync errors because gradient is not synced
properly. Before we found the root cause, let's give users an option to raise
OOMs.

Reviewed By: jmp84

Differential Revision: D15170357

fbshipit-source-id: 3e15e4e111a8380612157955509c39821a216ec4

Summary:
Added bert_large architecture
Pull Request resolved: https://github.com/pytorch/fairseq/pull/698

Differential Revision: D15198698

Pulled By: myleott

fbshipit-source-id: 1dc9e8d4c8c877d15afffe5fe581b4b93eefbc66

Summary:
- Add learned positional embedding binary flag to masked LM model.
- Add base arch config for masked LM model which sets all the binary parameters to False. Otherwise some of the binary flag parameters will always be override by config in `xlm_architecture` (e.g. encoder_learned_pos)

Reviewed By: liezl200

Differential Revision: D15054487

fbshipit-source-id: d78827f352b9160a89c9dc4f45b9fce15a2f234d

Summary:
This should make rendezvous happen as lazily as possible.
Pull Request resolved: https://github.com/pytorch/fairseq/pull/687

Differential Revision: D15151145

Pulled By: myleott

fbshipit-source-id: d70816a85414c5d509a6b12e2b339b4736db2c88

Summary: Pull Request resolved: https://github.com/pytorch/fairseq/pull/693

Differential Revision: D15174831

fbshipit-source-id: 98688b1269ead5694e5116659ff64507d3c0d1c0

Summary: Pull Request resolved: https://github.com/pytorch/fairseq/pull/692

Differential Revision: D15174954

fbshipit-source-id: 1a7bff9aeed3e2cc658577be9d79e8c9f72314c2

Summary:
Changes include:
1. Added get_normalized_probabilities to the encoder-only base class FairseqEncoderModel
2. Made CTCCriterion work for both batch_first (LSTMSubsampleEncoderModel) and batch_second (LSTMEncoderOnly) encoder types
3. Added tests for different encoder and CTC combinations.

TODO:
CTC still doesn't work for VGGLSTMEncoderModel so I have disabled that. Will debug and send out fix in another diff.

Reviewed By: jay-mahadeokar

Differential Revision: D15158818

fbshipit-source-id: acb484bad705c937d676d2c3dcde3e3562d68ed9

Summary: Pull Request resolved: https://github.com/pytorch/fairseq/pull/691

Differential Revision: D15172543

Pulled By: myleott

fbshipit-source-id: f2b626ff7f5e95f0ddc83c105af7ab9d092a135e

Summary: Pull Request resolved: https://github.com/pytorch/fairseq/pull/684

Differential Revision: D15154631

Pulled By: myleott

fbshipit-source-id: 5e7dd9651d9ed239b60c51b9a11d08c80307d3ba

Summary:
Pull Request resolved: https://github.com/pytorch/translate/pull/494

Pull Request resolved: https://github.com/pytorch/fairseq/pull/657

Library side change split from D14924942

Added 2 arguments for load_dataset in PytorchTranslateTask
1. dataset_upsampling. A nested dictionary {direction:{dataset: upsampling_ratio}}. Upsampling_ratio larger than one mean that the bitext is ob- served more often than actually present in the combined bitext and synthetic training corpus.

2. dataset_relative_ratio. A tuple (dataset, ratio). The ratio represents the frequency certain dataset gets sampled to the rest of corpora map.

At most one of them could be specified.

Reviewed By: liezl200

Differential Revision: D15041293

fbshipit-source-id: 92daad29895c234e26d1b19f121106118a3957ad

Summary: Pull Request resolved: https://github.com/pytorch/fairseq/pull/685

Differential Revision: D15154647

Pulled By: myleott

fbshipit-source-id: 36c72359755192a4a53367e19f8dd006791d483c

Summary:
Pull Request resolved: https://github.com/pytorch/fairseq/pull/664

Previously arguments for noising (dropout_prob for WordDropout and max_shuffle_distance for WordShuffle) are only passed in noising() so it could not be customized in NoisingDataset.

Now add default argument in initializer so the value could be specified at construction.

Reviewed By: liezl200

Differential Revision: D15071632

fbshipit-source-id: 59a9bf5a5e6d03c1e74f1b31c1927e221cb11dfa

Summary:
Co-authored-by: jingfeidu <jingfeidu@fb.com>

The implementation is by Jingfei Du from branch "bigbert". Copied over to this CR to get it merged in isolation since other changes seem to be already in master.

**Small changes from original:**
Added following line in `__init__` as discovered by myleott :

```
self.optimizer.set_lr(self.warmup_factor * self.lr)
```
Pull Request resolved: https://github.com/pytorch/fairseq/pull/683

Reviewed By: myleott

Differential Revision: D15149628

Pulled By: myleott

fbshipit-source-id: 5f715611182cdd111e636c66d5f24aa88fa03e29

Summary: Pull Request resolved: https://github.com/pytorch/fairseq/pull/682

Differential Revision: D15147735

Pulled By: myleott

fbshipit-source-id: 4a5f12c0b24591f964fe1f465be3775a67578e79

Summary: Pull Request resolved: https://github.com/pytorch/fairseq/pull/681

Differential Revision: D15147107

fbshipit-source-id: 4452c98059586a4d748868a7659329285a76d5ef

Summary:
- Add --add-bos-token option to LM task
- Cleanup utils.py and options.py
Pull Request resolved: https://github.com/pytorch/fairseq/pull/654

Differential Revision: D15041794

Pulled By: myleott

fbshipit-source-id: 3ad00007769d5f48308052cfd40de39c5ffa1a6e

Summary:
Pull Request resolved: https://github.com/pytorch/fairseq/pull/672

title

Reviewed By: jmp84, pipibjc

Differential Revision: D15094977

fbshipit-source-id: c24e4ec9355b53e1585ac4da32809f1c339c7364

Summary:
Pull Request resolved: https://github.com/pytorch/fairseq/pull/680

Some embedding names were renamed but this one was missed

So far I've only seen this affect our runs during continuing training. If you encountered any errors when continuing training from an XLM save_dir, rebasing past this diff (or patching this and canarying) should fix the problem

Reviewed By: pipibjc

Differential Revision: D15137463

fbshipit-source-id: c72067f16aaf1ba2b8286938bd25a19b70ae8712

Summary:
Add missing backslash.
Pull Request resolved: https://github.com/pytorch/fairseq/pull/679

Differential Revision: D15122270

Pulled By: myleott

fbshipit-source-id: fbdfde648051294eaa9f7a4e0c4cfbc57491a718

Summary: Pull Request resolved: https://github.com/pytorch/fairseq/pull/676

Differential Revision: D15114128

Pulled By: myleott

fbshipit-source-id: b11dde77b2f2610d33649101aea03fb5a3eeb56a

Summary:
Log fairseq's `args` and `sys.argv` in tensorboard to easily identify run hyperparameters from within tensorboard.

The idea was suggested in https://twitter.com/Thom_Wolf/status/1106300583835766786
Pull Request resolved: https://github.com/pytorch/fairseq/pull/673

Differential Revision: D15114159

Pulled By: myleott

fbshipit-source-id: d48133a7f629dffe984836712390c317916cf413

Summary: Pull Request resolved: https://github.com/pytorch/fairseq/pull/669

Differential Revision: D15114160

Pulled By: myleott

fbshipit-source-id: 64f4a8154c8931ddbbe459d4d4a54c46680ad6b6

Summary:
Pull Request resolved: https://github.com/pytorch/fairseq/pull/670

Pytorch-translate task needs to use extra arguments (such as vocabulary objects). By passing kwargs, we are able to have the ability to have extra arguments in setup_task

Reviewed By: akinh, pipibjc

Differential Revision: D15086810

fbshipit-source-id: 555f7976020eaac1febb8226f5a0055af0407ea6

Summary:
Pull Request resolved: https://github.com/pytorch/fairseq/pull/667

Use smaller models so that unittests won't timeout

Reviewed By: pipibjc

Differential Revision: D15056894

fbshipit-source-id: af9fbda6ea6e56cf82d52555620121b189e2f013

Summary:
Pull Request resolved: https://github.com/pytorch/fairseq/pull/666

Option to load the XLM weights into only the encoder or the decoder

Reviewed By: pipibjc

Differential Revision: D14881004

fbshipit-source-id: 6d0d598ea9c445ec468f71b8e855712de89a5dac

Summary:
Pull Request resolved: https://github.com/pytorch/fairseq/pull/629

Use GeLU as an alternate activation layer for ReLU.

Reviewed By: lematt1991

Differential Revision: D14689851

fbshipit-source-id: 7ec81fa34bc7bd0e1e43b337847ae932dcbf8b15

Summary:
Pull Request resolved: https://github.com/pytorch/fairseq/pull/653

After this diff, you can train a transformer model with --activation-fn 'relu', 'gelu', or 'gelu_fast'

gelu_fast is the default implementation in https://github.com/hendrycks/GELUs/blob/master/mnist_fcn.py#L72-L77
gelu is the alternate implementation in https://github.com/hendrycks/GELUs/blob/master/mnist_fcn.py#L72-L77 and the default implementation in https://github.com/facebookresearch/XLM

Reviewed By: pipibjc

Differential Revision: D14966006

fbshipit-source-id: 94e95fb99bd548ba47cf23b4999265c7b6833fc1