GitLab

Reason for issue was that optimzation steps where computed from example size, which is different from actual size of dataloader when an example is chunked into multiple instances.

Solution in this pull request is to compute num_optimization_steps directly from len(data_loader).

Updating learning rate with special warm up in examples

Make the epsilon of LayerNorm configurable.

move pytroch_pretrained_bert cache folder under same path as torch

Create optimizer only when args.do_train is True

Fix documentation typo

Small fix to remove shifting of lm labels during pre process of RocStories.

Fix #537

small fix to remove shifting of lm labels during pre process of roc stories, as this shifting happens interanlly in the model

Update example files so that tr_loss is not affected by args.gradient…

fixed BertForMultipleChoice model init and forward pass

Docs for new learning rate code

Fix training schedules in examples to match new API

- added some images for illustration
- updated comments in optimization

Hubconf

Fix --reduce_memory in finetune_on_pregenerated

Fix indentation weirdness in GPT-2 example.

Learning rate schedules improvement + extension

Pulling commits from main repo

- removed unused plotting code
- using ABC for LRSchedule
- added some schedule object init tests