GitLab

Add nlp dataset used

Add dataset used as it is now part of nlp package

* Add ComVE model cards

* Apply suggestions from code review
Co-authored-by: Julien Chaumond <chaumond@gmail.com>

* Created model card for my extreme summarization model

* Update model_cards/yuvraj/xSumm/README.md
Co-authored-by: Julien Chaumond <chaumond@gmail.com>

* Created model card for my summarization model

* Update model_cards/yuvraj/summarizer-cnndm/README.md
Co-authored-by: Julien Chaumond <chaumond@gmail.com>

- Maybe the result of this query answers the question You did some days ago @julien-c ;-)

The DataParallel training was fixed in https://github.com/huggingface/transformers/pull/5733, this commit also fixes the evaluation. It's more convenient when the user enables both `do_train` and `do_eval`.

Huge MT speedup!

* Improve doc of use_cache

* Update src/transformers/configuration_xlnet.py
Co-authored-by: Teven <teven.lescao@gmail.com>
Co-authored-by: Teven <teven.lescao@gmail.com>

* DataParallel fixes:

1. switched to a more precise check
-        if self.args.n_gpu > 1:
+        if isinstance(model, nn.DataParallel):

2. fix tests - require the same fixup under DataParallel as the training module

* another fix

* Don't pass sampler for iterable dataset

* Added check for test and eval dataloaders.

* Formatting

* Don't pass sampler for iterable dataset

* Added check for test and eval dataloaders.

* Formatting

* Cleaner if nesting.

* Added test for trainer and iterable dataset

* Formatting for test

* Fixed import when torch is available only.

* Added require torch decorator to helper class

* Moved dataset class inside unittest

* Removed nested if and changed model in test

* Checking torch availability for IterableDataset

cc @lhoestq @thomwolf

Also cc'ing model author @nreimers => Model pages now properly link to the dataset pages (and in the future, eval results, etc.)

Co-authored-by: Pradhy729 <49659913+Pradhy729@users.noreply.github.com>

Slightly breaking change, changes functionality for `use_cache` in XLNet: if use_cache is True and mem_len is 0 or None (which is the case in the base model config), the model behaves like GPT-2 and returns mems to be used as past in generation. At training time `use_cache` is overriden and always True.

This reverts commit 13be4872.

Slightly breaking change, changes functionality for `use_cache` in XLNet: if use_cache is True and mem_len is 0 or None (which is the case in the base model config), the model behaves like GPT-2 and returns mems to be used as past in generation. At training time `use_cache` is overriden and always True.

Co-authored-by: Sam Shleifer <sshleifer@gmail.com>

This reverts commit 0b2da0e5.

Slightly breaking change, changes functionality for `use_cache` in XLNet: if use_cache is True and mem_len is 0 or None (which is the case in the base model config), the model behaves like GPT-2 and returns mems to be used as past in generation. At training time `use_cache` is overriden and always True.