GitLab

* BertJapaneseTokenizer accept options for mecab

* black

* fix mecab_option to Option[str]

* add bert bahasa readme

* update readme

* update readme

* added xlnet

* added tiny-bert and fix xlnet readme

* added albert base

Update AutoModel & AutoTokernizer loading.

* added model_cards for polish squad models

* corrected mistake in polish design cards
Co-authored-by: Henryk Borzymowski <henryk.borzymowski@pwc.com>

* Create README.md

* added meta block (language: german)

* Added additional information about test data

* Resizing embedding matrix after sending it to the optimizer prevents from updating the newly resized matrix.

* Remove space for style matter

* solve conflicts

* improve comments

* replace heavy t5 models with tiny random models as was done by sshleifer

* fix isort

* change tf t5 argument naming for TF 2.2

* correct bug in testing

[T5, Testst] Add extensive hard-coded integration tests and make sure PT and TF give equal results (#3550)

* add some t5 integration tests

* finish summarization and translation integration tests for T5 - results loook good

* add tf test

* fix == vs is bug

* fix tf beam search error and make tf t5 tests pass

* add bert bahasa readme

* update readme

* update readme

* added xlnet

* added tiny-bert and fix xlnet readme

Create model card for: distilbert-multi-finetuned-for-xqua-on-tydiqa

* Start cleaning examples

* Fixup

* add bad words list

* make style

* add bad_words_tokens

* make style

* better naming

* make style

* fix typo

* fix conflicts

* add model size argument to summarization

* correct wrong import

* fix isort

* correct imports

* other isort make style

* make style

- Show that the last uploaded version was trained on more data (custom_license files)

Fix typo

* Create README.md

* Update README.md

* feat: add model card bert-imdb

* feat: add model card gpt2-imdb-pos

* feat: add model card gpt2-imdb

* Using loaded checkpoint with --do_predict

Without this fix, I'm getting near-random validation performance for a trained model, and the validation performance differs per validation run. I think this happens since the `model` variable isn't set with the loaded checkpoint, so I'm using a randomly initialized model. Looking at the model activations, they differ each time I run evaluation (but they don't with this fix).

* Update checkpoint loading

* Fixing model loading

Co-authored-by: Ioannis Douratsos <ioannisd@amazon.com>