GitLab

As pointed out in #1545, when using an uncased model, and adding
a new uncased token, the tokenizer does not correctly identify this
in the case that the input text contains the token in a cased format.

For instance, if we load bert-base-uncased into BertTokenizer, and
then use .add_tokens() to add "cool-token", we get the expected
result for .tokenize('this is a cool-token'). However, we get a
possibly unexpected result for .tokenize('this is a cOOl-Token'),
which in fact mirrors the result for the former from before the new
token was added.

This commit adds
- functionality to PreTrainedTokenizer to handle this
situation in case a tokenizer (currently Bert, DistilBert,
and XLNet) has the do_lower_case=True kwarg by:
    1) lowercasing tokens added with .add_tokens()
    2) lowercasing text at the beginning of .tokenize()
- new common test case for tokenizers

https://github.com/huggingface/transformers/issues/1545

Make imports and args.overwrite_cache match run_glue.py

converted from https://github.com/openai/gpt-2-output-dataset/tree/master/detector

Co-Authored-By: Lysandre Debut <lysandre.debut@reseau.eseo.fr>
Co-Authored-By: Jong Wook Kim <jongwook@nyu.edu>
Co-Authored-By: Jeff Wu <wuthefwasthat@gmail.com>

cf. https://docs.pytest.org/en/latest/example/simple.html

model forwards can take an inputs_embeds param

add progress bar to convert_examples_to_features

Add speed log to examples/run_squad.py

Fixing mode in evaluate during training

It takes considerate amount of time (~10 min) to parse the examples to features, it is good to have a progress-bar to track this

Fix #1623

Co-Authored-By: Santosh-Gupta <San.Gupta.ML@gmail.com>

Add common getter and setter for input_embeddings & output_embeddings

Fix token order in xlnet preprocessing for SQuAD

Fix https://github.com/huggingface/transformers/issues/1673