GitLab

:zip: #2106 tokenizer.tokenize speed improvement (3-8x) by caching added_tokens in a Set

[WIP] Sampling sequence generator for transformers

fix run_squad.py during fine-tuning xlnet on squad2.0

Fix TFXLNet

Add special tokens to documentation for bert examples to resolve issue: #1561

Bug-fix: Roberta Embeddings Not Masked

Support running tests in parallel

This looks like debug code accidentally committed in b18509c2.

Refs #2250.

This looks like a copy/paste mistake. Probably this test was never run.

Refs #2250.

This optimizes the total run time of the Circle CI test suite.

We're already using as many processes in parallel as we have CPU cores.
Furthermore, the number of core may be incorrectly calculated as 36
(we've seen this in pytest-xdist) which make compound the problem.

PyTorch performance craters without this.

This should prevent two issues:

- hitting API rate limits for tests that hit the HF API
- multiplying the cost of expensive test setups

Set the number of CPUs manually based on the Circle CI resource class,
or else we're getting 36 CPUs, which is far too much (perhaps that's
the underlying hardware and not what Circle CI allocates to us).

Don't parallelize the custom tokenizers tests because they take less
than one second to run and parallelization actually makes them slower.

Since the file is written to the filesystem, a filesystem lock is the
way to go here. Add a dependency on the third-party filelock library to
get cross-platform functionality.

Caching models across test cases and across runs of the test suite makes
slow tests somewhat more bearable.

Use gettempdir() instead of /tmp in tests. This makes it easier to
change the location of the cache with semi-standard TMPDIR/TEMP/TMP
environment variables.

Fix #2222.

This allows moving the file instead of copying it, which is more
reliable. Also it avoids writing large amounts of data to /tmp,
which may not be large enough to accomodate it.

Refs #2222.

This looks like it could be expensive, so don't run it twice.

cc @lysandrejik

- Create the output directory (whose name is passed by the user in the "save_directory" parameter) where it will be saved encoder and decoder, if not exists.
- Empty the output directory, if it contains any files or subdirectories.
- Create the "encoder" directory inside "save_directory", if not exists.
- Create the "decoder" directory inside "save_directory", if not exists.
- Save the encoder and the decoder in the previous two directories, respectively.

Fix Camembert and XLM-R `decode` method- Fix NER pipeline alignement

Numpy compatibility for sentence piece

convert to int earlier