GitLab

fix repetition penalty error in modeling_utils.py

Fix F841 flake8 warning

fix bug in prepare inputs for language generation for xlm for effective batch_size > 1

Correct tokenization for special and added tokens

revert erroneous fix #2276

Improve setup and requirements

Use -e only in docs targeted at contributors.

If a user copy-pastes  command line with [--editable], they will hit
an error. If they don't know the --editable option, we're giving them
a choice to make before they can move forwards, but this isn't a choice
they need to make right now.

Improve handling of optional imports

This is really just for consistency.

Else, the next line fails with a confusion exception because the spm
variable isn't defined.

ImportError isn't really appropriate when there's no import involved.

Most module use the simpler `raise` version. Normalize those that don't.

Most users don't need it.

This will improve version coverage.

If a user or contributor ran `pip install -e .` on transformers < 3.0,
pip created a transformers.egg-info directory next to the transformers
directory at the root of the repository.

In transformers 3.0, the source is in a `src` subdirectory.
`pip install -e .` creates a transformers.egg-info directory there.
However, pip will still pick transformers.egg-info from the previous
location. This is a bug: https://github.com/pypa/pip/issues/5466

Users and contributors are likely to hit this problem because the
documentation for transformers 3.0 relies heavily on extra_requires
which didn't exist in earlier versions, so aren't defined in a stale
transformers.egg-info directory.

If such a directory exists, remove it. It's autogenerated, gitignored
and not supposed to contain anything of value.

Also provide shortcuts in a Makefile.

It's superseded by the "docs" extras.

It's redundant with setup.py and, also, incomplete (e.g. numpy).

It was generated once, likely in a non-reproducible way (pip freeze
in a contributor's local environment), and never updated.