- 30 Nov, 2020 1 commit
-
-
Sylvain Gugger authored
-
- 29 Nov, 2020 1 commit
-
-
Stas Bekman authored
* implement job skipping for doc-only PRs * silent grep is crucial * wip * wip * wip * wip * wip * wip * wip * wip * let's add doc * let's add code * revert test commits * restore * Better name * Better name * Better name * some more testing * some more testing * some more testing * finish testing
-
- 23 Nov, 2020 1 commit
-
-
Julien Chaumond authored
* Make ci fail * Try to make tests actually run? * CI finally failing? * Fix CI * Revert "Fix CI" This reverts commit ca7923be7334d4e571b023478ebdd6b33dfd0ebb. * Ooops wrong one * one more try * Ok ok let's move this elsewhere * Alternative to globals() (#8667) * Alternative to globals() * Error is raised later so return None * Sentencepiece not installed make some tokenizers None * Apply Lysandre wisdom * Slightly clearer comment? cc @sgugger Co-authored-by:Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
-
- 19 Nov, 2020 1 commit
-
-
Sylvain Gugger authored
* Fix the CI and tests * Fix quality * Remove that m form nowhere
-
- 11 Nov, 2020 2 commits
-
-
Funtowicz Morgan authored
* Update deploy-docs dependencies on CI to enable Flax Signed-off-by:
Morgan Funtowicz <morgan@huggingface.co> * Added pair of "" Signed-off-by:
-
Funtowicz Morgan authored
* First addition of Flax/Jax documentation Signed-off-by:
Lysandre <lysandre.debut@reseau.eseo.fr>
-
- 04 Nov, 2020 2 commits
-
-
Sylvain Gugger authored
-
Sylvain Gugger authored
-
- 03 Nov, 2020 2 commits
-
-
Stas Bekman authored
* make it possible to invoke testconf.py in both test suites without crashing on having the same option added * perl -pi -e 's|--make_reports|--make-reports|' to be consistent with other opts * add `pytest --make-reports` to all CIs (and artifacts) * fix
-
Sylvain Gugger authored
-
- 29 Oct, 2020 1 commit
-
-
Sylvain Gugger authored
* Add a template for example scripts and apply it to mlm * Formatting * Fix test * Add plm script * Add a template for example scripts and apply it to mlm * Formatting * Fix test * Add plm script * Add a template for example scripts and apply it to mlm * Formatting * Fix test * Add plm script * Styling
-
- 28 Oct, 2020 1 commit
-
-
Lysandre Debut authored
-
- 27 Oct, 2020 2 commits
-
-
Lysandre Debut authored
-
Stas Bekman authored
* better reports * a whole bunch of reports in their own files * clean up * improvements * github artifacts experiment * style * complete the report generator with multiple improvements/fixes * fix * save all reports under one dir to easy upload * can remove temp failing tests * doc fix * some cleanup
-
- 26 Oct, 2020 1 commit
-
-
Sylvain Gugger authored
* Important files * Styling them all * Revert "Styling them all" This reverts commit 7d029395fdae8513b8281cbc2a6c239f8093503e. * Syling them for realsies * Fix syntax error * Fix benchmark_utils * More fixes * Fix modeling auto and script * Remove new line * Fixes * More fixes * Fix more files * Style * Add FSMT * More fixes * More fixes * More fixes * More fixes * Fixes * More fixes * More fixes * Last fixes * Make sphinx happy
-
- 23 Oct, 2020 1 commit
-
-
Thomas Wolf authored
[tests|tokenizers] Refactoring pipelines test backbone - Small tokenizers improvements - General tests speedups (#7970) * WIP refactoring pipeline tests - switching to fast tokenizers * fix dialog pipeline and fill-mask * refactoring pipeline tests backbone * make large tests slow * fix tests (tf Bart inactive for now) * fix doc... * clean up for merge * fixing tests - remove bart from summarization until there is TF * fix quality and RAG * Add new translation pipeline tests - fix JAX tests * only slow for dialog * Fixing the missing TF-BART imports in modeling_tf_auto * spin out pipeline tests in separate CI job * adding pipeline test to CI YAML * add slow pipeline tests * speed up tf and pt join test to avoid redoing all the standalone pt and tf tests * Update src/transformers/tokenization_utils_base.py Co-authored-by:
Sam Shleifer <sshleifer@gmail.com> * Update src/transformers/pipelines.py Co-authored-by:
Lysandre Debut <lysandre@huggingface.co> * Update src/transformers/testing_utils.py Co-authored-by:
-
- 20 Oct, 2020 1 commit
-
-
Stas Bekman authored
* [flax] fix repo_check Unless, this is actually a problem, this adds `modeling_flax_utils` to ignore list. otherwise currently it expects to have a 'tests/test_modeling_flax_utils.py' for it. for context please see: https://github.com/huggingface/transformers/pull/3722#issuecomment-712360415 * fix 2 more issues * merge https://github.com/huggingface/transformers/pull/7919/
-
- 19 Oct, 2020 2 commits
-
-
Funtowicz Morgan authored
* WIP flax bert * Initial commit Bert Jax/Flax implementation. * Embeddings working and equivalent to PyTorch. * Move embeddings in its own module BertEmbeddings * Added jax.jit annotation on forward call * BertEncoder on par with PyTorch ! :D * Add BertPooler on par with PyTorch !! * Working Jax+Flax implementation of BertModel with < 1e-5 differences on the last layer. * Fix pooled output to take only the first token of the sequence. * Refactoring to use BertConfig from transformers. * Renamed FXBertModel to FlaxBertModel * Model is now initialized in FlaxBertModel constructor and reused. * WIP JaxPreTrainedModel * Cleaning up the code of FlaxBertModel * Added ability to load Flax model saved through save_pretrained() * Added ability to convert Pytorch Bert model to FlaxBert * FlaxBert can now load every Pytorch Bert model with on-the-fly conversion * Fix hardcoded shape values in conversion scripts. * Improve the way we handle LayerNorm conversion from PyTorch to Flax. * Added positional embeddings as parameter of BertModel with default to np.arange. * Let's roll FlaxRoberta ! * Fix missing position_ids parameters on predict for Bert * Flax backend now supports batched inputs Signed-off-by:
Stefan Schweter <stefan@schweter.it> * Attempt to make style * Attempt to make style in tests too * Added jax & jaxlib to the flax optional dependencies. * Attempt to fix flake8 warnings ... * Redo black again and again * When black and flake8 fight each other for a space ...
馃挜 馃挜 馃挜 * Try removing trailing comma to make both black and flake happy! * Fix invalid is_<framework>_available call, thanks @LysandreJik馃帀 * Fix another invalid import in flax_roberta test * Bump and pin flax release to 0.1.0. * Make flake8 happy, remove unused jax import * Change the type of the catch for msgpack. * Remove unused import. * Put seed as optional constructor parameter. * trigger ci again * Fix too much parameters in BertAttention. * Formatting. * Simplify Flax unittests to avoid machine crashes. * Fix invalid number of arguments when raising issue for an unknown model. * Address @bastings comment in PR, moving jax.jit decorated outside of __call__ * Fix incorrect path to require_flax/require_pytorch functions. Signed-off-by:Morgan Funtowicz <funtowiczmo@gmail.com> * Attempt to make style. Signed-off-by:
馃幎 Signed-off-by:Marc van Zee <marcvanzee@gmail.com> * Bump flax to it's latest version Co-authored-by:
-
Stas Bekman authored
* add --durations=50 to some pytest runs * report all tests
-
- 18 Oct, 2020 1 commit
-
-
Thomas Wolf authored
* splitting fast and slow tokenizers [WIP] * [WIP] splitting sentencepiece and tokenizers dependencies * update dummy objects * add name_or_path to models and tokenizers * prefix added to file names * prefix * styling + quality * spliting all the tokenizer files - sorting sentencepiece based ones * update tokenizer version up to 0.9.0 * remove hard dependency on sentencepiece
馃帀 * and removed hard dependency on tokenizers馃帀 * update conversion script * update missing models * fixing tests * move test_tokenization_fast to main tokenization tests - fix bugs * bump up tokenizers * fix bert_generation * update ad fix several tokenizers * keep sentencepiece in deps for now * fix funnel and deberta tests * fix fsmt * fix marian tests * fix layoutlm * fix squeezebert and gpt2 * fix T5 tokenization * fix xlnet tests * style * fix mbart * bump up tokenizers to 0.9.2 * fix model tests * fix tf models * fix seq2seq examples * fix tests without sentencepiece * fix slow => fast conversion without sentencepiece * update auto and bert generation tests * fix mbart tests * fix auto and common test without tokenizers * fix tests without tokenizers * clean up tests lighten up when tokenizers + sentencepiece are both off * style quality and tests fixing * add sentencepiece to doc/examples reqs * leave sentencepiece on for now * style quality split hebert and fix pegasus * WIP Herbert fast * add sample_text_no_unicode and fix hebert tokenization * skip FSMT example test for now * fix style * fix fsmt in example tests * update following Lysandre and Sylvain's comments * Update src/transformers/testing_utils.py Co-authored-by:
-
- 05 Oct, 2020 1 commit
-
-
Sylvain Gugger authored
* PoC on RAG * Format class name/obj name * Better name in message * PoC on one TF model * Add PyTorch and TF dummy objects + script * Treat scikit-learn * Bad copy pastes * Typo
-
- 23 Sep, 2020 1 commit
-
-
Stas Bekman authored
* fix confused flake We run `black --target-version py35 ...` but flake8 doesn't know that, so currently with py38 flake8 fails suggesting that black should have reformatted 63 files. Indeed if I run: ``` black --line-length 119 --target-version py38 examples templates tests src utils ``` it indeed reformats 63 files. The only solution I found is to create a black config file as explained at https://github.com/psf/black#configuration-format, which is what this PR adds. Now flake8 knows that py35 is the standard and no longer gets confused regardless of the user's python version. * adjust the other files that will now rely on black's config file
-
- 22 Sep, 2020 1 commit
-
-
Sylvain Gugger authored
* Copy code from Bert to Roberta and add safeguard script * Fix docstring * Comment code * Formatting * Update src/transformers/modeling_roberta.py Co-authored-by:
-
- 17 Sep, 2020 1 commit
-
-
Stas Bekman authored
``` /home/circleci/.local/lib/python3.6/site-packages/isort/main.py:915: UserWarning: W0501: The following deprecated CLI flags were used and ignored: --recursive! "W0501: The following deprecated CLI flags were used and ignored: " ```
-
- 10 Sep, 2020 1 commit
-
-
Sylvain Gugger authored
* nlp -> datasets * More nlp -> datasets * Woopsie * More nlp -> datasets * One last
-
- 01 Sep, 2020 1 commit
-
-
Stas Bekman authored
we had it added for one job, please add it to all pytest jobs - we need the output of what tests were run to debug the codecov issue. thank you!
-
- 25 Aug, 2020 1 commit
-
-
Sylvain Gugger authored
* More tests to Trainer * Add warning in the doc
-
- 24 Aug, 2020 1 commit
-
-
Sylvain Gugger authored
* Run new isort * More changes * Update CI, CONTRIBUTING and benchmarks
-
- 17 Aug, 2020 1 commit
-
-
Masatoshi Suzuki authored
* Update BERT Japanese tokenizers * Update CircleCI config to download unidic * Specify to use the latest dictionary packages
-
- 12 Aug, 2020 2 commits
-
-
zcain117 authored
-
Sylvain Gugger authored
* Activate check on the CI * Fix repo inconsistencies * Don't document too much
-
- 11 Aug, 2020 2 commits
- 10 Aug, 2020 1 commit
-
-
Lysandre authored
-
- 07 Aug, 2020 2 commits
-
-
zcain117 authored
* Add setup for TPU CI to run every hour. * Re-organize config.yml Co-authored-by: -
Lysandre Debut authored
* Single workflow cache test Remove cache dir, re-trigger cache Only pip archives Not sudo when pip * All workflow cache Remove no-cache-dir instruction Remove last sudo occurrences v0.3
-
- 04 Aug, 2020 2 commits
-
-
Lysandre Debut authored
-
Lysandre Debut authored
* Upgrade pip when doing CI * Don't forget Github CI
-
- 31 Jul, 2020 1 commit
-
-
Paul O'Leary McCann authored
* Replace mecab-python3 with fugashi This replaces mecab-python3 with fugashi for Japanese tokenization. I am the maintainer of both projects. Both projects are MeCab wrappers, so the underlying C++ code is the same. fugashi is the newer wrapper and doesn't use SWIG, so for basic use of the MeCab API it's easier to use. This code insures the use of a version of ipadic installed via pip, which should make versioning and tracking down issues easier. fugashi has wheels for Windows, OSX, and Linux, which will help with issues with installing old versions of mecab-python3 on Windows. Compared to mecab-python3, because fugashi doesn't use SWIG, it doesn't require a C++ runtime to be installed on Windows. In adding this change I removed some code dealing with `cursor`, `token_start`, and `token_end` variables. These variables didn't seem to be used for anything, it is unclear to me why they were there. I ran the tests and they passed, though I couldn't figure out how to run the slow tests (`--runslow` gave an error) and didn't try testing with Tensorflow. * Style fix * Remove unused variable Forgot to delete this... * Adapt doc with install instructions * Fix typo Co-authored-by:
sgugger <sylvain.gugger@gmail.com> Co-authored-by:
-
- 26 Jul, 2020 1 commit
-
-
Stas Bekman authored
Currently, it's hard to derive which example tests were run on CI, and which weren't. Adding `-rA` flag to `pytest`, will now include a summary like: ``` ==================================================================== short test summary info ===================================================================== PASSED examples/test_examples.py::ExamplesTests::test_generation PASSED examples/test_examples.py::ExamplesTests::test_run_glue PASSED examples/test_examples.py::ExamplesTests::test_run_language_modeling PASSED examples/test_examples.py::ExamplesTests::test_run_squad FAILED examples/test_examples.py::ExamplesTests::test_run_pl_glue - AttributeError: 'Namespace' object has no attribute 'gpus' ============================================================ 1 failed, 4 passed, 8 warnings in 42.96s ============================================================ ``` which makes it easier to validate whether some example is being covered by CI or not.
-
