- 03 Jun, 2024 10 commits
-
-
Isotr0py authored
* add qwen2 gguf support * Update docs * fix qwen2 tokenizer * add qwen2 gguf test * fix typo in qwen2 gguf test * format code * Remove mistral, clarify the error message * format code * add typing and update docstring
-
Yih-Dar authored
* fix * fix * fix * fix --------- Co-authored-by:ydshieh <ydshieh@users.noreply.github.com>
-
NielsRogge authored
Update MLP
-
Joao Gante authored
* tmp commit * sliding window with fewer differences * make fixup + rebase * missing overwrite
-
fxmarty authored
* update non-causal mask for sdpa * add test * update docstrings * add one more test * fix cross attention bug * gentler atol/rtol
-
Pavithra Devi M authored
While running the model.prepare_tf_dataset() method, it raises the error below: ``` TypeError: Cannot convert [array([322., 1.])] to EagerTensor of dtype int64 ``` This happens, in "DataCollatorForSeq2Seq" function when we are try to convert the labels to tensors. While converting the labels to tensors, the labels can be in the format of list of list or list of ndarrays. There is no problem converting the list of list lables. There is a problem when the list of ndarrays are float values(like below). ``` [array([322., 1.])] ``` so the exception raises while trying to convert this label to tensors using below code. ``` batch["labels"] = tf.constant(batch["labels"], dtype=tf.int64) ``` The labels are always integer values, so this got converted to float values in the label padding operation below. ``` batch["labels"] = [ call(label) if padding_side == "right" else np.concatenate([[self.label_pad_token_id] * (max_label_length - len(label)), label]) for label in labels ] ``` Here we have 2 cases: 1 - Concatenating an array having integer padding token value with labels. 2 - Concatenating an empty array with labels. ---------------------------------------------------------------------------------------- case 1: Concatenating an array having integer padding token value with labels. WORKS EXPECTED: ---------------------------------------------------------------------------------------- ``` label = np.array([233, 1]) max_label_length = 4 label_pad_token_id = -100 np.concatenate([[label_pad_token_id] * (max_label_length - len(label)), label]) o/p: array([-100, -100, 233, 1]) ``` ---------------------------------------------------------------------------------------- Case 2: Concatenating an empty array with labels. GIVES THE ISSUE: This scenorio can happen when the label has the maximum label length -- No padding needed. ---------------------------------------------------------------------------------------- ``` label = np.array([233, 1]) max_label_length = 2 label_pad_token_id = -100 np.concatenate([[label_pad_token_id] * (max_label_length - len(label)), label]) o/p: array([233., 1.]) ``` ---------------------------------------------------------------------------------------- Solution: ---------------------------------------------------------------------------------------- We need to concatenate a ndarray of dtype int with labels. AFTER FIX: ---------- case 1: ``` label = np.array([233, 1]) max_label_length = 4 label_pad_token_id = -100 np.concatenate([np.array([label_pad_token_id] * (max_label_length - len(label)), dtype=np.int64),label]) o/p: array([-100, -100, 233, 1]) ``` case 2: ``` label = np.array([233, 1]) max_label_length = 2 label_pad_token_id = -100 np.concatenate([np.array([label_pad_token_id] * (max_label_length - len(label)), dtype=np.int64),label]) o/p: array([233, 1]) ``` -
Arthur authored
* fixes * fix-copies
-
Ahmed Moubtahij authored
* token healing impl + trie with extensions * make fixup * prefix-robust space tokenization * examples readme and requirements * make fixup * allow input prompt and model * redundant defaults * Specialized Trie * make fixup * updated tests with new inherited Tree * input ids to auto device_map * rm unused import * Update src/transformers/generation/utils.py Co-authored-by:
Arthur <48595927+ArthurZucker@users.noreply.github.com> * naming convention * Revert "naming convention" This reverts commit dd39d9c5b7a969e2d8a8d2a8e54f121b82dc44f0. * naming convention * last -hopefully- changes --------- Co-authored-by:
-
amyeroberts authored
* Remove copied froms for deprecated models * Remove automatically in script
-
CharlesCNorton authored
Corrected a typo in security.md. Changed `use_safetenstors` to `use_safetensors` in the section discussing the usage of safe formats for loading models to prevent arbitrary code execution.
-
- 31 May, 2024 10 commits
-
-
Arthur authored
* current working example! * commit regex and result file * update * nit * push the conversion file * oups * roadmap and nits * attempt diffs for 3 files * persimmon * nit * add diff file that is the same as the modeling_llama.py * fix rope nits * updates * updates with converted versions * give some breathing space to the code * delete * update * update * push the actual result * update regex patterns * update regex patterns * fix some issues * fix some issues * fix some issues * updates * updates * updates * updates * updates * revert changes done to llama * updates * update gemma * updates * oups * current state * current state * update * ouiiii * nit * clear diffs * nit * fixup * update * doc
馃殌 *馃敟 * for now use gemma * deal with comments * style * handle funtions * deal with assigns * todos * process inheritage * keep decorators? *馃 * deal with duplicates * fixup * correctly remove duplicate code * run ruff post script * ruff deals pretty well with imports, let's leave it to him * ah maybe not lol * for now remove all imports from child. * nit * conversion of llama * okay * convert starcoder2 * synch with main * update llama diff * updates * https://docs.astral.sh/ruff/rules/redefined-while-unused/ fixes the imports, bit needs later version of ruff * updates * okay actual state * non zero exit * update! * revert unrelated * remove other diff files * updates * cleanup * update * less diff! * stash * current updates * updates * No need for call * finished fining deps * update * current changes * current state * current state * new status * nit * finally * fixes * nits * order is now expected * use logger info instead of prints * fixup * up * nit * update * nits * update * correct merge * update * update * update * add warning * update caution message * update * better merging strategy * copy class statements :wink * fixups * nits * update * Apply suggestions from code review Co-authored-by:amyeroberts <22614925+amyeroberts@users.noreply.github.com> * nits * smaller header * do cleanup some stuff * even simpler header? * fixup * updates * ruff * update examples * nit * TODO * state * OUUUUUUF * current state * nits * final state * add a readme * fixup * remove diff llama * fix * nit * dummy noy funny * ruff format tests src utils --check * everless diffs * less diffs and fix test * fixes * naming nit? * update converter and add supper example * nits * updated for function signatures * update * update * add converted dummies * autoformat * single target assign fix * fixup * fix some imports * fixes * don't push them * `# noqa: F841` --------- Co-authored-by:
-
Vallepu Vamsi Krishna authored
* Description of quantization_config Added missing description about quantization_config in replace_with_bnb_linear for better readability. * Removed trailing spaces
-
Pavel Iakubovskii authored
* Initial setup * Metrics * Overfit on two batches * Train 40 epochs * Memory leak debugging * Trainer fine-tuning * Draft * Fixup * Trained end-to-end * Add requirements * Rewrite evaluator * nits * Add readme * Add instance-segmentation to the table * Support void masks * Remove sh * Update docs * Add pytorch test * Add accelerate test * Update examples/pytorch/instance-segmentation/README.md * Update examples/pytorch/instance-segmentation/run_instance_segmentation.py * Update examples/pytorch/instance-segmentation/run_instance_segmentation_no_trainer.py * Update examples/pytorch/instance-segmentation/run_instance_segmentation_no_trainer.py * Update examples/pytorch/instance-segmentation/run_instance_segmentation.py * Fix consistency oneformer * Fix imports * Fix imports sort * Apply suggestions from code review Co-authored-by:
NielsRogge <48327001+NielsRogge@users.noreply.github.com> * Update examples/pytorch/instance-segmentation/run_instance_segmentation.py Co-authored-by:
Sangbum Daniel Choi <34004152+SangbumChoi@users.noreply.github.com> * Add resources to docs * Update examples/pytorch/instance-segmentation/README.md Co-authored-by:
-
Aymeric Roucher authored
* Implement streaming run in ReAct agents * Allow additional imports in code agents * Python interpreter: support classes and exceptions, fixes
-
Marc Sun authored
* add sanity evaluation * fix * Apply suggestions from code review Co-authored-by:
Zach Mueller <muellerzr@gmail.com> * fix --------- Co-authored-by:
-
Younes Belkada authored
enhance error message
-
Asif Ajrof authored
`mask` variable is not defined. probably a writing mistake. it should be `segmentation_map`. `segmentation_map` should be a `1` channel image rather than `RGB`. [on a different note, the `mask_url` is the same as `raw_image`. could provide a better example.
-
Marc Sun authored
-
Yih-Dar authored
Co-authored-by: -
Arthur authored
* helper * Apply suggestions from code review Co-authored-by:
-
- 30 May, 2024 4 commits
-
-
Younes Belkada authored
remove `IS_GITHUB_CI`
-
Younes Belkada authored
Replace all occurences of `load_in_8bit` with bnb config
-
zspo authored
fix get_scheduler args
-
Younes Belkada authored
add validation for bnb config
-
- 29 May, 2024 12 commits
-
-
Yih-Dar authored
* remove * build --------- Co-authored-by: -
Dhruv Pai authored
* Modified test * Added on_optimizer_step to callbacks * Move callback after step is called * Added on optimizer step callback
-
Joao Gante authored
* add Raushan * add Raushan
-
Younes Belkada authored
Update overview.md
-
Yih-Dar authored
* fix * fix --------- Co-authored-by: -
Zach Mueller authored
-
Matt authored
-
Matt authored
* Fix env.py in cases where torch is not present * Simplify the fix (and avoid some issues)
-
Huazhong Ji authored
* Improve `transformers-cli env` reporting * move the line `"Using GPU in script?": "<fill in>"` to in if conditional statement * same option for npu
-
Lucain authored
* Fix has_file in offline mode * harmonize env variable for offline mode * Switch to HF_HUB_OFFLINE * fix test * revert test_offline to test TRANSFORMERS_OFFLINE * Add new offline test * merge conflicts * docs
-
Younes Belkada authored
* add mistral v3 conversion script * Update src/transformers/models/mistral/convert_mistral_weights_to_hf.py Co-authored-by:
-
Raushan Turganbay authored
* quanto latest version was refactored * add error msg * incorrect compare sign * Update src/transformers/cache_utils.py Co-authored-by:
-
- 28 May, 2024 4 commits
-
-
amyeroberts authored
* Deprecate models - graphormer - time_series_transformer - xlm_prophetnet - qdqbert - nat - ernie_m - tvlt - nezha - mega - jukebox - vit_hybrid - x_clip - deta - speech_to_text_2 - efficientformer - realm - gptsan_japanese * Fix up * Fix speech2text2 imports * Make sure message isn't indented * Fix docstrings * Correctly map for deprecated models from model_type * Uncomment out * Add back time series transformer and x-clip * Import fix and fix-up * Fix up with updated ruff
-
Younes Belkada authored
Update _redirects.yml
-
Younes Belkada authored
* fix flan t5 tests * better format
-
Jonny Li authored
-
