Skip to content

GitLab

"sgl-router/src/routers/vscode:/vscode.git/clone" did not exist on "d05a968ba8aa563355300df335f19f9f05ed7fa7"
Switch branch/tag
Find file BlameHistoryPermalink
  • Funtowicz Morgan's avatar
    Introduce a new tensor type for return_tensors on tokenizer for NumPy (#4585) · 5bf9afbf
    Funtowicz Morgan authored 
    * Refactor tensor creation in tokenizers.

* Make sure to convert string to TensorType

* Refactor convert_to_tensors_

* Introduce numpy tensor creation

* Format

* Add unittest for TensorType creation from str

* sorting imports

* Added unittests for numpy tensor conversion.

* Do not use in-place version for squeeze as numpy doesn't provide such feature.

* Added extra parameter prepend_batch_axis: bool on prepare_for_model.

* Ensure test_np_encode_plus_sent_to_model is not executed if encoder/decoder model.

* style.

* numpy tests require_torch for now while flax not merged.

* Hopefully will make flake8 happy.

* One more time 馃幎
    5bf9afbf
test_tokenization_utils.py 1.78 KB