Skip to content

GitLab

Switch branch/tag
Find file BlameHistoryPermalink
  • Funtowicz Morgan's avatar
    Introduce a new tensor type for return_tensors on tokenizer for NumPy (#4585) · 5bf9afbf
    Funtowicz Morgan authored 
    * Refactor tensor creation in tokenizers.

* Make sure to convert string to TensorType

* Refactor convert_to_tensors_

* Introduce numpy tensor creation

* Format

* Add unittest for TensorType creation from str

* sorting imports

* Added unittests for numpy tensor conversion.

* Do not use in-place version for squeeze as numpy doesn't provide such feature.

* Added extra parameter prepend_batch_axis: bool on prepare_for_model.

* Ensure test_np_encode_plus_sent_to_model is not executed if encoder/decoder model.

* style.

* numpy tests require_torch for now while flax not merged.

* Hopefully will make flake8 happy.

* One more time 馃幎
    5bf9afbf
test_tokenization_common.py 37.4 KB