examples/cpp/llama/tokenizer.py · 62b60db72a8e968dd74720d203a964a5ecb1df8d · OpenDAS / Lmdeploy · GitLab

"vscode:/vscode.git/clone" did not exist on "fa31da29e591ed2e64a7c6ba9153c0b2e5a0ddc2"

Find file Blame History Permalink

support 'input_tokens' in triton_example (#49) · 62e0fa9a
lvhan028 authored Jul 04, 2023
```
* check-in script for tokenizing a file

* use max_input_len
```
62e0fa9a

tokenizer.py 2.59 KB