Commit 0a52ac24 authored by zhuwenwen's avatar zhuwenwen
Browse files

add parse out

parent d8f03ffe
...@@ -132,6 +132,12 @@ vocab_size=vocab_size ...@@ -132,6 +132,12 @@ vocab_size=vocab_size
build/ build/
out out
``` ```
执行一下命令可以解析out结果:
```bash
pip install sentencepiece
python ../examples/cpp/llama/llama_tokenizer.py
其中,`tokenizer`为原模型路径
```
## 精度 ## 精度
测试数据:"I believe the meaning of life is" (token id: 306, 4658, 278, 6593, 310, 2834, 338),使用的加速卡:1张 DCU-Z100L-32G 测试数据:"I believe the meaning of life is" (token id: 306, 4658, 278, 6593, 310, 2834, 338),使用的加速卡:1张 DCU-Z100L-32G
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment