Skip to content

GitLab

Unverified Commit 2bfa234f authored by Casper's avatar Casper Committed by GitHub
Browse files

Benchmark info (#138)

parent 0f3214b1
Hide whitespace changes
Inline Side-by-side
Showing with 3 additions and 0 deletions
README.md
View file @ 2bfa234f
...@@ -197,6 +197,9 @@ generation_output = model.generate( ...@@ -197,6 +197,9 @@ generation_output = model.generate(
## Benchmarks ## Benchmarks
- GPU: RTX 3090
- Command: `python examples/benchmark --model_path <hf_model>`
| Model Name | Version | Batch Size | Prefill Length | Decode Length | Prefill tokens/s | Decode tokens/s | Memory (VRAM) | | Model Name | Version | Batch Size | Prefill Length | Decode Length | Prefill tokens/s | Decode tokens/s | Memory (VRAM) |
|---------------|---------|------------|----------------|---------------|------------------|-----------------|------------------| |---------------|---------|------------|----------------|---------------|------------------|-----------------|------------------|
| Vicuna 7B | GEMM | 1 | 64 | 64 | 2618.88 | 125.428 | 4.57 GB (19.31%) | | Vicuna 7B | GEMM | 1 | 64 | 64 | 2618.88 | 125.428 | 4.57 GB (19.31%) |
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment