Skip to content

GitLab

Unverified Commit 3114df41 authored by Loubna Ben Allal's avatar Loubna Ben Allal Committed by GitHub
Browse files

update README.md (#17657) 

- use CodeParrot scores of v1.1
- change evaluation command to use accelerate
parent c99ddcc4
Hide whitespace changes
Inline Side-by-side
Showing with 2 additions and 2 deletions
examples/research_projects/codeparrot/README.md
View file @ 3114df41
......@@ -165,7 +165,7 @@ python scripts/validation_loss.py \
In addition we evaluate the model on OpenAI's _HumanEval_ benchmark. You can run the evaluation with the following command:
```bash
python scripts/human_eval.py --model_ckpt lvwerra/codeparrot \
accelerate launch scripts/human_eval.py --model_ckpt lvwerra/codeparrot \
--do_sample True \
--temperature 0.2 \
--top_p 0.95 \
......@@ -178,7 +178,7 @@ The results as well as reference values are shown in the following table:
| Model | pass@1 | pass@10 | pass@100|
|-------|--------|---------|---------|
|CodeParrot 🦜 (110M) | 3.80% | 6.57% | 12.78% |
|CodeParrot 🦜 (1.5B) | 3.58% | 8.03% | 14.96% |
|CodeParrot 🦜 (1.5B) | 3.99% | 8.69% | 17.88% |
|||||
|Codex (25M)| 3.21% | 7.1% | 12.89%|
|Codex (85M)| 8.22% | 12.81% | 22.40% |
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment