Unverified Commit 725d7e3e authored by Lintang Sutawika's avatar Lintang Sutawika Committed by GitHub
Browse files

Added info on `lm-eval` command.

parent 94dce753
...@@ -99,6 +99,8 @@ python main.py \ ...@@ -99,6 +99,8 @@ python main.py \
--batch_size auto:4 --batch_size auto:4
``` ```
Alternatively, you can use `lm-eval` instead of `python main.py` to call lm eval from anywhere.
### Multi-GPU Evaluation with Hugging Face `accelerate` ### Multi-GPU Evaluation with Hugging Face `accelerate`
To parallelize evaluation of HuggingFace models across multiple GPUs, we allow for two different types of multi-GPU evaluation. To parallelize evaluation of HuggingFace models across multiple GPUs, we allow for two different types of multi-GPU evaluation.
...@@ -134,6 +136,11 @@ Using this setting helps for massive models like BLOOM which require, or to avoi ...@@ -134,6 +136,11 @@ Using this setting helps for massive models like BLOOM which require, or to avoi
**Note that this option requires launching evaluation via `python main.py` rather than `accelerate launch main.py`.** **Note that this option requires launching evaluation via `python main.py` rather than `accelerate launch main.py`.**
To use `accelerate` with the `lm-eval` command, use
```
accelerate launch --no_python lm-eval --model ...
```
### Commercial APIs ### Commercial APIs
Our library also supports the evaluation of models served via several commercial APIs, and hope to implement support for common performant local/self-hosted inference servers. Our library also supports the evaluation of models served via several commercial APIs, and hope to implement support for common performant local/self-hosted inference servers.
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment