Commit d4093886 authored by baberabb's avatar baberabb
Browse files

nits

parent d4edd280
...@@ -121,12 +121,12 @@ To run with vLLM, first install the vllm library, externally or via the lm_eval[ ...@@ -121,12 +121,12 @@ To run with vLLM, first install the vllm library, externally or via the lm_eval[
pip install -e .[vllm] pip install -e .[vllm]
``` ```
Then, you can run the library as normal. For single-GPU or multi-GPU — tensor parallel, data parallel or a combination of both — inference, for example: Then, you can run the library as normal. For single-GPU or multi-GPU — tensor parallel, data parallel, or a combination of both — inference, for example:
```bash ```bash
python -m lm_eval \ python -m lm_eval \
--model vllm \ --model vllm \
--model_args pretrained={model_name},tensor_parallel_size={number of GPUs to use},data_parallel={number of model replicas},dtype=auto,gpu_memory_utilization=0.8 --model_args pretrained={model_name},tensor_parallel_size={number of GPUs per model},data_parallel={number of model replicas},dtype=auto,gpu_memory_utilization=0.8
--tasks lambada_openai --tasks lambada_openai
--batch_size auto --batch_size auto
``` ```
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment