README.md 20.9 KB
Newer Older
Leo Gao's avatar
Leo Gao committed
1
# Language Model Evaluation Harness
Anish Thite's avatar
Anish Thite committed
2

3
[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.10256836.svg)](https://doi.org/10.5281/zenodo.10256836)
Hailey Schoelkopf's avatar
Hailey Schoelkopf committed
4

Stella Biderman's avatar
Stella Biderman committed
5
## Announcement
lintangsutawika's avatar
lintangsutawika committed
6
**A new v0.4.0 release of lm-evaluation-harness is available** !
Hailey Schoelkopf's avatar
Hailey Schoelkopf committed
7
8
9
10
11

New updates and features include:

- Internal refactoring
- Config-based task creation and configuration
Hailey Schoelkopf's avatar
Hailey Schoelkopf committed
12
- Easier import and sharing of externally-defined task config YAMLs
Hailey Schoelkopf's avatar
Hailey Schoelkopf committed
13
14
15
16
17
18
19
20
21
22
- Support for Jinja2 prompt design, easy modification of prompts + prompt imports from Promptsource
- More advanced configuration options, including output post-processing, answer extraction, and multiple LM generations per document, configurable fewshot settings, and more
- Speedups and new modeling libraries supported, including: faster data-parallel HF model usage, vLLM support, MPS support with HuggingFace, and more
- Logging and usability changes
- New tasks including CoT BIG-Bench-Hard, Belebele, user-defined task groupings, and more

Please see our updated documentation pages in `docs/` for more details.

Development will be continuing on the `main` branch, and we encourage you to give us feedback on what features are desired and how to improve the library further, or ask questions, either in issues or PRs on GitHub, or in the [EleutherAI discord](discord.gg/eleutherai)!

Fabrizio Milo's avatar
Fabrizio Milo committed
23
## Overview
Anish Thite's avatar
Anish Thite committed
24

Stella Biderman's avatar
Stella Biderman committed
25
This project provides a unified framework to test generative language models on a large number of different evaluation tasks.
Leo Gao's avatar
Leo Gao committed
26

Stella Biderman's avatar
Stella Biderman committed
27
**Features:**
Stella Biderman's avatar
Stella Biderman committed
28
- Over 60 standard academic benchmarks for LLMs, with hundreds of subtasks and variants implemented.
Stella Biderman's avatar
Stella Biderman committed
29
- Support for models loaded via [transformers](https://github.com/huggingface/transformers/) (including quantization via [AutoGPTQ](https://github.com/PanQiWei/AutoGPTQ)), [GPT-NeoX](https://github.com/EleutherAI/gpt-neox), and [Megatron-DeepSpeed](https://github.com/microsoft/Megatron-DeepSpeed/), with a flexible tokenization-agnostic interface.
30
- Support for fast and memory-efficient inference with [vLLM](https://github.com/vllm-project/vllm).
31
- Support for commercial APIs including [OpenAI](https://openai.com), and [TextSynth](https://textsynth.com/).
Stella Biderman's avatar
Stella Biderman committed
32
33
34
- Support for evaluation on adapters (e.g. LoRA) supported in [HuggingFace's PEFT library](https://github.com/huggingface/peft).
- Support for local models and benchmarks.
- Evaluation with publicly available prompts ensures reproducibility and comparability between papers.
Stella Biderman's avatar
Stella Biderman committed
35
- Easy support for custom prompts and evaluation metrics.
Stella Biderman's avatar
Stella Biderman committed
36

Stella Biderman's avatar
Stella Biderman committed
37
The Language Model Evaluation Harness is the backend for 🤗 Hugging Face's popular [Open LLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard), has been used in [hundreds of papers](https://scholar.google.com/scholar?oi=bibs&hl=en&authuser=2&cites=15052937328817631261,4097184744846514103,17476825572045927382,18443729326628441434,12854182577605049984) is used internally by dozens of companies including NVIDIA, Cohere, Nous Research, Booz Allen Hamilton, and Mosaic ML.
38

Leo Gao's avatar
Leo Gao committed
39
40
## Install

Hailey Schoelkopf's avatar
Hailey Schoelkopf committed
41
To install the `lm-eval` package from the github repository, run:
42

Leo Gao's avatar
Leo Gao committed
43
```bash
44
45
46
git clone https://github.com/EleutherAI/lm-evaluation-harness
cd lm-evaluation-harness
pip install -e .
Leo Gao's avatar
Leo Gao committed
47
```
Leo Gao's avatar
Leo Gao committed
48

baberabb's avatar
typo  
baberabb committed
49
We also provide a number of optional dependencies for extended functionality. Extras can be installed via `pip install -e ".[NAME]"`
Stella Biderman's avatar
Stella Biderman committed
50

Stella Biderman's avatar
Stella Biderman committed
51
| Name          | Use                                   |
52
|---------------|---------------------------------------|
Stella Biderman's avatar
Stella Biderman committed
53
54
| anthropic     | For using Anthropic's models          |
| gptq          | For loading models with GPTQ          |
55
| dev           | You probably don't want to use this   |
Stella Biderman's avatar
Stella Biderman committed
56
57
58
59
60
| multilingual  | For multilingual tokenizers           |
| openai        | For using OpenAI's models             |
| promptsource  | For using PromtSource prompts         |
| sentencepiece | For using the sentencepiece tokenizer |
| vllm          | For loading models with vLLM          |
61
| zeno          | For visualizing results with Zeno     |
Stella Biderman's avatar
Stella Biderman committed
62
| all           | Loads all extras                      |
haileyschoelkopf's avatar
haileyschoelkopf committed
63

Leo Gao's avatar
Leo Gao committed
64
65
## Basic Usage

Stella Biderman's avatar
Stella Biderman committed
66
67
68
### Hugging Face `transformers`

To evaluate a model hosted on the [HuggingFace Hub](https://huggingface.co/models) (e.g. GPT-J-6B) on `hellaswag` you can use the following command:
jon-tow's avatar
jon-tow committed
69

Leo Gao's avatar
Leo Gao committed
70
```bash
Stella Biderman's avatar
Stella Biderman committed
71
lm_eval --model hf \
Stella Biderman's avatar
Stella Biderman committed
72
    --model_args pretrained=EleutherAI/gpt-j-6B \
Stella Biderman's avatar
Stella Biderman committed
73
    --tasks hellaswag \
Hailey Schoelkopf's avatar
Hailey Schoelkopf committed
74
75
    --device cuda:0 \
    --batch_size 8
Leo Gao's avatar
Leo Gao committed
76
77
```

Stella Biderman's avatar
Stella Biderman committed
78
Additional arguments can be provided to the model constructor using the `--model_args` flag. Most notably, this supports the common practice of using the `revisions` feature on the Hub to store partially trained checkpoints, or to specify the datatype for running a model:
Leo Gao's avatar
Leo Gao committed
79
80

```bash
Stella Biderman's avatar
Stella Biderman committed
81
lm_eval --model hf \
Stella Biderman's avatar
Stella Biderman committed
82
    --model_args pretrained=EleutherAI/pythia-160m,revision=step100000,dtype="float" \
jon-tow's avatar
jon-tow committed
83
    --tasks lambada_openai,hellaswag \
Hailey Schoelkopf's avatar
Hailey Schoelkopf committed
84
85
86
87
    --device cuda:0 \
    --batch_size 8
```

Lenni Justen's avatar
Lenni Justen committed
88
Models that are loaded via both `transformers.AutoModelForCausalLM` (autoregressive, decoder-only GPT style models) and `transformers.AutoModelForSeq2SeqLM` (such as encoder-decoder models like T5) in Huggingface are supported.
Hailey Schoelkopf's avatar
Hailey Schoelkopf committed
89

90
91
92
Batch size selection can be automated by setting the  ```--batch_size``` flag to ```auto```. This will perform automatic detection of the largest batch size that will fit on your device. On tasks where there is a large difference between the longest and shortest example, it can be helpful to periodically recompute the largest batch size, to gain a further speedup. To do this, append ```:N``` to above flag to automatically recompute the largest batch size ```N``` times. For example, to recompute the batch size 4 times, the command would be:

```bash
Stella Biderman's avatar
Stella Biderman committed
93
lm_eval --model hf \
94
95
96
97
98
99
    --model_args pretrained=EleutherAI/pythia-160m,revision=step100000,dtype="float" \
    --tasks lambada_openai,hellaswag \
    --device cuda:0 \
    --batch_size auto:4
```

100
The full list of supported arguments are provided [here](./docs/interface.md), and on the terminal by calling `lm_eval -h`. Alternatively, you can use `lm-eval` instead of `lm_eval`.
101

Stella Biderman's avatar
Stella Biderman committed
102
> [!Note]
103
104
105
> Just like you can provide a local path to `transformers.AutoModel`, you can also provide a local path to `lm_eval` via `--model_args pretrained=/path/to/model`

#### Multi-GPU Evaluation with Hugging Face `accelerate`
106

Stella Biderman's avatar
Stella Biderman committed
107
To parallelize evaluation of HuggingFace models across multiple GPUs, we leverage the [accelerate 🚀](https://github.com/huggingface/accelerate) library as follows:
Hailey Schoelkopf's avatar
Hailey Schoelkopf committed
108
109

```
Stella Biderman's avatar
Stella Biderman committed
110
accelerate launch -m lm_eval --model hf \
Hailey Schoelkopf's avatar
Hailey Schoelkopf committed
111
    --tasks lambada_openai,arc_easy \
112
    --batch_size 16
Leo Gao's avatar
Leo Gao committed
113
114
```

115
This will perform *data-parallel evaluation*: that is, placing a **single full copy** of your model onto each available GPU and *splitting batches across GPUs* to evaluate on K GPUs K times faster than on one.
Stella Biderman's avatar
Stella Biderman committed
116

117
If your model is *is too large to be run on a single one of your GPUs* then you can use `accelerate` with Fully Sharded Data Parallel (FSDP) that splits the weights of the model across your data parallel ranks. To enable this, ensure you select `YES` when asked ```Do you want to use FullyShardedDataParallel?``` when running `accelerate config`. To enable memory-efficient loading, select `YES` when asked `Do you want each individually wrapped FSDP unit to broadcast module parameters from rank 0 at the start?`. This will ensure only the rank 0 process loads the model and then broadcasts the parameters to the other ranks instead of having each rank load all parameters which can lead to large RAM usage spikes around the start of the script that may cause errors.
118
119
120
121
122
123
124

To pass even more advanced keyword arguments to `accelerate`, we allow for the following arguments as well:
- `device_map_option`: How to split model weights across available GPUs. defaults to "auto".
- `max_memory_per_gpu`: the max GPU memory to use per GPU in loading the model.
- `max_cpu_memory`: the max amount of CPU memory to use when offloading the model weights to RAM.
- `offload_folder`: a folder where model weights will be offloaded to disk if needed.

lintangsutawika's avatar
lintangsutawika committed
125
To use `accelerate` with the `lm-eval` command, use
126
127
128
129
```
accelerate launch --no_python lm-eval --model ...
```

Zach Nussbaum's avatar
Zach Nussbaum committed
130

131
### Tensor + Data Parallel and Optimized Inference with `vLLM`
Hailey Schoelkopf's avatar
Hailey Schoelkopf committed
132

133
We also support vLLM for faster inference on [supported model types](https://docs.vllm.ai/en/latest/models/supported_models.html). For single-GPU or multi-GPU — tensor parallel, data parallel, or a combination of both — inference, for example:
Hailey Schoelkopf's avatar
Hailey Schoelkopf committed
134
135

```bash
Stella Biderman's avatar
Stella Biderman committed
136
lm_eval --model vllm \
Hailey Schoelkopf's avatar
Hailey Schoelkopf committed
137
    --model_args pretrained={model_name},tensor_parallel_size={GPUs_per_model},dtype=auto,gpu_memory_utilization=0.8,data_parallel_size={model_replicas} \
138
    --tasks lambada_openai \
Hailey Schoelkopf's avatar
Hailey Schoelkopf committed
139
140
141
142
    --batch_size auto
```
For a full list of supported vLLM configurations, please reference our vLLM integration and the vLLM documentation.

143
vLLM occasionally differs in output from Huggingface. We treat Huggingface as the reference implementation, and provide a [script](./scripts/model_comparator.py) for checking the validity of vllm results against HF.
144

145
### Model APIs and Inference Servers
Hailey Schoelkopf's avatar
Hailey Schoelkopf committed
146
147

Our library also supports the evaluation of models served via several commercial APIs, and we hope to implement support for the most commonly used performant local/self-hosted inference servers.
Hailey Schoelkopf's avatar
Hailey Schoelkopf committed
148

149
150
151
To call a hosted model, use:

```bash
152
export OPENAI_API_KEY=YOUR_KEY_HERE
153
lm_eval --model openai-completions \
Anjor Kanekar's avatar
Anjor Kanekar committed
154
    --model_args model=davinci \
155
156
157
    --tasks lambada_openai,hellaswag
```

158
159
160
161
162
We also support using your own local inference server with an implemented version of the OpenAI ChatCompletions endpoint and passing trained HuggingFace artifacts and tokenizers.

```bash
lm_eval --model local-chat-completions --tasks gsm8k --model_args model=facebook/opt-125m,base_url=http://{yourip}:8000/v1
```
163
164
Note that for externally hosted models, configs such as `--device` and `--batch_size` should not be used and do not function. Just like you can use `--model_args` to pass arbitrary arguments to the model constructor for local models, you can use it to pass arbitrary arguments to the model API for hosted models. See the documentation of the hosting service for information on what arguments they support.

Hailey Schoelkopf's avatar
Hailey Schoelkopf committed
165

166
167
168
169
170
171
172
173
174
175
| API or Inference Server                                                                                                   | Implemented?                    | `--model <xxx>` name                                                | Models supported:                                                                             | Request Types:                                             |
|---------------------------------------------------------------------------------------------------------------------------|---------------------------------|---------------------------------------------------------------------|-----------------------------------------------------------------------------------------------|------------------------------------------------------------|
| OpenAI Completions                                                                                                        | :heavy_check_mark:              | `openai-completions` | up to `code-davinci-002`                                                                      | `generate_until`, `loglikelihood`, `loglikelihood_rolling` |
| OpenAI ChatCompletions                                                                                                    | :heavy_check_mark:        | `openai-chat-completions`, `local-chat-completions`                                                               | [All ChatCompletions API models](https://platform.openai.com/docs/guides/gpt)                 | `generate_until` (no logprobs)                             |
| Anthropic                                                                                                                 | :heavy_check_mark:              | `anthropic`                                                         | [Supported Anthropic Engines](https://docs.anthropic.com/claude/reference/selecting-a-model)  | `generate_until` (no logprobs)                             |
| Textsynth                                                                                                                 | :heavy_check_mark:                   | `textsynth`                                                         | [All supported engines](https://textsynth.com/documentation.html#engines)                     | `generate_until`, `loglikelihood`, `loglikelihood_rolling` |
| Cohere                                                                                                                    | [:hourglass: - blocked on Cohere API bug](https://github.com/EleutherAI/lm-evaluation-harness/pull/395) | N/A                                                                 | [All `cohere.generate()` engines](https://docs.cohere.com/docs/models)                        | `generate_until`, `loglikelihood`, `loglikelihood_rolling` |
| [Llama.cpp](https://github.com/ggerganov/llama.cpp) (via [llama-cpp-python](https://github.com/abetlen/llama-cpp-python)) | :heavy_check_mark:              | `gguf`, `ggml`                                                      | [All models supported by llama.cpp](https://github.com/ggerganov/llama.cpp)                   | `generate_until`, `loglikelihood`, `loglikelihood_rolling` |
| vLLM                                                                                                                      | :heavy_check_mark:       | `vllm`                                                              | [Most HF Causal Language Models](https://docs.vllm.ai/en/latest/models/supported_models.html) | `generate_until`, `loglikelihood`, `loglikelihood_rolling` |
| Your local inference server!                                                                                              | :heavy_check_mark:                             | `local-chat-completions` (using `openai-completions` model type)    | Any server address that accepts GET requests using HF models and mirror's OpenAI's ChatCompletions interface                                  | `generate_until`                                           |                                | ...                                                      |
Hailey Schoelkopf's avatar
Hailey Schoelkopf committed
176
177
178

It is on our roadmap to create task variants designed to enable models which do not serve logprobs/loglikelihoods to be compared with generation performance of open-source models.

Stella Biderman's avatar
Stella Biderman committed
179
180
### Other Frameworks

lintangsutawika's avatar
lintangsutawika committed
181
A number of other libraries contain scripts for calling the eval harness through their library. These include [GPT-NeoX](https://github.com/EleutherAI/gpt-neox/blob/main/eval_tasks/eval_adapter.py), [Megatron-DeepSpeed](https://github.com/microsoft/Megatron-DeepSpeed/blob/main/examples/MoE/readme_evalharness.md), and [mesh-transformer-jax](https://github.com/kingoflolz/mesh-transformer-jax/blob/master/eval_harness.py).
Jason Phang's avatar
Jason Phang committed
182

Stella Biderman's avatar
Stella Biderman committed
183
184
### Additional Features

baberabb's avatar
baberabb committed
185
If you have a Metal compatible Mac, you can run the eval harness using the MPS back-end by replacing `--device cuda:0` with `--device mps` (requires PyTorch version 2.1 or higher).
Stella Biderman's avatar
Stella Biderman committed
186

187
188
189
190
191
192
193
194
195
196
> [!Note]
> You can inspect what the LM inputs look like by running the following command:
> ```bash
> python write_out.py \
>     --tasks all_tasks \
>     --num_fewshot 5 \
>     --num_examples 10 \
>     --output_base_path /path/to/output/folder
> ```
> This will write out one text file for each task.
jon-tow's avatar
jon-tow committed
197

Hailey Schoelkopf's avatar
Hailey Schoelkopf committed
198
199
200
To verify the data integrity of the tasks you're performing in addition to running the tasks themselves, you can use the `--check_integrity` flag:

```bash
Stella Biderman's avatar
Stella Biderman committed
201
lm_eval --model openai \
Hailey Schoelkopf's avatar
Hailey Schoelkopf committed
202
203
204
205
206
    --model_args engine=davinci \
    --tasks lambada_openai,hellaswag \
    --check_integrity
```

Hailey Schoelkopf's avatar
Hailey Schoelkopf committed
207
## Advanced Usage Tips
Stella Biderman's avatar
Stella Biderman committed
208
209
210

For models loaded with the HuggingFace  `transformers` library, any arguments provided via `--model_args` get passed to the relevant constructor directly. This means that anything you can do with `AutoModel` can be done with our library. For example, you can pass a local path via `pretrained=` or use models finetuned with [PEFT](https://github.com/huggingface/peft) by taking the call you would run to evaluate the base model and add `,peft=PATH` to the `model_args` argument:
```bash
Stella Biderman's avatar
Stella Biderman committed
211
lm_eval --model hf \
212
    --model_args pretrained=EleutherAI/gpt-j-6b,parallelize=True,load_in_4bit=True,peft=nomic-ai/gpt4all-j-lora \
Stella Biderman's avatar
Stella Biderman committed
213
214
215
    --tasks openbookqa,arc_easy,winogrande,hellaswag,arc_challenge,piqa,boolq \
    --device cuda:0
```
216

217
[GPTQ](https://github.com/PanQiWei/AutoGPTQ) quantized models can be loaded by specifying their file names in `,gptq=NAME` (or `,gptq=True` for default names) in the `model_args` argument:
218
219

```bash
Stella Biderman's avatar
Stella Biderman committed
220
lm_eval --model hf \
221
    --model_args pretrained=model-name-or-path,gptq=model.safetensors,gptq_use_triton=True \
Stella Biderman's avatar
Stella Biderman committed
222
    --tasks hellaswag
223
224
```

Stella Biderman's avatar
Stella Biderman committed
225
226
We support wildcards in task names, for example you can run all of the machine-translated lambada tasks via `--task lambada_openai_mt_*`.

227
228
To save evaluation results provide an `--output_path`. We also support logging model responses with the `--log_samples` flag for post-hoc analysis.

Hailey Schoelkopf's avatar
Hailey Schoelkopf committed
229
Additionally, one can provide a directory with `--use_cache` to cache the results of prior runs. This allows you to avoid repeated execution of the same (model, task) pairs for re-scoring.
230

231
For a full list of supported arguments, check out the [interface](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/interface.md) guide in our documentation!
Hailey Schoelkopf's avatar
Hailey Schoelkopf committed
232

233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
## Visualizing Results

You can use [Zeno](https://zenoml.com) to visualize the results of your eval harness runs.

First, head to [hub.zenoml.com](hub.zenoml.com) to create an account and get an API key [on your account page](hub.zenoml.com/account).
Add this key as an environment variable:

```bash
export ZENO_API_KEY=[your api key]
```

You'll also need to install the `lm_eval[zeno]` package extra.

To visualize the results, run the eval harness with the `log_samples` and `output_path` flags.
We expect `output_path` to contain multiple folders that represent individual model names.
You can thus run your evaluation on any number of tasks and models and upload all of the results as projects on Zeno.

```bash
lm_eval \
    --model hf \
    --model_args pretrained=EleutherAI/gpt-j-6B \
    --tasks hellaswag \
    --device cuda:0 \
    --batch_size 8 \
    --log_samples \
    --output_path output/gpt-j-6B
```

Then, you can upload the resulting data using the `zeno_visualize` script:

```bash
python scripts/zeno_visualize.py \
    --data_path output \
    --project_name "Eleuther Project"
```

This will use all subfolders in `data_path` as different models and upload all tasks within these model folders to Zeno.
If you run the eval harness on multiple tasks, the `project_name` will be used as a prefix and one project will be created per task.

Hailey Schoelkopf's avatar
Hailey Schoelkopf committed
272
273
274
275
## How to Contribute or Learn More?

For more information on the library and how everything fits together, check out all of our [documentation pages](https://github.com/EleutherAI/lm-evaluation-harness/tree/big-refactor/docs)! We plan to post a larger roadmap of desired + planned library improvements soon, with more information on how contributors can help.

Stella Biderman's avatar
Stella Biderman committed
276
277
278
279
### Implementing new tasks

To implement a new task in the eval harness, see [this guide](./docs/new_task_guide.md).

280
In general, we follow this priority list for addressing concerns about prompting and other eval details:
Stella Biderman's avatar
Stella Biderman committed
281
282
283
284
1. If there is widespread agreement among people who train LLMs, use the agreed upon procedure.
2. If there is a clear and unambiguous official implementation, use that procedure.
3. If there is widespread agreement among people who evaluate LLMs, use the agreed upon procedure.
4. If there are multiple common implementations but not universal or widespread agreement, use our preferred option among the common implementations. As before, prioritize choosing from among the implementations found in LLM training papers.
Stella Biderman's avatar
Stella Biderman committed
285

Stella Biderman's avatar
Stella Biderman committed
286
These are guidelines and not rules, and can be overruled in special circumstances.
Stella Biderman's avatar
Stella Biderman committed
287

baberabb's avatar
baberabb committed
288
We try to prioritize agreement with the procedures used by other groups to decrease the harm when people inevitably compare runs across different papers despite our discouragement of the practice. Historically, we also prioritized the implementation from [Language Models are Few Shot Learners](https://arxiv.org/abs/2005.14165) as our original goal was specifically to compare results with that paper.
Hailey Schoelkopf's avatar
Hailey Schoelkopf committed
289

290
291
### Support

baberabb's avatar
baberabb committed
292
The best way to get support is to open an issue on this repo or join the [EleutherAI Discord server](https://discord.gg/eleutherai). The `#lm-thunderdome` channel is dedicated to developing this project and the `#release-discussion` channel is for receiving support for our releases. If you've used the library and have had a positive (or negative) experience, we'd love to hear from you!
293

Leo Gao's avatar
Leo Gao committed
294
295
296
## Cite as

```
Stella Biderman's avatar
Stella Biderman committed
297
@misc{eval-harness,
Stella Biderman's avatar
Stella Biderman committed
298
  author       = {Gao, Leo and Tow, Jonathan and Abbasi, Baber and Biderman, Stella and Black, Sid and DiPofi, Anthony and Foster, Charles and Golding, Laurence and Hsu, Jeffrey and Le Noac'h, Alain and Li, Haonan and McDonell, Kyle and Muennighoff, Niklas and Ociepa, Chris and Phang, Jason and Reynolds, Laria and Schoelkopf, Hailey and Skowron, Aviya and Sutawika, Lintang and Tang, Eric and Thite, Anish and Wang, Ben and Wang, Kevin and Zou, Andy},
Leo Gao's avatar
Leo Gao committed
299
  title        = {A framework for few-shot language model evaluation},
300
301
  month        = 12,
  year         = 2023,
Leo Gao's avatar
Leo Gao committed
302
  publisher    = {Zenodo},
303
304
305
  version      = {v0.4.0},
  doi          = {10.5281/zenodo.10256836},
  url          = {https://zenodo.org/records/10256836}
Leo Gao's avatar
Leo Gao committed
306
307
}
```