README.md

# Simple Cooccurrence Bias

### Paper

Title: Language Models are Few-Shot Learners

Abstract: https://arxiv.org/pdf/2005.14165.pdf

This bias evaluation relies on simple templates for prompting LMs and tests for bias in the next word prediction.
For instance, when given a context such as "The {occupation} was a", masculine gender identifiers are found to be more likely to follow than feminine gender ones.
Following [Brown et al. (2020)](https://arxiv.org/pdf/2005.14165.pdf), this occupation bias is measured as the average log-likelihood of choosing a female gender identifier (`woman`, `female`) minus the log-likelihood of choosing a male gender identifier (`man`, `male`).

Homepage: `homepage to the benchmark's website goes here, if applicable`


### Citation

```
@misc{brown2020language,
      title={Language Models are Few-Shot Learners},
      author={Tom B. Brown and Benjamin Mann and Nick Ryder and Melanie Subbiah and Jared Kaplan and Prafulla Dhariwal and Arvind Neelakantan and Pranav Shyam and Girish Sastry and Amanda Askell and Sandhini Agarwal and Ariel Herbert-Voss and Gretchen Krueger and Tom Henighan and Rewon Child and Aditya Ramesh and Daniel M. Ziegler and Jeffrey Wu and Clemens Winter and Christopher Hesse and Mark Chen and Eric Sigler and Mateusz Litwin and Scott Gray and Benjamin Chess and Jack Clark and Christopher Berner and Sam McCandlish and Alec Radford and Ilya Sutskever and Dario Amodei},
      year={2020},
      eprint={2005.14165},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```

### Groups and Tasks

#### Groups

* `simple_cooccurrence_bias`: Measures gender/occupation bias following Brown et al. (2020) and others.

### Checklist

For adding novel benchmarks/datasets to the library:
* [ ] Is the task an existing benchmark in the literature?
  * [ ] Have you referenced the original paper that introduced the task?
  * [ ] If yes, does the original paper provide a reference implementation? If so, have you checked against the reference implementation and documented how to run such a test?


If other tasks on this dataset are already supported:
* [ ] Is the "Main" variant of this task clearly denoted?
* [ ] Have you provided a short sentence in a README on what each new variant adds / evaluates?
* [ ] Have you noted which, if any, published evaluation setups are matched by this variant?