Merge branch 'master' into auto-batching

.github/workflows/pull_request.yml

-name: Pull Request
-
-on: [pull_request]
-
-jobs:
-  pre-commit:
-    runs-on: ubuntu-20.04
-    steps:
-      - uses: actions/checkout@v3
-      - uses: actions/setup-python@v4
-        with:
-          python-version: 3.8
-      - uses: pre-commit/action@v2.0.3

.github/workflows/python-app.yml

-# This workflow will install Python dependencies, run tests and lint with a single version of Python
-# For more information see: https://help.github.com/actions/language-and-framework-guides/using-python-with-github-actions
-
-name: Build
-
-on:
-  push:
-    branches: [ master ]
-  pull_request:
-    branches: [ master ]
-
-jobs:
-  build:
-
-    runs-on: ubuntu-latest
-
-    steps:
-    - uses: actions/checkout@v3
-    - name: Cache
-      uses: actions/cache@v2.1.3
-      with:
-        # A list of files, directories, and wildcard patterns to cache and restore
-        path: |
-          ~/.cache
-        # An explicit key for restoring and saving the cache
-        key: evaldata-cache-4
-    - name: Set up Python 3.9
-      uses: actions/setup-python@v4
-      with:
-        python-version: 3.9
-    - name: Install dependencies
-      run: |
-        python -m pip install --upgrade pip
-        pip install flake8 pytest pytest-cov
-        pip install -e .[dev,multilingual]
-        # Install optional git dependencies
-        pip install bleurt@https://github.com/google-research/bleurt/archive/b610120347ef22b494b6d69b4316e303f5932516.zip#egg=bleurt
-        if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
-    - name: Lint with flake8
-      run: |
-        # stop the build if there are Python syntax errors or undefined names
-        flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
-        # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
-        flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
-    - name: Test with pytest
-      run: |
-        pytest -vv --cov=lm_eval/ tests/
-    - name: Upload to codecov
-      run: |
-        bash <(curl -s https://codecov.io/bash) -t $CODECOV_TOKEN

README.md

@@ -5,55 +5,69 @@
 
 ## Overview
 
-This project provides a unified framework to test autoregressive language models (GPT-2, GPT-3, GPTNeo, etc) on a large number of different evaluation tasks.
+This project provides a unified framework to test generative language models on a large number of different evaluation tasks.
 
 Features:
 
 - 200+ tasks implemented. See the [task-table](./docs/task_table.md) for a complete list.
-- Support for GPT-2, GPT-3, GPT-Neo, GPT-NeoX, and GPT-J, with flexible tokenization-agnostic interface.
+- Support for the Hugging Face `transformers` library, GPT-NeoX, Megatron-DeepSpeed, and the OpenAI API, with flexible tokenization-agnostic interface.
+- Support for evaluation on adapters (e.g. LoRa) supported in [HuggingFace's PEFT library](https://github.com/huggingface/peft).
 - Task versioning to ensure reproducibility.
 
 ## Install
 
+To install `lm-eval` from the github repository main branch, run:
+
 ```bash
-pip install lm-eval
+git clone https://github.com/EleutherAI/lm-evaluation-harness
+cd lm-evaluation-harness
+pip install -e .
 ```
 
-To install additional multlingual tokenization and text segmenation packages, you must install the package with the `multilingual` extra:
+To install additional multilingual tokenization and text segmentation packages, you must install the package with the `multilingual` extra:
 
 ```bash
-pip install "lm-eval[multilingual]"
+pip install -e ".[multilingual]"
 ```
 
 ## Basic Usage
 
 > **Note**: When reporting results from eval harness, please include the task versions (shown in `results["versions"]`) for reproducibility. This allows bug fixes to tasks while also ensuring that previously reported scores are reproducible. See the [Task Versioning](#task-versioning) section for more info.
 
-To evaluate a model hosted on the [HuggingFace Hub](https://huggingface.co/models) (e.g. GPT-J-6B) you can use the following command:
+To evaluate a model hosted on the [HuggingFace Hub](https://huggingface.co/models) (e.g. GPT-J-6B) on tasks with names matching the pattern `lambada_*` and `hellaswag` you can use the following command:
 
 
 ```bash
 python main.py \
     --model hf-causal \
     --model_args pretrained=EleutherAI/gpt-j-6B \
-    --tasks lambada_openai,hellaswag \
-    --device 0
+    --tasks lambada_*,hellaswag \
+    --device cuda:0
 ```
 
-Additional arguments can be provided to the model constructor using the `--model_args` flag. Most notably, this supports the common practice of using the `revisions` feature on the Hub to store partialy trained checkpoints:
+Additional arguments can be provided to the model constructor using the `--model_args` flag. Most notably, this supports the common practice of using the `revisions` feature on the Hub to store partially trained checkpoints:
 
 ```bash
 python main.py \
     --model hf-causal \
     --model_args pretrained=EleutherAI/pythia-160m,revision=step100000 \
     --tasks lambada_openai,hellaswag \
-    --device 0
+    --device cuda:0
 ```
 
-To evaluate models that are called via `AutoSeq2SeqLM`, you instead use `hf-seq2seq`.
+To evaluate models that are loaded via `AutoSeq2SeqLM` in Huggingface, you instead use `hf-seq2seq`. *To evaluate (causal) models across multiple GPUs, use `--model hf-causal-experimental`*
 
 > **Warning**: Choosing the wrong model may result in erroneous outputs despite not erroring.
 
+To use with [PEFT](https://github.com/huggingface/peft), take the call you would run to evaluate the base model and add `,peft=PATH` to the `model_args` argument as shown below:
+```bash
+python main.py \
+    --model hf-causal-experimental \
+    --model_args pretrained=EleutherAI/gpt-j-6b,peft=nomic-ai/gpt4all-j-lora \
+    --tasks openbookqa,arc_easy,winogrande,hellaswag,arc_challenge,piqa,boolq \
+    --device cuda:0
+```
+
 Our library also supports the OpenAI API:
 
 ```bash
@@ -64,7 +78,7 @@ python main.py \
     --tasks lambada_openai,hellaswag
 ```
 
-While this functionality is only officially mantained for the official OpenAI API, it tends to also work for other hosting services that use the same API such as [goose.ai](goose.ai) with minor modification. We also have an implementation for the [TextSynth](https://textsynth.com/index.html) API, using `--model textsynth`.
+While this functionality is only officially maintained for the official OpenAI API, it tends to also work for other hosting services that use the same API such as [goose.ai](goose.ai) with minor modification. We also have an implementation for the [TextSynth](https://textsynth.com/index.html) API, using `--model textsynth`.
 
 To verify the data integrity of the tasks you're performing in addition to running the tasks themselves, you can use the `--check_integrity` flag:
 
@@ -102,6 +116,8 @@ When reporting eval harness results, please also report the version of each task
 
 ## Test Set Decontamination
 
+To address concerns about train / test contamination, we provide utilities for comparing results on a benchmark using only the data points nto found in the model training set. Unfortunately, outside of models trained on the Pile and C4, its very rare that people who train models disclose the contents of the training data. However this utility can be useful to evaluate models you have trained on private data, provided you are willing to pre-compute the necessary indices. We provide computed indices for 13-gram exact match deduplication against the Pile, and plan to add additional precomputed dataset indices in the future (including C4 and min-hash LSH deduplication).
+
 For details on text decontamination, see the [decontamination guide](./docs/decontamination.md).
 
 Note that the directory provided to the `--decontamination_ngrams_path` argument should contain the ngram files and info.json. See the above guide for ngram generation for the pile, this could be adapted for other training sets.
@@ -111,7 +127,7 @@ python main.py \
     --model gpt2 \
     --tasks sciq \
     --decontamination_ngrams_path path/containing/training/set/ngrams \
-    --device 0
+    --device cuda:0
 ```
 
 ## Cite as

lm_eval/base.py

@@ -381,18 +381,25 @@ class BaseLM(LM):
 
         re_ord = utils.Reorderer(requests, _collate)
 
-        for context, until in tqdm(re_ord.get_reordered()):
+        for context, request_args in tqdm(re_ord.get_reordered()):
+            until = request_args['until']
             if isinstance(until, str):
                 until = [until]
 
-            (primary_until,) = self.tok_encode(until[0])
+            if until:
+                (primary_until,) = self.tok_encode(until[0])
+            else:
+                primary_until = None
 
             context_enc = torch.tensor(
                 [self.tok_encode(context)[self.max_gen_toks - self.max_length :]]
             ).to(self.device)
 
+            max_gen_tokens = min(
+                self.max_gen_toks, request_args.get('max_length', self.max_gen_toks)
+            )
             cont = self._model_generate(
-                context_enc, context_enc.shape[1] + self.max_gen_toks, primary_until
+                context_enc, context_enc.shape[1] + max_gen_tokens, primary_until
             )
 
             s = self.tok_decode(cont[0].tolist()[context_enc.shape[1] :])

lm_eval/datasets/arithmetic/arithmetic.py

-# Copyright 2020 The HuggingFace Datasets Authors and the current dataset script contributor.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-"""GPT-3 Arithmetic Test Dataset."""
-
-
-import json
-
-import datasets
-
-
-_CITATION = """\
-@inproceedings{NEURIPS2020_1457c0d6,
-    author = {Brown, Tom and Mann, Benjamin and Ryder, Nick and Subbiah, Melanie and Kaplan, Jared D and Dhariwal, Prafulla and Neelakantan, Arvind and Shyam, Pranav and Sastry, Girish and Askell, Amanda and Agarwal, Sandhini and Herbert-Voss, Ariel and Krueger, Gretchen and Henighan, Tom and Child, Rewon and Ramesh, Aditya and Ziegler, Daniel and Wu, Jeffrey and Winter, Clemens and Hesse, Chris and Chen, Mark and Sigler, Eric and Litwin, Mateusz and Gray, Scott and Chess, Benjamin and Clark, Jack and Berner, Christopher and McCandlish, Sam and Radford, Alec and Sutskever, Ilya and Amodei, Dario},
-    booktitle = {Advances in Neural Information Processing Systems},
-    editor = {H. Larochelle and M. Ranzato and R. Hadsell and M. F. Balcan and H. Lin},
-    pages = {1877--1901},
-    publisher = {Curran Associates, Inc.},
-    title = {Language Models are Few-Shot Learners},
-    url = {https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf},
-    volume = {33},
-    year = {2020}
-}
-"""
-
-_DESCRIPTION = """\
-A small battery of 10 tests that involve asking language models a simple arithmetic
-problem in natural language.
-"""
-
-_HOMEPAGE = "https://github.com/openai/gpt-3/tree/master/data"
-
-# TODO: Add the licence for the dataset here if you can find it
-_LICENSE = ""
-
-
-class ArithmeticConfig(datasets.BuilderConfig):
-    """BuilderConfig for GPT3 Arithmetic Test Dataset."""
-
-    def __init__(self, url, features, **kwargs):
-        """BuilderConfig for GPT3 Arithmetic dataset.
-
-        Args:
-        url: *string*, the url to the specific subset of the GPT3 Arithmetic dataset.
-        features: *list[string]*, list of the features that will appear in the
-            feature dict.
-        """
-        # Version history:
-        super().__init__(version=datasets.Version("0.0.1"), **kwargs)
-        self.url = url
-        self.features = features
-
-
-class Arithmetic(datasets.GeneratorBasedBuilder):
-    """A small battery of 10 tests involving simple arithmetic problems."""
-
-    BUILDER_CONFIGS = [
-        ArithmeticConfig(
-            name="arithmetic_2da",
-            url="https://raw.githubusercontent.com/openai/gpt-3/master/data/two_digit_addition.jsonl",
-            features=datasets.Features(
-                {
-                    "context": datasets.Value("string"),
-                    "completion": datasets.Value("string"),
-                }
-            ),
-            description="2-digit addition",
-        ),
-        ArithmeticConfig(
-            name="arithmetic_2ds",
-            url="https://raw.githubusercontent.com/openai/gpt-3/master/data/two_digit_subtraction.jsonl",
-            features=datasets.Features(
-                {
-                    "context": datasets.Value("string"),
-                    "completion": datasets.Value("string"),
-                }
-            ),
-            description="2-digit subtraction",
-        ),
-        ArithmeticConfig(
-            name="arithmetic_3da",
-            url="https://raw.githubusercontent.com/openai/gpt-3/master/data/three_digit_addition.jsonl",
-            features=datasets.Features(
-                {
-                    "context": datasets.Value("string"),
-                    "completion": datasets.Value("string"),
-                }
-            ),
-            description="3-digit addition",
-        ),
-        ArithmeticConfig(
-            name="arithmetic_3ds",
-            url="https://raw.githubusercontent.com/openai/gpt-3/master/data/three_digit_subtraction.jsonl",
-            features=datasets.Features(
-                {
-                    "context": datasets.Value("string"),
-                    "completion": datasets.Value("string"),
-                }
-            ),
-            description="3-digit subtraction",
-        ),
-        ArithmeticConfig(
-            name="arithmetic_4da",
-            url="https://raw.githubusercontent.com/openai/gpt-3/master/data/four_digit_addition.jsonl",
-            features=datasets.Features(
-                {
-                    "context": datasets.Value("string"),
-                    "completion": datasets.Value("string"),
-                }
-            ),
-            description="4-digit addition",
-        ),
-        ArithmeticConfig(
-            name="arithmetic_4ds",
-            url="https://raw.githubusercontent.com/openai/gpt-3/master/data/four_digit_subtraction.jsonl",
-            features=datasets.Features(
-                {
-                    "context": datasets.Value("string"),
-                    "completion": datasets.Value("string"),
-                }
-            ),
-            description="4-digit subtraction",
-        ),
-        ArithmeticConfig(
-            name="arithmetic_5da",
-            url="https://raw.githubusercontent.com/openai/gpt-3/master/data/five_digit_addition.jsonl",
-            features=datasets.Features(
-                {
-                    "context": datasets.Value("string"),
-                    "completion": datasets.Value("string"),
-                }
-            ),
-            description="5-digit addition",
-        ),
-        ArithmeticConfig(
-            name="arithmetic_5ds",
-            url="https://raw.githubusercontent.com/openai/gpt-3/master/data/five_digit_subtraction.jsonl",
-            features=datasets.Features(
-                {
-                    "context": datasets.Value("string"),
-                    "completion": datasets.Value("string"),
-                }
-            ),
-            description="5-digit subtraction",
-        ),
-        ArithmeticConfig(
-            name="arithmetic_2dm",
-            url="https://raw.githubusercontent.com/openai/gpt-3/master/data/two_digit_multiplication.jsonl",
-            features=datasets.Features(
-                {
-                    "context": datasets.Value("string"),
-                    "completion": datasets.Value("string"),
-                }
-            ),
-            description="2-digit multiplication",
-        ),
-        ArithmeticConfig(
-            name="arithmetic_1dc",
-            url="https://raw.githubusercontent.com/openai/gpt-3/master/data/single_digit_three_ops.jsonl",
-            features=datasets.Features(
-                {
-                    "context": datasets.Value("string"),
-                    "completion": datasets.Value("string"),
-                }
-            ),
-            description="Single digit 3 operations",
-        ),
-    ]
-
-    def _info(self):
-        return datasets.DatasetInfo(
-            description=f"{_DESCRIPTION}\n{self.config.description}",
-            features=self.config.features,
-            homepage=_HOMEPAGE,
-            license=_LICENSE,
-            citation=_CITATION,
-        )
-
-    def _split_generators(self, dl_manager):
-        urls = self.config.url
-        data_dir = dl_manager.download_and_extract(urls)
-        return [
-            datasets.SplitGenerator(
-                name=datasets.Split.VALIDATION,
-                # These kwargs will be passed to _generate_examples
-                gen_kwargs={
-                    "filepath": data_dir,
-                    "split": datasets.Split.VALIDATION,
-                },
-            ),
-        ]
-
-    # method parameters are unpacked from `gen_kwargs` as given in `_split_generators`
-    def _generate_examples(self, filepath, split):
-        with open(filepath, encoding="utf-8") as f:
-            for key, row in enumerate(f):
-                data = json.loads(row)
-                context = (
-                    data["context"]
-                    .strip()
-                    .replace("\n\n", "\n")
-                    .replace("Q:", "Question:")
-                    .replace("A:", "Answer:")
-                )
-                completion = data["completion"]
-                yield key, {"context": context, "completion": completion}