add readme

lm_eval/tasks/ruler/README.md

+# Task-name
+
+### Paper
+
+Title: `RULER: What’s the Real Context Size of Your Long-Context Language Models?`
+
+Abstract: `https://arxiv.org/abs/2404.06654`
+
+`RULER expands upon the vanilla NIAH test to encompass variations with diverse types and quantities of needles. Moreover, RULER introduces new task categories multi-hop tracing and aggregation to test behaviors beyond searching from context. We evaluate 17 long-context LMs with 13 representative tasks in RULER.`
+
+Homepage: `https://github.com/NVIDIA/RULER`
+
+> [!NOTE]
+> When using Ruler tasks, please note:
+> 1. A tokenizer is required for data processing. The system will use the tokenizer from model_args, or fall back to the tokenizer associated with the pretrained model name.
+> 2. The default maximum sequence length is 4096. For processing longer sequences, specify additional lengths using the metadata parameter:
+>   `--metadata=max_seq_length=4096,8192,16384,32768,65536,131072`. The metadata parameter can also be passed to the TaskManager.
+> 3. To prevent truncation of longer sequences, we recommend setting the max_length parameter in model_args:
+>   `--model_args=pretrained=...,max_length=32768`
+
+### Citation
+
+```
+@article{hsieh2024ruler,
+  title={RULER: What's the Real Context Size of Your Long-Context Language Models?},
+  author={Cheng-Ping Hsieh and Simeng Sun and Samuel Kriman and Shantanu Acharya and Dima Rekesh and Fei Jia and Yang Zhang and Boris Ginsburg},
+  year={2024},
+  journal={arXiv preprint arXiv:2404.06654},
+}
+```
+
+### Groups, Tags, and Tasks
+
+#### Groups
+
+* `ruler`: `All 13 tasks in the RULER benchmark`
+
+#### Tags
+
+`longcxt`: `Long-context tasks`
+
+#### Tasks
+
+* `niah_single_1`: `NIAH single needle; key=word,value=number,haystack=repeat ∼passkey retrieval`
+* `niah_single_2`: `NIAH single needle; key=word,value=number,haystack=essay ∼vanilla NIAH`
+* `niah_single_3`: `NIAH single needle; key=word,value=uuid,haystack=essay`
+* `niah_multikey_1`: `NIAH multi-key, ∼line retrieval`
+* `niah_multikey_2`: `NIAH multi-key, ∼KV retrieval`
+* `niah_multikey_3`: `NIAH multi-key, `
+* `niah_multiquery`: `NIA multi-query`
+* `niah_multivalue`: `NIAH multi-value`
+* `ruler_vt`: `Variation tracing`
+* `ruler_cwe`: `Common word extraction`
+* `ruler_fwe`: `Frequent word extraction`
+* `ruler_qa_hotpot`: `QA Hotpot`
+* `ruler_qa_squad`: `QA SQuADv2`
+
+### Checklist
+
+For adding novel benchmarks/datasets to the library:
+* [x] Is the task an existing benchmark in the literature?
+  * [x] Have you referenced the original paper that introduced the task?
+  * [x] If yes, does the original paper provide a reference implementation? If so, have you checked against the reference implementation and documented how to run such a test?
+
+
+If other tasks on this dataset are already supported:
+* [x] Is the "Main" variant of this task clearly denoted?
+* [x] Have you provided a short sentence in a README on what each new variant adds / evaluates?
+* [x] Have you noted which, if any, published evaluation setups are matched by this variant?