Skip to content

GitLab

"vscode:/vscode.git/clone" did not exist on "2f0d3864962343ce894b7abc705b876028f2c668"
Unverified Commit fa837646 authored by LSinev's avatar LSinev Committed by GitHub
Browse files

Fix typos in multiple places (#2244) 

ACLUE bibtex typo reported to ACL Anthology and fixed here as title in pdf is correct.
parent 259b756a
Hide whitespace changes
Inline Side-by-side
Showing with 60 additions and 60 deletions
lm_eval/tasks/README.md
View file @ fa837646
......@@ -69,7 +69,7 @@
| [mgsm](mgsm/README.md) | Benchmark of multilingual grade-school math problems. | Spanish, French, German, Russian, Chinese, Japanese, Thai, Swahili, Bengali, Telugu |
| [minerva_math](minerva_math/README.md) | Mathematics-focused tasks requiring numerical reasoning and problem-solving skills. | English |
| mmlu | Massive Multitask Language Understanding benchmark for broad domain language evaluation. Several variants are supported. | English |
| [mmlusr](mmlusr/README.md) | Variation of MMLU designed to be more rigourous. | English |
| [mmlusr](mmlusr/README.md) | Variation of MMLU designed to be more rigorous. | English |
| model_written_evals | Evaluation tasks auto-generated for evaluating a collection of AI Safety concerns. | |
| [mutual](mutual/README.md) | A retrieval-based dataset for multi-turn dialogue reasoning. | English |
| [nq_open](nq_open/README.md) | Open domain question answering tasks based on the Natural Questions dataset. | English |
......
lm_eval/tasks/__init__.py
View file @ fa837646
......@@ -492,7 +492,7 @@ class TaskManager:
"`group` and `group_alias` keys in tasks' configs will no longer be used in the next release of lm-eval. "
"`tag` will be used to allow to call a collection of tasks just like `group`. "
"`group` will be removed in order to not cause confusion with the new ConfigurableGroup "
"which will be the offical way to create groups with addition of group-wide configuations."
"which will be the official way to create groups with addition of group-wide configurations."
)
print_info = False
# attr = "tag"
......
lm_eval/tasks/aclue/README.md
View file @ fa837646
......@@ -14,7 +14,7 @@ Homepage: https://github.com/isen-zhang/ACLUE
```bibtex
@inproceedings{zhang-li-2023-large,
title = "Can Large Langauge Model Comprehend {A}ncient {C}hinese? A Preliminary Test on {ACLUE}",
title = "Can Large Language Model Comprehend {A}ncient {C}hinese? A Preliminary Test on {ACLUE}",
author = "Zhang, Yixuan and Li, Haonan",
booktitle = "Proceedings of the Ancient Language Processing Workshop",
month = sep,
......
lm_eval/tasks/eq_bench/README.md
View file @ fa837646
......@@ -16,8 +16,8 @@ Homepage: https://eqbench.com/
NOTE: There are some key differences between the lm-evaluation-harness version and the implementation described in the EQ-Bench paper (These have been OK'd by the author):
- The lm-eval version uses the EQ-Bench v2 test set (171 questions) and score calculation. It does not incorporate the revision part of the prompt, as per v2.1 (https://github.com/EQ-bench/EQ-Bench)
- No retries in lm-eval version (EQ-Bench pipeline retries with successively higher temps if it encounters unparseable answers)
- In the original implementation, unparseable answers are excluded from the final score, and 83% of answers have to be parseable or a fail is returned. The lm-eval version instead assigns 0 to unparsable answers and has no fail criteria. So for lower performing models, there may be differences with the EQ-Bench leaderboard.
- No retries in lm-eval version (EQ-Bench pipeline retries with successively higher temps if it encounters unparsable answers)
- In the original implementation, unparsable answers are excluded from the final score, and 83% of answers have to be parseable or a fail is returned. The lm-eval version instead assigns 0 to unparsable answers and has no fail criteria. So for lower performing models, there may be differences with the EQ-Bench leaderboard.
### Citation
......
lm_eval/tasks/ifeval/instructions.py
View file @ fa837646
......@@ -78,7 +78,7 @@ _ENDING_OPTIONS = ("Any other questions?", "Is there anything else I can help wi
# The number of highlighted sections.
_NUM_HIGHLIGHTED_SECTIONS = 4
# The section spliter.
# The section splitter.
_SECTION_SPLITER = ("Section", "SECTION")
# The number of sections.
......@@ -153,7 +153,7 @@ class ResponseLanguageChecker(Instruction):
return self._description_pattern.format(language=_LANGUAGES[self._language])
def get_instruction_args(self):
"""Returns the keyward args of `build_description`."""
"""Returns the keyword args of `build_description`."""
return {"language": self._language}
def get_instruction_args_keys(self):
......@@ -223,7 +223,7 @@ class NumberOfSentences(Instruction):
)
def get_instruction_args(self):
"""Returns the keyward args of `build_description`."""
"""Returns the keyword args of `build_description`."""
return {
"num_sentences": self._num_sentences_threshold,
"relation": self._comparison_relation,
......@@ -276,7 +276,7 @@ class PlaceholderChecker(Instruction):
return self._description_pattern.format(num_placeholders=self._num_placeholders)
def get_instruction_args(self):
"""Returns the keyward args of `build_description`."""
"""Returns the keyword args of `build_description`."""
return {"num_placeholders": self._num_placeholders}
def get_instruction_args_keys(self):
......@@ -323,7 +323,7 @@ class BulletListChecker(Instruction):
return self._description_pattern.format(num_bullets=self._num_bullets)
def get_instruction_args(self):
"""Returns the keyward args of `build_description`."""
"""Returns the keyword args of `build_description`."""
return {"num_bullets": self._num_bullets}
def get_instruction_args_keys(self):
......@@ -362,7 +362,7 @@ class ConstrainedResponseChecker(Instruction):
)
def get_instruction_args(self):
"""Returns the keyward args of `build_description`."""
"""Returns the keyword args of `build_description`."""
return None
def get_instruction_args_keys(self):
......@@ -393,7 +393,7 @@ class ConstrainedStartChecker(Instruction):
"""Build the instruction description.
Args:
starter: A string representing the keyward that the response should start
starter: A string representing the keyword that the response should start
with.
Returns:
......@@ -409,7 +409,7 @@ class ConstrainedStartChecker(Instruction):
return self._description_pattern.format(starter=self._starter)
def get_instruction_args(self):
"""Returns the keyward args of `build_description`."""
"""Returns the keyword args of `build_description`."""
return {"starter": self._starter}
def get_instruction_args_keys(self):
......@@ -458,7 +458,7 @@ class HighlightSectionChecker(Instruction):
return self._description_pattern.format(num_highlights=self._num_highlights)
def get_instruction_args(self):
"""Returns the keyward args of `build_description`."""
"""Returns the keyword args of `build_description`."""
return {"num_highlights": self._num_highlights}
def get_instruction_args_keys(self):
......@@ -469,12 +469,12 @@ class HighlightSectionChecker(Instruction):
"""Checks if the number of highlighted sections meets the requirement.
Args:
value: a string repesenting the response. The response is expected to
value: a string representing the response. The response is expected to
contain highlighted sections in the format of *highlighted*.
Returns:
True if the actual number of highlighted sections in the format of
*highlighed sections* meets the minimum requirement; otherwise False.
*highlighted sections* meets the minimum requirement; otherwise False.
"""
num_highlights = 0
highlights = re.findall(r"\*[^\n\*]*\*", value)
......@@ -529,7 +529,7 @@ class SectionChecker(Instruction):
)
def get_instruction_args(self):
"""Returns the keyward args of `build_description`."""
"""Returns the keyword args of `build_description`."""
return {
"section_spliter": self._section_spliter,
"num_sections": self._num_sections,
......@@ -582,7 +582,7 @@ class ParagraphChecker(Instruction):
return self._description_pattern.format(num_paragraphs=self._num_paragraphs)
def get_instruction_args(self):
"""Returns the keyward args of `build_description`."""
"""Returns the keyword args of `build_description`."""
return {"num_paragraphs": self._num_paragraphs}
def get_instruction_args_keys(self):
......@@ -642,7 +642,7 @@ class PostscriptChecker(Instruction):
return self._description_pattern.format(postscript=self._postscript_marker)
def get_instruction_args(self):
"""Returns the keyward args of `build_description`."""
"""Returns the keyword args of `build_description`."""
return {"postscript_marker": self._postscript_marker}
def get_instruction_args_keys(self):
......@@ -672,7 +672,7 @@ class PostscriptChecker(Instruction):
class RephraseChecker(Instruction):
"""Checks the repharse."""
"""Checks the rephrase."""
def build_description(self, *, original_message):
"""Build the instruction description.
......@@ -701,7 +701,7 @@ class RephraseChecker(Instruction):
return self._description
def get_instruction_args(self):
"""Returns the keyward args of `build_description`."""
"""Returns the keyword args of `build_description`."""
return {"original_message": self._reference_without_change}
def get_instruction_args_keys(self):
......@@ -766,7 +766,7 @@ class KeywordChecker(Instruction):
return self._description_pattern.format(keywords=self._keywords)
def get_instruction_args(self):
"""Returns the keyward args of `build_description`."""
"""Returns the keyword args of `build_description`."""
return {"keywords": self._keywords}
def get_instruction_args_keys(self):
......@@ -831,7 +831,7 @@ class KeywordFrequencyChecker(Instruction):
)
def get_instruction_args(self):
"""Returns the keyward args of `build_description`."""
"""Returns the keyword args of `build_description`."""
return {
"keyword": self._keyword,
"frequency": self._frequency,
......@@ -894,7 +894,7 @@ class NumberOfWords(Instruction):
)
def get_instruction_args(self):
"""Returns the keyward args of `build_description`."""
"""Returns the keyword args of `build_description`."""
return {"num_words": self._num_words, "relation": self._comparison_relation}
def get_instruction_args_keys(self):
......@@ -922,7 +922,7 @@ class JsonFormat(Instruction):
return self._description_pattern
def get_instruction_args(self):
"""Returns the keyward args of `build_description`."""
"""Returns the keyword args of `build_description`."""
return None
def get_instruction_args_keys(self):
......@@ -996,7 +996,7 @@ class ParagraphFirstWordCheck(Instruction):
)
def get_instruction_args(self):
"""Returns the keyward args of `build_description`."""
"""Returns the keyword args of `build_description`."""
return {
"num_paragraphs": self._num_paragraphs,
"nth_paragraph": self._nth_paragraph,
......@@ -1089,7 +1089,7 @@ class KeySentenceChecker(Instruction):
)
def get_instruction_args(self):
"""Returns the keyward args of `build_description`."""
"""Returns the keyword args of `build_description`."""
return {
"num_sentences": self._num_sentences,
"key_sentences": list(self._key_sentences),
......@@ -1117,7 +1117,7 @@ class ForbiddenWords(Instruction):
"""Build the instruction description.
Args:
forbidden_words: A sequences of strings respresenting words that are not
forbidden_words: A sequences of strings representing words that are not
allowed in the response.
Returns:
......@@ -1138,7 +1138,7 @@ class ForbiddenWords(Instruction):
return self._description_pattern.format(forbidden_words=self._forbidden_words)
def get_instruction_args(self):
"""Returns the keyward args of `build_description`."""
"""Returns the keyword args of `build_description`."""
return {"forbidden_words": self._forbidden_words}
def get_instruction_args_keys(self):
......@@ -1188,7 +1188,7 @@ class RephraseParagraph(Instruction):
)
def get_instruction_args(self):
"""Returns the keyward args of `build_description`."""
"""Returns the keyword args of `build_description`."""
return {
"original_paragraph": self._original_paragraph,
"low": self._low,
......@@ -1225,7 +1225,7 @@ class TwoResponsesChecker(Instruction):
return self._description_pattern
def get_instruction_args(self):
"""Returns the keyward args of `build_description`."""
"""Returns the keyword args of `build_description`."""
return None
def get_instruction_args_keys(self):
......
lm_eval/tasks/leaderboard/ifeval/instructions.py
View file @ fa837646
......@@ -78,7 +78,7 @@ _ENDING_OPTIONS = ("Any other questions?", "Is there anything else I can help wi
# The number of highlighted sections.
_NUM_HIGHLIGHTED_SECTIONS = 4
# The section spliter.
# The section splitter.
_SECTION_SPLITER = ("Section", "SECTION")
# The number of sections.
......@@ -153,7 +153,7 @@ class ResponseLanguageChecker(Instruction):
return self._description_pattern.format(language=_LANGUAGES[self._language])
def get_instruction_args(self):
"""Returns the keyward args of `build_description`."""
"""Returns the keyword args of `build_description`."""
return {"language": self._language}
def get_instruction_args_keys(self):
......@@ -223,7 +223,7 @@ class NumberOfSentences(Instruction):
)
def get_instruction_args(self):
"""Returns the keyward args of `build_description`."""
"""Returns the keyword args of `build_description`."""
return {
"num_sentences": self._num_sentences_threshold,
"relation": self._comparison_relation,
......@@ -276,7 +276,7 @@ class PlaceholderChecker(Instruction):
return self._description_pattern.format(num_placeholders=self._num_placeholders)
def get_instruction_args(self):
"""Returns the keyward args of `build_description`."""
"""Returns the keyword args of `build_description`."""
return {"num_placeholders": self._num_placeholders}
def get_instruction_args_keys(self):
......@@ -323,7 +323,7 @@ class BulletListChecker(Instruction):
return self._description_pattern.format(num_bullets=self._num_bullets)
def get_instruction_args(self):
"""Returns the keyward args of `build_description`."""
"""Returns the keyword args of `build_description`."""
return {"num_bullets": self._num_bullets}
def get_instruction_args_keys(self):
......@@ -362,7 +362,7 @@ class ConstrainedResponseChecker(Instruction):
)
def get_instruction_args(self):
"""Returns the keyward args of `build_description`."""
"""Returns the keyword args of `build_description`."""
return None
def get_instruction_args_keys(self):
......@@ -393,7 +393,7 @@ class ConstrainedStartChecker(Instruction):
"""Build the instruction description.
Args:
starter: A string representing the keyward that the response should start
starter: A string representing the keyword that the response should start
with.
Returns:
......@@ -409,7 +409,7 @@ class ConstrainedStartChecker(Instruction):
return self._description_pattern.format(starter=self._starter)
def get_instruction_args(self):
"""Returns the keyward args of `build_description`."""
"""Returns the keyword args of `build_description`."""
return {"starter": self._starter}
def get_instruction_args_keys(self):
......@@ -458,7 +458,7 @@ class HighlightSectionChecker(Instruction):
return self._description_pattern.format(num_highlights=self._num_highlights)
def get_instruction_args(self):
"""Returns the keyward args of `build_description`."""
"""Returns the keyword args of `build_description`."""
return {"num_highlights": self._num_highlights}
def get_instruction_args_keys(self):
......@@ -469,12 +469,12 @@ class HighlightSectionChecker(Instruction):
"""Checks if the number of highlighted sections meets the requirement.
Args:
value: a string repesenting the response. The response is expected to
value: a string representing the response. The response is expected to
contain highlighted sections in the format of *highlighted*.
Returns:
True if the actual number of highlighted sections in the format of
*highlighed sections* meets the minimum requirement; otherwise False.
*highlighted sections* meets the minimum requirement; otherwise False.
"""
num_highlights = 0
highlights = re.findall(r"\*[^\n\*]*\*", value)
......@@ -529,7 +529,7 @@ class SectionChecker(Instruction):
)
def get_instruction_args(self):
"""Returns the keyward args of `build_description`."""
"""Returns the keyword args of `build_description`."""
return {
"section_spliter": self._section_spliter,
"num_sections": self._num_sections,
......@@ -582,7 +582,7 @@ class ParagraphChecker(Instruction):
return self._description_pattern.format(num_paragraphs=self._num_paragraphs)
def get_instruction_args(self):
"""Returns the keyward args of `build_description`."""
"""Returns the keyword args of `build_description`."""
return {"num_paragraphs": self._num_paragraphs}
def get_instruction_args_keys(self):
......@@ -642,7 +642,7 @@ class PostscriptChecker(Instruction):
return self._description_pattern.format(postscript=self._postscript_marker)
def get_instruction_args(self):
"""Returns the keyward args of `build_description`."""
"""Returns the keyword args of `build_description`."""
return {"postscript_marker": self._postscript_marker}
def get_instruction_args_keys(self):
......@@ -672,7 +672,7 @@ class PostscriptChecker(Instruction):
class RephraseChecker(Instruction):
"""Checks the repharse."""
"""Checks the rephrase."""
def build_description(self, *, original_message):
"""Build the instruction description.
......@@ -701,7 +701,7 @@ class RephraseChecker(Instruction):
return self._description
def get_instruction_args(self):
"""Returns the keyward args of `build_description`."""
"""Returns the keyword args of `build_description`."""
return {"original_message": self._reference_without_change}
def get_instruction_args_keys(self):
......@@ -766,7 +766,7 @@ class KeywordChecker(Instruction):
return self._description_pattern.format(keywords=self._keywords)
def get_instruction_args(self):
"""Returns the keyward args of `build_description`."""
"""Returns the keyword args of `build_description`."""
return {"keywords": self._keywords}
def get_instruction_args_keys(self):
......@@ -831,7 +831,7 @@ class KeywordFrequencyChecker(Instruction):
)
def get_instruction_args(self):
"""Returns the keyward args of `build_description`."""
"""Returns the keyword args of `build_description`."""
return {
"keyword": self._keyword,
"frequency": self._frequency,
......@@ -894,7 +894,7 @@ class NumberOfWords(Instruction):
)
def get_instruction_args(self):
"""Returns the keyward args of `build_description`."""
"""Returns the keyword args of `build_description`."""
return {"num_words": self._num_words, "relation": self._comparison_relation}
def get_instruction_args_keys(self):
......@@ -922,7 +922,7 @@ class JsonFormat(Instruction):
return self._description_pattern
def get_instruction_args(self):
"""Returns the keyward args of `build_description`."""
"""Returns the keyword args of `build_description`."""
return None
def get_instruction_args_keys(self):
......@@ -996,7 +996,7 @@ class ParagraphFirstWordCheck(Instruction):
)
def get_instruction_args(self):
"""Returns the keyward args of `build_description`."""
"""Returns the keyword args of `build_description`."""
return {
"num_paragraphs": self._num_paragraphs,
"nth_paragraph": self._nth_paragraph,
......@@ -1089,7 +1089,7 @@ class KeySentenceChecker(Instruction):
)
def get_instruction_args(self):
"""Returns the keyward args of `build_description`."""
"""Returns the keyword args of `build_description`."""
return {
"num_sentences": self._num_sentences,
"key_sentences": list(self._key_sentences),
......@@ -1117,7 +1117,7 @@ class ForbiddenWords(Instruction):
"""Build the instruction description.
Args:
forbidden_words: A sequences of strings respresenting words that are not
forbidden_words: A sequences of strings representing words that are not
allowed in the response.
Returns:
......@@ -1138,7 +1138,7 @@ class ForbiddenWords(Instruction):
return self._description_pattern.format(forbidden_words=self._forbidden_words)
def get_instruction_args(self):
"""Returns the keyward args of `build_description`."""
"""Returns the keyword args of `build_description`."""
return {"forbidden_words": self._forbidden_words}
def get_instruction_args_keys(self):
......@@ -1188,7 +1188,7 @@ class RephraseParagraph(Instruction):
)
def get_instruction_args(self):
"""Returns the keyward args of `build_description`."""
"""Returns the keyword args of `build_description`."""
return {
"original_paragraph": self._original_paragraph,
"low": self._low,
......@@ -1225,7 +1225,7 @@ class TwoResponsesChecker(Instruction):
return self._description_pattern
def get_instruction_args(self):
"""Returns the keyward args of `build_description`."""
"""Returns the keyword args of `build_description`."""
return None
def get_instruction_args_keys(self):
......
lm_eval/tasks/scrolls/README.md
View file @ fa837646
......@@ -11,7 +11,7 @@ Homepage: https://www.scrolls-benchmark.com/
Since SCROLLS tasks are generally longer than the maximum sequence length of many models,
it is possible to create "subset" tasks that contain only those samples whose tokenized length
is less than some pre-defined limit. For example, to create a subset of "Qasper" that would
be suitable for a model using the GPTNeoX tokenizer and a 4K maximium sequence length:
be suitable for a model using the GPTNeoX tokenizer and a 4K maximum sequence length:
```
class QasperGPTNeoX4K(Qasper):
......
lm_eval/tasks/scrolls/task.py
View file @ fa837646
......@@ -439,7 +439,7 @@ class GovReport(_SCROLLSSummaryTask):
Note: The average length of the reference summaries is ~3,000
characters, or ~600 tokens as tokenized by GPT-NeoX. For causal models,
it is recommended to set `max_gen_toks` sufficently large (e.g. 1024)
it is recommended to set `max_gen_toks` sufficiently large (e.g. 1024)
to allow a full summary to be generated.
"""
......
lm_eval/tasks/tmmluplus/default/_generate_configs.py
View file @ fa837646
......@@ -11,7 +11,7 @@ from tqdm import tqdm
# Copy from https://github.com/iKala/ievals/blob/main/ievals/settings.py
# from TMMLU+ offical example
# from TMMLU+ official example
categories = {
"STEM": [
"physics",
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment