Skip to content

GitLab

Switch branch/tag
  1. 28 Jun, 2024 2 commits
    • Baber Abbasi's avatar
      Add chat template to `vllm` (#2034) · cc2d3463
      Baber Abbasi authored 
      

* add chat template

* refactor token padding

* nit

* nit

* check on failing test

* check transformers version

* remove transformers pin

* add ids to test

* nit

* fixup

* fix bos bug

* nit

* fixup! fix bos bug

* increase tolerance for table test

* don't detokenize vllm logprobs

* Update lm_eval/models/utils.py
Co-authored-by: default avatarHailey Schoelkopf <65563625+haileyschoelkopf@users.noreply.github.com>

* pre-commit run --all-files

---------
Co-authored-by: default avatarHailey Schoelkopf <65563625+haileyschoelkopf@users.noreply.github.com>
      cc2d3463
    • Steven Basart's avatar
      Fixes scrolls task bug with few_shot examples (#2003) · 801322e0
      Steven Basart authored 
      Bug:

```
python -m scripts.write_out --task scrolls_quality --output_base_path ~/workspace/
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/lm-evaluation-harness/scripts/write_out.py", line 92, in <module>
    main()
  File "/lm-evaluation-harness/scripts/write_out.py", line 51, in main
    task_dict = tasks.get_task_dict(task_names, task_manager)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 423, in get_task_dict
    task_name_from_string_dict = task_manager.load_task_or_group(
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 271, in load_task_or_group
    collections.ChainMap(*map(self._load_individual_task_or_group, task_list))
  File "/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 162, in _load_individual_task_or_group
    return load_task(task_config, task=name_or_config, group=parent_name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lm-evaluation-harness/lm_eval/tasks/__init__.py", line 148, in load_task
    task_object = config["class"]()
                  ^^^^^^^^^^^^^^^^^
  File "/lm-evaluation-harness/lm_eval/tasks/scrolls/task.py", line 120, in __init__
    super().__init__()
  File "/lm-evaluation-harness/lm_eval/api/task.py", line 703, in __init__
    self._config = TaskConfig(**config)
                   ^^^^^^^^^^^^^^^^^^^^
TypeError: lm_eval.api.task.TaskConfig() argument after ** must be a mapping, not NoneType
```
      801322e0
  3. 26 Jun, 2024 1 commit
  5. 25 Jun, 2024 3 commits
  7. 20 Jun, 2024 1 commit
  9. 19 Jun, 2024 3 commits
  11. 18 Jun, 2024 1 commit
  13. 13 Jun, 2024 2 commits
  15. 11 Jun, 2024 2 commits
  17. 10 Jun, 2024 1 commit
  19. 07 Jun, 2024 3 commits
  21. 06 Jun, 2024 3 commits
  23. 05 Jun, 2024 1 commit
  25. 03 Jun, 2024 1 commit
  27. 31 May, 2024 1 commit
  29. 24 May, 2024 2 commits
  31. 22 May, 2024 1 commit
  33. 21 May, 2024 1 commit
  35. 13 May, 2024 1 commit
  37. 09 May, 2024 1 commit
    • Edd's avatar
      Copal task (#1803) · 1980a13c
      Edd authored 
      * add copal

* change name to copal id for clarity and the task name

* remove `copal_id...` to yaml to make it work

* checkmark on README

* change group name to `copal_id`
      1980a13c
  39. 08 May, 2024 1 commit
  41. 07 May, 2024 3 commits
    • Yoav Katz's avatar
      Initial integration of the Unitxt to LM eval harness (#1615) · 885f48d6
      Yoav Katz authored 
      * Initial support for Unitxt datasets in LM Eval Harness

See  https://github.com/IBM/unitxt



The script 'generate_yamls.py' creates LM Eval Harness yaml files corresponding to Unitxt datasets specified in the 'unitxt_datasets' file.

The glue code required to register Unitxt metrics is in 'unitxt_wrapper.py'.

* Added dataset loading check to generate_yaml

Improved error messages.

* Speed up generate_yaml

Added printouts and improved error message

* Added output printout

* Simplified integration of unitxt datasets

Store all the common yaml configuration in a yaml include shared by all datasets of the same task.

* Post code review comments - part 1

1. Made sure include files don't end wth 'yaml' so they won't be marked as tasks
2. Added more datasets and tasks (NER, GEC)
3. Added README

* Post code review comments - part 2

1. Added install unitxt install option in pyproject.toml:
pip install 'lm_eval[unitxt]'
2. Added a check that unitxt is installed and print a clear error message if not

* Commited missing pyproject change

* Added documentation on adding datasets

* More doc changes

* add unitxt extra to readme

* run precommit

---------
Co-authored-by: default avatarhaileyschoelkopf <hailey@eleuther.ai>
      885f48d6
    • Hailey Schoelkopf's avatar
      Re-add Hendrycks MATH (no sympy checking, no Minerva hardcoded prompt) variant (#1793) · d42a3e44
      Hailey Schoelkopf authored 
      * add Hendrycks MATH (no sympy checking) variant

* add readmes for MATH tasks
      d42a3e44
    • Hailey Schoelkopf's avatar
      Fix Caching Tests ; Remove `pretrained=gpt2` default (#1775) · 7fe2b93c
      Hailey Schoelkopf authored
      7fe2b93c
  43. 01 May, 2024 4 commits
  45. 26 Apr, 2024 1 commit