修改readme

evaluate-0.4.2/measurements/word_length/requirements.txt

+git+https://github.com/huggingface/evaluate.git@{COMMIT_PLACEHOLDER}
+nltk~=3.7

evaluate-0.4.2/measurements/word_length/word_length.py

+# Copyright 2022 The HuggingFace Team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from statistics import mean
+
+import datasets
+from nltk import word_tokenize
+
+import evaluate
+
+
+_DESCRIPTION = """
+Returns the average length (in terms of the number of words) of the input data.
+"""
+
+_KWARGS_DESCRIPTION = """
+Args:
+    `data`: a list of `str` for which the word length is calculated.
+    `tokenizer` (`Callable`) : the approach used for tokenizing `data` (optional).
+        The default tokenizer is `word_tokenize` from NLTK: https://www.nltk.org/api/nltk.tokenize.html
+        This can be replaced by any function that takes a string as input and returns a list of tokens as output.
+
+Returns:
+    `average_word_length` (`float`) : the average number of words in the input list of strings.
+
+Examples:
+    >>> data = ["hello world"]
+    >>> wordlength = evaluate.load("word_length", module_type="measurement")
+    >>> results = wordlength.compute(data=data)
+    >>> print(results)
+    {'average_word_length': 2}
+"""
+
+# TODO: Add BibTeX citation
+_CITATION = """\
+@InProceedings{huggingface:module,
+title = {A great new module},
+authors={huggingface, Inc.},
+year={2020}
+}
+"""
+
+
+@evaluate.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
+class WordLength(evaluate.Measurement):
+    """This measurement returns the average number of words in the input string(s)."""
+
+    def _info(self):
+        # TODO: Specifies the evaluate.MeasurementInfo object
+        return evaluate.MeasurementInfo(
+            # This is the description that will appear on the modules page.
+            module_type="measurement",
+            description=_DESCRIPTION,
+            citation=_CITATION,
+            inputs_description=_KWARGS_DESCRIPTION,
+            # This defines the format of each prediction and reference
+            features=datasets.Features(
+                {
+                    "data": datasets.Value("string"),
+                }
+            ),
+        )
+
+    def _download_and_prepare(self, dl_manager):
+        import nltk
+
+        nltk.download("punkt")
+
+    def _compute(self, data, tokenizer=word_tokenize):
+        """Returns the average word length of the input data"""
+        lengths = [len(tokenizer(d)) for d in data]
+        average_length = mean(lengths)
+        return {"average_word_length": average_length}

evaluate-0.4.2/metrics/accuracy/README.md

+---
+title: Accuracy
+emoji: 🤗 
+colorFrom: blue
+colorTo: red
+sdk: gradio
+sdk_version: 3.19.1
+app_file: app.py
+pinned: false
+tags:
+- evaluate
+- metric
+description: >-
+  Accuracy is the proportion of correct predictions among the total number of cases processed. It can be computed with:
+  Accuracy = (TP + TN) / (TP + TN + FP + FN)
+  Where:
+  TP: True positive
+  TN: True negative
+  FP: False positive
+  FN: False negative
+---
+
+# Metric Card for Accuracy
+
+
+## Metric Description
+
+Accuracy is the proportion of correct predictions among the total number of cases processed. It can be computed with:
+Accuracy = (TP + TN) / (TP + TN + FP + FN)
+ Where:
+TP: True positive
+TN: True negative
+FP: False positive
+FN: False negative
+
+
+## How to Use
+
+At minimum, this metric requires predictions and references as inputs.
+
+```python
+>>> accuracy_metric = evaluate.load("accuracy")
+>>> results = accuracy_metric.compute(references=[0, 1], predictions=[0, 1])
+>>> print(results)
+{'accuracy': 1.0}
+```
+
+
+### Inputs
+- **predictions** (`list` of `int`): Predicted labels.
+- **references** (`list` of `int`): Ground truth labels.
+- **normalize** (`boolean`): If set to False, returns the number of correctly classified samples. Otherwise, returns the fraction of correctly classified samples. Defaults to True.
+- **sample_weight** (`list` of `float`): Sample weights Defaults to None.
+
+
+### Output Values
+- **accuracy**(`float` or `int`): Accuracy score. Minimum possible value is 0. Maximum possible value is 1.0, or the number of examples input, if `normalize` is set to `True`. A higher score means higher accuracy.
+
+Output Example(s):
+```python
+{'accuracy': 1.0}
+```
+
+This metric outputs a dictionary, containing the accuracy score.
+
+
+#### Values from Popular Papers
+
+Top-1 or top-5 accuracy is often used to report performance on supervised classification tasks such as image classification (e.g. on [ImageNet](https://paperswithcode.com/sota/image-classification-on-imagenet)) or sentiment analysis (e.g. on [IMDB](https://paperswithcode.com/sota/text-classification-on-imdb)). 
+
+
+### Examples
+
+Example 1-A simple example
+```python
+>>> accuracy_metric = evaluate.load("accuracy")
+>>> results = accuracy_metric.compute(references=[0, 1, 2, 0, 1, 2], predictions=[0, 1, 1, 2, 1, 0])
+>>> print(results)
+{'accuracy': 0.5}
+```
+
+Example 2-The same as Example 1, except with `normalize` set to `False`.
+```python
+>>> accuracy_metric = evaluate.load("accuracy")
+>>> results = accuracy_metric.compute(references=[0, 1, 2, 0, 1, 2], predictions=[0, 1, 1, 2, 1, 0], normalize=False)
+>>> print(results)
+{'accuracy': 3.0}
+```
+
+Example 3-The same as Example 1, except with `sample_weight` set.
+```python
+>>> accuracy_metric = evaluate.load("accuracy")
+>>> results = accuracy_metric.compute(references=[0, 1, 2, 0, 1, 2], predictions=[0, 1, 1, 2, 1, 0], sample_weight=[0.5, 2, 0.7, 0.5, 9, 0.4])
+>>> print(results)
+{'accuracy': 0.8778625954198473}
+```
+
+
+## Limitations and Bias
+This metric can be easily misleading, especially in the case of unbalanced classes. For example, a high accuracy might be because a model is doing well, but if the data is unbalanced, it might also be because the model is only accurately labeling the high-frequency class. In such cases, a more detailed analysis of the model's behavior, or the use of a different metric entirely, is necessary to determine how well the model is actually performing.
+
+
+## Citation(s)
+```bibtex
+@article{scikit-learn,
+  title={Scikit-learn: Machine Learning in {P}ython},
+  author={Pedregosa, F. and Varoquaux, G. and Gramfort, A. and Michel, V.
+         and Thirion, B. and Grisel, O. and Blondel, M. and Prettenhofer, P.
+         and Weiss, R. and Dubourg, V. and Vanderplas, J. and Passos, A. and
+         Cournapeau, D. and Brucher, M. and Perrot, M. and Duchesnay, E.},
+  journal={Journal of Machine Learning Research},
+  volume={12},
+  pages={2825--2830},
+  year={2011}
+}
+```
+
+
+## Further References

evaluate-0.4.2/metrics/accuracy/accuracy.py

+# Copyright 2020 The HuggingFace Datasets Authors and the current dataset script contributor.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Accuracy metric."""
+
+import datasets
+from sklearn.metrics import accuracy_score
+
+import evaluate
+
+
+_DESCRIPTION = """
+Accuracy is the proportion of correct predictions among the total number of cases processed. It can be computed with:
+Accuracy = (TP + TN) / (TP + TN + FP + FN)
+ Where:
+TP: True positive
+TN: True negative
+FP: False positive
+FN: False negative
+"""
+
+
+_KWARGS_DESCRIPTION = """
+Args:
+    predictions (`list` of `int`): Predicted labels.
+    references (`list` of `int`): Ground truth labels.
+    normalize (`boolean`): If set to False, returns the number of correctly classified samples. Otherwise, returns the fraction of correctly classified samples. Defaults to True.
+    sample_weight (`list` of `float`): Sample weights Defaults to None.
+
+Returns:
+    accuracy (`float` or `int`): Accuracy score. Minimum possible value is 0. Maximum possible value is 1.0, or the number of examples input, if `normalize` is set to `True`.. A higher score means higher accuracy.
+
+Examples:
+
+    Example 1-A simple example
+        >>> accuracy_metric = evaluate.load("accuracy")
+        >>> results = accuracy_metric.compute(references=[0, 1, 2, 0, 1, 2], predictions=[0, 1, 1, 2, 1, 0])
+        >>> print(results)
+        {'accuracy': 0.5}
+
+    Example 2-The same as Example 1, except with `normalize` set to `False`.
+        >>> accuracy_metric = evaluate.load("accuracy")
+        >>> results = accuracy_metric.compute(references=[0, 1, 2, 0, 1, 2], predictions=[0, 1, 1, 2, 1, 0], normalize=False)
+        >>> print(results)
+        {'accuracy': 3.0}
+
+    Example 3-The same as Example 1, except with `sample_weight` set.
+        >>> accuracy_metric = evaluate.load("accuracy")
+        >>> results = accuracy_metric.compute(references=[0, 1, 2, 0, 1, 2], predictions=[0, 1, 1, 2, 1, 0], sample_weight=[0.5, 2, 0.7, 0.5, 9, 0.4])
+        >>> print(results)
+        {'accuracy': 0.8778625954198473}
+"""
+
+
+_CITATION = """
+@article{scikit-learn,
+  title={Scikit-learn: Machine Learning in {P}ython},
+  author={Pedregosa, F. and Varoquaux, G. and Gramfort, A. and Michel, V.
+         and Thirion, B. and Grisel, O. and Blondel, M. and Prettenhofer, P.
+         and Weiss, R. and Dubourg, V. and Vanderplas, J. and Passos, A. and
+         Cournapeau, D. and Brucher, M. and Perrot, M. and Duchesnay, E.},
+  journal={Journal of Machine Learning Research},
+  volume={12},
+  pages={2825--2830},
+  year={2011}
+}
+"""
+
+
+@evaluate.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
+class Accuracy(evaluate.Metric):
+    def _info(self):
+        return evaluate.MetricInfo(
+            description=_DESCRIPTION,
+            citation=_CITATION,
+            inputs_description=_KWARGS_DESCRIPTION,
+            features=datasets.Features(
+                {
+                    "predictions": datasets.Sequence(datasets.Value("int32")),
+                    "references": datasets.Sequence(datasets.Value("int32")),
+                }
+                if self.config_name == "multilabel"
+                else {
+                    "predictions": datasets.Value("int32"),
+                    "references": datasets.Value("int32"),
+                }
+            ),
+            reference_urls=["https://scikit-learn.org/stable/modules/generated/sklearn.metrics.accuracy_score.html"],
+        )
+
+    def _compute(self, predictions, references, normalize=True, sample_weight=None):
+        return {
+            "accuracy": float(
+                accuracy_score(references, predictions, normalize=normalize, sample_weight=sample_weight)
+            )
+        }

evaluate-0.4.2/metrics/accuracy/app.py

+import evaluate
+from evaluate.utils import launch_gradio_widget
+
+
+module = evaluate.load("accuracy")
+launch_gradio_widget(module)

evaluate-0.4.2/metrics/accuracy/requirements.txt

+git+https://github.com/huggingface/evaluate@{COMMIT_PLACEHOLDER}
+scikit-learn
\ No newline at end of file

evaluate-0.4.2/metrics/bertscore/README.md

+---
+title: BERT Score
+emoji: 🤗 
+colorFrom: blue
+colorTo: red
+sdk: gradio
+sdk_version: 3.19.1
+app_file: app.py
+pinned: false
+tags:
+- evaluate
+- metric
+description: >-
+  BERTScore leverages the pre-trained contextual embeddings from BERT and matches words in candidate and reference
+  sentences by cosine similarity.
+  It has been shown to correlate with human judgment on sentence-level and system-level evaluation.
+  Moreover, BERTScore computes precision, recall, and F1 measure, which can be useful for evaluating different language
+  generation tasks.
+  
+  See the project's README at https://github.com/Tiiiger/bert_score#readme for more information.
+---
+
+# Metric Card for BERT Score
+
+## Metric description
+
+BERTScore is an automatic evaluation metric for text generation that computes a similarity score for each token in the candidate sentence with each token in the reference sentence. It leverages the pre-trained contextual embeddings from [BERT](https://huggingface.co/bert-base-uncased) models and matches words in candidate and reference sentences by cosine similarity. 
+
+Moreover, BERTScore computes precision, recall, and F1 measure, which can be useful for evaluating different language generation tasks. 
+
+## How to use 
+
+BERTScore takes 3 mandatory arguments : `predictions` (a list of string of candidate sentences), `references` (a list of strings or list of list of strings of reference sentences) and either `lang` (a string of two letters indicating the language of the sentences, in [ISO 639-1 format](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes)) or `model_type` (a string specififying which model to use, according to the BERT specification). The default behavior of the metric is to use the suggested model for the target language when one is specified, otherwise to use the `model_type` indicated.
+
+```python
+from evaluate import load
+bertscore = load("bertscore")
+predictions = ["hello there", "general kenobi"]
+references = ["hello there", "general kenobi"]
+results = bertscore.compute(predictions=predictions, references=references, lang="en")
+```
+
+BERTScore also accepts multiple optional arguments: 
+
+
+`num_layers` (int): The layer of representation to use. The default is the number of layers tuned on WMT16 correlation data, which depends on the `model_type` used.
+
+`verbose` (bool): Turn on intermediate status update. The default value is `False`. 
+
+`idf` (bool or dict): Use idf weighting; can also be a precomputed idf_dict. 
+
+`device` (str): On which the contextual embedding model will be allocated on. If this argument is `None`, the model lives on `cuda:0` if cuda is available.
+
+`nthreads` (int): Number of threads used for computation. The default value is `4`. 
+
+`rescale_with_baseline` (bool): Rescale BERTScore with the  pre-computed baseline. The default value is `False`. 
+
+`batch_size` (int): BERTScore processing batch size, at least one of `model_type` or `lang`. `lang` needs to be specified when `rescale_with_baseline` is `True`.
+
+`baseline_path` (str): Customized baseline file.
+
+`use_fast_tokenizer` (bool): `use_fast` parameter passed to HF tokenizer. The default value is `False`. 
+
+
+## Output values
+
+BERTScore outputs a dictionary with the following values:
+
+`precision`: The [precision](https://huggingface.co/metrics/precision) for each sentence from the `predictions` + `references` lists, which ranges from 0.0 to 1.0. 
+
+`recall`: The [recall](https://huggingface.co/metrics/recall) for each sentence from the `predictions` + `references` lists, which ranges from 0.0 to 1.0.
+
+`f1`: The [F1 score](https://huggingface.co/metrics/f1) for each sentence from the `predictions` + `references` lists, which ranges from 0.0 to 1.0.
+ 
+`hashcode:` The hashcode of the library.
+
+
+### Values from popular papers
+The [original BERTScore paper](https://openreview.net/pdf?id=SkeHuCVFDr) reported average model selection accuracies (Hits@1) on WMT18 hybrid systems for different language pairs, which ranged from 0.004 for `en<->tr` to 0.824 for `en<->de`.
+
+For more recent model performance, see the [metric leaderboard](https://paperswithcode.com/paper/bertscore-evaluating-text-generation-with).
+
+## Examples 
+
+Maximal values with the `distilbert-base-uncased` model:
+
+```python
+from evaluate import load
+bertscore = load("bertscore")
+predictions = ["hello world", "general kenobi"]
+references = ["hello world", "general kenobi"]
+results = bertscore.compute(predictions=predictions, references=references, model_type="distilbert-base-uncased")
+print(results)
+{'precision': [1.0, 1.0], 'recall': [1.0, 1.0], 'f1': [1.0, 1.0], 'hashcode': 'distilbert-base-uncased_L5_no-idf_version=0.3.10(hug_trans=4.10.3)'}
+```
+
+Partial match with the `distilbert-base-uncased` model:
+
+```python
+from evaluate import load
+bertscore = load("bertscore")
+predictions = ["hello world", "general kenobi"]
+references = ["goodnight moon", "the sun is shining"]
+results = bertscore.compute(predictions=predictions, references=references, model_type="distilbert-base-uncased")
+print(results)
+{'precision': [0.7380737066268921, 0.5584042072296143], 'recall': [0.7380737066268921, 0.5889028906822205], 'f1': [0.7380737066268921, 0.5732481479644775], 'hashcode': 'bert-base-uncased_L5_no-idf_version=0.3.10(hug_trans=4.10.3)'}
+```
+
+## Limitations and bias
+
+The [original BERTScore paper](https://openreview.net/pdf?id=SkeHuCVFDr) showed that BERTScore correlates well with human judgment on sentence-level and system-level evaluation, but this depends on the model and language pair selected.
+
+Furthermore, not all languages are supported by the metric -- see the [BERTScore supported language list](https://github.com/google-research/bert/blob/master/multilingual.md#list-of-languages) for more information.
+
+Finally, calculating the BERTScore metric involves downloading the BERT model that is used to compute the score-- the default model for `en`, `roberta-large`, takes over 1.4GB of storage space and downloading it can take a significant amount of time depending on the speed of your internet connection. If this is an issue, choose a smaller model; for instance `distilbert-base-uncased` is 268MB. A full list of compatible models can be found [here](https://docs.google.com/spreadsheets/d/1RKOVpselB98Nnh_EOC4A2BYn8_201tmPODpNWu4w7xI/edit#gid=0).  
+
+
+## Citation
+
+```bibtex
+@inproceedings{bert-score,
+  title={BERTScore: Evaluating Text Generation with BERT},
+  author={Tianyi Zhang* and Varsha Kishore* and Felix Wu* and Kilian Q. Weinberger and Yoav Artzi},
+  booktitle={International Conference on Learning Representations},
+  year={2020},
+  url={https://openreview.net/forum?id=SkeHuCVFDr}
+}
+```
+    
+## Further References 
+
+- [BERTScore Project README](https://github.com/Tiiiger/bert_score#readme)
+- [BERTScore ICLR 2020 Poster Presentation](https://iclr.cc/virtual_2020/poster_SkeHuCVFDr.html)

evaluate-0.4.2/metrics/bertscore/app.py

+import evaluate
+from evaluate.utils import launch_gradio_widget
+
+
+module = evaluate.load("bertscore")
+launch_gradio_widget(module)

evaluate-0.4.2/metrics/bertscore/bertscore.py

+# Copyright 2020 The HuggingFace Evaluate Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+""" BERTScore metric. """
+
+import functools
+from contextlib import contextmanager
+
+import bert_score
+import datasets
+from packaging import version
+
+import evaluate
+
+
+@contextmanager
+def filter_logging_context():
+    def filter_log(record):
+        return False if "This IS expected if you are initializing" in record.msg else True
+
+    logger = datasets.utils.logging.get_logger("transformers.modeling_utils")
+    logger.addFilter(filter_log)
+    try:
+        yield
+    finally:
+        logger.removeFilter(filter_log)
+
+
+_CITATION = """\
+@inproceedings{bert-score,
+  title={BERTScore: Evaluating Text Generation with BERT},
+  author={Tianyi Zhang* and Varsha Kishore* and Felix Wu* and Kilian Q. Weinberger and Yoav Artzi},
+  booktitle={International Conference on Learning Representations},
+  year={2020},
+  url={https://openreview.net/forum?id=SkeHuCVFDr}
+}
+"""
+
+_DESCRIPTION = """\
+BERTScore leverages the pre-trained contextual embeddings from BERT and matches words in candidate and reference
+sentences by cosine similarity.
+It has been shown to correlate with human judgment on sentence-level and system-level evaluation.
+Moreover, BERTScore computes precision, recall, and F1 measure, which can be useful for evaluating different language
+generation tasks.
+
+See the project's README at https://github.com/Tiiiger/bert_score#readme for more information.
+"""
+
+_KWARGS_DESCRIPTION = """
+BERTScore Metrics with the hashcode from a source against one or more references.
+
+Args:
+    predictions (list of str): Prediction/candidate sentences.
+    references (list of str or list of list of str): Reference sentences.
+    lang (str): Language of the sentences; required (e.g. 'en').
+    model_type (str): Bert specification, default using the suggested
+        model for the target language; has to specify at least one of
+        `model_type` or `lang`.
+    num_layers (int): The layer of representation to use,
+        default using the number of layers tuned on WMT16 correlation data.
+    verbose (bool): Turn on intermediate status update.
+    idf (bool or dict): Use idf weighting; can also be a precomputed idf_dict.
+    device (str): On which the contextual embedding model will be allocated on.
+        If this argument is None, the model lives on cuda:0 if cuda is available.
+    nthreads (int): Number of threads.
+    batch_size (int): Bert score processing batch size,
+        at least one of `model_type` or `lang`. `lang` needs to be
+        specified when `rescale_with_baseline` is True.
+    rescale_with_baseline (bool): Rescale bertscore with pre-computed baseline.
+    baseline_path (str): Customized baseline file.
+    use_fast_tokenizer (bool): `use_fast` parameter passed to HF tokenizer. New in version 0.3.10.
+
+Returns:
+    precision: Precision.
+    recall: Recall.
+    f1: F1 score.
+    hashcode: Hashcode of the library.
+
+Examples:
+
+    >>> predictions = ["hello there", "general kenobi"]
+    >>> references = ["hello there", "general kenobi"]
+    >>> bertscore = evaluate.load("bertscore")
+    >>> results = bertscore.compute(predictions=predictions, references=references, lang="en")
+    >>> print([round(v, 2) for v in results["f1"]])
+    [1.0, 1.0]
+"""
+
+
+@evaluate.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
+class BERTScore(evaluate.Metric):
+    def _info(self):
+        return evaluate.MetricInfo(
+            description=_DESCRIPTION,
+            citation=_CITATION,
+            homepage="https://github.com/Tiiiger/bert_score",
+            inputs_description=_KWARGS_DESCRIPTION,
+            features=[
+                datasets.Features(
+                    {
+                        "predictions": datasets.Value("string", id="sequence"),
+                        "references": datasets.Sequence(datasets.Value("string", id="sequence"), id="references"),
+                    }
+                ),
+                datasets.Features(
+                    {
+                        "predictions": datasets.Value("string", id="sequence"),
+                        "references": datasets.Value("string", id="sequence"),
+                    }
+                ),
+            ],
+            codebase_urls=["https://github.com/Tiiiger/bert_score"],
+            reference_urls=[
+                "https://github.com/Tiiiger/bert_score",
+                "https://arxiv.org/abs/1904.09675",
+            ],
+        )
+
+    def _compute(
+        self,
+        predictions,
+        references,
+        lang=None,
+        model_type=None,
+        num_layers=None,
+        verbose=False,
+        idf=False,
+        device=None,
+        batch_size=64,
+        nthreads=4,
+        all_layers=False,
+        rescale_with_baseline=False,
+        baseline_path=None,
+        use_fast_tokenizer=False,
+    ):
+
+        if isinstance(references[0], str):
+            references = [[ref] for ref in references]
+
+        if idf:
+            idf_sents = [r for ref in references for r in ref]
+        else:
+            idf_sents = None
+
+        get_hash = bert_score.utils.get_hash
+        scorer = bert_score.BERTScorer
+
+        if version.parse(bert_score.__version__) >= version.parse("0.3.10"):
+            get_hash = functools.partial(get_hash, use_fast_tokenizer=use_fast_tokenizer)
+            scorer = functools.partial(scorer, use_fast_tokenizer=use_fast_tokenizer)
+        elif use_fast_tokenizer:
+            raise ImportWarning(
+                "To use a fast tokenizer, the module `bert-score>=0.3.10` is required, and the current version of "
+                "`bert-score` doesn't match this condition.\n"
+                'You can install it with `pip install "bert-score>=0.3.10"`.'
+            )
+
+        if model_type is None:
+            if lang is None:
+                raise ValueError(
+                    "Either 'lang' (e.g. 'en') or 'model_type' (e.g. 'microsoft/deberta-xlarge-mnli')"
+                    " must be specified"
+                )
+            model_type = bert_score.utils.lang2model[lang.lower()]
+
+        if num_layers is None:
+            num_layers = bert_score.utils.model2layers[model_type]
+
+        hashcode = get_hash(
+            model=model_type,
+            num_layers=num_layers,
+            idf=idf,
+            rescale_with_baseline=rescale_with_baseline,
+            use_custom_baseline=baseline_path is not None,
+        )
+
+        with filter_logging_context():
+            if not hasattr(self, "cached_bertscorer") or self.cached_bertscorer.hash != hashcode:
+                self.cached_bertscorer = scorer(
+                    model_type=model_type,
+                    num_layers=num_layers,
+                    batch_size=batch_size,
+                    nthreads=nthreads,
+                    all_layers=all_layers,
+                    idf=idf,
+                    idf_sents=idf_sents,
+                    device=device,
+                    lang=lang,
+                    rescale_with_baseline=rescale_with_baseline,
+                    baseline_path=baseline_path,
+                )
+
+        (P, R, F) = self.cached_bertscorer.score(
+            cands=predictions,
+            refs=references,
+            verbose=verbose,
+            batch_size=batch_size,
+        )
+        output_dict = {
+            "precision": P.tolist(),
+            "recall": R.tolist(),
+            "f1": F.tolist(),
+            "hashcode": hashcode,
+        }
+        return output_dict

evaluate-0.4.2/metrics/bertscore/requirements.txt

+git+https://github.com/huggingface/evaluate@{COMMIT_PLACEHOLDER}
+bert_score
\ No newline at end of file

evaluate-0.4.2/metrics/bleu/README.md

+---
+title: BLEU
+emoji: 🤗 
+colorFrom: blue
+colorTo: red
+sdk: gradio
+sdk_version: 3.19.1
+app_file: app.py
+pinned: false
+tags:
+- evaluate
+- metric
+description: >-
+  BLEU (Bilingual Evaluation Understudy) is an algorithm for evaluating the quality of text which has been machine-translated from one natural language to another.
+  Quality is considered to be the correspondence between a machine's output and that of a human: "the closer a machine translation is to a professional human translation, the better it is"
+  – this is the central idea behind BLEU. BLEU was one of the first metrics to claim a high correlation with human judgements of quality, and remains one of the most popular automated and inexpensive metrics.
+
+  Scores are calculated for individual translated segments—generally sentences—by comparing them with a set of good quality reference translations.
+  Those scores are then averaged over the whole corpus to reach an estimate of the translation's overall quality.
+  Neither intelligibility nor grammatical correctness are not taken into account.
+---
+
+# Metric Card for BLEU
+
+
+## Metric Description
+BLEU (Bilingual Evaluation Understudy) is an algorithm for evaluating the quality of text which has been machine-translated from one natural language to another. Quality is considered to be the correspondence between a machine's output and that of a human: "the closer a machine translation is to a professional human translation, the better it is" – this is the central idea behind BLEU. BLEU was one of the first metrics to claim a high correlation with human judgements of quality, and remains one of the most popular automated and inexpensive metrics.
+
+Scores are calculated for individual translated segments—generally sentences—by comparing them with a set of good quality reference translations. Those scores are then averaged over the whole corpus to reach an estimate of the translation's overall quality. Neither intelligibility nor grammatical correctness are not taken into account.
+
+## Intended Uses
+BLEU and BLEU-derived metrics are most often used for machine translation.
+
+## How to Use
+
+This metric takes as input a list of predicted sentences and a list of lists of reference sentences (since each predicted sentence can have multiple references):
+
+```python
+>>> predictions = ["hello there general kenobi", "foo bar foobar"]
+>>> references = [
+...     ["hello there general kenobi", "hello there !"],
+...     ["foo bar foobar"]
+... ]
+>>> bleu = evaluate.load("bleu")
+>>> results = bleu.compute(predictions=predictions, references=references)
+>>> print(results)
+{'bleu': 1.0, 'precisions': [1.0, 1.0, 1.0, 1.0], 'brevity_penalty': 1.0, 'length_ratio': 1.1666666666666667, 'translation_length': 7, 'reference_length': 6}
+```
+
+### Inputs
+- **predictions** (`list` of `str`s): Translations to score.
+- **references** (`list` of `list`s of `str`s): references for each translation.
+- ** tokenizer** : approach used for standardizing `predictions` and `references`.
+    The default tokenizer is `tokenizer_13a`, a relatively minimal tokenization approach that is however equivalent to `mteval-v13a`, used by WMT.
+    This can be replaced by another tokenizer from a source such as [SacreBLEU](https://github.com/mjpost/sacrebleu/tree/master/sacrebleu/tokenizers).
+
+The default tokenizer is based on whitespace and regexes. It can be replaced by any function that takes a string as input and returns a list of tokens as output. E.g. `word_tokenize()` from [NLTK](https://www.nltk.org/api/nltk.tokenize.html) or pretrained tokenizers from the [Tokenizers library](https://huggingface.co/docs/tokenizers/index).
+- **max_order** (`int`): Maximum n-gram order to use when computing BLEU score. Defaults to `4`.
+- **smooth** (`boolean`): Whether or not to apply Lin et al. 2004 smoothing. Defaults to `False`.
+
+### Output Values
+- **bleu** (`float`): bleu score
+- **precisions** (`list` of `float`s): geometric mean of n-gram precisions,
+- **brevity_penalty** (`float`): brevity penalty,
+- **length_ratio** (`float`): ratio of lengths,
+- **translation_length** (`int`): translation_length,
+- **reference_length** (`int`): reference_length
+
+Output Example:
+```python
+{'bleu': 1.0, 'precisions': [1.0, 1.0, 1.0, 1.0], 'brevity_penalty': 1.0, 'length_ratio': 1.1666666666666667, 'translation_length': 7, 'reference_length': 6}
+```
+
+BLEU's output is always a number between 0 and 1. This value indicates how similar the candidate text is to the reference texts, with values closer to 1 representing more similar texts. Few human translations will attain a score of 1, since this would indicate that the candidate is identical to one of the reference translations. For this reason, it is not necessary to attain a score of 1. Because there are more opportunities to match, adding additional reference translations will increase the BLEU score.
+
+#### Values from Popular Papers
+The [original BLEU paper](https://aclanthology.org/P02-1040/) (Papineni et al. 2002) compares BLEU scores of five different models on the same 500-sentence corpus. These scores ranged from 0.0527 to 0.2571.
+
+The [Attention is All you Need paper](https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf) (Vaswani et al. 2017) got a BLEU score of 0.284 on the WMT 2014 English-to-German translation task, and 0.41 on the WMT 2014 English-to-French translation task.
+
+### Examples
+
+Example where each prediction has 1 reference:
+```python
+>>> predictions = ["hello there general kenobi","foo bar foobar"]
+>>> references = [
+...     ["hello there general kenobi"],
+...     ["foo bar foobar"]
+... ]
+>>> bleu = evaluate.load("bleu")
+>>> results = bleu.compute(predictions=predictions, references=references)
+>>> print(results)
+{'bleu': 1.0, 'precisions': [1.0, 1.0, 1.0, 1.0], 'brevity_penalty': 1.0, 'length_ratio': 1.0, 'translation_length': 7, 'reference_length': 7}
+```
+
+Example where the second prediction has 2 references:
+```python
+>>> predictions = [
+...     ["hello there general kenobi",
+...     ["foo bar foobar"]
+... ]
+>>> references = [
+...     [["hello there general kenobi"], ["hello there!"]],
+...     [["foo bar foobar"]]
+... ]
+>>> bleu = evaluate.load("bleu")
+>>> results = bleu.compute(predictions=predictions, references=references)
+>>> print(results)
+{'bleu': 1.0, 'precisions': [1.0, 1.0, 1.0, 1.0], 'brevity_penalty': 1.0, 'length_ratio': 1.1666666666666667, 'translation_length': 7, 'reference_length': 6}
+```
+
+Example with the word tokenizer from NLTK:
+```python
+>>> bleu = evaluate.load("bleu")
+>>> from nltk.tokenize import word_tokenize
+>>> predictions = [
+...     ["hello there general kenobi",
+...     ["foo bar foobar"]
+... ]
+>>> references = [
+...     [["hello there general kenobi"], ["hello there!"]],
+...     [["foo bar foobar"]]
+... ]
+>>> results = bleu.compute(predictions=predictions, references=references, tokenizer=word_tokenize)
+>>> print(results)
+{'bleu': 1.0, 'precisions': [1.0, 1.0, 1.0, 1.0], 'brevity_penalty': 1.0, 'length_ratio': 1.1666666666666667, 'translation_length': 7, 'reference_length': 6}
+```
+
+## Limitations and Bias
+This metric has multiple known limitations:
+- BLEU compares overlap in tokens from the predictions and references, instead of comparing meaning. This can lead to discrepancies between BLEU scores and human ratings.
+- Shorter predicted translations achieve higher scores than longer ones, simply due to how the score is calculated. A brevity penalty is introduced to attempt to counteract this.
+- BLEU scores are not comparable across different datasets, nor are they comparable across different languages.
+- BLEU scores can vary greatly depending on which parameters are used to generate the scores, especially when different tokenization and normalization techniques are used. It is therefore not possible to compare BLEU scores generated using different parameters, or when these parameters are unknown. For more discussion around this topic, see the following [issue](https://github.com/huggingface/datasets/issues/137).
+
+## Citation
+```bibtex
+@INPROCEEDINGS{Papineni02bleu:a,
+    author = {Kishore Papineni and Salim Roukos and Todd Ward and Wei-jing Zhu},
+    title = {BLEU: a Method for Automatic Evaluation of Machine Translation},
+    booktitle = {},
+    year = {2002},
+    pages = {311--318}
+}
+@inproceedings{lin-och-2004-orange,
+    title = "{ORANGE}: a Method for Evaluating Automatic Evaluation Metrics for Machine Translation",
+    author = "Lin, Chin-Yew  and
+      Och, Franz Josef",
+    booktitle = "{COLING} 2004: Proceedings of the 20th International Conference on Computational Linguistics",
+    month = "aug 23{--}aug 27",
+    year = "2004",
+    address = "Geneva, Switzerland",
+    publisher = "COLING",
+    url = "https://www.aclweb.org/anthology/C04-1072",
+    pages = "501--507",
+}
+```
+
+## Further References
+- This Hugging Face implementation uses [this Tensorflow implementation](https://github.com/tensorflow/nmt/blob/master/nmt/scripts/bleu.py)

evaluate-0.4.2/metrics/bleu/app.py

+import evaluate
+from evaluate.utils import launch_gradio_widget
+
+
+module = evaluate.load("bleu")
+launch_gradio_widget(module)

evaluate-0.4.2/metrics/bleu/bleu.py

+# Copyright 2020 The HuggingFace Evaluate Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+""" BLEU metric. """
+
+import datasets
+
+import evaluate
+
+from .nmt_bleu import compute_bleu  # From: https://github.com/tensorflow/nmt/blob/master/nmt/scripts/bleu.py
+from .tokenizer_13a import Tokenizer13a
+
+
+_CITATION = """\
+@INPROCEEDINGS{Papineni02bleu:a,
+    author = {Kishore Papineni and Salim Roukos and Todd Ward and Wei-jing Zhu},
+    title = {BLEU: a Method for Automatic Evaluation of Machine Translation},
+    booktitle = {},
+    year = {2002},
+    pages = {311--318}
+}
+@inproceedings{lin-och-2004-orange,
+    title = "{ORANGE}: a Method for Evaluating Automatic Evaluation Metrics for Machine Translation",
+    author = "Lin, Chin-Yew  and
+      Och, Franz Josef",
+    booktitle = "{COLING} 2004: Proceedings of the 20th International Conference on Computational Linguistics",
+    month = "aug 23{--}aug 27",
+    year = "2004",
+    address = "Geneva, Switzerland",
+    publisher = "COLING",
+    url = "https://www.aclweb.org/anthology/C04-1072",
+    pages = "501--507",
+}
+"""
+
+_DESCRIPTION = """\
+BLEU (Bilingual Evaluation Understudy) is an algorithm for evaluating the quality of text which has been machine-translated from one natural language to another.
+Quality is considered to be the correspondence between a machine's output and that of a human: "the closer a machine translation is to a professional human translation, the better it is"
+– this is the central idea behind BLEU. BLEU was one of the first metrics to claim a high correlation with human judgements of quality, and remains one of the most popular automated and inexpensive metrics.
+
+Scores are calculated for individual translated segments—generally sentences—by comparing them with a set of good quality reference translations.
+Those scores are then averaged over the whole corpus to reach an estimate of the translation's overall quality.
+Neither intelligibility nor grammatical correctness are not taken into account.
+"""
+
+_KWARGS_DESCRIPTION = """
+Computes BLEU score of translated segments against one or more references.
+Args:
+    predictions: list of translations to score.
+    references: list of lists of or just a list of references for each translation.
+    tokenizer : approach used for tokenizing `predictions` and `references`.
+        The default tokenizer is `tokenizer_13a`, a minimal tokenization approach that is equivalent to `mteval-v13a`, used by WMT.
+        This can be replaced by any function that takes a string as input and returns a list of tokens as output.
+    max_order: Maximum n-gram order to use when computing BLEU score.
+    smooth: Whether or not to apply Lin et al. 2004 smoothing.
+Returns:
+    'bleu': bleu score,
+    'precisions': geometric mean of n-gram precisions,
+    'brevity_penalty': brevity penalty,
+    'length_ratio': ratio of lengths,
+    'translation_length': translation_length,
+    'reference_length': reference_length
+Examples:
+
+    >>> predictions = ["hello there general kenobi", "foo bar foobar"]
+    >>> references = [
+    ...     ["hello there general kenobi", "hello there!"],
+    ...     ["foo bar foobar"]
+    ... ]
+    >>> bleu = evaluate.load("bleu")
+    >>> results = bleu.compute(predictions=predictions, references=references)
+    >>> print(results["bleu"])
+    1.0
+"""
+
+
+@evaluate.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
+class Bleu(evaluate.Metric):
+    def _info(self):
+        return evaluate.MetricInfo(
+            description=_DESCRIPTION,
+            citation=_CITATION,
+            inputs_description=_KWARGS_DESCRIPTION,
+            features=[
+                datasets.Features(
+                    {
+                        "predictions": datasets.Value("string", id="sequence"),
+                        "references": datasets.Sequence(datasets.Value("string", id="sequence"), id="references"),
+                    }
+                ),
+                datasets.Features(
+                    {
+                        "predictions": datasets.Value("string", id="sequence"),
+                        "references": datasets.Value("string", id="sequence"),
+                    }
+                ),
+            ],
+            codebase_urls=["https://github.com/tensorflow/nmt/blob/master/nmt/scripts/bleu.py"],
+            reference_urls=[
+                "https://en.wikipedia.org/wiki/BLEU",
+                "https://towardsdatascience.com/evaluating-text-output-in-nlp-bleu-at-your-own-risk-e8609665a213",
+            ],
+        )
+
+    def _compute(self, predictions, references, tokenizer=Tokenizer13a(), max_order=4, smooth=False):
+        # if only one reference is provided make sure we still use list of lists
+        if isinstance(references[0], str):
+            references = [[ref] for ref in references]
+
+        references = [[tokenizer(r) for r in ref] for ref in references]
+        predictions = [tokenizer(p) for p in predictions]
+        score = compute_bleu(
+            reference_corpus=references, translation_corpus=predictions, max_order=max_order, smooth=smooth
+        )
+        (bleu, precisions, bp, ratio, translation_length, reference_length) = score
+        return {
+            "bleu": bleu,
+            "precisions": precisions,
+            "brevity_penalty": bp,
+            "length_ratio": ratio,
+            "translation_length": translation_length,
+            "reference_length": reference_length,
+        }

evaluate-0.4.2/metrics/bleu/requirements.txt

+git+https://github.com/huggingface/evaluate@{COMMIT_PLACEHOLDER}
\ No newline at end of file

evaluate-0.4.2/metrics/bleu/tokenizer_13a.py

+# Source: https://github.com/mjpost/sacrebleu/blob/master/sacrebleu/tokenizers/tokenizer_13a.py
+# Copyright 2020 SacreBLEU Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import re
+from functools import lru_cache
+
+
+class BaseTokenizer:
+    """A base dummy tokenizer to derive from."""
+
+    def signature(self):
+        """
+        Returns a signature for the tokenizer.
+        :return: signature string
+        """
+        return "none"
+
+    def __call__(self, line):
+        """
+        Tokenizes an input line with the tokenizer.
+        :param line: a segment to tokenize
+        :return: the tokenized line
+        """
+        return line
+
+
+class TokenizerRegexp(BaseTokenizer):
+    def signature(self):
+        return "re"
+
+    def __init__(self):
+        self._re = [
+            # language-dependent part (assuming Western languages)
+            (re.compile(r"([\{-\~\[-\` -\&\(-\+\:-\@\/])"), r" \1 "),
+            # tokenize period and comma unless preceded by a digit
+            (re.compile(r"([^0-9])([\.,])"), r"\1 \2 "),
+            # tokenize period and comma unless followed by a digit
+            (re.compile(r"([\.,])([^0-9])"), r" \1 \2"),
+            # tokenize dash when preceded by a digit
+            (re.compile(r"([0-9])(-)"), r"\1 \2 "),
+            # one space only between words
+            # NOTE: Doing this in Python (below) is faster
+            # (re.compile(r'\s+'), r' '),
+        ]
+
+    @lru_cache(maxsize=2**16)
+    def __call__(self, line):
+        """Common post-processing tokenizer for `13a` and `zh` tokenizers.
+        :param line: a segment to tokenize
+        :return: the tokenized line
+        """
+        for (_re, repl) in self._re:
+            line = _re.sub(repl, line)
+
+        # no leading or trailing spaces, single space within words
+        # return ' '.join(line.split())
+        # This line is changed with regards to the original tokenizer (seen above) to return individual words
+        return line.split()
+
+
+class Tokenizer13a(BaseTokenizer):
+    def signature(self):
+        return "13a"
+
+    def __init__(self):
+        self._post_tokenizer = TokenizerRegexp()
+
+    @lru_cache(maxsize=2**16)
+    def __call__(self, line):
+        """Tokenizes an input line using a relatively minimal tokenization
+        that is however equivalent to mteval-v13a, used by WMT.
+
+        :param line: a segment to tokenize
+        :return: the tokenized line
+        """
+
+        # language-independent part:
+        line = line.replace("<skipped>", "")
+        line = line.replace("-\n", "")
+        line = line.replace("\n", " ")
+
+        if "&" in line:
+            line = line.replace("&quot;", '"')
+            line = line.replace("&amp;", "&")
+            line = line.replace("&lt;", "<")
+            line = line.replace("&gt;", ">")
+
+        return self._post_tokenizer(f" {line} ")

evaluate-0.4.2/metrics/bleurt/README.md

+---
+title: BLEURT
+emoji: 🤗 
+colorFrom: blue
+colorTo: red
+sdk: gradio
+sdk_version: 3.19.1
+app_file: app.py
+pinned: false
+tags:
+- evaluate
+- metric
+description: >-
+  BLEURT a learnt evaluation metric for Natural Language Generation. It is built using multiple phases of transfer learning starting from a pretrained BERT model (Devlin et al. 2018)
+  and then employing another pre-training phrase using synthetic data. Finally it is trained on WMT human annotations. You may run BLEURT out-of-the-box or fine-tune
+  it for your specific application (the latter is expected to perform better).
+
+  See the project's README at https://github.com/google-research/bleurt#readme for more information.
+---
+
+# Metric Card for BLEURT
+
+
+## Metric Description
+BLEURT is a learned evaluation metric for Natural Language Generation. It is built using multiple phases of transfer learning starting from a pretrained BERT model [Devlin et al. 2018](https://arxiv.org/abs/1810.04805), employing another pre-training phrase using synthetic data, and finally trained on WMT human annotations. 
+
+It is possible to run BLEURT out-of-the-box or fine-tune it for your specific application (the latter is expected to perform better).
+See the project's [README](https://github.com/google-research/bleurt#readme) for more information.
+
+## Intended Uses
+BLEURT is intended to be used for evaluating text produced by language models. 
+
+## How to Use
+
+This metric takes as input lists of predicted sentences and reference sentences:
+
+```python
+>>> predictions = ["hello there", "general kenobi"]
+>>> references = ["hello there", "general kenobi"]
+>>> bleurt = load("bleurt", module_type="metric")
+>>> results = bleurt.compute(predictions=predictions, references=references)
+```
+
+### Inputs
+- **predictions** (`list` of `str`s): List of generated sentences to score.
+- **references** (`list` of `str`s): List of references to compare to.
+- **checkpoint** (`str`): BLEURT checkpoint. Will default to `BLEURT-tiny` if not specified. Other models that can be chosen are: `"bleurt-tiny-128"`, `"bleurt-tiny-512"`, `"bleurt-base-128"`, `"bleurt-base-512"`, `"bleurt-large-128"`, `"bleurt-large-512"`, `"BLEURT-20-D3"`, `"BLEURT-20-D6"`, `"BLEURT-20-D12"` and `"BLEURT-20"`. 
+
+### Output Values
+- **scores** : a `list` of scores, one per prediction. 
+
+Output Example:
+```python
+{'scores': [1.0295498371124268, 1.0445425510406494]}
+
+```
+
+BLEURT's output is always a number between 0 and (approximately 1). This value indicates how similar the generated text is to the reference texts, with values closer to 1 representing more similar texts. 
+
+#### Values from Popular Papers
+
+The [original BLEURT paper](https://arxiv.org/pdf/2004.04696.pdf) reported that the metric is better correlated with human judgment compared to similar metrics such as BERT and BERTscore.
+
+BLEURT is used to compare models across different asks (e.g. (Table to text generation)[https://paperswithcode.com/sota/table-to-text-generation-on-dart?metric=BLEURT]).
+
+### Examples
+
+Example with the default model:
+```python
+>>> predictions = ["hello there", "general kenobi"]
+>>> references = ["hello there", "general kenobi"]
+>>> bleurt = load("bleurt", module_type="metric")
+>>> results = bleurt.compute(predictions=predictions, references=references)
+>>> print(results)
+{'scores': [1.0295498371124268, 1.0445425510406494]}
+```
+
+Example with the `"bleurt-base-128"` model checkpoint:
+```python
+>>> predictions = ["hello there", "general kenobi"]
+>>> references = ["hello there", "general kenobi"]
+>>> bleurt = load("bleurt", module_type="metric", checkpoint="bleurt-base-128")
+>>> results = bleurt.compute(predictions=predictions, references=references)
+>>> print(results)
+{'scores': [1.0295498371124268, 1.0445425510406494]}
+```
+
+## Limitations and Bias
+The [original BLEURT paper](https://arxiv.org/pdf/2004.04696.pdf) showed that BLEURT correlates well with human judgment, but this depends on the model and language pair selected.
+
+Furthermore, currently BLEURT only supports English-language scoring, given that it leverages models trained on English corpora. It may also reflect, to a certain extent, biases and correlations that were present in the model training data. 
+
+Finally, calculating the BLEURT metric involves downloading the BLEURT model that is used to compute the score, which can take a significant amount of time depending on the model chosen. Starting with the default model, `bleurt-tiny`, and testing out larger models if necessary can be a useful approach if memory or internet speed is an issue.
+
+
+## Citation
+```bibtex
+@inproceedings{bleurt,
+  title={BLEURT: Learning Robust Metrics for Text Generation},
+  author={Thibault Sellam and Dipanjan Das and Ankur P. Parikh},
+  booktitle={ACL},
+  year={2020},
+  url={https://arxiv.org/abs/2004.04696}
+}
+```
+
+## Further References
+- The original [BLEURT GitHub repo](https://github.com/google-research/bleurt/)

evaluate-0.4.2/metrics/bleurt/app.py

+import sys
+
+import evaluate
+from evaluate.utils import launch_gradio_widget
+
+
+sys.path = [p for p in sys.path if p != "/home/user/app"]
+module = evaluate.load("bleurt")
+sys.path = ["/home/user/app"] + sys.path
+
+launch_gradio_widget(module)

evaluate-0.4.2/metrics/bleurt/bleurt.py

+# Copyright 2020 The HuggingFace Evaluate Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+""" BLEURT metric. """
+
+import os
+
+import datasets
+from bleurt import score  # From: git+https://github.com/google-research/bleurt.git
+
+import evaluate
+
+
+logger = evaluate.logging.get_logger(__name__)
+
+
+_CITATION = """\
+@inproceedings{bleurt,
+  title={BLEURT: Learning Robust Metrics for Text Generation},
+  author={Thibault Sellam and Dipanjan Das and Ankur P. Parikh},
+  booktitle={ACL},
+  year={2020},
+  url={https://arxiv.org/abs/2004.04696}
+}
+"""
+
+_DESCRIPTION = """\
+BLEURT a learnt evaluation metric for Natural Language Generation. It is built using multiple phases of transfer learning starting from a pretrained BERT model (Devlin et al. 2018)
+and then employing another pre-training phrase using synthetic data. Finally it is trained on WMT human annotations. You may run BLEURT out-of-the-box or fine-tune
+it for your specific application (the latter is expected to perform better).
+
+See the project's README at https://github.com/google-research/bleurt#readme for more information.
+"""
+
+_KWARGS_DESCRIPTION = """
+BLEURT score.
+
+Args:
+    `predictions` (list of str): prediction/candidate sentences
+    `references` (list of str): reference sentences
+    `checkpoint` BLEURT checkpoint. Will default to BLEURT-tiny if None.
+
+Returns:
+    'scores': List of scores.
+Examples:
+
+    >>> predictions = ["hello there", "general kenobi"]
+    >>> references = ["hello there", "general kenobi"]
+    >>> bleurt = evaluate.load("bleurt")
+    >>> results = bleurt.compute(predictions=predictions, references=references)
+    >>> print([round(v, 2) for v in results["scores"]])
+    [1.03, 1.04]
+"""
+
+CHECKPOINT_URLS = {
+    "bleurt-tiny-128": "https://storage.googleapis.com/bleurt-oss/bleurt-tiny-128.zip",
+    "bleurt-tiny-512": "https://storage.googleapis.com/bleurt-oss/bleurt-tiny-512.zip",
+    "bleurt-base-128": "https://storage.googleapis.com/bleurt-oss/bleurt-base-128.zip",
+    "bleurt-base-512": "https://storage.googleapis.com/bleurt-oss/bleurt-base-512.zip",
+    "bleurt-large-128": "https://storage.googleapis.com/bleurt-oss/bleurt-large-128.zip",
+    "bleurt-large-512": "https://storage.googleapis.com/bleurt-oss/bleurt-large-512.zip",
+    "BLEURT-20-D3": "https://storage.googleapis.com/bleurt-oss-21/BLEURT-20-D3.zip",
+    "BLEURT-20-D6": "https://storage.googleapis.com/bleurt-oss-21/BLEURT-20-D6.zip",
+    "BLEURT-20-D12": "https://storage.googleapis.com/bleurt-oss-21/BLEURT-20-D12.zip",
+    "BLEURT-20": "https://storage.googleapis.com/bleurt-oss-21/BLEURT-20.zip",
+}
+
+
+@evaluate.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
+class BLEURT(evaluate.Metric):
+    def _info(self):
+
+        return evaluate.MetricInfo(
+            description=_DESCRIPTION,
+            citation=_CITATION,
+            homepage="https://github.com/google-research/bleurt",
+            inputs_description=_KWARGS_DESCRIPTION,
+            features=datasets.Features(
+                {
+                    "predictions": datasets.Value("string", id="sequence"),
+                    "references": datasets.Value("string", id="sequence"),
+                }
+            ),
+            codebase_urls=["https://github.com/google-research/bleurt"],
+            reference_urls=["https://github.com/google-research/bleurt", "https://arxiv.org/abs/2004.04696"],
+        )
+
+    def _download_and_prepare(self, dl_manager):
+
+        # check that config name specifies a valid BLEURT model
+        if self.config_name == "default":
+            logger.warning(
+                "Using default BLEURT-Base checkpoint for sequence maximum length 128. "
+                "You can use a bigger model for better results with e.g.: evaluate.load('bleurt', 'bleurt-large-512')."
+            )
+            self.config_name = "bleurt-base-128"
+
+        if self.config_name.lower() in CHECKPOINT_URLS:
+            checkpoint_name = self.config_name.lower()
+
+        elif self.config_name.upper() in CHECKPOINT_URLS:
+            checkpoint_name = self.config_name.upper()
+
+        else:
+            raise KeyError(
+                f"{self.config_name} model not found. You should supply the name of a model checkpoint for bleurt in {CHECKPOINT_URLS.keys()}"
+            )
+
+        # download the model checkpoint specified by self.config_name and set up the scorer
+        model_path = dl_manager.download_and_extract(CHECKPOINT_URLS[checkpoint_name])
+        self.scorer = score.BleurtScorer(os.path.join(model_path, checkpoint_name))
+
+    def _compute(self, predictions, references):
+        scores = self.scorer.score(references=references, candidates=predictions)
+        return {"scores": scores}

evaluate-0.4.2/metrics/bleurt/requirements.txt

+git+https://github.com/huggingface/evaluate@{COMMIT_PLACEHOLDER}
+git+https://github.com/google-research/bleurt.git
\ No newline at end of file

evaluate-0.4.2/metrics/brier_score/README.md

+---
+title: Brier Score
+emoji: 🤗 
+colorFrom: blue
+colorTo: red
+sdk: gradio
+sdk_version: 3.19.1
+app_file: app.py
+pinned: false
+tags:
+- evaluate
+- metric
+description: >-
+  The Brier score is a measure of the error between two probability distributions.
+---
+
+# Metric Card for Brier Score
+
+
+## Metric Description
+Brier score is a type of evaluation metric for classification tasks, where you predict outcomes such as win/lose, spam/ham, click/no-click etc.
+`BrierScore = 1/N * sum( (p_i - o_i)^2 )`
+
+Where `p_i` is the prediction probability of occurrence of the event, and the term `o_i` is equal to 1 if the event occurred and 0 if not. Which means: the lower the value of this score, the better the prediction. 
+## How to Use
+
+At minimum, this metric requires predictions and references as inputs.
+
+```python
+>>> brier_score = evaluate.load("brier_score")
+>>> predictions = np.array([0, 0, 1, 1])
+>>> references = np.array([0.1, 0.9, 0.8, 0.3])
+>>> results = brier_score.compute(predictions=predictions, references=references)
+```
+
+### Inputs
+
+Mandatory inputs: 
+- `predictions`: numeric array-like of shape (`n_samples,`) or (`n_samples`, `n_outputs`), representing the estimated target values.
+
+- `references`: numeric array-like of shape (`n_samples,`) or (`n_samples`, `n_outputs`), representing the ground truth (correct) target values.
+
+Optional arguments:
+- `sample_weight`: numeric array-like of shape (`n_samples,`) representing sample weights. The default is `None`.
+- `pos_label`: the label of the positive class. The default is `1`.
+        
+
+### Output Values
+This metric returns a dictionary with the following keys:
+- `brier_score (float)`: the computed Brier score.
+
+
+Output Example(s):
+```python
+{'brier_score': 0.5}
+```
+
+#### Values from Popular Papers
+
+
+### Examples
+```python
+>>> brier_score = evaluate.load("brier_score")
+>>> predictions = np.array([0, 0, 1, 1])
+>>> references = np.array([0.1, 0.9, 0.8, 0.3])
+>>> results = brier_score.compute(predictions=predictions, references=references)
+>>> print(results)
+{'brier_score': 0.3375}
+```
+Example with `y_true` contains string, an error will be raised and `pos_label` should be explicitly specified.
+```python
+>>> brier_score_metric = evaluate.load("brier_score")
+>>> predictions =  np.array(["spam", "ham", "ham", "spam"])
+>>> references = np.array([0.1, 0.9, 0.8, 0.3])
+>>> results = brier_score.compute(predictions, references, pos_label="ham")
+>>> print(results) 
+{'brier_score': 0.0374}
+```
+## Limitations and Bias
+The [brier_score](https://huggingface.co/metrics/brier_score) is appropriate for binary and categorical outcomes that can be structured as true or false, but it is inappropriate for ordinal variables which can take on three or more values.
+## Citation(s)
+```bibtex
+@article{scikit-learn,
+  title={Scikit-learn: Machine Learning in {P}ython},
+  author={Pedregosa, F. and Varoquaux, G. and Gramfort, A. and Michel, V.
+         and Thirion, B. and Grisel, O. and Blondel, M. and Prettenhofer, P.
+         and Weiss, R. and Dubourg, V. and Vanderplas, J. and Passos, A. and
+         Cournapeau, D. and Brucher, M. and Perrot, M. and Duchesnay, E.},
+  journal={Journal of Machine Learning Research},
+  volume={12},
+  pages={2825--2830},
+  year={2011}
+}
+
+@Article{brier1950verification,
+  title={Verification of forecasts expressed in terms of probability},
+  author={Brier, Glenn W and others},
+  journal={Monthly weather review},
+  volume={78},
+  number={1},
+  pages={1--3},
+  year={1950}
+}
+```
+## Further References
+- [Brier Score - Wikipedia](https://en.wikipedia.org/wiki/Brier_score)
\ No newline at end of file