Duplicate from evaluate-metric/bertscore

Co-authored-by: Mamta Narang <[email protected]>

.gitattributes

@@ -0,0 +1,27 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zstandard filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,136 @@
+---
+title: BERT Score
+emoji: 🤗
+colorFrom: blue
+colorTo: red
+sdk: gradio
+sdk_version: 3.0.2
+app_file: app.py
+pinned: false
+tags:
+- evaluate
+- metric
+description: >-
+  BERTScore leverages the pre-trained contextual embeddings from BERT and
+  matches words in candidate and reference sentences by cosine similarity. It
+  has been shown to correlate with human judgment on sentence-level and
+  system-level evaluation. Moreover, BERTScore computes precision, recall, and
+  F1 measure, which can be useful for evaluating different language generation
+  tasks.
+
+  See the project's README at https://github.com/Tiiiger/bert_score#readme for
+  more information.
+duplicated_from: evaluate-metric/bertscore
+---
+
+# Metric Card for BERT Score
+
+## Metric description
+
+BERTScore is an automatic evaluation metric for text generation that computes a similarity score for each token in the candidate sentence with each token in the reference sentence. It leverages the pre-trained contextual embeddings from [BERT](https://huggingface.co/bert-base-uncased) models and matches words in candidate and reference sentences by cosine similarity. 
+
+Moreover, BERTScore computes precision, recall, and F1 measure, which can be useful for evaluating different language generation tasks. 
+
+## How to use 
+
+BERTScore takes 3 mandatory arguments : `predictions` (a list of string of candidate sentences), `references` (a list of strings or list of list of strings of reference sentences) and either `lang` (a string of two letters indicating the language of the sentences, in [ISO 639-1 format](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes)) or `model_type` (a string specififying which model to use, according to the BERT specification). The default behavior of the metric is to use the suggested model for the target language when one is specified, otherwise to use the `model_type` indicated.
+
+```python
+from evaluate import load
+bertscore = load("bertscore")
+predictions = ["hello there", "general kenobi"]
+references = ["hello there", "general kenobi"]
+results = bertscore.compute(predictions=predictions, references=references, lang="en")
+```
+
+BERTScore also accepts multiple optional arguments: 
+
+
+`num_layers` (int): The layer of representation to use. The default is the number of layers tuned on WMT16 correlation data, which depends on the `model_type` used.
+
+`verbose` (bool): Turn on intermediate status update. The default value is `False`. 
+
+`idf` (bool or dict): Use idf weighting; can also be a precomputed idf_dict. 
+
+`device` (str): On which the contextual embedding model will be allocated on. If this argument is `None`, the model lives on `cuda:0` if cuda is available.
+
+`nthreads` (int): Number of threads used for computation. The default value is `4`. 
+
+`rescale_with_baseline` (bool): Rescale BERTScore with the  pre-computed baseline. The default value is `False`. 
+
+`batch_size` (int): BERTScore processing batch size, at least one of `model_type` or `lang`. `lang` needs to be specified when `rescale_with_baseline` is `True`.
+
+`baseline_path` (str): Customized baseline file.
+
+`use_fast_tokenizer` (bool): `use_fast` parameter passed to HF tokenizer. The default value is `False`. 
+
+
+## Output values
+
+BERTScore outputs a dictionary with the following values:
+
+`precision`: The [precision](https://huggingface.co/metrics/precision) for each sentence from the `predictions` + `references` lists, which ranges from 0.0 to 1.0. 
+
+`recall`: The [recall](https://huggingface.co/metrics/recall) for each sentence from the `predictions` + `references` lists, which ranges from 0.0 to 1.0.
+
+`f1`: The [F1 score](https://huggingface.co/metrics/f1) for each sentence from the `predictions` + `references` lists, which ranges from 0.0 to 1.0.
+ 
+`hashcode:` The hashcode of the library.
+
+
+### Values from popular papers
+The [original BERTScore paper](https://openreview.net/pdf?id=SkeHuCVFDr) reported average model selection accuracies (Hits@1) on WMT18 hybrid systems for different language pairs, which ranged from 0.004 for `en<->tr` to 0.824 for `en<->de`.
+
+For more recent model performance, see the [metric leaderboard](https://paperswithcode.com/paper/bertscore-evaluating-text-generation-with).
+
+## Examples 
+
+Maximal values with the `distilbert-base-uncased` model:
+
+```python
+from evaluate import load
+bertscore = load("bertscore")
+predictions = ["hello world", "general kenobi"]
+references = ["hello world", "general kenobi"]
+results = bertscore.compute(predictions=predictions, references=references, model_type="distilbert-base-uncased")
+print(results)
+{'precision': [1.0, 1.0], 'recall': [1.0, 1.0], 'f1': [1.0, 1.0], 'hashcode': 'distilbert-base-uncased_L5_no-idf_version=0.3.10(hug_trans=4.10.3)'}
+```
+
+Partial match with the `bert-base-uncased` model:
+
+```python
+from evaluate import load
+bertscore = load("bertscore")
+predictions = ["hello world", "general kenobi"]
+references = ["goodnight moon", "the sun is shining"]
+results = bertscore.compute(predictions=predictions, references=references, model_type="distilbert-base-uncased")
+print(results)
+{'precision': [0.7380737066268921, 0.5584042072296143], 'recall': [0.7380737066268921, 0.5889028906822205], 'f1': [0.7380737066268921, 0.5732481479644775], 'hashcode': 'bert-base-uncased_L5_no-idf_version=0.3.10(hug_trans=4.10.3)'}
+```
+
+## Limitations and bias
+
+The [original BERTScore paper](https://openreview.net/pdf?id=SkeHuCVFDr) showed that BERTScore correlates well with human judgment on sentence-level and system-level evaluation, but this depends on the model and language pair selected.
+
+Furthermore, not all languages are supported by the metric -- see the [BERTScore supported language list](https://github.com/google-research/bert/blob/master/multilingual.md#list-of-languages) for more information.
+
+Finally, calculating the BERTScore metric involves downloading the BERT model that is used to compute the score-- the default model for `en`, `roberta-large`, takes over 1.4GB of storage space and downloading it can take a significant amount of time depending on the speed of your internet connection. If this is an issue, choose a smaller model; for instance `distilbert-base-uncased` is 268MB. A full list of compatible models can be found [here](https://docs.google.com/spreadsheets/d/1RKOVpselB98Nnh_EOC4A2BYn8_201tmPODpNWu4w7xI/edit#gid=0).  
+
+
+## Citation
+
+```bibtex
+@inproceedings{bert-score,
+  title={BERTScore: Evaluating Text Generation with BERT},
+  author={Tianyi Zhang* and Varsha Kishore* and Felix Wu* and Kilian Q. Weinberger and Yoav Artzi},
+  booktitle={International Conference on Learning Representations},
+  year={2020},
+  url={https://openreview.net/forum?id=SkeHuCVFDr}
+}
+```
+    
+## Further References 
+
+- [BERTScore Project README](https://github.com/Tiiiger/bert_score#readme)
+- [BERTScore ICLR 2020 Poster Presentation](https://iclr.cc/virtual_2020/poster_SkeHuCVFDr.html)

app.py

@@ -0,0 +1,6 @@
+import evaluate
+from evaluate.utils import launch_gradio_widget
+
+
+module = evaluate.load("bertscore")
+launch_gradio_widget(module)

bertscore.py

@@ -0,0 +1,215 @@
+# Copyright 2020 The HuggingFace Evaluate Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+""" BERTScore metric. """
+
+import functools
+from contextlib import contextmanager
+
+import bert_score
+import datasets
+from packaging import version
+
+import evaluate
+
+
+@contextmanager
+def filter_logging_context():
+    def filter_log(record):
+        return False if "This IS expected if you are initializing" in record.msg else True
+
+    logger = datasets.utils.logging.get_logger("transformers.modeling_utils")
+    logger.addFilter(filter_log)
+    try:
+        yield
+    finally:
+        logger.removeFilter(filter_log)
+
+
+_CITATION = """\
+@inproceedings{bert-score,
+  title={BERTScore: Evaluating Text Generation with BERT},
+  author={Tianyi Zhang* and Varsha Kishore* and Felix Wu* and Kilian Q. Weinberger and Yoav Artzi},
+  booktitle={International Conference on Learning Representations},
+  year={2020},
+  url={https://openreview.net/forum?id=SkeHuCVFDr}
+}
+"""
+
+_DESCRIPTION = """\
+BERTScore leverages the pre-trained contextual embeddings from BERT and matches words in candidate and reference
+sentences by cosine similarity.
+It has been shown to correlate with human judgment on sentence-level and system-level evaluation.
+Moreover, BERTScore computes precision, recall, and F1 measure, which can be useful for evaluating different language
+generation tasks.
+
+See the project's README at https://github.com/Tiiiger/bert_score#readme for more information.
+"""
+
+_KWARGS_DESCRIPTION = """
+BERTScore Metrics with the hashcode from a source against one or more references.
+
+Args:
+    predictions (list of str): Prediction/candidate sentences.
+    references (list of str or list of list of str): Reference sentences.
+    lang (str): Language of the sentences; required (e.g. 'en').
+    model_type (str): Bert specification, default using the suggested
+        model for the target language; has to specify at least one of
+        `model_type` or `lang`.
+    num_layers (int): The layer of representation to use,
+        default using the number of layers tuned on WMT16 correlation data.
+    verbose (bool): Turn on intermediate status update.
+    idf (bool or dict): Use idf weighting; can also be a precomputed idf_dict.
+    device (str): On which the contextual embedding model will be allocated on.
+        If this argument is None, the model lives on cuda:0 if cuda is available.
+    nthreads (int): Number of threads.
+    batch_size (int): Bert score processing batch size,
+        at least one of `model_type` or `lang`. `lang` needs to be
+        specified when `rescale_with_baseline` is True.
+    rescale_with_baseline (bool): Rescale bertscore with pre-computed baseline.
+    baseline_path (str): Customized baseline file.
+    use_fast_tokenizer (bool): `use_fast` parameter passed to HF tokenizer. New in version 0.3.10.
+
+Returns:
+    precision: Precision.
+    recall: Recall.
+    f1: F1 score.
+    hashcode: Hashcode of the library.
+
+Examples:
+
+    >>> predictions = ["hello there", "general kenobi"]
+    >>> references = ["hello there", "general kenobi"]
+    >>> bertscore = evaluate.load("bertscore")
+    >>> results = bertscore.compute(predictions=predictions, references=references, lang="en")
+    >>> print([round(v, 2) for v in results["f1"]])
+    [1.0, 1.0]
+"""
+
+
+@evaluate.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
+class BERTScore(evaluate.Metric):
+    def _info(self):
+        return evaluate.MetricInfo(
+            description=_DESCRIPTION,
+            citation=_CITATION,
+            homepage="https://github.com/Tiiiger/bert_score",
+            inputs_description=_KWARGS_DESCRIPTION,
+            features=[
+                datasets.Features(
+                    {
+                        "predictions": datasets.Value("string", id="sequence"),
+                        "references": datasets.Sequence(datasets.Value("string", id="sequence"), id="references"),
+                    }
+                ),
+                datasets.Features(
+                    {
+                        "predictions": datasets.Value("string", id="sequence"),
+                        "references": datasets.Value("string", id="sequence"),
+                    }
+                ),
+            ],
+            codebase_urls=["https://github.com/Tiiiger/bert_score"],
+            reference_urls=[
+                "https://github.com/Tiiiger/bert_score",
+                "https://arxiv.org/abs/1904.09675",
+            ],
+        )
+
+    def _compute(
+        self,
+        predictions,
+        references,
+        lang=None,
+        model_type=None,
+        num_layers=None,
+        verbose=False,
+        idf=False,
+        device=None,
+        batch_size=64,
+        nthreads=4,
+        all_layers=False,
+        rescale_with_baseline=False,
+        baseline_path=None,
+        use_fast_tokenizer=False,
+    ):
+
+        if isinstance(references[0], str):
+            references = [[ref] for ref in references]
+
+        if idf:
+            idf_sents = [r for ref in references for r in ref]
+        else:
+            idf_sents = None
+
+        get_hash = bert_score.utils.get_hash
+        scorer = bert_score.BERTScorer
+
+        if version.parse(bert_score.__version__) >= version.parse("0.3.10"):
+            get_hash = functools.partial(get_hash, use_fast_tokenizer=use_fast_tokenizer)
+            scorer = functools.partial(scorer, use_fast_tokenizer=use_fast_tokenizer)
+        elif use_fast_tokenizer:
+            raise ImportWarning(
+                "To use a fast tokenizer, the module `bert-score>=0.3.10` is required, and the current version of "
+                "`bert-score` doesn't match this condition.\n"
+                'You can install it with `pip install "bert-score>=0.3.10"`.'
+            )
+
+        if model_type is None:
+            if lang is None:
+                raise ValueError(
+                    "Either 'lang' (e.g. 'en') or 'model_type' (e.g. 'microsoft/deberta-xlarge-mnli')"
+                    " must be specified"
+                )
+            model_type = bert_score.utils.lang2model[lang.lower()]
+
+        if num_layers is None:
+            num_layers = bert_score.utils.model2layers[model_type]
+
+        hashcode = get_hash(
+            model=model_type,
+            num_layers=num_layers,
+            idf=idf,
+            rescale_with_baseline=rescale_with_baseline,
+            use_custom_baseline=baseline_path is not None,
+        )
+
+        with filter_logging_context():
+            if not hasattr(self, "cached_bertscorer") or self.cached_bertscorer.hash != hashcode:
+                self.cached_bertscorer = scorer(
+                    model_type=model_type,
+                    num_layers=num_layers,
+                    batch_size=batch_size,
+                    nthreads=nthreads,
+                    all_layers=all_layers,
+                    idf=idf,
+                    idf_sents=idf_sents,
+                    device=device,
+                    lang=lang,
+                    rescale_with_baseline=rescale_with_baseline,
+                    baseline_path=baseline_path,
+                )
+
+        (P, R, F) = self.cached_bertscorer.score(
+            cands=predictions,
+            refs=references,
+            verbose=verbose,
+            batch_size=batch_size,
+        )
+        output_dict = {
+            "precision": P.tolist(),
+            "recall": R.tolist(),
+            "f1": F.tolist(),
+            "hashcode": hashcode,
+        }
+        return output_dict

requirements.txt

@@ -0,0 +1,2 @@
+git+https://github.com/huggingface/evaluate@6abb0d53b82b1e5efea5d683b91d7990a653c78d
+bert_score