ChrF Score¶

Module Interface¶

class torchmetrics.CHRFScore(n_char_order=6, n_word_order=2, beta=2.0, lowercase=False, whitespace=False, return_sentence_level_score=False, **kwargs)[source]

Calculate chrf score of machine translated text with one or more references.

This implementation supports both ChrF score computation introduced in chrF score and chrF++ score introduced in chrF++ score. This implementation follows the implmenetaions from https://github.com/m-popovic/chrF and https://github.com/mjpost/sacrebleu/blob/master/sacrebleu/metrics/chrf.py.

As input to forward and update the metric accepts the following input:

preds (Sequence): An iterable of hypothesis corpus
target (Sequence): An iterable of iterables of reference corpus

As output of forward and compute the metric returns the following output:

chrf (Tensor): If return_sentence_level_score=True return a list of sentence-level chrF/chrF++ scores, else return a corpus-level chrF/chrF++ score

Parameters

n_char_order¶ (int) – A character n-gram order. If n_char_order=6, the metrics refers to the official chrF/chrF++.
n_word_order¶ (int) – A word n-gram order. If n_word_order=2, the metric refers to the official chrF++. If n_word_order=0, the metric is equivalent to the original ChrF.
beta¶ (float) – parameter determining an importance of recall w.r.t. precision. If beta=1, their importance is equal.
lowercase¶ (bool) – An indication whether to enable case-insesitivity.
whitespace¶ (bool) – An indication whether keep whitespaces during n-gram extraction.
return_sentence_level_score¶ (bool) – An indication whether a sentence-level chrF/chrF++ score to be returned.
kwargs¶ (Any) – Additional keyword arguments, see Advanced metric settings for more info.

Raises

ValueError – If n_char_order is not an integer greater than or equal to 1.
ValueError – If n_word_order is not an integer greater than or equal to 0.
ValueError – If beta is smaller than 0.

Example

>>> from torchmetrics import CHRFScore
>>> preds = ['the cat is on the mat']
>>> target = [['there is a cat on the mat', 'a cat is on the mat']]
>>> chrf = CHRFScore()
>>> chrf(preds, target)
tensor(0.8640)

Initializes internal Module state, shared by both nn.Module and ScriptModule.

Functional Interface¶

torchmetrics.functional.chrf_score(preds, target, n_char_order=6, n_word_order=2, beta=2.0, lowercase=False, whitespace=False, return_sentence_level_score=False)[source]

Calculate chrF score of machine translated text with one or more references. This implementation supports both chrF score computation introduced in [1] and chrF++ score introduced in chrF++ score. This implementation follows the implmenetaions from https://github.com/m-popovic/chrF and https://github.com/mjpost/sacrebleu/blob/master/sacrebleu/metrics/chrf.py.

Parameters

preds¶ (Union[str, Sequence[str]]) – An iterable of hypothesis corpus.
target¶ (Sequence[Union[str, Sequence[str]]]) – An iterable of iterables of reference corpus.
n_char_order¶ (int) – A character n-gram order. If n_char_order=6, the metrics refers to the official chrF/chrF++.
n_word_order¶ (int) – A word n-gram order. If n_word_order=2, the metric refers to the official chrF++. If n_word_order=0, the metric is equivalent to the original chrF.
beta¶ (float) – A parameter determining an importance of recall w.r.t. precision. If beta=1, their importance is equal.
lowercase¶ (bool) – An indication whether to enable case-insesitivity.
whitespace¶ (bool) – An indication whether to keep whitespaces during character n-gram extraction.
return_sentence_level_score¶ (bool) – An indication whether a sentence-level chrF/chrF++ score to be returned.

Return type

Union[Tensor, Tuple[Tensor, Tensor]]

Returns

A corpus-level chrF/chrF++ score. (Optionally) A list of sentence-level chrF/chrF++ scores if return_sentence_level_score=True.

Raises

ValueError – If n_char_order is not an integer greater than or equal to 1.
ValueError – If n_word_order is not an integer greater than or equal to 0.
ValueError – If beta is smaller than 0.

Example

>>> from torchmetrics.functional import chrf_score
>>> preds = ['the cat is on the mat']
>>> target = [['there is a cat on the mat', 'a cat is on the mat']]
>>> chrf_score(preds, target)
tensor(0.8640)

References

[1] chrF: character n-gram F-score for automatic MT evaluation by Maja Popović chrF score

[2] chrF++: words helping character n-grams by Maja Popović chrF++ score