Shortcuts

ChrF Score

Module Interface

class torchmetrics.CHRFScore(n_char_order=6, n_word_order=2, beta=2.0, lowercase=False, whitespace=False, return_sentence_level_score=False, compute_on_step=None, **kwargs)[source]

Calculate chrf score of machine translated text with one or more references.

This implementation supports both ChrF score computation introduced in [1] and chrF++ score introduced in chrF++ score_. This implementation follows the implmenetaions from https://github.com/m-popovic/chrF and https://github.com/mjpost/sacrebleu/blob/master/sacrebleu/metrics/chrf.py.

Parameters
  • n_char_order (int) – A character n-gram order. If n_char_order=6, the metrics refers to the official chrF/chrF++.

  • n_word_order (int) – A word n-gram order. If n_word_order=2, the metric refers to the official chrF++. If n_word_order=0, the metric is equivalent to the original ChrF.

  • beta (float) – parameter determining an importance of recall w.r.t. precision. If beta=1, their importance is equal.

  • lowercase (bool) – An indication whether to enable case-insesitivity.

  • whitespace (bool) – An indication whether keep whitespaces during n-gram extraction.

  • return_sentence_level_score (bool) – An indication whether a sentence-level chrF/chrF++ score to be returned.

  • compute_on_step (Optional[bool]) –

    Forward only calls update() and returns None if this is set to False.

    Deprecated since version v0.8: Argument has no use anymore and will be removed v0.9.

  • kwargs (Dict[str, Any]) – Additional keyword arguments, see Advanced metric settings for more info.

Raises
  • ValueError – If n_char_order is not an integer greater than or equal to 1.

  • ValueError – If n_word_order is not an integer greater than or equal to 0.

  • ValueError – If beta is smaller than 0.

Example

>>> from torchmetrics import CHRFScore
>>> preds = ['the cat is on the mat']
>>> target = [['there is a cat on the mat', 'a cat is on the mat']]
>>> metric = CHRFScore()
>>> metric(preds, target)
tensor(0.8640)

References

[1] chrF: character n-gram F-score for automatic MT evaluation by Maja Popović chrF score

[2] chrF++: words helping character n-grams by Maja Popović chrF++ score

Initializes internal Module state, shared by both nn.Module and ScriptModule.

compute()[source]

Calculate chrF/chrF++ score.

Return type

Union[Tensor, Tuple[Tensor, Tensor]]

Returns

A corpus-level chrF/chrF++ score. (Optionally) A list of sentence-level chrF/chrF++ scores if return_sentence_level_score=True.

update(preds, target)[source]

Compute Precision Scores.

Parameters
Return type

None

Functional Interface

torchmetrics.functional.chrf_score(preds, target, n_char_order=6, n_word_order=2, beta=2.0, lowercase=False, whitespace=False, return_sentence_level_score=False)[source]

Calculate chrF score of machine translated text with one or more references. This implementation supports both chrF score computation introduced in [1] and chrF++ score introduced in chrF++ score. This implementation follows the implmenetaions from https://github.com/m-popovic/chrF and https://github.com/mjpost/sacrebleu/blob/master/sacrebleu/metrics/chrf.py.

Parameters
  • preds (Union[str, Sequence[str]]) – An iterable of hypothesis corpus.

  • target (Sequence[Union[str, Sequence[str]]]) – An iterable of iterables of reference corpus.

  • n_char_order (int) – A character n-gram order. If n_char_order=6, the metrics refers to the official chrF/chrF++.

  • n_word_order (int) – A word n-gram order. If n_word_order=2, the metric refers to the official chrF++. If n_word_order=0, the metric is equivalent to the original chrF.

  • beta (float) – A parameter determining an importance of recall w.r.t. precision. If beta=1, their importance is equal.

  • lowercase (bool) – An indication whether to enable case-insesitivity.

  • whitespace (bool) – An indication whether to keep whitespaces during character n-gram extraction.

  • return_sentence_level_score (bool) – An indication whether a sentence-level chrF/chrF++ score to be returned.

Return type

Union[Tensor, Tuple[Tensor, Tensor]]

Returns

A corpus-level chrF/chrF++ score. (Optionally) A list of sentence-level chrF/chrF++ scores if return_sentence_level_score=True.

Raises
  • ValueError – If n_char_order is not an integer greater than or equal to 1.

  • ValueError – If n_word_order is not an integer greater than or equal to 0.

  • ValueError – If beta is smaller than 0.

Example

>>> from torchmetrics.functional import chrf_score
>>> preds = ['the cat is on the mat']
>>> target = [['there is a cat on the mat', 'a cat is on the mat']]
>>> chrf_score(preds, target)
tensor(0.8640)

References

[1] chrF: character n-gram F-score for automatic MT evaluation by Maja Popović chrF score

[2] chrF++: words helping character n-grams by Maja Popović chrF++ score

Read the Docs v: v0.8.0
Versions
latest
stable
v0.8.0
v0.7.3
v0.7.2
v0.7.1
v0.7.0
v0.6.2
v0.6.1
v0.6.0
v0.5.1
v0.5.0
v0.4.1
v0.4.0
v0.3.2
v0.3.1
v0.3.0
v0.2.0
v0.1.0
Downloads
On Read the Docs
Project Home
Builds

Free document hosting provided by Read the Docs.