Stat Scores

Module Interface

class torchmetrics.StatScores(threshold=0.5, top_k=None, reduce='micro', num_classes=None, ignore_index=None, mdmc_reduce=None, multiclass=None, **kwargs)

Computes the number of true positives, false positives, true negatives, false negatives. Related to Type I and Type II errors and the confusion matrix.

The reduction method (how the statistics are aggregated) is controlled by the reduce parameter, and additionally by the mdmc_reduce parameter in the multi-dimensional multi-class case.

Accepts all inputs listed in Input types.

Parameters

• threshold (float) – Threshold for transforming probability or logit predictions to binary (0,1) predictions, in the case of binary or multi-label inputs. Default value of 0.5 corresponds to input being probabilities.

• top_k (Optional[int]) – Number of the highest probability or logit score predictions considered finding the correct label, relevant only for (multi-dimensional) multi-class inputs. The default value (None) will be interpreted as 1 for these inputs. Should be left at default (None) for all other types of inputs.

• reduce (str) –

  Defines the reduction that is applied. Should be one of the following:

  • 'micro' [default]: Counts the statistics by summing over all [sample, class] combinations (globally). Each statistic is represented by a single integer.

  • 'macro': Counts the statistics for each class separately (over all samples). Each statistic is represented by a (C,) tensor. Requires num_classes to be set.

  • 'samples': Counts the statistics for each sample separately (over all classes). Each statistic is represented by a (N, ) 1d tensor.

  Note

  What is considered a sample in the multi-dimensional multi-class case depends on the value of mdmc_reduce.

• num_classes (Optional[int]) – Number of classes. Necessary for (multi-dimensional) multi-class or multi-label data.

• ignore_index (Optional[int]) – Specify a class (label) to ignore. If given, this class index does not contribute to the returned score, regardless of reduction method. If an index is ignored, and reduce='macro', the class statistics for the ignored class will all be returned as -1.

• mdmc_reduce (Optional[str]) –

  Defines how the multi-dimensional multi-class inputs are handeled. Should be one of the following:

  • None [default]: Should be left unchanged if your data is not multi-dimensional multi-class (see Input types for the definition of input types).

  • 'samplewise': In this case, the statistics are computed separately for each sample on the N axis, and then the outputs are concatenated together. In each sample the extra axes ... are flattened to become the sub-sample axis, and statistics for each sample are computed by treating the sub-sample axis as the N axis for that sample.

  • 'global': In this case the N and ... dimensions of the inputs are flattened into a new N_X sample axis, i.e. the inputs are treated as if they were (N_X, C). From here on the reduce parameter applies as usual.

• multiclass (Optional[bool]) – Used only in certain special cases, where you want to treat inputs as a different type than what they appear to be. See the parameter’s documentation section for a more detailed explanation and examples.

• kwargs (Dict[str, Any]) – Additional keyword arguments, see Advanced metric settings for more info.

Raises

• ValueError – If reduce is none of "micro", "macro" or "samples".

• ValueError – If mdmc_reduce is none of None, "samplewise", "global".

• ValueError – If reduce is set to "macro" and num_classes is not provided.

• ValueError – If num_classes is set and ignore_index is not in the range 0 <= ignore_index < num_classes.

Example

>>> from torchmetrics.classification import StatScores
>>> preds  = torch.tensor([1, 0, 2, 1])
>>> target = torch.tensor([1, 1, 2, 0])
>>> stat_scores = StatScores(reduce='macro', num_classes=3)
>>> stat_scores(preds, target)
tensor([[0, 1, 2, 1, 1],
        [1, 1, 1, 1, 2],
        [1, 0, 3, 0, 1]])
>>> stat_scores = StatScores(reduce='micro')
>>> stat_scores(preds, target)
tensor([2, 2, 6, 2, 4])

Initializes internal Module state, shared by both nn.Module and ScriptModule.

compute()

Computes the stat scores based on inputs passed in to update previously.

Return type

Tensor

Returns

The metric returns a tensor of shape (..., 5), where the last dimension corresponds to [tp, fp, tn, fn, sup] (sup stands for support and equals tp + fn). The shape depends on the reduce and mdmc_reduce (in case of multi-dimensional multi-class data) parameters:

• If the data is not multi-dimensional multi-class, then

  • If reduce='micro', the shape will be (5, )

  • If reduce='macro', the shape will be (C, 5), where C stands for the number of classes

  • If reduce='samples', the shape will be (N, 5), where N stands for the number of samples

• If the data is multi-dimensional multi-class and mdmc_reduce='global', then

  • If reduce='micro', the shape will be (5, )

  • If reduce='macro', the shape will be (C, 5)

  • If reduce='samples', the shape will be (N*X, 5), where X stands for the product of sizes of all “extra” dimensions of the data (i.e. all dimensions except for C and N)

• If the data is multi-dimensional multi-class and mdmc_reduce='samplewise', then

  • If reduce='micro', the shape will be (N, 5)

  • If reduce='macro', the shape will be (N, C, 5)

  • If reduce='samples', the shape will be (N, X, 5)

update(preds, target)

Update state with predictions and targets.

See Input types for more information on input types.

Parameters

• preds (Tensor) – Predictions from model (probabilities, logits or labels)

• target (Tensor) – Ground truth values

Return type

None

Functional Interface

torchmetrics.functional.stat_scores(preds, target, reduce='micro', mdmc_reduce=None, num_classes=None, top_k=None, threshold=0.5, multiclass=None, ignore_index=None)

Computes the number of true positives, false positives, true negatives, false negatives. Related to Type I and Type II errors and the confusion matrix.

The reduction method (how the statistics are aggregated) is controlled by the reduce parameter, and additionally by the mdmc_reduce parameter in the multi-dimensional multi-class case. Accepts all inputs listed in Input types.

Parameters

• preds (Tensor) – Predictions from model (probabilities, logits or labels)

• target (Tensor) – Ground truth values

• threshold (float) – Threshold for transforming probability or logit predictions to binary (0,1) predictions, in the case of binary or multi-label inputs. Default value of 0.5 corresponds to input being probabilities.

• top_k (Optional[int]) –

  Number of highest probability or logit score predictions considered to find the correct label, relevant only for (multi-dimensional) multi-class inputs. The default value (None) will be interpreted as 1 for these inputs.

  Should be left at default (None) for all other types of inputs.

• reduce (str) –

  Defines the reduction that is applied. Should be one of the following:

  • 'micro' [default]: Counts the statistics by summing over all [sample, class] combinations (globally). Each statistic is represented by a single integer.

  • 'macro': Counts the statistics for each class separately (over all samples). Each statistic is represented by a (C,) tensor. Requires num_classes to be set.

  • 'samples': Counts the statistics for each sample separately (over all classes). Each statistic is represented by a (N, ) 1d tensor.

  Note

  What is considered a sample in the multi-dimensional multi-class case depends on the value of mdmc_reduce.

• num_classes (Optional[int]) – Number of classes. Necessary for (multi-dimensional) multi-class or multi-label data.

• ignore_index (Optional[int]) – Specify a class (label) to ignore. If given, this class index does not contribute to the returned score, regardless of reduction method. If an index is ignored, and reduce='macro', the class statistics for the ignored class will all be returned as -1.

• mdmc_reduce (Optional[str]) –

  Defines how the multi-dimensional multi-class inputs are handeled. Should be one of the following:

  • None [default]: Should be left unchanged if your data is not multi-dimensional multi-class (see Input types for the definition of input types).

  • 'samplewise': In this case, the statistics are computed separately for each sample on the N axis, and then the outputs are concatenated together. In each sample the extra axes ... are flattened to become the sub-sample axis, and statistics for each sample are computed by treating the sub-sample axis as the N axis for that sample.

  • 'global': In this case the N and ... dimensions of the inputs are flattened into a new N_X sample axis, i.e. the inputs are treated as if they were (N_X, C). From here on the reduce parameter applies as usual.

• multiclass (Optional[bool]) – Used only in certain special cases, where you want to treat inputs as a different type than what they appear to be. See the parameter’s documentation section for a more detailed explanation and examples.

Return type

Tensor

Returns

The metric returns a tensor of shape (..., 5), where the last dimension corresponds to [tp, fp, tn, fn, sup] (sup stands for support and equals tp + fn). The shape depends on the reduce and mdmc_reduce (in case of multi-dimensional multi-class data) parameters:

• If the data is not multi-dimensional multi-class, then

  • If reduce='micro', the shape will be (5, )

  • If reduce='macro', the shape will be (C, 5), where C stands for the number of classes

  • If reduce='samples', the shape will be (N, 5), where N stands for the number of samples

• If the data is multi-dimensional multi-class and mdmc_reduce='global', then

  • If reduce='micro', the shape will be (5, )

  • If reduce='macro', the shape will be (C, 5)

  • If reduce='samples', the shape will be (N*X, 5), where X stands for the product of sizes of all “extra” dimensions of the data (i.e. all dimensions except for C and N)

• If the data is multi-dimensional multi-class and mdmc_reduce='samplewise', then

  • If reduce='micro', the shape will be (N, 5)

  • If reduce='macro', the shape will be (N, C, 5)

  • If reduce='samples', the shape will be (N, X, 5)

Raises

• ValueError – If reduce is none of "micro", "macro" or "samples".

• ValueError – If mdmc_reduce is none of None, "samplewise", "global".

• ValueError – If reduce is set to "macro" and num_classes is not provided.

• ValueError – If num_classes is set and ignore_index is not in the range [0, num_classes).

• ValueError – If ignore_index is used with binary data.

• ValueError – If inputs are multi-dimensional multi-class and mdmc_reduce is not provided.

Example

>>> from torchmetrics.functional import stat_scores
>>> preds  = torch.tensor([1, 0, 2, 1])
>>> target = torch.tensor([1, 1, 2, 0])
>>> stat_scores(preds, target, reduce='macro', num_classes=3)
tensor([[0, 1, 2, 1, 1],
        [1, 1, 1, 1, 2],
        [1, 0, 3, 0, 1]])
>>> stat_scores(preds, target, reduce='micro')
tensor([2, 2, 6, 2, 4])