Table of Contents
Shortcuts

Group Fairness

Module Interface

BinaryFairness

class torchmetrics.classification.BinaryFairness(num_groups, task='all', threshold=0.5, ignore_index=None, validate_args=True, **kwargs)[source]

Computes Demographic parity and Equal opportunity ratio for binary classification problems.

Accepts the following input tensors:

  • preds (int or float tensor): (N, ...). If preds is a floating point tensor with values outside [0,1] range we consider the input to be logits and will auto apply sigmoid per element. Additionally, we convert to int tensor with thresholding using the value in threshold.

  • groups (int tensor): (N, ...). The group identifiers should be 0, 1, ..., (num_groups - 1).

  • target (int tensor): (N, ...).

The additional dimensions are flatted along the batch dimension.

This class computes the ratio between positivity rates and true positives rates for different groups. If more than two groups are present, the disparity between the lowest and highest group is reported. A disparity between positivity rates indicates a potential violation of demographic parity, and between true positive rates indicates a potential violation of equal opportunity.

The lowest rate is divided by the highest, so a lower value means more discrimination against the numerator. In the results this is also indicated as the key of dict is {metric}_{identifier_low_group}_{identifier_high_group}.

Parameters:

  • num_groups (int) – The number of groups.

  • task (Literal['demographic_parity', 'equal_opportunity', 'all']) – The task to compute. Can be either demographic_parity or equal_opportunity or all.

  • threshold (float) – Threshold for transforming probability to binary {0,1} predictions.

  • ignore_index (Optional[int]) – Specifies a target value that is ignored and does not contribute to the metric calculation

  • validate_args (bool) – bool indicating if input arguments and tensors should be validated for correctness. Set to False for faster computations.

  • kwargs (Any) – Additional keyword arguments, see Advanced metric settings for more info.

Returns:

The metric returns a dict where the key identifies the metric and groups with the lowest and highest true positives rates as follows: {metric}__{identifier_low_group}_{identifier_high_group}. The value is a tensor with the disparity rate.

Example (preds is int tensor):
>>> from torchmetrics.classification import BinaryFairness
>>> target = torch.tensor([0, 1, 0, 1, 0, 1])
>>> preds = torch.tensor([0, 1, 0, 1, 0, 1])
>>> groups = torch.tensor([0, 1, 0, 1, 0, 1])
>>> metric = BinaryFairness(2)
>>> metric(preds, target, groups)
{'DP_0_1': tensor(0.), 'EO_0_1': tensor(0.)}
Example (preds is float tensor):
>>> from torchmetrics.classification import BinaryFairness
>>> target = torch.tensor([0, 1, 0, 1, 0, 1])
>>> preds = torch.tensor([0.11, 0.84, 0.22, 0.73, 0.33, 0.92])
>>> groups = torch.tensor([0, 1, 0, 1, 0, 1])
>>> metric = BinaryFairness(2)
>>> metric(preds, target, groups)
{'DP_0_1': tensor(0.), 'EO_0_1': tensor(0.)}
plot(val=None, ax=None)[source]

Plot a single or multiple values from the metric.

Parameters:

  • val (Union[Tensor, Sequence[Tensor], None]) – Either a single result from calling metric.forward or metric.compute or a list of these results. If no value is provided, will automatically call metric.compute and plot that result.

  • ax (Optional[Axes]) – An matplotlib axis object. If provided will add plot to that axis

Return type:

Tuple[Figure, Union[Axes, ndarray]]

Returns:

Figure object and Axes object

Raises:

ModuleNotFoundError – If matplotlib is not installed

>>> import torch
>>> _ = torch.manual_seed(42)
>>> # Example plotting a single value
>>> from torchmetrics.classification import BinaryFairness
>>> metric = BinaryFairness(2)
>>> metric.update(torch.rand(20), torch.randint(2,(20,)), torch.randint(2,(20,)))
>>> fig_, ax_ = metric.plot()
../_images/group_fairness-1.png 
>>> import torch
>>> _ = torch.manual_seed(42)
>>> # Example plotting multiple values
>>> from torchmetrics.classification import BinaryFairness
>>> metric = BinaryFairness(2)
>>> values = [ ]
>>> for _ in range(10):
...     values.append(metric(torch.rand(20), torch.randint(2,(20,)), torch.ones(20).long()))
>>> fig_, ax_ = metric.plot(values)
../_images/group_fairness-2.png

BinaryGroupStatRates

class torchmetrics.classification.BinaryGroupStatRates(num_groups, threshold=0.5, ignore_index=None, validate_args=True, **kwargs)[source]

Computes the true/false positives and true/false negatives rates for binary classification by group.

Related to Type I and Type II errors.

Accepts the following input tensors:

  • preds (int or float tensor): (N, ...). If preds is a floating point tensor with values outside [0,1] range we consider the input to be logits and will auto apply sigmoid per element. Additionally, we convert to int tensor with thresholding using the value in threshold.

  • target (int tensor): (N, ...).

  • groups (int tensor): (N, ...). The group identifiers should be 0, 1, ..., (num_groups - 1).

The additional dimensions are flatted along the batch dimension.

Parameters:

  • num_groups (int) – The number of groups.

  • threshold (float) – Threshold for transforming probability to binary {0,1} predictions.

  • ignore_index (Optional[int]) – Specifies a target value that is ignored and does not contribute to the metric calculation

  • validate_args (bool) – bool indicating if input arguments and tensors should be validated for correctness. Set to False for faster computations.

  • kwargs (Any) – Additional keyword arguments, see Advanced metric settings for more info.

Returns:

The metric returns a dict with a group identifier as key and a tensor with the tp, fp, tn and fn rates as value.

Example (preds is int tensor):
>>> from torchmetrics.classification import BinaryGroupStatRates
>>> target = torch.tensor([0, 1, 0, 1, 0, 1])
>>> preds = torch.tensor([0, 1, 0, 1, 0, 1])
>>> groups = torch.tensor([0, 1, 0, 1, 0, 1])
>>> metric = BinaryGroupStatRates(num_groups=2)
>>> metric(preds, target, groups)
{'group_0': tensor([0., 0., 1., 0.]), 'group_1': tensor([1., 0., 0., 0.])}
Example (preds is float tensor):
>>> from torchmetrics.classification import BinaryGroupStatRates
>>> target = torch.tensor([0, 1, 0, 1, 0, 1])
>>> preds = torch.tensor([0.11, 0.84, 0.22, 0.73, 0.33, 0.92])
>>> groups = torch.tensor([0, 1, 0, 1, 0, 1])
>>> metric = BinaryGroupStatRates(num_groups=2)
>>> metric(preds, target, groups)
{'group_0': tensor([0., 0., 1., 0.]), 'group_1': tensor([1., 0., 0., 0.])}

Functional Interface

binary_fairness

torchmetrics.functional.classification.binary_fairness(preds, target, groups, task='all', threshold=0.5, ignore_index=None, validate_args=True)[source]

Compute either Demographic parity and Equal opportunity ratio for binary classification problems.

This is done by setting the task argument to either 'demographic_parity', 'equal_opportunity' or all. See the documentation of demographic_parity() and equal_opportunity() for the specific details of each argument influence and examples.

Parameters:

  • preds (Tensor) – Tensor with predictions.

  • target (Tensor) – Tensor with true labels (not required for demographic_parity).

  • groups (Tensor) – Tensor with group identifiers. The group identifiers should be 0, 1, ..., (num_groups - 1).

  • task (Literal['demographic_parity', 'equal_opportunity', 'all']) – The task to compute. Can be either demographic_parity or equal_opportunity or all.

  • threshold (float) – Threshold for transforming probability to binary {0,1} predictions.

  • ignore_index (Optional[int]) – Specifies a target value that is ignored and does not contribute to the metric calculation

  • validate_args (bool) – bool indicating if input arguments and tensors should be validated for correctness. Set to False for faster computations.

Return type:

Dict[str, Tensor]

demographic_parity

torchmetrics.functional.classification.demographic_parity(preds, groups, threshold=0.5, ignore_index=None, validate_args=True)[source]

Demographic parity compares the positivity rates between all groups.

If more than two groups are present, the disparity between the lowest and highest group is reported. The lowest positivity rate is divided by the highest, so a lower value means more discrimination against the numerator. In the results this is also indicated as the key of dict is DP_{identifier_low_group}_{identifier_high_group}.

\[\text{DP} = \dfrac{\min_a PR_a}{\max_a PR_a}.\]

where \(\text{PR}\) represents the positivity rate for group \(\text{a}\).

Accepts the following input tensors:

  • preds (int or float tensor): (N, ...). If preds is a floating point tensor with values outside [0,1] range we consider the input to be logits and will auto apply sigmoid per element. Additionally, we convert to int tensor with thresholding using the value in threshold.

  • groups (int tensor): (N, ...). The group identifiers should be 0, 1, ..., (num_groups - 1).

  • target (int tensor): (N, ...).

The additional dimensions are flatted along the batch dimension.

Parameters:

  • preds (Tensor) – Tensor with predictions.

  • groups (Tensor) – Tensor with group identifiers. The group identifiers should be 0, 1, ..., (num_groups - 1).

  • threshold (float) – Threshold for transforming probability to binary {0,1} predictions.

  • ignore_index (Optional[int]) – Specifies a target value that is ignored and does not contribute to the metric calculation

  • validate_args (bool) – bool indicating if input arguments and tensors should be validated for correctness. Set to False for faster computations.

Return type:

Dict[str, Tensor]

Returns:

The metric returns a dict where the key identifies the group with the lowest and highest positivity rates as follows: DP_{identifier_low_group}_{identifier_high_group}. The value is a tensor with the DP rate.

Example (preds is int tensor):
>>> from torchmetrics.functional.classification import demographic_parity
>>> preds = torch.tensor([0, 1, 0, 1, 0, 1])
>>> groups = torch.tensor([0, 1, 0, 1, 0, 1])
>>> demographic_parity(preds, groups)
{'DP_0_1': tensor(0.)}
Example (preds is float tensor):
>>> from torchmetrics.functional.classification import demographic_parity
>>> preds = torch.tensor([0.11, 0.84, 0.22, 0.73, 0.33, 0.92])
>>> groups = torch.tensor([0, 1, 0, 1, 0, 1])
>>> demographic_parity(preds, groups)
{'DP_0_1': tensor(0.)}

equal_opportunity

torchmetrics.functional.classification.equal_opportunity(preds, target, groups, threshold=0.5, ignore_index=None, validate_args=True)[source]

Equal opportunity compares the true positive rates between all groups.

If more than two groups are present, the disparity between the lowest and highest group is reported. The lowest true positive rate is divided by the highest, so a lower value means more discrimination against the numerator. In the results this is also indicated as the key of dict is EO_{identifier_low_group}_{identifier_high_group}.

\[\text{DP} = \dfrac{\min_a TPR_a}{\max_a TPR_a}.\]

where \(\text{TPR}\) represents the true positives rate for group \(\text{a}\).

Accepts the following input tensors:

  • preds (int or float tensor): (N, ...). If preds is a floating point tensor with values outside [0,1] range we consider the input to be logits and will auto apply sigmoid per element. Additionally, we convert to int tensor with thresholding using the value in threshold.

  • target (int tensor): (N, ...).

  • groups (int tensor): (N, ...). The group identifiers should be 0, 1, ..., (num_groups - 1).

The additional dimensions are flatted along the batch dimension.

Parameters:

  • preds (Tensor) – Tensor with predictions.

  • target (Tensor) – Tensor with true labels.

  • groups (Tensor) – Tensor with group identifiers. The group identifiers should be 0, 1, ..., (num_groups - 1).

  • threshold (float) – Threshold for transforming probability to binary {0,1} predictions.

  • ignore_index (Optional[int]) – Specifies a target value that is ignored and does not contribute to the metric calculation

  • validate_args (bool) – bool indicating if input arguments and tensors should be validated for correctness. Set to False for faster computations.

Return type:

Dict[str, Tensor]

Returns:

The metric returns a dict where the key identifies the group with the lowest and highest true positives rates as follows: EO_{identifier_low_group}_{identifier_high_group}. The value is a tensor with the EO rate.

Example (preds is int tensor):
>>> from torchmetrics.functional.classification import equal_opportunity
>>> target = torch.tensor([0, 1, 0, 1, 0, 1])
>>> preds = torch.tensor([0, 1, 0, 1, 0, 1])
>>> groups = torch.tensor([0, 1, 0, 1, 0, 1])
>>> equal_opportunity(preds, target, groups)
{'EO_0_1': tensor(0.)}
Example (preds is float tensor):
>>> from torchmetrics.functional.classification import equal_opportunity
>>> target = torch.tensor([0, 1, 0, 1, 0, 1])
>>> preds = torch.tensor([0.11, 0.84, 0.22, 0.73, 0.33, 0.92])
>>> groups = torch.tensor([0, 1, 0, 1, 0, 1])
>>> equal_opportunity(preds, target, groups)
{'EO_0_1': tensor(0.)}

binary_groups_stat_rates

torchmetrics.functional.classification.binary_groups_stat_rates(preds, target, groups, num_groups, threshold=0.5, ignore_index=None, validate_args=True)[source]

Compute the true/false positives and true/false negatives rates for binary classification by group.

Related to Type I and Type II errors.

Accepts the following input tensors:

  • preds (int or float tensor): (N, ...). If preds is a floating point tensor with values outside [0,1] range we consider the input to be logits and will auto apply sigmoid per element. Additionally, we convert to int tensor with thresholding using the value in threshold.

  • target (int tensor): (N, ...).

  • groups (int tensor): (N, ...). The group identifiers should be 0, 1, ..., (num_groups - 1).

The additional dimensions are flatted along the batch dimension.

Parameters:

  • preds (Tensor) – Tensor with predictions.

  • target (Tensor) – Tensor with true labels.

  • groups (Tensor) – Tensor with group identifiers. The group identifiers should be 0, 1, ..., (num_groups - 1).

  • num_groups (int) – The number of groups.

  • threshold (float) – Threshold for transforming probability to binary {0,1} predictions.

  • ignore_index (Optional[int]) – Specifies a target value that is ignored and does not contribute to the metric calculation

  • validate_args (bool) – bool indicating if input arguments and tensors should be validated for correctness. Set to False for faster computations.

Return type:

Dict[str, Tensor]

Returns:

The metric returns a dict with a group identifier as key and a tensor with the tp, fp, tn and fn rates as value.

Example (preds is int tensor):
>>> from torchmetrics.functional.classification import binary_groups_stat_rates
>>> target = torch.tensor([0, 1, 0, 1, 0, 1])
>>> preds = torch.tensor([0, 1, 0, 1, 0, 1])
>>> groups = torch.tensor([0, 1, 0, 1, 0, 1])
>>> binary_groups_stat_rates(preds, target, groups, 2)
{'group_0': tensor([0., 0., 1., 0.]), 'group_1': tensor([1., 0., 0., 0.])}
Example (preds is float tensor):
>>> from torchmetrics.functional.classification import binary_groups_stat_rates
>>> target = torch.tensor([0, 1, 0, 1, 0, 1])
>>> preds = torch.tensor([0.11, 0.84, 0.22, 0.73, 0.33, 0.92])
>>> groups = torch.tensor([0, 1, 0, 1, 0, 1])
>>> binary_groups_stat_rates(preds, target, groups, 2)
{'group_0': tensor([0., 0., 1., 0.]), 'group_1': tensor([1., 0., 0., 0.])}

To analyze traffic and optimize your experience, we serve cookies on this site. By clicking or navigating, you agree to allow our usage of cookies. Read PyTorch Lightning's Privacy Policy.