Precision Recall Curve¶

Module Interface¶

class torchmetrics.RetrievalPrecisionRecallCurve(max_k=None, adaptive_k=False, empty_target_action='neg', ignore_index=None, **kwargs)[source]

Computes precision-recall pairs for different k (from 1 to max_k).

In a ranked retrieval context, appropriate sets of retrieved documents are naturally given by the top k retrieved documents.

Recall is the fraction of relevant documents retrieved among all the relevant documents. Precision is the fraction of relevant documents among all the retrieved documents.

For each such set, precision and recall values can be plotted to give a recall-precision curve.

Forward accepts:

preds (float tensor): (N, ...)
target (long or bool tensor): (N, ...)
indexes (long tensor): (N, ...)

indexes, preds and target must have the same dimension. indexes indicate to which query a prediction belongs. Predictions will be first grouped by indexes and then RetrievalRecallAtFixedPrecision will be computed as the mean of the RetrievalRecallAtFixedPrecision over each query.

Parameters

max_k¶ (Optional[int]) – Calculate recall and precision for all possible top k from 1 to max_k (default: None, which considers all possible top k)
adaptive_k¶ (bool) – adjust k to min(k, number of documents) for each query
empty_target_action¶ (str) –
Specify what to do with queries that do not have at least a positive target. Choose from:
- 'neg': those queries count as 0.0 (default)
- 'pos': those queries count as 1.0
- 'skip': skip those queries; if all queries are skipped, 0.0 is returned
- 'error': raise a ValueError
ignore_index¶ (Optional[int]) – Ignore predictions where the target is equal to this number.
kwargs¶ (Any) – Additional keyword arguments, see Advanced metric settings for more info.

Raises

ValueError – If empty_target_action is not one of error, skip, neg or pos.
ValueError – If ignore_index is not None or an integer.
ValueError – If max_k parameter is not None or an integer larger than 0.

Example

>>> from torchmetrics import RetrievalPrecisionRecallCurve
>>> indexes = tensor([0, 0, 0, 0, 1, 1, 1])
>>> preds = tensor([0.4, 0.01, 0.5, 0.6, 0.2, 0.3, 0.5])
>>> target = tensor([True, False, False, True, True, False, True])
>>> r = RetrievalPrecisionRecallCurve(max_k=4)
>>> precisions, recalls, top_k = r(preds, target, indexes=indexes)
>>> precisions
tensor([1.0000, 0.5000, 0.6667, 0.5000])
>>> recalls
tensor([0.5000, 0.5000, 1.0000, 1.0000])
>>> top_k
tensor([1, 2, 3, 4])

Initializes internal Module state, shared by both nn.Module and ScriptModule.

compute()[source]

Override this method to compute the final metric value from state variables synchronized across the distributed backend.

Return type: Tuple[Tensor, Tensor, Tensor]

update(preds, target, indexes)[source]

Check shape, check and convert dtypes, flatten and add to accumulators.

Return type: None

Functional Interface¶

torchmetrics.functional.retrieval_precision_recall_curve(preds, target, max_k=None, adaptive_k=False)[source]

Computes precision-recall pairs for different k (from 1 to max_k).

In a ranked retrieval context, appropriate sets of retrieved documents are naturally given by the top k retrieved documents.

Recall is the fraction of relevant documents retrieved among all the relevant documents. Precision is the fraction of relevant documents among all the retrieved documents.

For each such set, precision and recall values can be plotted to give a recall-precision curve.

preds and target should be of the same shape and live on the same device. If no target is True, 0 is returned. target must be either bool or integers and preds must be float, otherwise an error is raised.

Parameters

preds¶ (Tensor) – estimated probabilities of each document to be relevant.
target¶ (Tensor) – ground truth about each document being relevant or not.
max_k¶ (Optional[int]) – Calculate recall and precision for all possible top k from 1 to max_k (default: None, which considers all possible top k)
adaptive_k¶ (bool) – adjust max_k to min(max_k, number of documents) for each query

Return type

Tuple[Tensor, Tensor, Tensor]

Returns

tensor with the precision values for each k (at k) from 1 to max_k tensor with the recall values for each k (at k) from 1 to max_k tensor with all possibles k

Raises

ValueError – If max_k is not None or an integer larger than 0.
ValueError – If adaptive_k is not boolean.

Example

>>> from  torchmetrics.functional import retrieval_precision_recall_curve
>>> preds = tensor([0.2, 0.3, 0.5])
>>> target = tensor([True, False, True])
>>> precisions, recalls, top_k = retrieval_precision_recall_curve(preds, target, max_k=2)
>>> precisions
tensor([1.0000, 0.5000])
>>> recalls
tensor([0.5000, 0.5000])
>>> top_k
tensor([1, 2])