Shortcuts

SQuAD

Module Interface

class torchmetrics.SQuAD(**kwargs)[source]

Calculate SQuAD Metric which corresponds to the scoring script for version 1 of the Stanford Question Answering Dataset (SQuAD).

As input to forward and update the metric accepts the following input:

  • preds (Dict): A Dictionary or List of Dictionary-s that map id and prediction_text to the respective values

    Example prediction:

    {"prediction_text": "TorchMetrics is awesome", "id": "123"}
    
  • target (Dict): A Dictionary or List of Dictionary-s that contain the answers and id in the SQuAD Format.

    Example target:

    {
        'answers': [{'answer_start': [1], 'text': ['This is a test answer']}],
        'id': '1',
    }
    

    Reference SQuAD Format:

    {
        'answers': {'answer_start': [1], 'text': ['This is a test text']},
        'context': 'This is a test context.',
        'id': '1',
        'question': 'Is this a test?',
        'title': 'train test'
    }
    

As output of forward and compute the metric returns the following output:

  • squad (Dict): A dictionary containing the F1 score (key: “f1”),

    and Exact match score (key: “exact_match”) for the batch.

Parameters

kwargs (Any) – Additional keyword arguments, see Advanced metric settings for more info.

Example

>>> from torchmetrics import SQuAD
>>> preds = [{"prediction_text": "1976", "id": "56e10a3be3433e1400422b22"}]
>>> target = [{"answers": {"answer_start": [97], "text": ["1976"]}, "id": "56e10a3be3433e1400422b22"}]
>>> squad = SQuAD()
>>> squad(preds, target)
{'exact_match': tensor(100.), 'f1': tensor(100.)}

Initializes internal Module state, shared by both nn.Module and ScriptModule.

Functional Interface

torchmetrics.functional.squad(preds, target)[source]

Calculate SQuAD Metric .

Parameters
  • preds (Union[Dict[str, str], List[Dict[str, str]]]) –

    A Dictionary or List of Dictionary-s that map id and prediction_text to the respective values.

    Example prediction:

    {"prediction_text": "TorchMetrics is awesome", "id": "123"}
    

  • target (Union[Dict[str, Union[str, Dict[str, Union[List[str], List[int]]]]], List[Dict[str, Union[str, Dict[str, Union[List[str], List[int]]]]]]]) –

    A Dictionary or List of Dictionary-s that contain the answers and id in the SQuAD Format.

    Example target:

    {
        'answers': [{'answer_start': [1], 'text': ['This is a test answer']}],
        'id': '1',
    }
    

    Reference SQuAD Format:

    {
        'answers': {'answer_start': [1], 'text': ['This is a test text']},
        'context': 'This is a test context.',
        'id': '1',
        'question': 'Is this a test?',
        'title': 'train test'
    }
    

Return type

Dict[str, Tensor]

Returns

Dictionary containing the F1 score, Exact match score for the batch.

Example

>>> from torchmetrics.functional.text.squad import squad
>>> preds = [{"prediction_text": "1976", "id": "56e10a3be3433e1400422b22"}]
>>> target = [{"answers": {"answer_start": [97], "text": ["1976"]},"id": "56e10a3be3433e1400422b22"}]
>>> squad(preds, target)
{'exact_match': tensor(100.), 'f1': tensor(100.)}
Raises

KeyError – If the required keys are missing in either predictions or targets.

References

[1] SQuAD: 100,000+ Questions for Machine Comprehension of Text by Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, Percy Liang SQuAD Metric .

Read the Docs v: latest
Versions
latest
stable
v0.11.1
v0.11.0
v0.10.3
v0.10.2
v0.10.1
v0.10.0
v0.9.3
v0.9.2
v0.9.1
v0.9.0
v0.8.2
v0.8.1
v0.8.0
v0.7.3
v0.7.2
v0.7.1
v0.7.0
v0.6.2
v0.6.1
v0.6.0
v0.5.1
v0.5.0
v0.4.1
v0.4.0
v0.3.2
v0.3.1
v0.3.0
v0.2.0
v0.1.0
Downloads
pdf
html
On Read the Docs
Project Home
Builds

Free document hosting provided by Read the Docs.