Shortcuts

SQuAD

Module Interface

class torchmetrics.SQuAD(compute_on_step=None, **kwargs)[source]

Calculate SQuAD Metric which corresponds to the scoring script for version 1 of the Stanford Question Answering Dataset (SQuAD).

Parameters
  • compute_on_step (Optional[bool]) –

    Forward only calls update() and returns None if this is set to False.

    Deprecated since version v0.8: Argument has no use anymore and will be removed v0.9.

  • kwargs (Dict[str, Any]) – Additional keyword arguments, see Advanced metric settings for more info.

Example

>>> from torchmetrics import SQuAD
>>> preds = [{"prediction_text": "1976", "id": "56e10a3be3433e1400422b22"}]
>>> target = [{"answers": {"answer_start": [97], "text": ["1976"]}, "id": "56e10a3be3433e1400422b22"}]
>>> squad = SQuAD()
>>> squad(preds, target)
{'exact_match': tensor(100.), 'f1': tensor(100.)}

References

[1] SQuAD: 100,000+ Questions for Machine Comprehension of Text by Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, Percy Liang SQuAD Metric .

Initializes internal Module state, shared by both nn.Module and ScriptModule.

compute()[source]

Aggregate the F1 Score and Exact match for the batch.

Return type

Dict[str, Tensor]

Returns

Dictionary containing the F1 score, Exact match score for the batch.

update(preds, target)[source]

Compute F1 Score and Exact Match for a collection of predictions and references.

Parameters
  • preds (Union[Dict[str, str], List[Dict[str, str]]]) –

    A Dictionary or List of Dictionary-s that map id and prediction_text to the respective values. Example prediction:

    {"prediction_text": "TorchMetrics is awesome", "id": "123"}
    

  • target (Union[Dict[str, Union[str, Dict[str, Union[List[str], List[int]]]]], List[Dict[str, Union[str, Dict[str, Union[List[str], List[int]]]]]]]) –

    A Dictionary or List of Dictionary-s that contain the answers and id in the SQuAD Format. Example target:

    {
        'answers': [{'answer_start': [1], 'text': ['This is a test answer']}],
        'id': '1',
    }
    

    Reference SQuAD Format:

    {
        'answers': {'answer_start': [1], 'text': ['This is a test text']},
        'context': 'This is a test context.',
        'id': '1',
        'question': 'Is this a test?',
        'title': 'train test'
    }
    

Raises

KeyError – If the required keys are missing in either predictions or targets.

Return type

None

Functional Interface

torchmetrics.functional.squad(preds, target)[source]

Calculate SQuAD Metric .

Parameters
  • preds (Union[Dict[str, str], List[Dict[str, str]]]) –

    A Dictionary or List of Dictionary-s that map id and prediction_text to the respective values.

    Example prediction:

    {"prediction_text": "TorchMetrics is awesome", "id": "123"}
    

  • target (Union[Dict[str, Union[str, Dict[str, Union[List[str], List[int]]]]], List[Dict[str, Union[str, Dict[str, Union[List[str], List[int]]]]]]]) –

    A Dictionary or List of Dictionary-s that contain the answers and id in the SQuAD Format.

    Example target:

    {
        'answers': [{'answer_start': [1], 'text': ['This is a test answer']}],
        'id': '1',
    }
    

    Reference SQuAD Format:

    {
        'answers': {'answer_start': [1], 'text': ['This is a test text']},
        'context': 'This is a test context.',
        'id': '1',
        'question': 'Is this a test?',
        'title': 'train test'
    }
    

Return type

Dict[str, Tensor]

Returns

Dictionary containing the F1 score, Exact match score for the batch.

Example

>>> from torchmetrics.functional.text.squad import squad
>>> preds = [{"prediction_text": "1976", "id": "56e10a3be3433e1400422b22"}]
>>> target = [{"answers": {"answer_start": [97], "text": ["1976"]},"id": "56e10a3be3433e1400422b22"}]
>>> squad(preds, target)
{'exact_match': tensor(100.), 'f1': tensor(100.)}
Raises

KeyError – If the required keys are missing in either predictions or targets.

References

[1] SQuAD: 100,000+ Questions for Machine Comprehension of Text by Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, Percy Liang SQuAD Metric .

Read the Docs v: v0.8.2
Versions
latest
stable
v0.8.2
v0.8.1
v0.8.0
v0.7.3
v0.7.2
v0.7.1
v0.7.0
v0.6.2
v0.6.1
v0.6.0
v0.5.1
v0.5.0
v0.4.1
v0.4.0
v0.3.2
v0.3.1
v0.3.0
v0.2.0
v0.1.0
Downloads
On Read the Docs
Project Home
Builds

Free document hosting provided by Read the Docs.