SQuAD¶
Module Interface¶
- class torchmetrics.SQuAD(**kwargs)[source]
Calculate SQuAD Metric which corresponds to the scoring script for version 1 of the Stanford Question Answering Dataset (SQuAD).
- Parameters
kwargs¶ (
Any
) – Additional keyword arguments, see Advanced metric settings for more info.
Example
>>> from torchmetrics import SQuAD >>> preds = [{"prediction_text": "1976", "id": "56e10a3be3433e1400422b22"}] >>> target = [{"answers": {"answer_start": [97], "text": ["1976"]}, "id": "56e10a3be3433e1400422b22"}] >>> squad = SQuAD() >>> squad(preds, target) {'exact_match': tensor(100.), 'f1': tensor(100.)}
References
[1] SQuAD: 100,000+ Questions for Machine Comprehension of Text by Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, Percy Liang SQuAD Metric .
Initializes internal Module state, shared by both nn.Module and ScriptModule.
- compute()[source]
Aggregate the F1 Score and Exact match for the batch.
- update(preds, target)[source]
Compute F1 Score and Exact Match for a collection of predictions and references.
- Parameters
preds¶ (
Union
[Dict
[str
,str
],List
[Dict
[str
,str
]]]) –A Dictionary or List of Dictionary-s that map
id
andprediction_text
to the respective values. Example prediction:{"prediction_text": "TorchMetrics is awesome", "id": "123"}
target¶ (
Union
[Dict
[str
,Union
[str
,Dict
[str
,Union
[List
[str
],List
[int
]]]]],List
[Dict
[str
,Union
[str
,Dict
[str
,Union
[List
[str
],List
[int
]]]]]]]) –A Dictionary or List of Dictionary-s that contain the
answers
andid
in the SQuAD Format. Example target:{ 'answers': [{'answer_start': [1], 'text': ['This is a test answer']}], 'id': '1', }
Reference SQuAD Format:
{ 'answers': {'answer_start': [1], 'text': ['This is a test text']}, 'context': 'This is a test context.', 'id': '1', 'question': 'Is this a test?', 'title': 'train test' }
- Raises
KeyError – If the required keys are missing in either predictions or targets.
- Return type
Functional Interface¶
- torchmetrics.functional.squad(preds, target)[source]
Calculate SQuAD Metric .
- Parameters
preds¶ (
Union
[Dict
[str
,str
],List
[Dict
[str
,str
]]]) –A Dictionary or List of Dictionary-s that map id and prediction_text to the respective values.
Example prediction:
{"prediction_text": "TorchMetrics is awesome", "id": "123"}
target¶ (
Union
[Dict
[str
,Union
[str
,Dict
[str
,Union
[List
[str
],List
[int
]]]]],List
[Dict
[str
,Union
[str
,Dict
[str
,Union
[List
[str
],List
[int
]]]]]]]) –A Dictionary or List of Dictionary-s that contain the answers and id in the SQuAD Format.
Example target:
{ 'answers': [{'answer_start': [1], 'text': ['This is a test answer']}], 'id': '1', }
Reference SQuAD Format:
{ 'answers': {'answer_start': [1], 'text': ['This is a test text']}, 'context': 'This is a test context.', 'id': '1', 'question': 'Is this a test?', 'title': 'train test' }
- Return type
- Returns
Dictionary containing the F1 score, Exact match score for the batch.
Example
>>> from torchmetrics.functional.text.squad import squad >>> preds = [{"prediction_text": "1976", "id": "56e10a3be3433e1400422b22"}] >>> target = [{"answers": {"answer_start": [97], "text": ["1976"]},"id": "56e10a3be3433e1400422b22"}] >>> squad(preds, target) {'exact_match': tensor(100.), 'f1': tensor(100.)}
- Raises
KeyError – If the required keys are missing in either predictions or targets.
References
[1] SQuAD: 100,000+ Questions for Machine Comprehension of Text by Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, Percy Liang SQuAD Metric .