Shortcuts

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

Note: we move fast, but still we preserve 0.1 version (one feature release) back compatibility.

[0.11.0] - 2022-11-30

[0.11.0] - Added

  • Added MulticlassExactMatch to classification metrics (#1343)

  • Added TotalVariation to image package (#978)

  • Added CLIPScore to new multimodal package (#1314)

  • Added regression metrics:

    • KendallRankCorrCoef (#1271)

    • LogCoshError (#1316)

  • Added new nominal metrics:

  • Added option to pass distributed_available_fn to metrics to allow checks for custom communication backend for making dist_sync_fn actually useful (#1301)

  • Added normalize argument to Inception, FID, KID metrics (#1246)

[0.11.0] - Changed

  • Changed minimum Pytorch version to be 1.8 (#1263)

  • Changed interface for all functional and modular classification metrics after refactor (#1252)

[0.11.0] - Removed

  • Removed deprecated BinnedAveragePrecision, BinnedPrecisionRecallCurve, RecallAtFixedPrecision (#1251)

  • Removed deprecated LabelRankingAveragePrecision, LabelRankingLoss and CoverageError (#1251)

  • Removed deprecated KLDivergence and AUC (#1251)

[0.11.0] - Fixed

  • Fixed precision bug in pairwise_euclidean_distance (#1352)

[0.10.3] - 2022-11-16

[0.10.3] - Fixed

  • Fixed bug in Metrictracker.best_metric when return_step=False (#1306)

  • Fixed bug to prevent users from going into an infinite loop if trying to iterate of a single metric (#1320)

  • Fixed bug in Metrictracker.best_metric when return_step=False (#1306)

[0.10.2] - 2022-10-31

[0.10.2] - Changed

  • Changed in-place operation to out-of-place operation in pairwise_cosine_similarity (#1288)

[0.10.2] - Fixed

  • Fixed high memory usage for certain classification metrics when average='micro' (#1286)

  • Fixed precision problems when structural_similarity_index_measure was used with autocast (#1291)

  • Fixed slow performance for confusion matrix based metrics (#1302)

  • Fixed restrictive dtype checking in spearman_corrcoef when used with autocast (#1303)

[0.10.1] - 2022-10-21

[0.10.1] - Fixed

  • Fixed broken clone method for classification metrics (#1250)

  • Fixed unintentional downloading of nltk.punkt when lsum not in rouge_keys (#1258)

  • Fixed type casting in MAP metric between bool and float32 (#1150)

[0.10.0] - 2022-10-04

[0.10.0] - Added

  • Added a new NLP metric InfoLM (#915)

  • Added Perplexity metric (#922)

  • Added ConcordanceCorrCoef metric to regression package (#1201)

  • Added argument normalize to LPIPS metric (#1216)

  • Added support for multiprocessing of batches in PESQ metric (#1227)

  • Added support for multioutput in PearsonCorrCoef and SpearmanCorrCoef (#1200)

[0.10.0] - Changed

[0.10.0] - Deprecated

  • Deprecated BinnedAveragePrecision, BinnedPrecisionRecallCurve, BinnedRecallAtFixedPrecision (#1163)

    • BinnedAveragePrecision -> use AveragePrecision with thresholds arg

    • BinnedPrecisionRecallCurve -> use AveragePrecisionRecallCurve with thresholds arg

    • BinnedRecallAtFixedPrecision -> use RecallAtFixedPrecision with thresholds arg

  • Renamed and refactored LabelRankingAveragePrecision, LabelRankingLoss and CoverageError (#1167)

    • LabelRankingAveragePrecision -> MultilabelRankingAveragePrecision

    • LabelRankingLoss -> MultilabelRankingLoss

    • CoverageError -> MultilabelCoverageError

  • Deprecated KLDivergence and AUC from classification package (#1189)

    • KLDivergence moved to regression package

    • Instead of AUC use torchmetrics.utils.compute.auc

[0.10.0] - Fixed

  • Fixed a bug in ssim when return_full_image=True where the score was still reduced (#1204)

  • Fixed MPS support for:

  • Fixed bug in ClasswiseWrapper such that compute gave wrong result (#1225)

  • Fixed synchronization of empty list states (#1219)

[0.9.3] - 2022-08-22

[0.9.3] - Added

  • Added global option sync_on_compute to disable automatic synchronization when compute is called (#1107)

[0.9.3] - Fixed

  • Fixed missing reset in ClasswiseWrapper (#1129)

  • Fixed JaccardIndex multi-label compute (#1125)

  • Fix SSIM propagate device if gaussian_kernel is False, add test (#1149)

[0.9.2] - 2022-06-29

[0.9.2] - Fixed

  • Fixed mAP calculation for areas with 0 predictions (#1080)

  • Fixed bug where avg precision state and auroc state was not merge when using MetricCollections (#1086)

  • Skip box conversion if no boxes are present in MeanAveragePrecision (#1097)

  • Fixed inconsistency in docs and code when setting average="none" in AvaragePrecision metric (#1116)

[0.9.1] - 2022-06-08

[0.9.1] - Added

  • Added specific RuntimeError when metric object is on the wrong device (#1056)

  • Added an option to specify own n-gram weights for BLEUScore and SacreBLEUScore instead of using uniform weights only. (#1075)

[0.9.1] - Fixed

  • Fixed aggregation metrics when input only contains zero (#1070)

  • Fixed TypeError when providing superclass arguments as kwargs (#1069)

  • Fixed bug related to state reference in metric collection when using compute groups (#1076)

[0.9.0] - 2022-05-30

[0.9.0] - Added

  • Added RetrievalPrecisionRecallCurve and RetrievalRecallAtFixedPrecision to retrieval package (#951)

  • Added class property full_state_update that determines forward should call update once or twice ( #984, #1033)

  • Added support for nested metric collections (#1003)

  • Added Dice to classification package (#1021)

  • Added support to segmentation type segm as IOU for mean average precision (#822)

[0.9.0] - Changed

  • Renamed reduction argument to average in Jaccard score and added additional options (#874)

[0.9.0] - Removed

[0.9.0] - Fixed

  • Fixed non-empty state dict for a few metrics (#1012)

  • Fixed bug when comparing states while finding compute groups (#1022)

  • Fixed torch.double support in stat score metrics (#1023)

  • Fixed FID calculation for non-equal size real and fake input (#1028)

  • Fixed case where KLDivergence could output Nan (#1030)

  • Fixed deterministic for PyTorch<1.8 (#1035)

  • Fixed default value for mdmc_average in Accuracy (#1036)

  • Fixed missing copy of property when using compute groups in MetricCollection (#1052)

[0.8.2] - 2022-05-06

[0.8.2] - Fixed

  • Fixed multi device aggregation in PearsonCorrCoef (#998)

  • Fixed MAP metric when using custom list of thresholds (#995)

  • Fixed compatibility between compute groups in MetricCollection and prefix/postfix arg (#1007)

  • Fixed compatibility with future Pytorch 1.12 in safe_matmul (#1011, #1014)

[0.8.1] - 2022-04-27

[0.8.1] - Changed

  • Reimplemented the signal_distortion_ratio metric, which removed the absolute requirement of fast-bss-eval (#964)

[0.8.1] - Fixed

  • Fixed “Sort currently does not support bool dtype on CUDA” error in MAP for empty preds (#983)

  • Fixed BinnedPrecisionRecallCurve when thresholds argument is not provided (#968)

  • Fixed CalibrationError to work on logit input (#985)

[0.8.0] - 2022-04-14

[0.8.0] - Added

  • Added WeightedMeanAbsolutePercentageError to regression package (#948)

  • Added new classification metrics:

    • CoverageError (#787)

    • LabelRankingAveragePrecision and LabelRankingLoss (#787)

  • Added new image metric:

    • SpectralAngleMapper (#885)

    • ErrorRelativeGlobalDimensionlessSynthesis (#894)

    • UniversalImageQualityIndex (#824)

    • SpectralDistortionIndex (#873)

  • Added support for MetricCollection in MetricTracker (#718)

  • Added support for 3D image and uniform kernel in StructuralSimilarityIndexMeasure (#818)

  • Added smart update of MetricCollection (#709)

  • Added ClasswiseWrapper for better logging of classification metrics with multiple output values (#832)

  • Added **kwargs argument for passing additional arguments to base class (#833)

  • Added negative ignore_index for the Accuracy metric (#362)

  • Added adaptive_k for the RetrievalPrecision metric (#910)

  • Added reset_real_features argument image quality assessment metrics (#722)

  • Added new keyword argument compute_on_cpu to all metrics (#867)

[0.8.0] - Changed

  • Made num_classes in jaccard_index a required argument (#853, #914)

  • Added normalizer, tokenizer to ROUGE metric (#838)

  • Improved shape checking of permutation_invariant_training (#864)

  • Allowed reduction None (#891)

  • MetricTracker.best_metric will now give a warning when computing on metric that do not have a best (#913)

[0.8.0] - Deprecated

  • Deprecated argument compute_on_step (#792)

  • Deprecated passing in dist_sync_on_step, process_group, dist_sync_fn direct argument (#833)

[0.8.0] - Removed

  • Removed support for versions of Pytorch-Lightning lower than v1.5 (#788)

  • Removed deprecated functions, and warnings in Text (#773)

    • WER and functional.wer

  • Removed deprecated functions and warnings in Image (#796)

    • SSIM and functional.ssim

    • PSNR and functional.psnr

  • Removed deprecated functions, and warnings in classification and regression (#806)

    • FBeta and functional.fbeta

    • F1 and functional.f1

    • Hinge and functional.hinge

    • IoU and functional.iou

    • MatthewsCorrcoef

    • PearsonCorrcoef

    • SpearmanCorrcoef

  • Removed deprecated functions, and warnings in detection and pairwise (#804)

    • MAP and functional.pairwise.manhatten

  • Removed deprecated functions, and warnings in Audio (#805)

    • PESQ and functional.audio.pesq

    • PIT and functional.audio.pit

    • SDR and functional.audio.sdr and functional.audio.si_sdr

    • SNR and functional.audio.snr and functional.audio.si_snr

    • STOI and functional.audio.stoi

  • Removed unused get_num_classes from torchmetrics.utilities.data (#914)

[0.8.0] - Fixed

  • Fixed device mismatch for MAP metric in specific cases (#950)

  • Improved testing speed (#820)

  • Fixed compatibility of ClasswiseWrapper with the prefix argument of MetricCollection (#843)

  • Fixed BestScore on GPU (#912)

  • Fixed Lsum computation for ROUGEScore (#944)

[0.7.3] - 2022-03-23

[0.7.3] - Fixed

  • Fixed unsafe log operation in TweedieDeviace for power=1 (#847)

  • Fixed bug in MAP metric related to either no ground truth or no predictions (#884)

  • Fixed ConfusionMatrix, AUROC and AveragePrecision on GPU when running in deterministic mode (#900)

  • Fixed NaN or Inf results returned by signal_distortion_ratio (#899)

  • Fixed memory leak when using update method with tensor where requires_grad=True (#902)

[0.7.2] - 2022-02-10

[0.7.2] - Fixed

  • Minor patches in JOSS paper.

[0.7.1] - 2022-02-03

[0.7.1] - Changed

  • Used torch.bucketize in calibration error when torch>1.8 for faster computations (#769)

  • Improve mAP performance (#742)

[0.7.1] - Fixed

  • Fixed check for available modules (#772)

  • Fixed Matthews correlation coefficient when the denominator is 0 (#781)

[0.7.0] - 2022-01-17

[0.7.0] - Added

  • Added NLP metrics:

    • MatchErrorRate (#619)

    • WordInfoLost and WordInfoPreserved (#630)

    • SQuAD (#623)

    • CHRFScore (#641)

    • TranslationEditRate (#646)

    • ExtendedEditDistance (#668)

  • Added MultiScaleSSIM into image metrics (#679)

  • Added Signal to Distortion Ratio (SDR) to audio package (#565)

  • Added MinMaxMetric to wrappers (#556)

  • Added ignore_index to retrieval metrics (#676)

  • Added support for multi references in ROUGEScore (#680)

  • Added a default VSCode devcontainer configuration (#621)

[0.7.0] - Changed

  • Scalar metrics will now consistently have additional dimensions squeezed (#622)

  • Metrics having third party dependencies removed from global import (#463)

  • Untokenized for BLEUScore input stay consistent with all the other text metrics (#640)

  • Arguments reordered for TER, BLEUScore, SacreBLEUScore, CHRFScore now expect input order as predictions first and target second (#696)

  • Changed dtype of metric state from torch.float to torch.long in ConfusionMatrix to accommodate larger values (#715)

  • Unify preds, target input argument’s naming across all text metrics (#723, #727)

    • bert, bleu, chrf, sacre_bleu, wip, wil, cer, ter, wer, mer, rouge, squad

[0.7.0] - Deprecated

  • Renamed IoU -> Jaccard Index (#662)

  • Renamed text WER metric (#714)

    • functional.wer -> functional.word_error_rate

    • WER -> WordErrorRate

  • Renamed correlation coefficient classes: (#710)

    • MatthewsCorrcoef -> MatthewsCorrCoef

    • PearsonCorrcoef -> PearsonCorrCoef

    • SpearmanCorrcoef -> SpearmanCorrCoef

  • Renamed audio STOI metric: (#753, #758)

    • audio.STOI to audio.ShortTimeObjectiveIntelligibility

    • functional.audio.stoi to functional.audio.short_time_objective_intelligibility

  • Renamed audio PESQ metrics: (#751)

    • functional.audio.pesq -> functional.audio.perceptual_evaluation_speech_quality

    • audio.PESQ -> audio.PerceptualEvaluationSpeechQuality

  • Renamed audio SDR metrics: (#711)

    • functional.sdr -> functional.signal_distortion_ratio

    • functional.si_sdr -> functional.scale_invariant_signal_distortion_ratio

    • SDR -> SignalDistortionRatio

    • SI_SDR -> ScaleInvariantSignalDistortionRatio

  • Renamed audio SNR metrics: (#712)

    • functional.snr -> functional.signal_distortion_ratio

    • functional.si_snr -> functional.scale_invariant_signal_noise_ratio

    • SNR -> SignalNoiseRatio

    • SI_SNR -> ScaleInvariantSignalNoiseRatio

  • Renamed F-score metrics: (#731, #740)

    • functional.f1 -> functional.f1_score

    • F1 -> F1Score

    • functional.fbeta -> functional.fbeta_score

    • FBeta -> FBetaScore

  • Renamed Hinge metric: (#734)

    • functional.hinge -> functional.hinge_loss

    • Hinge -> HingeLoss

  • Renamed image PSNR metrics (#732)

    • functional.psnr -> functional.peak_signal_noise_ratio

    • PSNR -> PeakSignalNoiseRatio

  • Renamed image PIT metric: (#737)

    • functional.pit -> functional.permutation_invariant_training

    • PIT -> PermutationInvariantTraining

  • Renamed image SSIM metric: (#747)

    • functional.ssim -> functional.scale_invariant_signal_noise_ratio

    • SSIM -> StructuralSimilarityIndexMeasure

  • Renamed detection MAP to MeanAveragePrecision metric (#754)

  • Renamed Fidelity & LPIPS image metric: (#752)

    • image.FID -> image.FrechetInceptionDistance

    • image.KID -> image.KernelInceptionDistance

    • image.LPIPS -> image.LearnedPerceptualImagePatchSimilarity

[0.7.0] - Removed

  • Removed embedding_similarity metric (#638)

  • Removed argument concatenate_texts from wer metric (#638)

  • Removed arguments newline_sep and decimal_places from rouge metric (#638)

[0.7.0] - Fixed

  • Fixed MetricCollection kwargs filtering when no kwargs are present in update signature (#707)

[0.6.2] - 2021-12-15

[0.6.2] - Fixed

  • Fixed torch.sort currently does not support bool dtype on CUDA (#665)

  • Fixed mAP properly checks if ground truths are empty (#684)

  • Fixed initialization of tensors to be on correct device for MAP metric (#673)

[0.6.1] - 2021-12-06

[0.6.1] - Changed

  • Migrate MAP metrics from pycocotools to PyTorch (#632)

  • Use torch.topk instead of torch.argsort in retrieval precision for speedup (#627)

[0.6.1] - Fixed

  • Fix empty predictions in MAP metric (#594, #610, #624)

  • Fix edge case of AUROC with average=weighted on GPU (#606)

  • Fixed forward in compositional metrics (#645)

[0.6.0] - 2021-10-28

[0.6.0] - Added

  • Added audio metrics:

    • Perceptual Evaluation of Speech Quality (PESQ) (#353)

    • Short-Time Objective Intelligibility (STOI) (#353)

  • Added Information retrieval metrics:

    • RetrievalRPrecision (#577)

    • RetrievalHitRate (#576)

  • Added NLP metrics:

    • SacreBLEUScore (#546)

    • CharErrorRate (#575)

  • Added other metrics:

    • Tweedie Deviance Score (#499)

    • Learned Perceptual Image Patch Similarity (LPIPS) (#431)

  • Added MAP (mean average precision) metric to new detection package (#467)

  • Added support for float targets in nDCG metric (#437)

  • Added average argument to AveragePrecision metric for reducing multi-label and multi-class problems (#477)

  • Added MultioutputWrapper (#510)

  • Added metric sweeping:

    • higher_is_better as constant attribute (#544)

    • higher_is_better to rest of codebase (#584)

  • Added simple aggregation metrics: SumMetric, MeanMetric, CatMetric, MinMetric, MaxMetric (#506)

  • Added pairwise submodule with metrics (#553)

    • pairwise_cosine_similarity

    • pairwise_euclidean_distance

    • pairwise_linear_similarity

    • pairwise_manhatten_distance

[0.6.0] - Changed

  • AveragePrecision will now as default output the macro average for multilabel and multiclass problems (#477)

  • half, double, float will no longer change the dtype of the metric states. Use metric.set_dtype instead (#493)

  • Renamed AverageMeter to MeanMetric (#506)

  • Changed is_differentiable from property to a constant attribute (#551)

  • ROC and AUROC will no longer throw an error when either the positive or negative class is missing. Instead return 0 score and give a warning

[0.6.0] - Deprecated

  • Deprecated functional.self_supervised.embedding_similarity in favour of new pairwise submodule

[0.6.0] - Removed

  • Removed dtype property (#493)

[0.6.0] - Fixed

  • Fixed bug in F1 with average='macro' and ignore_index!=None (#495)

  • Fixed bug in pit by using the returned first result to initialize device and type (#533)

  • Fixed SSIM metric using too much memory (#539)

  • Fixed bug where device property was not properly update when metric was a child of a module (#542)

[0.5.1] - 2021-08-30

[0.5.1] - Added

  • Added device and dtype properties (#462)

  • Added TextTester class for robustly testing text metrics (#450)

[0.5.1] - Changed

  • Added support for float targets in nDCG metric (#437)

[0.5.1] - Removed

  • Removed rouge-score as dependency for text package (#443)

  • Removed jiwer as dependency for text package (#446)

  • Removed bert-score as dependency for text package (#473)

[0.5.1] - Fixed

  • Fixed ranking of samples in SpearmanCorrCoef metric (#448)

  • Fixed bug where compositional metrics where unable to sync because of type mismatch (#454)

  • Fixed metric hashing (#478)

  • Fixed BootStrapper metrics not working on GPU (#462)

  • Fixed the semantic ordering of kernel height and width in SSIM metric (#474)

[0.5.0] - 2021-08-09

[0.5.0] - Added

  • Added Text-related (NLP) metrics:

  • Added MetricTracker wrapper metric for keeping track of the same metric over multiple epochs (#238)

  • Added other metrics:

    • Symmetric Mean Absolute Percentage error (SMAPE) (#375)

    • Calibration error (#394)

    • Permutation Invariant Training (PIT) (#384)

  • Added support in nDCG metric for target with values larger than 1 (#349)

  • Added support for negative targets in nDCG metric (#378)

  • Added None as reduction option in CosineSimilarity metric (#400)

  • Allowed passing labels in (n_samples, n_classes) to AveragePrecision (#386)

[0.5.0] - Changed

  • Moved psnr and ssim from functional.regression.* to functional.image.* (#382)

  • Moved image_gradient from functional.image_gradients to functional.image.gradients (#381)

  • Moved R2Score from regression.r2score to regression.r2 (#371)

  • Pearson metric now only store 6 statistics instead of all predictions and targets (#380)

  • Use torch.argmax instead of torch.topk when k=1 for better performance (#419)

  • Moved check for number of samples in R2 score to support single sample updating (#426)

[0.5.0] - Deprecated

  • Rename r2score >> r2_score and kldivergence >> kl_divergence in functional (#371)

  • Moved bleu_score from functional.nlp to functional.text.bleu (#360)

[0.5.0] - Removed

  • Removed restriction that threshold has to be in (0,1) range to support logit input ( #351 #401)

  • Removed restriction that preds could not be bigger than num_classes to support logit input (#357)

  • Removed module regression.psnr and regression.ssim (#382):

  • Removed (#379):

    • function functional.mean_relative_error

    • num_thresholds argument in BinnedPrecisionRecallCurve

[0.5.0] - Fixed

  • Fixed bug where classification metrics with average='macro' would lead to wrong result if a class was missing (#303)

  • Fixed weighted, multi-class AUROC computation to allow for 0 observations of some class, as contribution to final AUROC is 0 (#376)

  • Fixed that _forward_cache and _computed attributes are also moved to the correct device if metric is moved (#413)

  • Fixed calculation in IoU metric when using ignore_index argument (#328)

[0.4.1] - 2021-07-05

[0.4.1] - Changed

[0.4.1] - Fixed

  • Fixed DDP by is_sync logic to Metric (#339)

[0.4.0] - 2021-06-29

[0.4.0] - Added

  • Added Image-related metrics:

    • Fréchet inception distance (FID) (#213)

    • Kernel Inception Distance (KID) (#301)

    • Inception Score (#299)

    • KL divergence (#247)

  • Added Audio metrics: SNR, SI_SDR, SI_SNR (#292)

  • Added other metrics:

    • Cosine Similarity (#305)

    • Specificity (#210)

    • Mean Absolute Percentage error (MAPE) (#248)

  • Added add_metrics method to MetricCollection for adding additional metrics after initialization (#221)

  • Added pre-gather reduction in the case of dist_reduce_fx="cat" to reduce communication cost (#217)

  • Added better error message for AUROC when num_classes is not provided for multiclass input (#244)

  • Added support for unnormalized scores (e.g. logits) in Accuracy, Precision, Recall, FBeta, F1, StatScore, Hamming, ConfusionMatrix metrics (#200)

  • Added squared argument to MeanSquaredError for computing RMSE (#249)

  • Added is_differentiable property to ConfusionMatrix, F1, FBeta, Hamming, Hinge, IOU, MatthewsCorrcoef, Precision, Recall, PrecisionRecallCurve, ROC, StatScores (#253)

  • Added sync and sync_context methods for manually controlling when metric states are synced (#302)

[0.4.0] - Changed

  • Forward cache is reset when reset method is called (#260)

  • Improved per-class metric handling for imbalanced datasets for precision, recall, precision_recall, fbeta, f1, accuracy, and specificity (#204)

  • Decorated torch.jit.unused to MetricCollection forward (#307)

  • Renamed thresholds argument to binned metrics for manually controlling the thresholds (#322)

  • Extend typing (#324, #326, #327)

[0.4.0] - Deprecated

  • Deprecated functional.mean_relative_error, use functional.mean_absolute_percentage_error (#248)

  • Deprecated num_thresholds argument in BinnedPrecisionRecallCurve (#322)

[0.4.0] - Removed

  • Removed argument is_multiclass (#319)

[0.4.0] - Fixed

  • AUC can also support more dimensional inputs when all but one dimension are of size 1 (#242)

  • Fixed dtype of modular metrics after reset has been called (#243)

  • Fixed calculation in matthews_corrcoef to correctly match formula (#321)

[0.3.2] - 2021-05-10

[0.3.2] - Added

  • Added is_differentiable property:

    • To AUC, AUROC, CohenKappa and AveragePrecision (#178)

    • To PearsonCorrCoef, SpearmanCorrcoef, R2Score and ExplainedVariance (#225)

[0.3.2] - Changed

  • MetricCollection should return metrics with prefix on items(), keys() (#209)

  • Calling compute before update will now give warning (#164)

[0.3.2] - Removed

  • Removed numpy as direct dependency (#212)

[0.3.2] - Fixed

  • Fixed auc calculation and add tests (#197)

  • Fixed loading persisted metric states using load_state_dict() (#202)

  • Fixed PSNR not working with DDP (#214)

  • Fixed metric calculation with unequal batch sizes (#220)

  • Fixed metric concatenation for list states for zero-dim input (#229)

  • Fixed numerical instability in AUROC metric for large input (#230)

[0.3.1] - 2021-04-21

  • Cleaning remaining inconsistency and fix PL develop integration ( #191, #192, #193, #194 )

[0.3.0] - 2021-04-20

[0.3.0] - Added

  • Added BootStrapper to easily calculate confidence intervals for metrics (#101)

  • Added Binned metrics (#128)

  • Added metrics for Information Retrieval ((PL^5032)):

    • RetrievalMAP (PL^5032)

    • RetrievalMRR (#119)

    • RetrievalPrecision (#139)

    • RetrievalRecall (#146)

    • RetrievalNormalizedDCG (#160)

    • RetrievalFallOut (#161)

  • Added other metrics:

    • CohenKappa (#69)

    • MatthewsCorrcoef (#98)

    • PearsonCorrcoef (#157)

    • SpearmanCorrcoef (#158)

    • Hinge (#120)

  • Added average='micro' as an option in AUROC for multilabel problems (#110)

  • Added multilabel support to ROC metric (#114)

  • Added testing for half precision (#77, #135 )

  • Added AverageMeter for ad-hoc averages of values (#138)

  • Added prefix argument to MetricCollection (#70)

  • Added __getitem__ as metric arithmetic operation (#142)

  • Added property is_differentiable to metrics and test for differentiability (#154)

  • Added support for average, ignore_index and mdmc_average in Accuracy metric (#166)

  • Added postfix arg to MetricCollection (#188)

[0.3.0] - Changed

  • Changed ExplainedVariance from storing all preds/targets to tracking 5 statistics (#68)

  • Changed behaviour of confusionmatrix for multilabel data to better match multilabel_confusion_matrix from sklearn (#134)

  • Updated FBeta arguments (#111)

  • Changed reset method to use detach.clone() instead of deepcopy when resetting to default (#163)

  • Metrics passed as dict to MetricCollection will now always be in deterministic order (#173)

  • Allowed MetricCollection pass metrics as arguments (#176)

[0.3.0] - Deprecated

  • Rename argument is_multiclass -> multiclass (#162)

[0.3.0] - Removed

  • Prune remaining deprecated (#92)

[0.3.0] - Fixed

  • Fixed when _stable_1d_sort to work when n>=N (PL^6177)

  • Fixed _computed attribute not being correctly reset (#147)

  • Fixed to Blau score (#165)

  • Fixed backwards compatibility for logging with older version of pytorch-lightning (#182)

[0.2.0] - 2021-03-12

[0.2.0] - Changed

  • Decoupled PL dependency (#13)

  • Refactored functional - mimic the module-like structure: classification, regression, etc. (#16)

  • Refactored utilities - split to topics/submodules (#14)

  • Refactored MetricCollection (#19)

[0.2.0] - Removed

  • Removed deprecated metrics from PL base (#12, #15)

[0.1.0] - 2021-02-22

  • Added Accuracy metric now generalizes to Top-k accuracy for (multi-dimensional) multi-class inputs using the top_k parameter (PL^4838)

  • Added Accuracy metric now enables the computation of subset accuracy for multi-label or multi-dimensional multi-class inputs with the subset_accuracy parameter (PL^4838)

  • Added HammingDistance metric to compute the hamming distance (loss) (PL^4838)

  • Added StatScores metric to compute the number of true positives, false positives, true negatives and false negatives (PL^4839)

  • Added R2Score metric (PL^5241)

  • Added MetricCollection (PL^4318)

  • Added .clone() method to metrics (PL^4318)

  • Added IoU class interface (PL^4704)

  • The Recall and Precision metrics (and their functional counterparts recall and precision) can now be generalized to Recall@K and Precision@K with the use of top_k parameter (PL^4842)

  • Added compositional metrics (PL^5464)

  • Added AUC/AUROC class interface (PL^5479)

  • Added QuantizationAwareTraining callback (PL^5706)

  • Added ConfusionMatrix class interface (PL^4348)

  • Added multiclass AUROC metric (PL^4236)

  • Added PrecisionRecallCurve, ROC, AveragePrecision class metric (PL^4549)

  • Classification metrics overhaul (PL^4837)

  • Added F1 class metric (PL^4656)

  • Added metrics aggregation in Horovod and fixed early stopping (PL^3775)

  • Added persistent(mode) method to metrics, to enable and disable metric states being added to state_dict (PL^4482)

  • Added unification of regression metrics (PL^4166)

  • Added persistent flag to Metric.add_state (PL^4195)

  • Added classification metrics (PL^4043)

  • Added new Metrics API. (PL^3868, PL^3921)

  • Added EMB similarity (PL^3349)

  • Added SSIM metrics (PL^2671)

  • Added BLEU metrics (PL^2535)