Shortcuts

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

Note: we move fast, but still we preserve 0.1 version (one feature release) back compatibility.

[0.8.2] - 2022-05-06

[0.8.2] - Fixed

  • Fixed multi device aggregation in PearsonCorrCoef (#998)

  • Fixed MAP metric when using custom list of thresholds (#995)

  • Fixed compatibility between compute groups in MetricCollection and prefix/postfix arg (#1007)

  • Fixed compatibility with future Pytorch 1.12 in safe_matmul (#1011, #1014)

[0.8.1] - 2022-04-27

[0.8.1] - Changed

  • Reimplemented the signal_distortion_ratio metric, which removed the absolute requirement of fast-bss-eval (#964)

[0.8.1] - Fixed

  • Fixed “Sort currently does not support bool dtype on CUDA” error in MAP for empty preds (#983)

  • Fixed BinnedPrecisionRecallCurve when thresholds argument is not provided (#968)

  • Fixed CalibrationError to work on logit input (#985)

[0.8.0] - 2022-04-14

[0.8.0] - Added

  • Added WeightedMeanAbsolutePercentageError to regression package (#948)

  • Added new classification metrics:

    • CoverageError (#787)

    • LabelRankingAveragePrecision and LabelRankingLoss (#787)

  • Added new image metric:

    • SpectralAngleMapper (#885)

    • ErrorRelativeGlobalDimensionlessSynthesis (#894)

    • UniversalImageQualityIndex (#824)

    • SpectralDistortionIndex (#873)

  • Added support for MetricCollection in MetricTracker (#718)

  • Added support for 3D image and uniform kernel in StructuralSimilarityIndexMeasure (#818)

  • Added smart update of MetricCollection (#709)

  • Added ClasswiseWrapper for better logging of classification metrics with multiple output values (#832)

  • Added **kwargs argument for passing additional arguments to base class (#833)

  • Added negative ignore_index for the Accuracy metric (#362)

  • Added adaptive_k for the RetrievalPrecision metric (#910)

  • Added reset_real_features argument image quality assessment metrics (#722)

  • Added new keyword argument compute_on_cpu to all metrics (#867)

[0.8.0] - Changed

  • Made num_classes in jaccard_index a required argument (#853, #914)

  • Added normalizer, tokenizer to ROUGE metric (#838)

  • Improved shape checking of permutation_invariant_training (#864)

  • Allowed reduction None (#891)

  • MetricTracker.best_metric will now give a warning when computing on metric that do not have a best (#913)

[0.8.0] - Deprecated

  • Deprecated argument compute_on_step (#792)

  • Deprecated passing in dist_sync_on_step, process_group, dist_sync_fn direct argument (#833)

[0.8.0] - Removed

  • Removed support for versions of Pytorch-Lightning lower than v1.5 (#788)

  • Removed deprecated functions, and warnings in Text (#773)

    • WER and functional.wer

  • Removed deprecated functions and warnings in Image (#796)

    • SSIM and functional.ssim

    • PSNR and functional.psnr

  • Removed deprecated functions, and warnings in classification and regression (#806)

    • FBeta and functional.fbeta

    • F1 and functional.f1

    • Hinge and functional.hinge

    • IoU and functional.iou

    • MatthewsCorrcoef

    • PearsonCorrcoef

    • SpearmanCorrcoef

  • Removed deprecated functions, and warnings in detection and pairwise (#804)

    • MAP and functional.pairwise.manhatten

  • Removed deprecated functions, and warnings in Audio (#805)

    • PESQ and functional.audio.pesq

    • PIT and functional.audio.pit

    • SDR and functional.audio.sdr and functional.audio.si_sdr

    • SNR and functional.audio.snr and functional.audio.si_snr

    • STOI and functional.audio.stoi

  • Removed unused get_num_classes from torchmetrics.utilities.data (#914)

[0.8.0] - Fixed

  • Fixed device mismatch for MAP metric in specific cases (#950)

  • Improved testing speed (#820)

  • Fixed compatibility of ClasswiseWrapper with the prefix argument of MetricCollection (#843)

  • Fixed BestScore on GPU (#912)

  • Fixed Lsum computation for ROUGEScore (#944)

[0.7.3] - 2022-03-23

[0.7.3] - Fixed

  • Fixed unsafe log operation in TweedieDeviace for power=1 (#847)

  • Fixed bug in MAP metric related to either no ground truth or no predictions (#884)

  • Fixed ConfusionMatrix, AUROC and AveragePrecision on GPU when running in deterministic mode (#900)

  • Fixed NaN or Inf results returned by signal_distortion_ratio (#899)

  • Fixed memory leak when using update method with tensor where requires_grad=True (#902)

[0.7.2] - 2022-02-10

[0.7.2] - Fixed

  • Minor patches in JOSS paper.

[0.7.1] - 2022-02-03

[0.7.1] - Changed

  • Used torch.bucketize in calibration error when torch>1.8 for faster computations (#769)

  • Improve mAP performance (#742)

[0.7.1] - Fixed

  • Fixed check for available modules (#772)

  • Fixed Matthews correlation coefficient when the denominator is 0 (#781)

[0.7.0] - 2022-01-17

[0.7.0] - Added

  • Added NLP metrics:

    • MatchErrorRate (#619)

    • WordInfoLost and WordInfoPreserved (#630)

    • SQuAD (#623)

    • CHRFScore (#641)

    • TranslationEditRate (#646)

    • ExtendedEditDistance (#668)

  • Added MultiScaleSSIM into image metrics (#679)

  • Added Signal to Distortion Ratio (SDR) to audio package (#565)

  • Added MinMaxMetric to wrappers (#556)

  • Added ignore_index to retrieval metrics (#676)

  • Added support for multi references in ROUGEScore (#680)

  • Added a default VSCode devcontainer configuration (#621)

[0.7.0] - Changed

  • Scalar metrics will now consistently have additional dimensions squeezed (#622)

  • Metrics having third party dependencies removed from global import (#463)

  • Untokenized for BLEUScore input stay consistent with all the other text metrics (#640)

  • Arguments reordered for TER, BLEUScore, SacreBLEUScore, CHRFScore now expect input order as predictions first and target second (#696)

  • Changed dtype of metric state from torch.float to torch.long in ConfusionMatrix to accommodate larger values (#715)

  • Unify preds, target input argument’s naming across all text metrics (#723, #727)

    • bert, bleu, chrf, sacre_bleu, wip, wil, cer, ter, wer, mer, rouge, squad

[0.7.0] - Deprecated

  • Renamed IoU -> Jaccard Index (#662)

  • Renamed text WER metric (#714)

    • functional.wer -> functional.word_error_rate

    • WER -> WordErrorRate

  • Renamed correlation coefficient classes: (#710)

    • MatthewsCorrcoef -> MatthewsCorrCoef

    • PearsonCorrcoef -> PearsonCorrCoef

    • SpearmanCorrcoef -> SpearmanCorrCoef

  • Renamed audio STOI metric: (#753, #758)

    • audio.STOI to audio.ShortTimeObjectiveIntelligibility

    • functional.audio.stoi to functional.audio.short_time_objective_intelligibility

  • Renamed audio PESQ metrics: (#751)

    • functional.audio.pesq -> functional.audio.perceptual_evaluation_speech_quality

    • audio.PESQ -> audio.PerceptualEvaluationSpeechQuality

  • Renamed audio SDR metrics: (#711)

    • functional.sdr -> functional.signal_distortion_ratio

    • functional.si_sdr -> functional.scale_invariant_signal_distortion_ratio

    • SDR -> SignalDistortionRatio

    • SI_SDR -> ScaleInvariantSignalDistortionRatio

  • Renamed audio SNR metrics: (#712)

    • functional.snr -> functional.signal_distortion_ratio

    • functional.si_snr -> functional.scale_invariant_signal_noise_ratio

    • SNR -> SignalNoiseRatio

    • SI_SNR -> ScaleInvariantSignalNoiseRatio

  • Renamed F-score metrics: (#731, #740)

    • functional.f1 -> functional.f1_score

    • F1 -> F1Score

    • functional.fbeta -> functional.fbeta_score

    • FBeta -> FBetaScore

  • Renamed Hinge metric: (#734)

    • functional.hinge -> functional.hinge_loss

    • Hinge -> HingeLoss

  • Renamed image PSNR metrics (#732)

    • functional.psnr -> functional.peak_signal_noise_ratio

    • PSNR -> PeakSignalNoiseRatio

  • Renamed image PIT metric: (#737)

    • functional.pit -> functional.permutation_invariant_training

    • PIT -> PermutationInvariantTraining

  • Renamed image SSIM metric: (#747)

    • functional.ssim -> functional.scale_invariant_signal_noise_ratio

    • SSIM -> StructuralSimilarityIndexMeasure

  • Renamed detection MAP to MeanAveragePrecision metric (#754)

  • Renamed Fidelity & LPIPS image metric: (#752)

    • image.FID -> image.FrechetInceptionDistance

    • image.KID -> image.KernelInceptionDistance

    • image.LPIPS -> image.LearnedPerceptualImagePatchSimilarity

[0.7.0] - Removed

  • Removed embedding_similarity metric (#638)

  • Removed argument concatenate_texts from wer metric (#638)

  • Removed arguments newline_sep and decimal_places from rouge metric (#638)

[0.7.0] - Fixed

  • Fixed MetricCollection kwargs filtering when no kwargs are present in update signature (#707)

[0.6.2] - 2021-12-15

[0.6.2] - Fixed

  • Fixed torch.sort currently does not support bool dtype on CUDA (#665)

  • Fixed mAP properly checks if ground truths are empty (#684)

  • Fixed initialization of tensors to be on correct device for MAP metric (#673)

[0.6.1] - 2021-12-06

[0.6.1] - Changed

  • Migrate MAP metrics from pycocotools to PyTorch (#632)

  • Use torch.topk instead of torch.argsort in retrieval precision for speedup (#627)

[0.6.1] - Fixed

  • Fix empty predictions in MAP metric (#594, #610, #624)

  • Fix edge case of AUROC with average=weighted on GPU (#606)

  • Fixed forward in compositional metrics (#645)

[0.6.0] - 2021-10-28

[0.6.0] - Added

  • Added audio metrics:

    • Perceptual Evaluation of Speech Quality (PESQ) (#353)

    • Short-Time Objective Intelligibility (STOI) (#353)

  • Added Information retrieval metrics:

    • RetrievalRPrecision (#577)

    • RetrievalHitRate (#576)

  • Added NLP metrics:

    • SacreBLEUScore (#546)

    • CharErrorRate (#575)

  • Added other metrics:

    • Tweedie Deviance Score (#499)

    • Learned Perceptual Image Patch Similarity (LPIPS) (#431)

  • Added MAP (mean average precision) metric to new detection package (#467)

  • Added support for float targets in nDCG metric (#437)

  • Added average argument to AveragePrecision metric for reducing multi-label and multi-class problems (#477)

  • Added MultioutputWrapper (#510)

  • Added metric sweeping:

    • higher_is_better as constant attribute (#544)

    • higher_is_better to rest of codebase (#584)

  • Added simple aggregation metrics: SumMetric, MeanMetric, CatMetric, MinMetric, MaxMetric (#506)

  • Added pairwise submodule with metrics (#553)

    • pairwise_cosine_similarity

    • pairwise_euclidean_distance

    • pairwise_linear_similarity

    • pairwise_manhatten_distance

[0.6.0] - Changed

  • AveragePrecision will now as default output the macro average for multilabel and multiclass problems (#477)

  • half, double, float will no longer change the dtype of the metric states. Use metric.set_dtype instead (#493)

  • Renamed AverageMeter to MeanMetric (#506)

  • Changed is_differentiable from property to a constant attribute (#551)

  • ROC and AUROC will no longer throw an error when either the positive or negative class is missing. Instead return 0 score and give a warning

[0.6.0] - Deprecated

  • Deprecated functional.self_supervised.embedding_similarity in favour of new pairwise submodule

[0.6.0] - Removed

  • Removed dtype property (#493)

[0.6.0] - Fixed

  • Fixed bug in F1 with average='macro' and ignore_index!=None (#495)

  • Fixed bug in pit by using the returned first result to initialize device and type (#533)

  • Fixed SSIM metric using too much memory (#539)

  • Fixed bug where device property was not properly update when metric was a child of a module (#542)

[0.5.1] - 2021-08-30

[0.5.1] - Added

  • Added device and dtype properties (#462)

  • Added TextTester class for robustly testing text metrics (#450)

[0.5.1] - Changed

  • Added support for float targets in nDCG metric (#437)

[0.5.1] - Removed

  • Removed rouge-score as dependency for text package (#443)

  • Removed jiwer as dependency for text package (#446)

  • Removed bert-score as dependency for text package (#473)

[0.5.1] - Fixed

  • Fixed ranking of samples in SpearmanCorrCoef metric (#448)

  • Fixed bug where compositional metrics where unable to sync because of type mismatch (#454)

  • Fixed metric hashing (#478)

  • Fixed BootStrapper metrics not working on GPU (#462)

  • Fixed the semantic ordering of kernel height and width in SSIM metric (#474)

[0.5.0] - 2021-08-09

[0.5.0] - Added

  • Added Text-related (NLP) metrics:

  • Added MetricTracker wrapper metric for keeping track of the same metric over multiple epochs (#238)

  • Added other metrics:

    • Symmetric Mean Absolute Percentage error (SMAPE) (#375)

    • Calibration error (#394)

    • Permutation Invariant Training (PIT) (#384)

  • Added support in nDCG metric for target with values larger than 1 (#349)

  • Added support for negative targets in nDCG metric (#378)

  • Added None as reduction option in CosineSimilarity metric (#400)

  • Allowed passing labels in (n_samples, n_classes) to AveragePrecision (#386)

[0.5.0] - Changed

  • Moved psnr and ssim from functional.regression.* to functional.image.* (#382)

  • Moved image_gradient from functional.image_gradients to functional.image.gradients (#381)

  • Moved R2Score from regression.r2score to regression.r2 (#371)

  • Pearson metric now only store 6 statistics instead of all predictions and targets (#380)

  • Use torch.argmax instead of torch.topk when k=1 for better performance (#419)

  • Moved check for number of samples in R2 score to support single sample updating (#426)

[0.5.0] - Deprecated

  • Rename r2score >> r2_score and kldivergence >> kl_divergence in functional (#371)

  • Moved bleu_score from functional.nlp to functional.text.bleu (#360)

[0.5.0] - Removed

  • Removed restriction that threshold has to be in (0,1) range to support logit input ( #351 #401)

  • Removed restriction that preds could not be bigger than num_classes to support logit input (#357)

  • Removed module regression.psnr and regression.ssim (#382):

  • Removed (#379):

    • function functional.mean_relative_error

    • num_thresholds argument in BinnedPrecisionRecallCurve

[0.5.0] - Fixed

  • Fixed bug where classification metrics with average='macro' would lead to wrong result if a class was missing (#303)

  • Fixed weighted, multi-class AUROC computation to allow for 0 observations of some class, as contribution to final AUROC is 0 (#376)

  • Fixed that _forward_cache and _computed attributes are also moved to the correct device if metric is moved (#413)

  • Fixed calculation in IoU metric when using ignore_index argument (#328)

[0.4.1] - 2021-07-05

[0.4.1] - Changed

[0.4.1] - Fixed

  • Fixed DDP by is_sync logic to Metric (#339)

[0.4.0] - 2021-06-29

[0.4.0] - Added

  • Added Image-related metrics:

    • Fréchet inception distance (FID) (#213)

    • Kernel Inception Distance (KID) (#301)

    • Inception Score (#299)

    • KL divergence (#247)

  • Added Audio metrics: SNR, SI_SDR, SI_SNR (#292)

  • Added other metrics:

    • Cosine Similarity (#305)

    • Specificity (#210)

    • Mean Absolute Percentage error (MAPE) (#248)

  • Added add_metrics method to MetricCollection for adding additional metrics after initialization (#221)

  • Added pre-gather reduction in the case of dist_reduce_fx="cat" to reduce communication cost (#217)

  • Added better error message for AUROC when num_classes is not provided for multiclass input (#244)

  • Added support for unnormalized scores (e.g. logits) in Accuracy, Precision, Recall, FBeta, F1, StatScore, Hamming, ConfusionMatrix metrics (#200)

  • Added squared argument to MeanSquaredError for computing RMSE (#249)

  • Added is_differentiable property to ConfusionMatrix, F1, FBeta, Hamming, Hinge, IOU, MatthewsCorrcoef, Precision, Recall, PrecisionRecallCurve, ROC, StatScores (#253)

  • Added sync and sync_context methods for manually controlling when metric states are synced (#302)

[0.4.0] - Changed

  • Forward cache is reset when reset method is called (#260)

  • Improved per-class metric handling for imbalanced datasets for precision, recall, precision_recall, fbeta, f1, accuracy, and specificity (#204)

  • Decorated torch.jit.unused to MetricCollection forward (#307)

  • Renamed thresholds argument to binned metrics for manually controlling the thresholds (#322)

  • Extend typing (#324, #326, #327)

[0.4.0] - Deprecated

  • Deprecated functional.mean_relative_error, use functional.mean_absolute_percentage_error (#248)

  • Deprecated num_thresholds argument in BinnedPrecisionRecallCurve (#322)

[0.4.0] - Removed

  • Removed argument is_multiclass (#319)

[0.4.0] - Fixed

  • AUC can also support more dimensional inputs when all but one dimension are of size 1 (#242)

  • Fixed dtype of modular metrics after reset has been called (#243)

  • Fixed calculation in matthews_corrcoef to correctly match formula (#321)

[0.3.2] - 2021-05-10

[0.3.2] - Added

  • Added is_differentiable property:

    • To AUC, AUROC, CohenKappa and AveragePrecision (#178)

    • To PearsonCorrCoef, SpearmanCorrcoef, R2Score and ExplainedVariance (#225)

[0.3.2] - Changed

  • MetricCollection should return metrics with prefix on items(), keys() (#209)

  • Calling compute before update will now give warning (#164)

[0.3.2] - Removed

  • Removed numpy as direct dependency (#212)

[0.3.2] - Fixed

  • Fixed auc calculation and add tests (#197)

  • Fixed loading persisted metric states using load_state_dict() (#202)

  • Fixed PSNR not working with DDP (#214)

  • Fixed metric calculation with unequal batch sizes (#220)

  • Fixed metric concatenation for list states for zero-dim input (#229)

  • Fixed numerical instability in AUROC metric for large input (#230)

[0.3.1] - 2021-04-21

  • Cleaning remaining inconsistency and fix PL develop integration ( #191, #192, #193, #194 )

[0.3.0] - 2021-04-20

[0.3.0] - Added

  • Added BootStrapper to easily calculate confidence intervals for metrics (#101)

  • Added Binned metrics (#128)

  • Added metrics for Information Retrieval ((PL^5032)):

    • RetrievalMAP (PL^5032)

    • RetrievalMRR (#119)

    • RetrievalPrecision (#139)

    • RetrievalRecall (#146)

    • RetrievalNormalizedDCG (#160)

    • RetrievalFallOut (#161)

  • Added other metrics:

    • CohenKappa (#69)

    • MatthewsCorrcoef (#98)

    • PearsonCorrcoef (#157)

    • SpearmanCorrcoef (#158)

    • Hinge (#120)

  • Added average='micro' as an option in AUROC for multilabel problems (#110)

  • Added multilabel support to ROC metric (#114)

  • Added testing for half precision (#77, #135 )

  • Added AverageMeter for ad-hoc averages of values (#138)

  • Added prefix argument to MetricCollection (#70)

  • Added __getitem__ as metric arithmetic operation (#142)

  • Added property is_differentiable to metrics and test for differentiability (#154)

  • Added support for average, ignore_index and mdmc_average in Accuracy metric (#166)

  • Added postfix arg to MetricCollection (#188)

[0.3.0] - Changed

  • Changed ExplainedVariance from storing all preds/targets to tracking 5 statistics (#68)

  • Changed behaviour of confusionmatrix for multilabel data to better match multilabel_confusion_matrix from sklearn (#134)

  • Updated FBeta arguments (#111)

  • Changed reset method to use detach.clone() instead of deepcopy when resetting to default (#163)

  • Metrics passed as dict to MetricCollection will now always be in deterministic order (#173)

  • Allowed MetricCollection pass metrics as arguments (#176)

[0.3.0] - Deprecated

  • Rename argument is_multiclass -> multiclass (#162)

[0.3.0] - Removed

  • Prune remaining deprecated (#92)

[0.3.0] - Fixed

  • Fixed when _stable_1d_sort to work when n>=N (PL^6177)

  • Fixed _computed attribute not being correctly reset (#147)

  • Fixed to Blau score (#165)

  • Fixed backwards compatibility for logging with older version of pytorch-lightning (#182)

[0.2.0] - 2021-03-12

[0.2.0] - Changed

  • Decoupled PL dependency (#13)

  • Refactored functional - mimic the module-like structure: classification, regression, etc. (#16)

  • Refactored utilities - split to topics/submodules (#14)

  • Refactored MetricCollection (#19)

[0.2.0] - Removed

  • Removed deprecated metrics from PL base (#12, #15)

[0.1.0] - 2021-02-22

  • Added Accuracy metric now generalizes to Top-k accuracy for (multi-dimensional) multi-class inputs using the top_k parameter (PL^4838)

  • Added Accuracy metric now enables the computation of subset accuracy for multi-label or multi-dimensional multi-class inputs with the subset_accuracy parameter (PL^4838)

  • Added HammingDistance metric to compute the hamming distance (loss) (PL^4838)

  • Added StatScores metric to compute the number of true positives, false positives, true negatives and false negatives (PL^4839)

  • Added R2Score metric (PL^5241)

  • Added MetricCollection (PL^4318)

  • Added .clone() method to metrics (PL^4318)

  • Added IoU class interface (PL^4704)

  • The Recall and Precision metrics (and their functional counterparts recall and precision) can now be generalized to Recall@K and Precision@K with the use of top_k parameter (PL^4842)

  • Added compositional metrics (PL^5464)

  • Added AUC/AUROC class interface (PL^5479)

  • Added QuantizationAwareTraining callback (PL^5706)

  • Added ConfusionMatrix class interface (PL^4348)

  • Added multiclass AUROC metric (PL^4236)

  • Added PrecisionRecallCurve, ROC, AveragePrecision class metric (PL^4549)

  • Classification metrics overhaul (PL^4837)

  • Added F1 class metric (PL^4656)

  • Added metrics aggregation in Horovod and fixed early stopping (PL^3775)

  • Added persistent(mode) method to metrics, to enable and disable metric states being added to state_dict (PL^4482)

  • Added unification of regression metrics (PL^4166)

  • Added persistent flag to Metric.add_state (PL^4195)

  • Added classification metrics (PL^4043)

  • Added new Metrics API. (PL^3868, PL^3921)

  • Added EMB similarity (PL^3349)

  • Added SSIM metrics (PL^2671)

  • Added BLEU metrics (PL^2535)

Read the Docs v: v0.8.2
Versions
latest
stable
v0.8.2
v0.8.1
v0.8.0
v0.7.3
v0.7.2
v0.7.1
v0.7.0
v0.6.2
v0.6.1
v0.6.0
v0.5.1
v0.5.0
v0.4.1
v0.4.0
v0.3.2
v0.3.1
v0.3.0
v0.2.0
v0.1.0
Downloads
On Read the Docs
Project Home
Builds

Free document hosting provided by Read the Docs.