chemprop.uncertainty ==================== .. py:module:: chemprop.uncertainty Submodules ---------- .. toctree:: :maxdepth: 1 /autoapi/chemprop/uncertainty/calibrator/index /autoapi/chemprop/uncertainty/estimator/index /autoapi/chemprop/uncertainty/evaluator/index Attributes ---------- .. autoapisummary:: chemprop.uncertainty.UncertaintyCalibratorRegistry chemprop.uncertainty.UncertaintyEstimatorRegistry chemprop.uncertainty.UncertaintyEvaluatorRegistry Classes ------- .. autoapisummary:: chemprop.uncertainty.AdaptiveMulticlassConformalCalibrator chemprop.uncertainty.BinaryClassificationCalibrator chemprop.uncertainty.CalibratorBase chemprop.uncertainty.IsotonicCalibrator chemprop.uncertainty.IsotonicMulticlassCalibrator chemprop.uncertainty.MulticlassClassificationCalibrator chemprop.uncertainty.MulticlassConformalCalibrator chemprop.uncertainty.MultilabelConformalCalibrator chemprop.uncertainty.MVEWeightingCalibrator chemprop.uncertainty.PlattCalibrator chemprop.uncertainty.RegressionCalibrator chemprop.uncertainty.RegressionConformalCalibrator chemprop.uncertainty.ZelikmanCalibrator chemprop.uncertainty.ZScalingCalibrator chemprop.uncertainty.ClassEstimator chemprop.uncertainty.ClassificationDirichletEstimator chemprop.uncertainty.DropoutEstimator chemprop.uncertainty.EnsembleEstimator chemprop.uncertainty.EvidentialAleatoricEstimator chemprop.uncertainty.EvidentialEpistemicEstimator chemprop.uncertainty.EvidentialTotalEstimator chemprop.uncertainty.MulticlassDirichletEstimator chemprop.uncertainty.MVEEstimator chemprop.uncertainty.NoUncertaintyEstimator chemprop.uncertainty.QuantileRegressionEstimator chemprop.uncertainty.UncertaintyEstimator chemprop.uncertainty.BinaryClassificationEvaluator chemprop.uncertainty.CalibrationAreaEvaluator chemprop.uncertainty.ExpectedNormalizedErrorEvaluator chemprop.uncertainty.MulticlassClassificationEvaluator chemprop.uncertainty.MulticlassConformalEvaluator chemprop.uncertainty.MultilabelConformalEvaluator chemprop.uncertainty.NLLClassEvaluator chemprop.uncertainty.NLLMulticlassEvaluator chemprop.uncertainty.NLLRegressionEvaluator chemprop.uncertainty.RegressionConformalEvaluator chemprop.uncertainty.RegressionEvaluator chemprop.uncertainty.SpearmanEvaluator Package Contents ---------------- .. py:class:: AdaptiveMulticlassConformalCalibrator(alpha) Bases: :py:obj:`MulticlassConformalCalibrator` Create a prediction sets of possible labels :math:`C(X_{\text{test}}) \subset \{1 \mathrel{.\,.} K\}` that follows: .. math:: 1 - \alpha \leq \Pr (Y_{\text{test}} \in C(X_{\text{test}})) \leq 1 - \alpha + \frac{1}{n + 1} In other words, the probability that the prediction set contains the correct label is almost exactly :math:`1-\alpha`. More detailes can be found in [1]_. :param alpha: Error rate, :math:`\alpha \in [0, 1]` :type alpha: float .. rubric:: References .. [1] Angelopoulos, A.N.; Bates, S.; "A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification." arXiv Preprint 2021, https://arxiv.org/abs/2107.07511 .. py:method:: nonconformity_scores(preds) :staticmethod: Compute nonconformity score by greedily including classes in the classification set until it reaches the true label. .. math:: s(x, y) = \sum_{j=1}^{k} \hat{f}(x)_{\pi_j(x)}, \text{ where } y = \pi_k(x) where :math:`\pi_k(x)` is the permutation of :math:`\{1 \mathrel{.\,.} K\}` that sorts :math:`\hat{f}(X_{test})` from most likely to least likely. .. py:class:: BinaryClassificationCalibrator Bases: :py:obj:`CalibratorBase` A class for calibrating the predicted uncertainties in binary classification tasks. .. py:method:: fit(uncs, targets, mask) :abstractmethod: Fit calibration method for the calibration data. :param uncs: the predicted uncertainties (i.e., the predicted probability of class 1) of the shape of ``n x t``, where ``n`` is the number of input molecules/reactions, and ``t`` is the number of tasks. :type uncs: Tensor :param targets: a tensor of the shape ``n x t`` :type targets: Tensor :param mask: a tensor of the shape ``n x t`` indicating whether the given values should be used in the fitting :type mask: Tensor :returns: **self** -- the fitted calibrator :rtype: BinaryClassificationCalibrator .. py:class:: CalibratorBase Bases: :py:obj:`abc.ABC` A base class for calibrating the predicted uncertainties. .. py:method:: fit(*args, **kwargs) :abstractmethod: Fit calibration method for the calibration data. .. py:method:: apply(uncs) :abstractmethod: Apply this calibrator to the input uncertainties. :param uncs: a tensor containinig uncalibrated uncertainties :type uncs: Tensor :returns: the calibrated uncertainties :rtype: Tensor .. py:class:: IsotonicCalibrator Bases: :py:obj:`BinaryClassificationCalibrator` Calibrate binary classification datasets using isotonic regression as discussed in [guo2017]_. In effect, the method transforms incoming uncalibrated confidences using a histogram-like function where the range of each transforming bin and its magnitude is learned. .. rubric:: References .. [guo2017] Guo, C.; Pleiss, G.; Sun, Y.; Weinberger, K. Q. "On calibration of modern neural networks". ICML, 2017. https://arxiv.org/abs/1706.04599 .. py:method:: fit(uncs, targets, mask) Fit calibration method for the calibration data. :param uncs: the predicted uncertainties (i.e., the predicted probability of class 1) of the shape of ``n x t``, where ``n`` is the number of input molecules/reactions, and ``t`` is the number of tasks. :type uncs: Tensor :param targets: a tensor of the shape ``n x t`` :type targets: Tensor :param mask: a tensor of the shape ``n x t`` indicating whether the given values should be used in the fitting :type mask: Tensor :returns: **self** -- the fitted calibrator :rtype: BinaryClassificationCalibrator .. py:method:: apply(uncs) Apply this calibrator to the input uncertainties. :param uncs: a tensor containinig uncalibrated uncertainties :type uncs: Tensor :returns: the calibrated uncertainties :rtype: Tensor .. py:class:: IsotonicMulticlassCalibrator Bases: :py:obj:`MulticlassClassificationCalibrator` Calibrate multiclass classification datasets using isotonic regression as discussed in [guo2017]_. It uses a one-vs-all aggregation scheme to extend isotonic regression from binary to multiclass classifiers. .. rubric:: References .. [guo2017] Guo, C.; Pleiss, G.; Sun, Y.; Weinberger, K. Q. "On calibration of modern neural networks". ICML, 2017. https://arxiv.org/abs/1706.04599 .. py:method:: fit(uncs, targets, mask) Fit calibration method for the calibration data. :param uncs: the predicted uncertainties (i.e., the predicted probabilities for each class) of the shape of ``n x t x c``, where ``n`` is the number of input molecules/reactions, ``t`` is the number of tasks, and ``c`` is the number of classes. :type uncs: Tensor :param targets: a tensor of the shape ``n x t`` :type targets: Tensor :param mask: a tensor of the shape ``n x t`` indicating whether the given values should be used in the fitting :type mask: Tensor :returns: **self** -- the fitted calibrator :rtype: MulticlassClassificationCalibrator .. py:method:: apply(uncs) Apply this calibrator to the input uncertainties. :param uncs: a tensor containinig uncalibrated uncertainties :type uncs: Tensor :returns: the calibrated uncertainties :rtype: Tensor .. py:class:: MulticlassClassificationCalibrator Bases: :py:obj:`CalibratorBase` A class for calibrating the predicted uncertainties in multiclass classification tasks. .. py:method:: fit(uncs, targets, mask) :abstractmethod: Fit calibration method for the calibration data. :param uncs: the predicted uncertainties (i.e., the predicted probabilities for each class) of the shape of ``n x t x c``, where ``n`` is the number of input molecules/reactions, ``t`` is the number of tasks, and ``c`` is the number of classes. :type uncs: Tensor :param targets: a tensor of the shape ``n x t`` :type targets: Tensor :param mask: a tensor of the shape ``n x t`` indicating whether the given values should be used in the fitting :type mask: Tensor :returns: **self** -- the fitted calibrator :rtype: MulticlassClassificationCalibrator .. py:class:: MulticlassConformalCalibrator(alpha) Bases: :py:obj:`MulticlassClassificationCalibrator` Create a prediction sets of possible labels :math:`C(X_{\text{test}}) \subset \{1 \mathrel{.\,.} K\}` that follows: .. math:: 1 - \alpha \leq \Pr (Y_{\text{test}} \in C(X_{\text{test}})) \leq 1 - \alpha + \frac{1}{n + 1} In other words, the probability that the prediction set contains the correct label is almost exactly :math:`1-\alpha`. More detailes can be found in [1]_. :param alpha: Error rate, :math:`\alpha \in [0, 1]` :type alpha: float .. rubric:: References .. [1] Angelopoulos, A.N.; Bates, S.; "A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification." arXiv Preprint 2021, https://arxiv.org/abs/2107.07511 .. py:attribute:: alpha .. py:method:: nonconformity_scores(preds) :staticmethod: Compute nonconformity score as the negative of the softmax output for the true class. .. math:: s_i = -\hat{f}(X_i)_{Y_i} .. py:method:: fit(uncs, targets, mask) Fit calibration method for the calibration data. :param uncs: the predicted uncertainties (i.e., the predicted probabilities for each class) of the shape of ``n x t x c``, where ``n`` is the number of input molecules/reactions, ``t`` is the number of tasks, and ``c`` is the number of classes. :type uncs: Tensor :param targets: a tensor of the shape ``n x t`` :type targets: Tensor :param mask: a tensor of the shape ``n x t`` indicating whether the given values should be used in the fitting :type mask: Tensor :returns: **self** -- the fitted calibrator :rtype: MulticlassClassificationCalibrator .. py:method:: apply(uncs) Apply this calibrator to the input uncertainties. :param uncs: a tensor containinig uncalibrated uncertainties :type uncs: Tensor :returns: the calibrated uncertainties :rtype: Tensor .. py:class:: MultilabelConformalCalibrator(alpha) Bases: :py:obj:`BinaryClassificationCalibrator` Creates conformal in-set and conformal out-set such that, for :math:`1-\alpha` proportion of datapoints, the set of labels is bounded by the in- and out-sets [1]_: .. math:: \Pr \left( \hat{\mathcal C}_{\text{in}}(X) \subseteq \mathcal Y \subseteq \hat{\mathcal C}_{\text{out}}(X) \right) \geq 1 - \alpha, where the in-set :math:`\hat{\mathcal C}_\text{in}` is contained by the set of true labels :math:`\mathcal Y` and :math:`\mathcal Y` is contained within the out-set :math:`\hat{\mathcal C}_\text{out}`. :param alpha: The error rate, :math:`\alpha \in [0, 1]` :type alpha: float .. rubric:: References .. [1] Cauchois, M.; Gupta, S.; Duchi, J.; "Knowing What You Know: Valid and Validated Confidence Sets in Multiclass and Multilabel Prediction." arXiv Preprint 2020, https://arxiv.org/abs/2004.10181 .. py:attribute:: alpha .. py:method:: nonconformity_scores(preds) :staticmethod: Compute nonconformity score as the negative of the predicted probability. .. math:: s_i = -\hat{f}(X_i)_{Y_i} .. py:method:: fit(uncs, targets, mask) Fit calibration method for the calibration data. :param uncs: the predicted uncertainties (i.e., the predicted probability of class 1) of the shape of ``n x t``, where ``n`` is the number of input molecules/reactions, and ``t`` is the number of tasks. :type uncs: Tensor :param targets: a tensor of the shape ``n x t`` :type targets: Tensor :param mask: a tensor of the shape ``n x t`` indicating whether the given values should be used in the fitting :type mask: Tensor :returns: **self** -- the fitted calibrator :rtype: BinaryClassificationCalibrator .. py:method:: apply(uncs) Apply this calibrator to the input uncertainties. :param uncs: a tensor containinig uncalibrated uncertainties :type uncs: Tensor :returns: the calibrated uncertainties of the shape of ``n x t x 2``, where ``n`` is the number of input molecules/reactions, ``t`` is the number of tasks, and the first element in the last dimension corresponds to the in-set :math:`\hat{\mathcal C}_\text{in}`, while the second corresponds to the out-set :math:`\hat{\mathcal C}_\text{out}`. :rtype: Tensor .. py:class:: MVEWeightingCalibrator Bases: :py:obj:`RegressionCalibrator` Calibrate regression datasets that have ensembles of individual models that make variance predictions. This method minimizes the negative log likelihood for the predictions versus the targets by applying a weighted average across the variance predictions of the ensemble. [wang2021]_ .. rubric:: References .. [wang2021] Wang, D.; Yu, J.; Chen, L.; Li, X.; Jiang, H.; Chen, K.; Zheng, M.; Luo, X. "A hybrid framework for improving uncertainty quantification in deep learning-based QSAR regression modeling." J. Cheminform., 2021, 13, 1-17. https://doi.org/10.1186/s13321-021-00551-x .. py:method:: fit(preds, uncs, targets, mask) Fit calibration method for the calibration data. :param preds: the predictions for regression tasks. It is a tensor of the shape of ``n x t``, where ``n`` is the number of input molecules/reactions, and ``t`` is the number of tasks. :type preds: Tensor :param uncs: the predicted uncertainties of the shape of ``m x n x t`` :type uncs: Tensor :param targets: a tensor of the shape ``n x t`` :type targets: Tensor :param mask: a tensor of the shape ``n x t`` indicating whether the given values should be used in the fitting :type mask: Tensor :returns: **self** -- the fitted calibrator :rtype: MVEWeightingCalibrator .. py:method:: apply(uncs) Apply this calibrator to the input uncertainties. :param uncs: a tensor containinig uncalibrated uncertainties of the shape of ``m x n x t`` :type uncs: Tensor :returns: the calibrated uncertainties of the shape of ``n x t`` :rtype: Tensor .. py:class:: PlattCalibrator Bases: :py:obj:`BinaryClassificationCalibrator` Calibrate classification datasets using the Platt scaling algorithm [guo2017]_, [platt1999]_. In [platt1999]_, Platt suggests using the number of positive and negative training examples to adjust the value of target probabilities used to fit the parameters. .. rubric:: References .. [guo2017] Guo, C.; Pleiss, G.; Sun, Y.; Weinberger, K. Q. "On calibration of modern neural networks". ICML, 2017. https://arxiv.org/abs/1706.04599 .. [platt1999] Platt, J. "Probabilistic Outputs for Support Vector Machines and Comparisons to Regularized Likelihood Methods." Adv. Large Margin Classif. 1999, 10 (3), 61–74. .. py:method:: fit(uncs, targets, mask, training_targets = None) Fit calibration method for the calibration data. :param uncs: the predicted uncertainties (i.e., the predicted probability of class 1) of the shape of ``n x t``, where ``n`` is the number of input molecules/reactions, and ``t`` is the number of tasks. :type uncs: Tensor :param targets: a tensor of the shape ``n x t`` :type targets: Tensor :param mask: a tensor of the shape ``n x t`` indicating whether the given values should be used in the fitting :type mask: Tensor :returns: **self** -- the fitted calibrator :rtype: BinaryClassificationCalibrator .. py:method:: apply(uncs) Apply this calibrator to the input uncertainties. :param uncs: a tensor containinig uncalibrated uncertainties :type uncs: Tensor :returns: the calibrated uncertainties :rtype: Tensor .. py:class:: RegressionCalibrator Bases: :py:obj:`CalibratorBase` A class for calibrating the predicted uncertainties in regressions tasks. .. py:method:: fit(preds, uncs, targets, mask) :abstractmethod: Fit calibration method for the calibration data. :param preds: the predictions for regression tasks. It is a tensor of the shape of ``n x t``, where ``n`` is the number of input molecules/reactions, and ``t`` is the number of tasks. :type preds: Tensor :param uncs: the predicted uncertainties of the shape of ``n x t`` :type uncs: Tensor :param targets: a tensor of the shape ``n x t`` :type targets: Tensor :param mask: a tensor of the shape ``n x t`` indicating whether the given values should be used in the fitting :type mask: Tensor :returns: **self** -- the fitted calibrator :rtype: RegressionCalibrator .. py:class:: RegressionConformalCalibrator(alpha) Bases: :py:obj:`RegressionCalibrator` Conformalize quantiles to make the interval :math:`[\hat{t}_{\alpha/2}(x),\hat{t}_{1-\alpha/2}(x)]` to have approximately :math:`1-\alpha` coverage. [angelopoulos2021]_ .. math:: s(x, y) &= \max \left\{ \hat{t}_{\alpha/2}(x) - y, y - \hat{t}_{1-\alpha/2}(x) \right\} \hat{q} &= Q(s_1, \ldots, s_n; \left\lceil \frac{(n+1)(1-\alpha)}{n} \right\rceil) C(x) &= \left[ \hat{t}_{\alpha/2}(x) - \hat{q}, \hat{t}_{1-\alpha/2}(x) + \hat{q} \right] where :math:`s` is the nonconformity score as the difference between :math:`y` and its nearest quantile. :math:`\hat{t}_{\alpha/2}(x)` and :math:`\hat{t}_{1-\alpha/2}(x)` are the predicted quantiles from a quantile regression model. .. note:: The algorithm is specifically designed for quantile regression model. Intuitively, the set :math:`C(x)` just grows or shrinks the distance between the quantiles by :math:`\hat{q}` to achieve coverage. However, this function can also be applied to regression model without quantiles being provided. In this case, both :math:`\hat{t}_{\alpha/2}(x)` and :math:`\hat{t}_{1-\alpha/2}(x)` are the same as :math:`\hat{y}`. Then, the interval would be the same for every data point (i.e., :math:`\left[-\hat{q}, \hat{q} \right]`). :param alpha: The error rate, :math:`\alpha \in [0, 1]` :type alpha: float .. rubric:: References .. [angelopoulos2021] Angelopoulos, A.N.; Bates, S.; "A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification." arXiv Preprint 2021, https://arxiv.org/abs/2107.07511 .. py:attribute:: alpha .. py:attribute:: bounds .. py:method:: fit(preds, uncs, targets, mask) Fit calibration method for the calibration data. :param preds: the predictions for regression tasks. It is a tensor of the shape of ``n x t``, where ``n`` is the number of input molecules/reactions, and ``t`` is the number of tasks. :type preds: Tensor :param uncs: the predicted uncertainties of the shape of ``n x t`` :type uncs: Tensor :param targets: a tensor of the shape ``n x t`` :type targets: Tensor :param mask: a tensor of the shape ``n x t`` indicating whether the given values should be used in the fitting :type mask: Tensor :returns: **self** -- the fitted calibrator :rtype: RegressionCalibrator .. py:method:: apply(uncs) Apply this calibrator to the input uncertainties (half intervals). :param uncs: a tensor containinig uncalibrated uncertainties :type uncs: Tensor :returns: the calibrated half intervals :rtype: Tensor .. py:data:: UncertaintyCalibratorRegistry .. py:class:: ZelikmanCalibrator(p) Bases: :py:obj:`RegressionCalibrator` Calibrate regression datasets using a method that does not depend on a particular probability function form. It uses the "CRUDE" method as described in [zelikman2020]_. We implemented this method to be used with variance as the uncertainty. :param p: The target qunatile, :math:`p \in [0, 1]` :type p: float .. rubric:: References .. [zelikman2020] Zelikman, E.; Healy, C.; Zhou, S.; Avati, A. "CRUDE: calibrating regression uncertainty distributions empirically." arXiv preprint arXiv:2005.12496. https://doi.org/10.48550/arXiv.2005.12496 .. py:attribute:: p .. py:method:: fit(preds, uncs, targets, mask) Fit calibration method for the calibration data. :param preds: the predictions for regression tasks. It is a tensor of the shape of ``n x t``, where ``n`` is the number of input molecules/reactions, and ``t`` is the number of tasks. :type preds: Tensor :param uncs: the predicted uncertainties of the shape of ``n x t`` :type uncs: Tensor :param targets: a tensor of the shape ``n x t`` :type targets: Tensor :param mask: a tensor of the shape ``n x t`` indicating whether the given values should be used in the fitting :type mask: Tensor :returns: **self** -- the fitted calibrator :rtype: RegressionCalibrator .. py:method:: apply(uncs) Apply this calibrator to the input uncertainties. :param uncs: a tensor containinig uncalibrated uncertainties :type uncs: Tensor :returns: the calibrated uncertainties :rtype: Tensor .. py:class:: ZScalingCalibrator Bases: :py:obj:`RegressionCalibrator` Calibrate regression datasets by applying a scaling value to the uncalibrated standard deviation, fitted by minimizing the negative-log-likelihood of a normal distribution around each prediction. [levi2022]_ .. rubric:: References .. [levi2022] Levi, D.; Gispan, L.; Giladi, N.; Fetaya, E. "Evaluating and Calibrating Uncertainty Prediction in Regression Tasks." Sensors, 2022, 22(15), 5540. https://www.mdpi.com/1424-8220/22/15/5540 .. py:method:: fit(preds, uncs, targets, mask) Fit calibration method for the calibration data. :param preds: the predictions for regression tasks. It is a tensor of the shape of ``n x t``, where ``n`` is the number of input molecules/reactions, and ``t`` is the number of tasks. :type preds: Tensor :param uncs: the predicted uncertainties of the shape of ``n x t`` :type uncs: Tensor :param targets: a tensor of the shape ``n x t`` :type targets: Tensor :param mask: a tensor of the shape ``n x t`` indicating whether the given values should be used in the fitting :type mask: Tensor :returns: **self** -- the fitted calibrator :rtype: RegressionCalibrator .. py:method:: apply(uncs) Apply this calibrator to the input uncertainties. :param uncs: a tensor containinig uncalibrated uncertainties :type uncs: Tensor :returns: the calibrated uncertainties :rtype: Tensor .. py:class:: ClassEstimator Bases: :py:obj:`UncertaintyEstimator` A helper class for making model predictions and associated uncertainty predictions. .. py:method:: __call__(dataloader, models, trainer) Calculate the uncalibrated predictions and uncertainties for the dataloader. dataloader: DataLoader the dataloader used for model predictions and uncertainty predictions models: Iterable[MPNN] | Iterable[MolAtomBondMPNN] the models used for model predictions and uncertainty predictions. If using MolAtomBondMPNN models, the uncertainty estimator will return preds and uncs for each of the mole, atom, and bond predictions and uncertainties. trainer: pl.Trainer an instance of the :class:`~lightning.pytorch.trainer.trainer.Trainer` used to manage model inference :returns: * **preds** (*Tensor*) -- the model predictions, with shape varying by task type: * regression/binary classification: ``m x n x t`` * multiclass classification: ``m x n x t x c``, where ``m`` is the number of models, ``n`` is the number of inputs, ``t`` is the number of tasks, and ``c`` is the number of classes. * **uncs** (*Tensor*) -- the predicted uncertainties, with shapes of ``m' x n x t``. * *.. note::* -- The ``m`` and ``m'`` are different by definition. The ``m`` is the number of models, while the ``m'`` is the number of uncertainty estimations. For example, if two MVE or evidential models are provided, both ``m`` and ``m'`` are two. However, for an ensemble of two models, ``m'`` would be one (even though ``m = 2``). .. py:class:: ClassificationDirichletEstimator Bases: :py:obj:`UncertaintyEstimator` A :class:`ClassificationDirichletEstimator` predicts an amount of 'evidence' for both the negative class and the positive class as described in [sensoy2018]_. The class probabilities and the uncertainty are calculated based on the evidence. .. math:: S = \sum_{i=1}^K \alpha_i p_i = \alpha_i / S u = K / S where :math:`K` is the number of classes, :math:`\alpha_i` is the evidence for class :math:`i`, :math:`p_i` is the probability of class :math:`i`, and :math:`u` is the uncertainty. .. rubric:: References .. [sensoy2018] Sensoy, M.; Kaplan, L.; Kandemir, M. "Evidential deep learning to quantify classification uncertainty." NeurIPS, 2018, 31. https://doi.org/10.48550/arXiv.1806.01768 .. py:method:: __call__(dataloader, models, trainer) Calculate the uncalibrated predictions and uncertainties for the dataloader. dataloader: DataLoader the dataloader used for model predictions and uncertainty predictions models: Iterable[MPNN] | Iterable[MolAtomBondMPNN] the models used for model predictions and uncertainty predictions. If using MolAtomBondMPNN models, the uncertainty estimator will return preds and uncs for each of the mole, atom, and bond predictions and uncertainties. trainer: pl.Trainer an instance of the :class:`~lightning.pytorch.trainer.trainer.Trainer` used to manage model inference :returns: * **preds** (*Tensor*) -- the model predictions, with shape varying by task type: * regression/binary classification: ``m x n x t`` * multiclass classification: ``m x n x t x c``, where ``m`` is the number of models, ``n`` is the number of inputs, ``t`` is the number of tasks, and ``c`` is the number of classes. * **uncs** (*Tensor*) -- the predicted uncertainties, with shapes of ``m' x n x t``. * *.. note::* -- The ``m`` and ``m'`` are different by definition. The ``m`` is the number of models, while the ``m'`` is the number of uncertainty estimations. For example, if two MVE or evidential models are provided, both ``m`` and ``m'`` are two. However, for an ensemble of two models, ``m'`` would be one (even though ``m = 2``). .. py:class:: DropoutEstimator(ensemble_size, dropout = None) Bases: :py:obj:`UncertaintyEstimator` A :class:`DropoutEstimator` creates a virtual ensemble of models via Monte Carlo dropout with the provided model [gal2016]_. :param ensemble_size: The number of samples to draw for the ensemble. :type ensemble_size: int :param dropout: The probability of dropping out units in the dropout layers. If unspecified, the training probability is used, which is prefered but not possible if the model was not trained with dropout (i.e. p=0). :type dropout: float | None .. rubric:: References .. [gal2016] Gal, Y.; Ghahramani, Z. "Dropout as a bayesian approximation: Representing model uncertainty in deep learning." International conference on machine learning. PMLR, 2016. https://arxiv.org/abs/1506.02142 .. py:attribute:: ensemble_size .. py:attribute:: dropout :value: None .. py:method:: __call__(dataloader, models, trainer) Calculate the uncalibrated predictions and uncertainties for the dataloader. dataloader: DataLoader the dataloader used for model predictions and uncertainty predictions models: Iterable[MPNN] | Iterable[MolAtomBondMPNN] the models used for model predictions and uncertainty predictions. If using MolAtomBondMPNN models, the uncertainty estimator will return preds and uncs for each of the mole, atom, and bond predictions and uncertainties. trainer: pl.Trainer an instance of the :class:`~lightning.pytorch.trainer.trainer.Trainer` used to manage model inference :returns: * **preds** (*Tensor*) -- the model predictions, with shape varying by task type: * regression/binary classification: ``m x n x t`` * multiclass classification: ``m x n x t x c``, where ``m`` is the number of models, ``n`` is the number of inputs, ``t`` is the number of tasks, and ``c`` is the number of classes. * **uncs** (*Tensor*) -- the predicted uncertainties, with shapes of ``m' x n x t``. * *.. note::* -- The ``m`` and ``m'`` are different by definition. The ``m`` is the number of models, while the ``m'`` is the number of uncertainty estimations. For example, if two MVE or evidential models are provided, both ``m`` and ``m'`` are two. However, for an ensemble of two models, ``m'`` would be one (even though ``m = 2``). .. py:class:: EnsembleEstimator Bases: :py:obj:`UncertaintyEstimator` Class that predicts the uncertainty of predictions based on the variance in predictions among an ensemble's submodels. .. py:method:: __call__(dataloader, models, trainer) Calculate the uncalibrated predictions and uncertainties for the dataloader. dataloader: DataLoader the dataloader used for model predictions and uncertainty predictions models: Iterable[MPNN] | Iterable[MolAtomBondMPNN] the models used for model predictions and uncertainty predictions. If using MolAtomBondMPNN models, the uncertainty estimator will return preds and uncs for each of the mole, atom, and bond predictions and uncertainties. trainer: pl.Trainer an instance of the :class:`~lightning.pytorch.trainer.trainer.Trainer` used to manage model inference :returns: * **preds** (*Tensor*) -- the model predictions, with shape varying by task type: * regression/binary classification: ``m x n x t`` * multiclass classification: ``m x n x t x c``, where ``m`` is the number of models, ``n`` is the number of inputs, ``t`` is the number of tasks, and ``c`` is the number of classes. * **uncs** (*Tensor*) -- the predicted uncertainties, with shapes of ``m' x n x t``. * *.. note::* -- The ``m`` and ``m'`` are different by definition. The ``m`` is the number of models, while the ``m'`` is the number of uncertainty estimations. For example, if two MVE or evidential models are provided, both ``m`` and ``m'`` are two. However, for an ensemble of two models, ``m'`` would be one (even though ``m = 2``). .. py:class:: EvidentialAleatoricEstimator Bases: :py:obj:`UncertaintyEstimator` Class that predicts the aleatoric evidential uncertainty based on hyperparameters of the evidential distribution. .. py:method:: __call__(dataloader, models, trainer) Calculate the uncalibrated predictions and uncertainties for the dataloader. dataloader: DataLoader the dataloader used for model predictions and uncertainty predictions models: Iterable[MPNN] | Iterable[MolAtomBondMPNN] the models used for model predictions and uncertainty predictions. If using MolAtomBondMPNN models, the uncertainty estimator will return preds and uncs for each of the mole, atom, and bond predictions and uncertainties. trainer: pl.Trainer an instance of the :class:`~lightning.pytorch.trainer.trainer.Trainer` used to manage model inference :returns: * **preds** (*Tensor*) -- the model predictions, with shape varying by task type: * regression/binary classification: ``m x n x t`` * multiclass classification: ``m x n x t x c``, where ``m`` is the number of models, ``n`` is the number of inputs, ``t`` is the number of tasks, and ``c`` is the number of classes. * **uncs** (*Tensor*) -- the predicted uncertainties, with shapes of ``m' x n x t``. * *.. note::* -- The ``m`` and ``m'`` are different by definition. The ``m`` is the number of models, while the ``m'`` is the number of uncertainty estimations. For example, if two MVE or evidential models are provided, both ``m`` and ``m'`` are two. However, for an ensemble of two models, ``m'`` would be one (even though ``m = 2``). .. py:class:: EvidentialEpistemicEstimator Bases: :py:obj:`UncertaintyEstimator` Class that predicts the epistemic evidential uncertainty based on hyperparameters of the evidential distribution. .. py:method:: __call__(dataloader, models, trainer) Calculate the uncalibrated predictions and uncertainties for the dataloader. dataloader: DataLoader the dataloader used for model predictions and uncertainty predictions models: Iterable[MPNN] | Iterable[MolAtomBondMPNN] the models used for model predictions and uncertainty predictions. If using MolAtomBondMPNN models, the uncertainty estimator will return preds and uncs for each of the mole, atom, and bond predictions and uncertainties. trainer: pl.Trainer an instance of the :class:`~lightning.pytorch.trainer.trainer.Trainer` used to manage model inference :returns: * **preds** (*Tensor*) -- the model predictions, with shape varying by task type: * regression/binary classification: ``m x n x t`` * multiclass classification: ``m x n x t x c``, where ``m`` is the number of models, ``n`` is the number of inputs, ``t`` is the number of tasks, and ``c`` is the number of classes. * **uncs** (*Tensor*) -- the predicted uncertainties, with shapes of ``m' x n x t``. * *.. note::* -- The ``m`` and ``m'`` are different by definition. The ``m`` is the number of models, while the ``m'`` is the number of uncertainty estimations. For example, if two MVE or evidential models are provided, both ``m`` and ``m'`` are two. However, for an ensemble of two models, ``m'`` would be one (even though ``m = 2``). .. py:class:: EvidentialTotalEstimator Bases: :py:obj:`UncertaintyEstimator` Class that predicts the total evidential uncertainty based on hyperparameters of the evidential distribution [amini2020]_. .. rubric:: References .. [amini2020] Amini, A.; Schwarting, W.; Soleimany, A.; Rus, D. "Deep Evidential Regression". NeurIPS, 2020. https://arxiv.org/abs/1910.02600 .. py:method:: __call__(dataloader, models, trainer) Calculate the uncalibrated predictions and uncertainties for the dataloader. dataloader: DataLoader the dataloader used for model predictions and uncertainty predictions models: Iterable[MPNN] | Iterable[MolAtomBondMPNN] the models used for model predictions and uncertainty predictions. If using MolAtomBondMPNN models, the uncertainty estimator will return preds and uncs for each of the mole, atom, and bond predictions and uncertainties. trainer: pl.Trainer an instance of the :class:`~lightning.pytorch.trainer.trainer.Trainer` used to manage model inference :returns: * **preds** (*Tensor*) -- the model predictions, with shape varying by task type: * regression/binary classification: ``m x n x t`` * multiclass classification: ``m x n x t x c``, where ``m`` is the number of models, ``n`` is the number of inputs, ``t`` is the number of tasks, and ``c`` is the number of classes. * **uncs** (*Tensor*) -- the predicted uncertainties, with shapes of ``m' x n x t``. * *.. note::* -- The ``m`` and ``m'`` are different by definition. The ``m`` is the number of models, while the ``m'`` is the number of uncertainty estimations. For example, if two MVE or evidential models are provided, both ``m`` and ``m'`` are two. However, for an ensemble of two models, ``m'`` would be one (even though ``m = 2``). .. py:class:: MulticlassDirichletEstimator Bases: :py:obj:`UncertaintyEstimator` A :class:`MulticlassDirichletEstimator` predicts an amount of 'evidence' for each class as described in [sensoy2018]_. The class probabilities and the uncertainty are calculated based on the evidence. .. math:: S = \sum_{i=1}^K \alpha_i p_i = \alpha_i / S u = K / S where :math:`K` is the number of classes, :math:`\alpha_i` is the evidence for class :math:`i`, :math:`p_i` is the probability of class :math:`i`, and :math:`u` is the uncertainty. .. rubric:: References .. [sensoy2018] Sensoy, M.; Kaplan, L.; Kandemir, M. "Evidential deep learning to quantify classification uncertainty." NeurIPS, 2018, 31. https://doi.org/10.48550/arXiv.1806.01768 .. py:method:: __call__(dataloader, models, trainer) Calculate the uncalibrated predictions and uncertainties for the dataloader. dataloader: DataLoader the dataloader used for model predictions and uncertainty predictions models: Iterable[MPNN] | Iterable[MolAtomBondMPNN] the models used for model predictions and uncertainty predictions. If using MolAtomBondMPNN models, the uncertainty estimator will return preds and uncs for each of the mole, atom, and bond predictions and uncertainties. trainer: pl.Trainer an instance of the :class:`~lightning.pytorch.trainer.trainer.Trainer` used to manage model inference :returns: * **preds** (*Tensor*) -- the model predictions, with shape varying by task type: * regression/binary classification: ``m x n x t`` * multiclass classification: ``m x n x t x c``, where ``m`` is the number of models, ``n`` is the number of inputs, ``t`` is the number of tasks, and ``c`` is the number of classes. * **uncs** (*Tensor*) -- the predicted uncertainties, with shapes of ``m' x n x t``. * *.. note::* -- The ``m`` and ``m'`` are different by definition. The ``m`` is the number of models, while the ``m'`` is the number of uncertainty estimations. For example, if two MVE or evidential models are provided, both ``m`` and ``m'`` are two. However, for an ensemble of two models, ``m'`` would be one (even though ``m = 2``). .. py:class:: MVEEstimator Bases: :py:obj:`UncertaintyEstimator` Class that estimates prediction means and variances (MVE). [nix1994]_ .. rubric:: References .. [nix1994] Nix, D. A.; Weigend, A. S. "Estimating the mean and variance of the target probability distribution." Proceedings of 1994 IEEE International Conference on Neural Networks, 1994 https://doi.org/10.1109/icnn.1994.374138 .. py:method:: __call__(dataloader, models, trainer) Calculate the uncalibrated predictions and uncertainties for the dataloader. dataloader: DataLoader the dataloader used for model predictions and uncertainty predictions models: Iterable[MPNN] | Iterable[MolAtomBondMPNN] the models used for model predictions and uncertainty predictions. If using MolAtomBondMPNN models, the uncertainty estimator will return preds and uncs for each of the mole, atom, and bond predictions and uncertainties. trainer: pl.Trainer an instance of the :class:`~lightning.pytorch.trainer.trainer.Trainer` used to manage model inference :returns: * **preds** (*Tensor*) -- the model predictions, with shape varying by task type: * regression/binary classification: ``m x n x t`` * multiclass classification: ``m x n x t x c``, where ``m`` is the number of models, ``n`` is the number of inputs, ``t`` is the number of tasks, and ``c`` is the number of classes. * **uncs** (*Tensor*) -- the predicted uncertainties, with shapes of ``m' x n x t``. * *.. note::* -- The ``m`` and ``m'`` are different by definition. The ``m`` is the number of models, while the ``m'`` is the number of uncertainty estimations. For example, if two MVE or evidential models are provided, both ``m`` and ``m'`` are two. However, for an ensemble of two models, ``m'`` would be one (even though ``m = 2``). .. py:class:: NoUncertaintyEstimator Bases: :py:obj:`UncertaintyEstimator` A helper class for making model predictions and associated uncertainty predictions. .. py:method:: __call__(dataloader, models, trainer) Calculate the uncalibrated predictions and uncertainties for the dataloader. dataloader: DataLoader the dataloader used for model predictions and uncertainty predictions models: Iterable[MPNN] | Iterable[MolAtomBondMPNN] the models used for model predictions and uncertainty predictions. If using MolAtomBondMPNN models, the uncertainty estimator will return preds and uncs for each of the mole, atom, and bond predictions and uncertainties. trainer: pl.Trainer an instance of the :class:`~lightning.pytorch.trainer.trainer.Trainer` used to manage model inference :returns: * **preds** (*Tensor*) -- the model predictions, with shape varying by task type: * regression/binary classification: ``m x n x t`` * multiclass classification: ``m x n x t x c``, where ``m`` is the number of models, ``n`` is the number of inputs, ``t`` is the number of tasks, and ``c`` is the number of classes. * **uncs** (*Tensor*) -- the predicted uncertainties, with shapes of ``m' x n x t``. * *.. note::* -- The ``m`` and ``m'`` are different by definition. The ``m`` is the number of models, while the ``m'`` is the number of uncertainty estimations. For example, if two MVE or evidential models are provided, both ``m`` and ``m'`` are two. However, for an ensemble of two models, ``m'`` would be one (even though ``m = 2``). .. py:class:: QuantileRegressionEstimator Bases: :py:obj:`UncertaintyEstimator` A helper class for making model predictions and associated uncertainty predictions. .. py:method:: __call__(dataloader, models, trainer) Calculate the uncalibrated predictions and uncertainties for the dataloader. dataloader: DataLoader the dataloader used for model predictions and uncertainty predictions models: Iterable[MPNN] | Iterable[MolAtomBondMPNN] the models used for model predictions and uncertainty predictions. If using MolAtomBondMPNN models, the uncertainty estimator will return preds and uncs for each of the mole, atom, and bond predictions and uncertainties. trainer: pl.Trainer an instance of the :class:`~lightning.pytorch.trainer.trainer.Trainer` used to manage model inference :returns: * **preds** (*Tensor*) -- the model predictions, with shape varying by task type: * regression/binary classification: ``m x n x t`` * multiclass classification: ``m x n x t x c``, where ``m`` is the number of models, ``n`` is the number of inputs, ``t`` is the number of tasks, and ``c`` is the number of classes. * **uncs** (*Tensor*) -- the predicted uncertainties, with shapes of ``m' x n x t``. * *.. note::* -- The ``m`` and ``m'`` are different by definition. The ``m`` is the number of models, while the ``m'`` is the number of uncertainty estimations. For example, if two MVE or evidential models are provided, both ``m`` and ``m'`` are two. However, for an ensemble of two models, ``m'`` would be one (even though ``m = 2``). .. py:class:: UncertaintyEstimator Bases: :py:obj:`abc.ABC` A helper class for making model predictions and associated uncertainty predictions. .. py:method:: __call__(dataloader, models, trainer) :abstractmethod: Calculate the uncalibrated predictions and uncertainties for the dataloader. dataloader: DataLoader the dataloader used for model predictions and uncertainty predictions models: Iterable[MPNN] | Iterable[MolAtomBondMPNN] the models used for model predictions and uncertainty predictions. If using MolAtomBondMPNN models, the uncertainty estimator will return preds and uncs for each of the mole, atom, and bond predictions and uncertainties. trainer: pl.Trainer an instance of the :class:`~lightning.pytorch.trainer.trainer.Trainer` used to manage model inference :returns: * **preds** (*Tensor*) -- the model predictions, with shape varying by task type: * regression/binary classification: ``m x n x t`` * multiclass classification: ``m x n x t x c``, where ``m`` is the number of models, ``n`` is the number of inputs, ``t`` is the number of tasks, and ``c`` is the number of classes. * **uncs** (*Tensor*) -- the predicted uncertainties, with shapes of ``m' x n x t``. * *.. note::* -- The ``m`` and ``m'`` are different by definition. The ``m`` is the number of models, while the ``m'`` is the number of uncertainty estimations. For example, if two MVE or evidential models are provided, both ``m`` and ``m'`` are two. However, for an ensemble of two models, ``m'`` would be one (even though ``m = 2``). .. py:data:: UncertaintyEstimatorRegistry .. py:class:: BinaryClassificationEvaluator Bases: :py:obj:`abc.ABC` Evaluates the quality of uncertainty estimates in binary classification tasks. .. py:method:: evaluate(uncs, targets, mask) :abstractmethod: Evaluate the performance of uncertainty predictions against the model target values. :param uncs: the predicted uncertainties (i.e., the predicted probability of class 1) of the shape of ``n x t``, where ``n`` is the number of input molecules/reactions, and ``t`` is the number of tasks. :type uncs: Tensor :param targets: a tensor of the shape ``n x t`` :type targets: Tensor :param mask: a tensor of the shape ``n x t`` indicating whether the given values should be used in the evaluation :type mask: Tensor :returns: a tensor of the shape ``t`` containing the evaluated metrics :rtype: Tensor .. py:class:: CalibrationAreaEvaluator Bases: :py:obj:`RegressionEvaluator` A class for evaluating regression uncertainty values based on how they deviate from perfect calibration on an observed-probability versus expected-probability plot. .. py:method:: evaluate(preds, uncs, targets, mask, num_bins = 100) Evaluate the performance of uncertainty predictions against the model target values. :param preds: the predictions for regression tasks. It is a tensor of the shape of ``n x t``, where ``n`` is the number of input molecules/reactions, and ``t`` is the number of tasks. :type preds: Tensor :param uncs: the predicted uncertainties (variance) of the shape of ``n x t`` :type uncs: Tensor :param targets: a tensor of the shape ``n x t`` :type targets: Tensor :param mask: a tensor of the shape ``n x t`` indicating whether the given values should be used in the evaluation :type mask: Tensor :param num_bins: the number of bins to discretize the ``[0, 1]`` interval :type num_bins: int, default=100 :returns: a tensor of the shape ``t`` containing the evaluated metrics :rtype: Tensor .. py:class:: ExpectedNormalizedErrorEvaluator Bases: :py:obj:`RegressionEvaluator` A class that evaluates uncertainty performance by binning together clusters of predictions and comparing the average predicted variance of the clusters against the RMSE of the cluster. [1]_ .. math:: \mathrm{ENCE} = \frac{1}{N} \sum_{i=1}^{N} \frac{|\mathrm{RMV}_i - \mathrm{RMSE}_i|}{\mathrm{RMV}_i} where :math:`N` is the number of bins, :math:`\mathrm{RMV}_i` is the root of the mean uncertainty over the :math:`i`-th bin and :math:`\mathrm{RMSE}_i` is the root mean square error over the :math:`i`-th bin. This discrepancy is further normalized by the uncertainty over the bin, :math:`\mathrm{RMV}_i`, because the error is expected to be naturally higher as the uncertainty increases. .. rubric:: References .. [1] Levi, D.; Gispan, L.; Giladi, N.; Fetaya, E. "Evaluating and Calibrating Uncertainty Prediction in Regression Tasks." Sensors, 2022, 22(15), 5540. https://www.mdpi.com/1424-8220/22/15/5540 .. py:method:: evaluate(preds, uncs, targets, mask, num_bins = 100) Evaluate the performance of uncertainty predictions against the model target values. :param preds: the predictions for regression tasks. It is a tensor of the shape of ``n x t``, where ``n`` is the number of input molecules/reactions, and ``t`` is the number of tasks. :type preds: Tensor :param uncs: the predicted uncertainties (variance) of the shape of ``n x t`` :type uncs: Tensor :param targets: a tensor of the shape ``n x t`` :type targets: Tensor :param mask: a tensor of the shape ``n x t`` indicating whether the given values should be used in the evaluation :type mask: Tensor :param num_bins: the number of bins the data are divided into :type num_bins: int, default=100 :returns: a tensor of the shape ``t`` containing the evaluated metrics :rtype: Tensor .. py:class:: MulticlassClassificationEvaluator Bases: :py:obj:`abc.ABC` Evaluates the quality of uncertainty estimates in multiclass classification tasks. .. py:method:: evaluate(uncs, targets, mask) :abstractmethod: Evaluate the performance of uncertainty predictions against the model target values. :param uncs: the predicted uncertainties (i.e., the predicted probabilities for each class) of the shape of ``n x t x c``, where ``n`` is the number of input molecules/reactions, ``t`` is the number of tasks, and ``c`` is the number of classes. :type uncs: Tensor :param targets: a tensor of the shape ``n x t`` :type targets: Tensor :param mask: a tensor of the shape ``n x t`` indicating whether the given values should be used in the evaluation :type mask: Tensor :returns: a tensor of the shape ``t`` containing the evaluated metrics :rtype: Tensor .. py:class:: MulticlassConformalEvaluator Bases: :py:obj:`MulticlassClassificationEvaluator` Evaluate the coverage of conformal prediction for multiclass classification datasets. .. math:: \Pr (Y_{\text{test}} \in C(X_{\text{test}})) where the :math:`C(X_{\text{test}}) \subset \{1 \mathrel{.\,.} K\}` is a prediction set of possible labels . .. py:method:: evaluate(uncs, targets, mask) Evaluate the performance of uncertainty predictions against the model target values. :param uncs: the predicted uncertainties (i.e., the predicted probabilities for each class) of the shape of ``n x t x c``, where ``n`` is the number of input molecules/reactions, ``t`` is the number of tasks, and ``c`` is the number of classes. :type uncs: Tensor :param targets: a tensor of the shape ``n x t`` :type targets: Tensor :param mask: a tensor of the shape ``n x t`` indicating whether the given values should be used in the evaluation :type mask: Tensor :returns: a tensor of the shape ``t`` containing the evaluated metrics :rtype: Tensor .. py:class:: MultilabelConformalEvaluator Bases: :py:obj:`BinaryClassificationEvaluator` Evaluate the coverage of conformal prediction for binary classification datasets with multiple labels. .. math:: \Pr \left( \hat{\mathcal C}_{\text{in}}(X) \subseteq \mathcal Y \subseteq \hat{\mathcal C}_{\text{out}}(X) \right) where the in-set :math:`\hat{\mathcal C}_\text{in}` is contained by the set of true labels :math:`\mathcal Y` and :math:`\mathcal Y` is contained within the out-set :math:`\hat{\mathcal C}_\text{out}`. .. py:method:: evaluate(uncs, targets, mask) Evaluate the performance of uncertainty predictions against the model target values. :param uncs: the predicted uncertainties (i.e., the predicted probability of class 1) of the shape of ``n x t``, where ``n`` is the number of input molecules/reactions, and ``t`` is the number of tasks. :type uncs: Tensor :param targets: a tensor of the shape ``n x t`` :type targets: Tensor :param mask: a tensor of the shape ``n x t`` indicating whether the given values should be used in the evaluation :type mask: Tensor :returns: a tensor of the shape ``t`` containing the evaluated metrics :rtype: Tensor .. py:class:: NLLClassEvaluator Bases: :py:obj:`BinaryClassificationEvaluator` Evaluate uncertainty values for binary classification datasets using the mean negative-log-likelihood of the targets given the assigned probabilities from the model: .. math:: \mathrm{NLL} = -\log(\hat{y} \cdot y + (1 - \hat{y}) \cdot (1 - y)) where :math:`y` is the true binary label (0 or 1), and :math:`\hat{y}` is the predicted probability associated with the class label 1. The function returns a tensor containing the mean NLL for each task. .. py:method:: evaluate(uncs, targets, mask) Evaluate the performance of uncertainty predictions against the model target values. :param uncs: the predicted uncertainties (i.e., the predicted probability of class 1) of the shape of ``n x t``, where ``n`` is the number of input molecules/reactions, and ``t`` is the number of tasks. :type uncs: Tensor :param targets: a tensor of the shape ``n x t`` :type targets: Tensor :param mask: a tensor of the shape ``n x t`` indicating whether the given values should be used in the evaluation :type mask: Tensor :returns: a tensor of the shape ``t`` containing the evaluated metrics :rtype: Tensor .. py:class:: NLLMulticlassEvaluator Bases: :py:obj:`MulticlassClassificationEvaluator` Evaluate uncertainty values for multiclass classification datasets using the mean negative-log-likelihood of the targets given the assigned probabilities from the model: .. math:: \mathrm{NLL} = -\log(p_{y_i}) where :math:`p_{y_i}` is the predicted probability for the true class :math:`y_i`, calculated as: .. math:: p_{y_i} = \sum_{k=1}^{K} \mathbb{1}(y_i = k) \cdot p_k Here: :math:`K` is the total number of classes, :math:`\mathbb{1}(y_i = k)` is the indicator function that is 1 when the true class :math:`y_i` equals class :math:`k`, and 0 otherwise, and :math:`p_k` is the predicted probability for class :math:`k`. The function returns a tensor containing the mean NLL for each task. .. py:method:: evaluate(uncs, targets, mask) Evaluate the performance of uncertainty predictions against the model target values. :param uncs: the predicted uncertainties (i.e., the predicted probabilities for each class) of the shape of ``n x t x c``, where ``n`` is the number of input molecules/reactions, ``t`` is the number of tasks, and ``c`` is the number of classes. :type uncs: Tensor :param targets: a tensor of the shape ``n x t`` :type targets: Tensor :param mask: a tensor of the shape ``n x t`` indicating whether the given values should be used in the evaluation :type mask: Tensor :returns: a tensor of the shape ``t`` containing the evaluated metrics :rtype: Tensor .. py:class:: NLLRegressionEvaluator Bases: :py:obj:`RegressionEvaluator` Evaluate uncertainty values for regression datasets using the mean negative-log-likelihood of the targets given the probability distributions estimated by the model: .. math:: \mathrm{NLL}(y, \hat y) = \frac{1}{2} \log(2 \pi \sigma^2) + \frac{(y - \hat{y})^2}{2 \sigma^2} where :math:`\hat{y}` is the predicted value, :math:`y` is the true value, and :math:`\sigma^2` is the predicted uncertainty (variance). The function returns a tensor containing the mean NLL for each task. .. py:method:: evaluate(preds, uncs, targets, mask) Evaluate the performance of uncertainty predictions against the model target values. :param preds: the predictions for regression tasks. It is a tensor of the shape of ``n x t``, where ``n`` is the number of input molecules/reactions, and ``t`` is the number of tasks. :type preds: Tensor :param uncs: the predicted uncertainties of the shape of ``n x t`` :type uncs: Tensor :param targets: a tensor of the shape ``n x t`` :type targets: Tensor :param mask: a tensor of the shape ``n x t`` indicating whether the given values should be used in the evaluation :type mask: Tensor :returns: a tensor of the shape ``t`` containing the evaluated metrics :rtype: Tensor .. py:class:: RegressionConformalEvaluator Bases: :py:obj:`RegressionEvaluator` Evaluate the coverage of conformal prediction for regression datasets. .. math:: \Pr (Y_{\text{test}} \in C(X_{\text{test}})) where the :math:`C(X_{\text{test}})` is the predicted interval. .. py:method:: evaluate(preds, uncs, targets, mask) Evaluate the performance of uncertainty predictions against the model target values. :param preds: the predictions for regression tasks. It is a tensor of the shape of ``n x t``, where ``n`` is the number of input molecules/reactions, and ``t`` is the number of tasks. :type preds: Tensor :param uncs: the predicted uncertainties of the shape of ``n x t`` :type uncs: Tensor :param targets: a tensor of the shape ``n x t`` :type targets: Tensor :param mask: a tensor of the shape ``n x t`` indicating whether the given values should be used in the evaluation :type mask: Tensor :returns: a tensor of the shape ``t`` containing the evaluated metrics :rtype: Tensor .. py:class:: RegressionEvaluator Bases: :py:obj:`abc.ABC` Evaluates the quality of uncertainty estimates in regression tasks. .. py:method:: evaluate(preds, uncs, targets, mask) :abstractmethod: Evaluate the performance of uncertainty predictions against the model target values. :param preds: the predictions for regression tasks. It is a tensor of the shape of ``n x t``, where ``n`` is the number of input molecules/reactions, and ``t`` is the number of tasks. :type preds: Tensor :param uncs: the predicted uncertainties of the shape of ``n x t`` :type uncs: Tensor :param targets: a tensor of the shape ``n x t`` :type targets: Tensor :param mask: a tensor of the shape ``n x t`` indicating whether the given values should be used in the evaluation :type mask: Tensor :returns: a tensor of the shape ``t`` containing the evaluated metrics :rtype: Tensor .. py:class:: SpearmanEvaluator Bases: :py:obj:`RegressionEvaluator` Evaluate the Spearman rank correlation coefficient between the uncertainties and errors in the model predictions. The correlation coefficient returns a value in the [-1, 1] range, with better scores closer to 1 observed when the uncertainty values are predictive of the rank ordering of the errors in the model prediction. .. py:method:: evaluate(preds, uncs, targets, mask) Evaluate the performance of uncertainty predictions against the model target values. :param preds: the predictions for regression tasks. It is a tensor of the shape of ``n x t``, where ``n`` is the number of input molecules/reactions, and ``t`` is the number of tasks. :type preds: Tensor :param uncs: the predicted uncertainties of the shape of ``n x t`` :type uncs: Tensor :param targets: a tensor of the shape ``n x t`` :type targets: Tensor :param mask: a tensor of the shape ``n x t`` indicating whether the given values should be used in the evaluation :type mask: Tensor :returns: a tensor of the shape ``t`` containing the evaluated metrics :rtype: Tensor .. py:data:: UncertaintyEvaluatorRegistry