chemprop.uncertainty

Submodules

• chemprop.uncertainty.calibrator
• chemprop.uncertainty.estimator
• chemprop.uncertainty.evaluator

Attributes

UncertaintyCalibratorRegistry
UncertaintyEstimatorRegistry
UncertaintyEvaluatorRegistry

Classes

AdaptiveMulticlassConformalCalibrator	Create a prediction sets of possible labels \(C(X_{\text{test}}) \subset \{1 \mathrel{.\,.} K\}\) that follows:
BinaryClassificationCalibrator	A class for calibrating the predicted uncertainties in binary classification tasks.
CalibratorBase	A base class for calibrating the predicted uncertainties.
IsotonicCalibrator	Calibrate binary classification datasets using isotonic regression as discussed in [guo2017].
IsotonicMulticlassCalibrator	Calibrate multiclass classification datasets using isotonic regression as discussed in
MulticlassClassificationCalibrator	A class for calibrating the predicted uncertainties in multiclass classification tasks.
MulticlassConformalCalibrator	Create a prediction sets of possible labels \(C(X_{\text{test}}) \subset \{1 \mathrel{.\,.} K\}\) that follows:
MultilabelConformalCalibrator	Creates conformal in-set and conformal out-set such that, for \(1-\alpha\) proportion of datapoints,
MVEWeightingCalibrator	Calibrate regression datasets that have ensembles of individual models that make variance predictions.
PlattCalibrator	Calibrate classification datasets using the Platt scaling algorithm [guo2017], [platt1999].
RegressionCalibrator	A class for calibrating the predicted uncertainties in regressions tasks.
RegressionConformalCalibrator	Conformalize quantiles to make the interval \([\hat{t}_{\alpha/2}(x),\hat{t}_{1-\alpha/2}(x)]\) to have
ZelikmanCalibrator	Calibrate regression datasets using a method that does not depend on a particular probability function form.
ZScalingCalibrator	Calibrate regression datasets by applying a scaling value to the uncalibrated standard deviation,
ClassEstimator	A helper class for making model predictions and associated uncertainty predictions.
ClassificationDirichletEstimator	A ClassificationDirichletEstimator predicts an amount of 'evidence' for both the
DropoutEstimator	A DropoutEstimator creates a virtual ensemble of models via Monte Carlo dropout with
EnsembleEstimator	Class that predicts the uncertainty of predictions based on the variance in predictions among
EvidentialAleatoricEstimator	Class that predicts the aleatoric evidential uncertainty based on hyperparameters of
EvidentialEpistemicEstimator	Class that predicts the epistemic evidential uncertainty based on hyperparameters of
EvidentialTotalEstimator	Class that predicts the total evidential uncertainty based on hyperparameters of
MulticlassDirichletEstimator	A MulticlassDirichletEstimator predicts an amount of 'evidence' for each class as
MVEEstimator	Class that estimates prediction means and variances (MVE). [nix1994]
NoUncertaintyEstimator	A helper class for making model predictions and associated uncertainty predictions.
QuantileRegressionEstimator	A helper class for making model predictions and associated uncertainty predictions.
UncertaintyEstimator	A helper class for making model predictions and associated uncertainty predictions.
BinaryClassificationEvaluator	Evaluates the quality of uncertainty estimates in binary classification tasks.
CalibrationAreaEvaluator	A class for evaluating regression uncertainty values based on how they deviate from perfect
ExpectedNormalizedErrorEvaluator	A class that evaluates uncertainty performance by binning together clusters of predictions
MulticlassClassificationEvaluator	Evaluates the quality of uncertainty estimates in multiclass classification tasks.
MulticlassConformalEvaluator	Evaluate the coverage of conformal prediction for multiclass classification datasets.
MultilabelConformalEvaluator	Evaluate the coverage of conformal prediction for binary classification datasets with multiple labels.
NLLClassEvaluator	Evaluate uncertainty values for binary classification datasets using the mean negative-log-likelihood
NLLMulticlassEvaluator	Evaluate uncertainty values for multiclass classification datasets using the mean negative-log-likelihood
NLLRegressionEvaluator	Evaluate uncertainty values for regression datasets using the mean negative-log-likelihood
RegressionConformalEvaluator	Evaluate the coverage of conformal prediction for regression datasets.
RegressionEvaluator	Evaluates the quality of uncertainty estimates in regression tasks.
SpearmanEvaluator	Evaluate the Spearman rank correlation coefficient between the uncertainties and errors in the model predictions.

Package Contents

class chemprop.uncertainty.AdaptiveMulticlassConformalCalibrator(alpha)

Bases: MulticlassConformalCalibrator

Create a prediction sets of possible labels \(C(X_{\text{test}}) \subset \{1 \mathrel{.\,.} K\}\) that follows:

\[1 - \alpha \leq \Pr (Y_{\text{test}} \in C(X_{\text{test}})) \leq 1 - \alpha + \frac{1}{n + 1}\]

In other words, the probability that the prediction set contains the correct label is almost exactly \(1-\alpha\). More detailes can be found in [1]_.

Parameters:

alpha (float) – Error rate, \(\alpha \in [0, 1]\)

References

[1]

Angelopoulos, A.N.; Bates, S.; “A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification.” arXiv Preprint 2021, https://arxiv.org/abs/2107.07511

static nonconformity_scores(preds)

Compute nonconformity score by greedily including classes in the classification set until it reaches the true label.

\[s(x, y) = \sum_{j=1}^{k} \hat{f}(x)_{\pi_j(x)}, \text{ where } y = \pi_k(x)\]

where \(\pi_k(x)\) is the permutation of \(\{1 \mathrel{.\,.} K\}\) that sorts \(\hat{f}(X_{test})\) from most likely to least likely.

class chemprop.uncertainty.BinaryClassificationCalibrator

Bases: CalibratorBase

A class for calibrating the predicted uncertainties in binary classification tasks.

abstractmethod fit(uncs, targets, mask)

Fit calibration method for the calibration data.

Parameters:

• uncs (Tensor) – the predicted uncertainties (i.e., the predicted probability of class 1) of the shape of n x t, where n is the number of input molecules/reactions, and t is the number of tasks.

• targets (Tensor) – a tensor of the shape n x t

• mask (Tensor) – a tensor of the shape n x t indicating whether the given values should be used in the fitting

Returns:

self – the fitted calibrator

Return type:

BinaryClassificationCalibrator

class chemprop.uncertainty.CalibratorBase

Bases: abc.ABC

A base class for calibrating the predicted uncertainties.

abstractmethod fit(*args, **kwargs)

Fit calibration method for the calibration data.

Return type:

Self

abstractmethod apply(uncs)

Apply this calibrator to the input uncertainties.

Parameters:

uncs (Tensor) – a tensor containinig uncalibrated uncertainties

Returns:

the calibrated uncertainties

Return type:

Tensor

class chemprop.uncertainty.IsotonicCalibrator

Bases: BinaryClassificationCalibrator

Calibrate binary classification datasets using isotonic regression as discussed in [guo2017]. In effect, the method transforms incoming uncalibrated confidences using a histogram-like function where the range of each transforming bin and its magnitude is learned.

References

[guo2017]

Guo, C.; Pleiss, G.; Sun, Y.; Weinberger, K. Q. “On calibration of modern neural networks”. ICML, 2017. https://arxiv.org/abs/1706.04599

fit(uncs, targets, mask)

Fit calibration method for the calibration data.

Parameters:

• uncs (Tensor) – the predicted uncertainties (i.e., the predicted probability of class 1) of the shape of n x t, where n is the number of input molecules/reactions, and t is the number of tasks.

• targets (Tensor) – a tensor of the shape n x t

• mask (Tensor) – a tensor of the shape n x t indicating whether the given values should be used in the fitting

Returns:

self – the fitted calibrator

Return type:

BinaryClassificationCalibrator

apply(uncs)

Apply this calibrator to the input uncertainties.

Parameters:

uncs (Tensor) – a tensor containinig uncalibrated uncertainties

Returns:

the calibrated uncertainties

Return type:

Tensor

class chemprop.uncertainty.IsotonicMulticlassCalibrator

Bases: MulticlassClassificationCalibrator

Calibrate multiclass classification datasets using isotonic regression as discussed in [guo2017]. It uses a one-vs-all aggregation scheme to extend isotonic regression from binary to multiclass classifiers.

References

[guo2017]

Guo, C.; Pleiss, G.; Sun, Y.; Weinberger, K. Q. “On calibration of modern neural networks”. ICML, 2017. https://arxiv.org/abs/1706.04599

fit(uncs, targets, mask)

Fit calibration method for the calibration data.

Parameters:

• uncs (Tensor) – the predicted uncertainties (i.e., the predicted probabilities for each class) of the shape of n x t x c, where n is the number of input molecules/reactions, t is the number of tasks, and c is the number of classes.

• targets (Tensor) – a tensor of the shape n x t

• mask (Tensor) – a tensor of the shape n x t indicating whether the given values should be used in the fitting

Returns:

self – the fitted calibrator

Return type:

MulticlassClassificationCalibrator

apply(uncs)

Apply this calibrator to the input uncertainties.

Parameters:

uncs (Tensor) – a tensor containinig uncalibrated uncertainties

Returns:

the calibrated uncertainties

Return type:

Tensor

class chemprop.uncertainty.MulticlassClassificationCalibrator

Bases: CalibratorBase

A class for calibrating the predicted uncertainties in multiclass classification tasks.

abstractmethod fit(uncs, targets, mask)

Fit calibration method for the calibration data.

Parameters:

• uncs (Tensor) – the predicted uncertainties (i.e., the predicted probabilities for each class) of the shape of n x t x c, where n is the number of input molecules/reactions, t is the number of tasks, and c is the number of classes.

• targets (Tensor) – a tensor of the shape n x t

• mask (Tensor) – a tensor of the shape n x t indicating whether the given values should be used in the fitting

Returns:

self – the fitted calibrator

Return type:

MulticlassClassificationCalibrator

class chemprop.uncertainty.MulticlassConformalCalibrator(alpha)

Bases: MulticlassClassificationCalibrator

Create a prediction sets of possible labels \(C(X_{\text{test}}) \subset \{1 \mathrel{.\,.} K\}\) that follows:

\[1 - \alpha \leq \Pr (Y_{\text{test}} \in C(X_{\text{test}})) \leq 1 - \alpha + \frac{1}{n + 1}\]

In other words, the probability that the prediction set contains the correct label is almost exactly \(1-\alpha\). More detailes can be found in [1]_.

Parameters:

alpha (float) – Error rate, \(\alpha \in [0, 1]\)

References

[1]

Angelopoulos, A.N.; Bates, S.; “A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification.” arXiv Preprint 2021, https://arxiv.org/abs/2107.07511

alpha

static nonconformity_scores(preds)

Compute nonconformity score as the negative of the softmax output for the true class.

\[s_i = -\hat{f}(X_i)_{Y_i}\]

Parameters:

preds (torch.Tensor)

fit(uncs, targets, mask)

Fit calibration method for the calibration data.

Parameters:

• uncs (Tensor) – the predicted uncertainties (i.e., the predicted probabilities for each class) of the shape of n x t x c, where n is the number of input molecules/reactions, t is the number of tasks, and c is the number of classes.

• targets (Tensor) – a tensor of the shape n x t

• mask (Tensor) – a tensor of the shape n x t indicating whether the given values should be used in the fitting

Returns:

self – the fitted calibrator

Return type:

MulticlassClassificationCalibrator

apply(uncs)

Apply this calibrator to the input uncertainties.

Parameters:

uncs (Tensor) – a tensor containinig uncalibrated uncertainties

Returns:

the calibrated uncertainties

Return type:

Tensor

class chemprop.uncertainty.MultilabelConformalCalibrator(alpha)

Bases: BinaryClassificationCalibrator

Creates conformal in-set and conformal out-set such that, for \(1-\alpha\) proportion of datapoints, the set of labels is bounded by the in- and out-sets [1]_:

\[\Pr \left( \hat{\mathcal C}_{\text{in}}(X) \subseteq \mathcal Y \subseteq \hat{\mathcal C}_{\text{out}}(X) \right) \geq 1 - \alpha,\]

where the in-set \(\hat{\mathcal C}_\text{in}\) is contained by the set of true labels \(\mathcal Y\) and \(\mathcal Y\) is contained within the out-set \(\hat{\mathcal C}_\text{out}\).

Parameters:

alpha (float) – The error rate, \(\alpha \in [0, 1]\)

References

[1]

Cauchois, M.; Gupta, S.; Duchi, J.; “Knowing What You Know: Valid and Validated Confidence Sets in Multiclass and Multilabel Prediction.” arXiv Preprint 2020, https://arxiv.org/abs/2004.10181

alpha

static nonconformity_scores(preds)

Compute nonconformity score as the negative of the predicted probability.

\[s_i = -\hat{f}(X_i)_{Y_i}\]

Parameters:

preds (torch.Tensor)

fit(uncs, targets, mask)

Fit calibration method for the calibration data.

Parameters:

• uncs (Tensor) – the predicted uncertainties (i.e., the predicted probability of class 1) of the shape of n x t, where n is the number of input molecules/reactions, and t is the number of tasks.

• targets (Tensor) – a tensor of the shape n x t

• mask (Tensor) – a tensor of the shape n x t indicating whether the given values should be used in the fitting

Returns:

self – the fitted calibrator

Return type:

BinaryClassificationCalibrator

apply(uncs)

Apply this calibrator to the input uncertainties.

Parameters:

uncs (Tensor) – a tensor containinig uncalibrated uncertainties

Returns:

the calibrated uncertainties of the shape of n x t x 2, where n is the number of input molecules/reactions, t is the number of tasks, and the first element in the last dimension corresponds to the in-set \(\hat{\mathcal C}_\text{in}\), while the second corresponds to the out-set \(\hat{\mathcal C}_\text{out}\).

Return type:

Tensor

class chemprop.uncertainty.MVEWeightingCalibrator

Bases: RegressionCalibrator

Calibrate regression datasets that have ensembles of individual models that make variance predictions.

This method minimizes the negative log likelihood for the predictions versus the targets by applying a weighted average across the variance predictions of the ensemble. [wang2021]

References

[wang2021]

Wang, D.; Yu, J.; Chen, L.; Li, X.; Jiang, H.; Chen, K.; Zheng, M.; Luo, X. “A hybrid framework for improving uncertainty quantification in deep learning-based QSAR regression modeling.” J. Cheminform., 2021, 13, 1-17. https://doi.org/10.1186/s13321-021-00551-x

fit(preds, uncs, targets, mask)

Fit calibration method for the calibration data.

Parameters:

• preds (Tensor) – the predictions for regression tasks. It is a tensor of the shape of n x t, where n is the number of input molecules/reactions, and t is the number of tasks.

• uncs (Tensor) – the predicted uncertainties of the shape of m x n x t

• targets (Tensor) – a tensor of the shape n x t

• mask (Tensor) – a tensor of the shape n x t indicating whether the given values should be used in the fitting

Returns:

self – the fitted calibrator

Return type:

MVEWeightingCalibrator

apply(uncs)

Apply this calibrator to the input uncertainties.

Parameters:

uncs (Tensor) – a tensor containinig uncalibrated uncertainties of the shape of m x n x t

Returns:

the calibrated uncertainties of the shape of n x t

Return type:

Tensor

class chemprop.uncertainty.PlattCalibrator

Bases: BinaryClassificationCalibrator

Calibrate classification datasets using the Platt scaling algorithm [guo2017], [platt1999].

In [platt1999], Platt suggests using the number of positive and negative training examples to adjust the value of target probabilities used to fit the parameters.

References

[guo2017]

Guo, C.; Pleiss, G.; Sun, Y.; Weinberger, K. Q. “On calibration of modern neural networks”. ICML, 2017. https://arxiv.org/abs/1706.04599

[platt1999] (1,2,3)

Platt, J. “Probabilistic Outputs for Support Vector Machines and Comparisons to Regularized Likelihood Methods.” Adv. Large Margin Classif. 1999, 10 (3), 61–74.

fit(uncs, targets, mask, training_targets=None)

Fit calibration method for the calibration data.

Parameters:

• uncs (Tensor) – the predicted uncertainties (i.e., the predicted probability of class 1) of the shape of n x t, where n is the number of input molecules/reactions, and t is the number of tasks.

• targets (Tensor) – a tensor of the shape n x t

• mask (Tensor) – a tensor of the shape n x t indicating whether the given values should be used in the fitting

• training_targets (torch.Tensor | None)

Returns:

self – the fitted calibrator

Return type:

BinaryClassificationCalibrator

apply(uncs)

Apply this calibrator to the input uncertainties.

Parameters:

uncs (Tensor) – a tensor containinig uncalibrated uncertainties

Returns:

the calibrated uncertainties

Return type:

Tensor

class chemprop.uncertainty.RegressionCalibrator

Bases: CalibratorBase

A class for calibrating the predicted uncertainties in regressions tasks.

abstractmethod fit(preds, uncs, targets, mask)

Fit calibration method for the calibration data.

Parameters:

• preds (Tensor) – the predictions for regression tasks. It is a tensor of the shape of n x t, where n is the number of input molecules/reactions, and t is the number of tasks.

• uncs (Tensor) – the predicted uncertainties of the shape of n x t

• targets (Tensor) – a tensor of the shape n x t

• mask (Tensor) – a tensor of the shape n x t indicating whether the given values should be used in the fitting

Returns:

self – the fitted calibrator

Return type:

RegressionCalibrator

class chemprop.uncertainty.RegressionConformalCalibrator(alpha)

Bases: RegressionCalibrator

Conformalize quantiles to make the interval \([\hat{t}_{\alpha/2}(x),\hat{t}_{1-\alpha/2}(x)]\) to have approximately \(1-\alpha\) coverage. [angelopoulos2021]

\[ \begin{align}\begin{aligned}s(x, y) &= \max \left\{ \hat{t}_{\alpha/2}(x) - y, y - \hat{t}_{1-\alpha/2}(x) \right\}\\\hat{q} &= Q(s_1, \ldots, s_n; \left\lceil \frac{(n+1)(1-\alpha)}{n} \right\rceil)\\C(x) &= \left[ \hat{t}_{\alpha/2}(x) - \hat{q}, \hat{t}_{1-\alpha/2}(x) + \hat{q} \right]\end{aligned}\end{align} \]

where \(s\) is the nonconformity score as the difference between \(y\) and its nearest quantile. \(\hat{t}_{\alpha/2}(x)\) and \(\hat{t}_{1-\alpha/2}(x)\) are the predicted quantiles from a quantile regression model.

Note

The algorithm is specifically designed for quantile regression model. Intuitively, the set \(C(x)\) just grows or shrinks the distance between the quantiles by \(\hat{q}\) to achieve coverage. However, this function can also be applied to regression model without quantiles being provided. In this case, both \(\hat{t}_{\alpha/2}(x)\) and \(\hat{t}_{1-\alpha/2}(x)\) are the same as \(\hat{y}\). Then, the interval would be the same for every data point (i.e., \(\left[-\hat{q}, \hat{q} \right]\)).

Parameters:

alpha (float) – The error rate, \(\alpha \in [0, 1]\)

References

[angelopoulos2021]

Angelopoulos, A.N.; Bates, S.; “A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification.” arXiv Preprint 2021, https://arxiv.org/abs/2107.07511

alpha

bounds

fit(preds, uncs, targets, mask)

Fit calibration method for the calibration data.

Parameters:

• preds (Tensor) – the predictions for regression tasks. It is a tensor of the shape of n x t, where n is the number of input molecules/reactions, and t is the number of tasks.

• uncs (Tensor) – the predicted uncertainties of the shape of n x t

• targets (Tensor) – a tensor of the shape n x t

• mask (Tensor) – a tensor of the shape n x t indicating whether the given values should be used in the fitting

Returns:

self – the fitted calibrator

Return type:

RegressionCalibrator

apply(uncs)

Apply this calibrator to the input uncertainties (half intervals).

Parameters:

uncs (Tensor) – a tensor containinig uncalibrated uncertainties

Returns:

the calibrated half intervals

Return type:

Tensor

chemprop.uncertainty.UncertaintyCalibratorRegistry

class chemprop.uncertainty.ZelikmanCalibrator(p)

Bases: RegressionCalibrator

Calibrate regression datasets using a method that does not depend on a particular probability function form.

It uses the “CRUDE” method as described in [zelikman2020]. We implemented this method to be used with variance as the uncertainty.

Parameters:

p (float) – The target qunatile, \(p \in [0, 1]\)

References

[zelikman2020]

Zelikman, E.; Healy, C.; Zhou, S.; Avati, A. “CRUDE: calibrating regression uncertainty distributions empirically.” arXiv preprint arXiv:2005.12496. https://doi.org/10.48550/arXiv.2005.12496

p

fit(preds, uncs, targets, mask)

Fit calibration method for the calibration data.

Parameters:

• preds (Tensor) – the predictions for regression tasks. It is a tensor of the shape of n x t, where n is the number of input molecules/reactions, and t is the number of tasks.

• uncs (Tensor) – the predicted uncertainties of the shape of n x t

• targets (Tensor) – a tensor of the shape n x t

• mask (Tensor) – a tensor of the shape n x t indicating whether the given values should be used in the fitting

Returns:

self – the fitted calibrator

Return type:

RegressionCalibrator

apply(uncs)

Apply this calibrator to the input uncertainties.

Parameters:

uncs (Tensor) – a tensor containinig uncalibrated uncertainties

Returns:

the calibrated uncertainties

Return type:

Tensor

class chemprop.uncertainty.ZScalingCalibrator

Bases: RegressionCalibrator

Calibrate regression datasets by applying a scaling value to the uncalibrated standard deviation, fitted by minimizing the negative-log-likelihood of a normal distribution around each prediction. [levi2022]

References

[levi2022]

Levi, D.; Gispan, L.; Giladi, N.; Fetaya, E. “Evaluating and Calibrating Uncertainty Prediction in Regression Tasks.” Sensors, 2022, 22(15), 5540. https://www.mdpi.com/1424-8220/22/15/5540

fit(preds, uncs, targets, mask)

Fit calibration method for the calibration data.

Parameters:

• preds (Tensor) – the predictions for regression tasks. It is a tensor of the shape of n x t, where n is the number of input molecules/reactions, and t is the number of tasks.

• uncs (Tensor) – the predicted uncertainties of the shape of n x t

• targets (Tensor) – a tensor of the shape n x t

• mask (Tensor) – a tensor of the shape n x t indicating whether the given values should be used in the fitting

Returns:

self – the fitted calibrator

Return type:

RegressionCalibrator

apply(uncs)

Apply this calibrator to the input uncertainties.

Parameters:

uncs (Tensor) – a tensor containinig uncalibrated uncertainties

Returns:

the calibrated uncertainties

Return type:

Tensor

class chemprop.uncertainty.ClassEstimator

Bases: UncertaintyEstimator

A helper class for making model predictions and associated uncertainty predictions.

__call__(dataloader, models, trainer)

Calculate the uncalibrated predictions and uncertainties for the dataloader.

dataloader: DataLoader

the dataloader used for model predictions and uncertainty predictions

models: Iterable[MPNN] | Iterable[MolAtomBondMPNN]

the models used for model predictions and uncertainty predictions. If using MolAtomBondMPNN models, the uncertainty estimator will return preds and uncs for each of the mole, atom, and bond predictions and uncertainties.

trainer: pl.Trainer

an instance of the Trainer used to manage model inference

Returns:

• preds (Tensor) – the model predictions, with shape varying by task type:

  • regression/binary classification: m x n x t

  • multiclass classification: m x n x t x c, where m is the number of models,

  n is the number of inputs, t is the number of tasks, and c is the number of classes.

• uncs (Tensor) – the predicted uncertainties, with shapes of m' x n x t.

• .. note:: – The m and m' are different by definition. The m is the number of models, while the m' is the number of uncertainty estimations. For example, if two MVE or evidential models are provided, both m and m' are two. However, for an ensemble of two models, m' would be one (even though m = 2).

Parameters:

• dataloader (torch.utils.data.DataLoader)

• models (Iterable[chemprop.models.MPNN] | Iterable[chemprop.models.MolAtomBondMPNN])

• trainer (lightning.pytorch.Trainer)

Return type:

tuple[torch.Tensor, torch.Tensor] | tuple[tuple[torch.Tensor | None, torch.Tensor | None, torch.Tensor | None], tuple[torch.Tensor | None, torch.Tensor | None, torch.Tensor | None]]

class chemprop.uncertainty.ClassificationDirichletEstimator

Bases: UncertaintyEstimator

A ClassificationDirichletEstimator predicts an amount of ‘evidence’ for both the negative class and the positive class as described in [sensoy2018]. The class probabilities and the uncertainty are calculated based on the evidence.

\[S = \sum_{i=1}^K \alpha_i p_i = \alpha_i / S u = K / S\]

where \(K\) is the number of classes, \(\alpha_i\) is the evidence for class \(i\), \(p_i\) is the probability of class \(i\), and \(u\) is the uncertainty.

References

[sensoy2018]

Sensoy, M.; Kaplan, L.; Kandemir, M. “Evidential deep learning to quantify classification uncertainty.” NeurIPS, 2018, 31. https://doi.org/10.48550/arXiv.1806.01768

__call__(dataloader, models, trainer)

Calculate the uncalibrated predictions and uncertainties for the dataloader.

dataloader: DataLoader

the dataloader used for model predictions and uncertainty predictions

models: Iterable[MPNN] | Iterable[MolAtomBondMPNN]

the models used for model predictions and uncertainty predictions. If using MolAtomBondMPNN models, the uncertainty estimator will return preds and uncs for each of the mole, atom, and bond predictions and uncertainties.

trainer: pl.Trainer

an instance of the Trainer used to manage model inference

Returns:

• preds (Tensor) – the model predictions, with shape varying by task type:

  • regression/binary classification: m x n x t

  • multiclass classification: m x n x t x c, where m is the number of models,

  n is the number of inputs, t is the number of tasks, and c is the number of classes.

• uncs (Tensor) – the predicted uncertainties, with shapes of m' x n x t.

• .. note:: – The m and m' are different by definition. The m is the number of models, while the m' is the number of uncertainty estimations. For example, if two MVE or evidential models are provided, both m and m' are two. However, for an ensemble of two models, m' would be one (even though m = 2).

Parameters:

• dataloader (torch.utils.data.DataLoader)

• models (Iterable[chemprop.models.MPNN] | Iterable[chemprop.models.MolAtomBondMPNN])

• trainer (lightning.pytorch.Trainer)

Return type:

tuple[torch.Tensor, torch.Tensor] | tuple[tuple[torch.Tensor | None, torch.Tensor | None, torch.Tensor | None], tuple[torch.Tensor | None, torch.Tensor | None, torch.Tensor | None]]

class chemprop.uncertainty.DropoutEstimator(ensemble_size, dropout=None)

Bases: UncertaintyEstimator

A DropoutEstimator creates a virtual ensemble of models via Monte Carlo dropout with the provided model [gal2016].

Parameters:

• ensemble_size (int) – The number of samples to draw for the ensemble.

• dropout (float | None) – The probability of dropping out units in the dropout layers. If unspecified, the training probability is used, which is prefered but not possible if the model was not trained with dropout (i.e. p=0).

References

[gal2016]

Gal, Y.; Ghahramani, Z. “Dropout as a bayesian approximation: Representing model uncertainty in deep learning.” International conference on machine learning. PMLR, 2016. https://arxiv.org/abs/1506.02142

ensemble_size

dropout = None

__call__(dataloader, models, trainer)

Calculate the uncalibrated predictions and uncertainties for the dataloader.

dataloader: DataLoader

the dataloader used for model predictions and uncertainty predictions

models: Iterable[MPNN] | Iterable[MolAtomBondMPNN]

the models used for model predictions and uncertainty predictions. If using MolAtomBondMPNN models, the uncertainty estimator will return preds and uncs for each of the mole, atom, and bond predictions and uncertainties.

trainer: pl.Trainer

an instance of the Trainer used to manage model inference

Returns:

• preds (Tensor) – the model predictions, with shape varying by task type:

  • regression/binary classification: m x n x t

  • multiclass classification: m x n x t x c, where m is the number of models,

  n is the number of inputs, t is the number of tasks, and c is the number of classes.

• uncs (Tensor) – the predicted uncertainties, with shapes of m' x n x t.

• .. note:: – The m and m' are different by definition. The m is the number of models, while the m' is the number of uncertainty estimations. For example, if two MVE or evidential models are provided, both m and m' are two. However, for an ensemble of two models, m' would be one (even though m = 2).

Parameters:

• dataloader (torch.utils.data.DataLoader)

• models (Iterable[chemprop.models.MPNN] | Iterable[chemprop.models.MolAtomBondMPNN])

• trainer (lightning.pytorch.Trainer)

Return type:

tuple[torch.Tensor, torch.Tensor] | tuple[tuple[torch.Tensor | None, torch.Tensor | None, torch.Tensor | None], tuple[torch.Tensor | None, torch.Tensor | None, torch.Tensor | None]]

class chemprop.uncertainty.EnsembleEstimator

Bases: UncertaintyEstimator

Class that predicts the uncertainty of predictions based on the variance in predictions among an ensemble’s submodels.

__call__(dataloader, models, trainer)

Calculate the uncalibrated predictions and uncertainties for the dataloader.

dataloader: DataLoader

the dataloader used for model predictions and uncertainty predictions

models: Iterable[MPNN] | Iterable[MolAtomBondMPNN]

the models used for model predictions and uncertainty predictions. If using MolAtomBondMPNN models, the uncertainty estimator will return preds and uncs for each of the mole, atom, and bond predictions and uncertainties.

trainer: pl.Trainer

an instance of the Trainer used to manage model inference

Returns:

• preds (Tensor) – the model predictions, with shape varying by task type:

  • regression/binary classification: m x n x t

  • multiclass classification: m x n x t x c, where m is the number of models,

  n is the number of inputs, t is the number of tasks, and c is the number of classes.

• uncs (Tensor) – the predicted uncertainties, with shapes of m' x n x t.

• .. note:: – The m and m' are different by definition. The m is the number of models, while the m' is the number of uncertainty estimations. For example, if two MVE or evidential models are provided, both m and m' are two. However, for an ensemble of two models, m' would be one (even though m = 2).

Parameters:

• dataloader (torch.utils.data.DataLoader)

• models (Iterable[chemprop.models.MPNN] | Iterable[chemprop.models.MolAtomBondMPNN])

• trainer (lightning.pytorch.Trainer)

Return type:

tuple[torch.Tensor, torch.Tensor] | tuple[tuple[torch.Tensor | None, torch.Tensor | None, torch.Tensor | None], tuple[torch.Tensor | None, torch.Tensor | None, torch.Tensor | None]]

class chemprop.uncertainty.EvidentialAleatoricEstimator

Bases: UncertaintyEstimator

Class that predicts the aleatoric evidential uncertainty based on hyperparameters of the evidential distribution.

__call__(dataloader, models, trainer)

Calculate the uncalibrated predictions and uncertainties for the dataloader.

dataloader: DataLoader

the dataloader used for model predictions and uncertainty predictions

models: Iterable[MPNN] | Iterable[MolAtomBondMPNN]

the models used for model predictions and uncertainty predictions. If using MolAtomBondMPNN models, the uncertainty estimator will return preds and uncs for each of the mole, atom, and bond predictions and uncertainties.

trainer: pl.Trainer

an instance of the Trainer used to manage model inference

Returns:

• preds (Tensor) – the model predictions, with shape varying by task type:

  • regression/binary classification: m x n x t

  • multiclass classification: m x n x t x c, where m is the number of models,

  n is the number of inputs, t is the number of tasks, and c is the number of classes.

• uncs (Tensor) – the predicted uncertainties, with shapes of m' x n x t.

• .. note:: – The m and m' are different by definition. The m is the number of models, while the m' is the number of uncertainty estimations. For example, if two MVE or evidential models are provided, both m and m' are two. However, for an ensemble of two models, m' would be one (even though m = 2).

Parameters:

• dataloader (torch.utils.data.DataLoader)

• models (Iterable[chemprop.models.MPNN] | Iterable[chemprop.models.MolAtomBondMPNN])

• trainer (lightning.pytorch.Trainer)

Return type:

tuple[torch.Tensor, torch.Tensor] | tuple[tuple[torch.Tensor | None, torch.Tensor | None, torch.Tensor | None], tuple[torch.Tensor | None, torch.Tensor | None, torch.Tensor | None]]

class chemprop.uncertainty.EvidentialEpistemicEstimator

Bases: UncertaintyEstimator

Class that predicts the epistemic evidential uncertainty based on hyperparameters of the evidential distribution.

__call__(dataloader, models, trainer)

Calculate the uncalibrated predictions and uncertainties for the dataloader.

dataloader: DataLoader

the dataloader used for model predictions and uncertainty predictions

models: Iterable[MPNN] | Iterable[MolAtomBondMPNN]

the models used for model predictions and uncertainty predictions. If using MolAtomBondMPNN models, the uncertainty estimator will return preds and uncs for each of the mole, atom, and bond predictions and uncertainties.

trainer: pl.Trainer

an instance of the Trainer used to manage model inference

Returns:

• preds (Tensor) – the model predictions, with shape varying by task type:

  • regression/binary classification: m x n x t

  • multiclass classification: m x n x t x c, where m is the number of models,

  n is the number of inputs, t is the number of tasks, and c is the number of classes.

• uncs (Tensor) – the predicted uncertainties, with shapes of m' x n x t.

• .. note:: – The m and m' are different by definition. The m is the number of models, while the m' is the number of uncertainty estimations. For example, if two MVE or evidential models are provided, both m and m' are two. However, for an ensemble of two models, m' would be one (even though m = 2).

Parameters:

• dataloader (torch.utils.data.DataLoader)

• models (Iterable[chemprop.models.MPNN] | Iterable[chemprop.models.MolAtomBondMPNN])

• trainer (lightning.pytorch.Trainer)

Return type:

tuple[torch.Tensor, torch.Tensor] | tuple[tuple[torch.Tensor | None, torch.Tensor | None, torch.Tensor | None], tuple[torch.Tensor | None, torch.Tensor | None, torch.Tensor | None]]

class chemprop.uncertainty.EvidentialTotalEstimator

Bases: UncertaintyEstimator

Class that predicts the total evidential uncertainty based on hyperparameters of the evidential distribution [amini2020].

References

[amini2020]

Amini, A.; Schwarting, W.; Soleimany, A.; Rus, D. “Deep Evidential Regression”. NeurIPS, 2020. https://arxiv.org/abs/1910.02600

__call__(dataloader, models, trainer)

Calculate the uncalibrated predictions and uncertainties for the dataloader.

dataloader: DataLoader

the dataloader used for model predictions and uncertainty predictions

models: Iterable[MPNN] | Iterable[MolAtomBondMPNN]

the models used for model predictions and uncertainty predictions. If using MolAtomBondMPNN models, the uncertainty estimator will return preds and uncs for each of the mole, atom, and bond predictions and uncertainties.

trainer: pl.Trainer

an instance of the Trainer used to manage model inference

Returns:

• preds (Tensor) – the model predictions, with shape varying by task type:

  • regression/binary classification: m x n x t

  • multiclass classification: m x n x t x c, where m is the number of models,

  n is the number of inputs, t is the number of tasks, and c is the number of classes.

• uncs (Tensor) – the predicted uncertainties, with shapes of m' x n x t.

• .. note:: – The m and m' are different by definition. The m is the number of models, while the m' is the number of uncertainty estimations. For example, if two MVE or evidential models are provided, both m and m' are two. However, for an ensemble of two models, m' would be one (even though m = 2).

Parameters:

• dataloader (torch.utils.data.DataLoader)

• models (Iterable[chemprop.models.MPNN] | Iterable[chemprop.models.MolAtomBondMPNN])

• trainer (lightning.pytorch.Trainer)

Return type:

tuple[torch.Tensor, torch.Tensor] | tuple[tuple[torch.Tensor | None, torch.Tensor | None, torch.Tensor | None], tuple[torch.Tensor | None, torch.Tensor | None, torch.Tensor | None]]

class chemprop.uncertainty.MulticlassDirichletEstimator

Bases: UncertaintyEstimator

A MulticlassDirichletEstimator predicts an amount of ‘evidence’ for each class as described in [sensoy2018]. The class probabilities and the uncertainty are calculated based on the evidence.

\[S = \sum_{i=1}^K \alpha_i p_i = \alpha_i / S u = K / S\]

where \(K\) is the number of classes, \(\alpha_i\) is the evidence for class \(i\), \(p_i\) is the probability of class \(i\), and \(u\) is the uncertainty.

References

[sensoy2018]

Sensoy, M.; Kaplan, L.; Kandemir, M. “Evidential deep learning to quantify classification uncertainty.” NeurIPS, 2018, 31. https://doi.org/10.48550/arXiv.1806.01768

__call__(dataloader, models, trainer)

Calculate the uncalibrated predictions and uncertainties for the dataloader.

dataloader: DataLoader

the dataloader used for model predictions and uncertainty predictions

models: Iterable[MPNN] | Iterable[MolAtomBondMPNN]

the models used for model predictions and uncertainty predictions. If using MolAtomBondMPNN models, the uncertainty estimator will return preds and uncs for each of the mole, atom, and bond predictions and uncertainties.

trainer: pl.Trainer

an instance of the Trainer used to manage model inference

Returns:

• preds (Tensor) – the model predictions, with shape varying by task type:

  • regression/binary classification: m x n x t

  • multiclass classification: m x n x t x c, where m is the number of models,

  n is the number of inputs, t is the number of tasks, and c is the number of classes.

• uncs (Tensor) – the predicted uncertainties, with shapes of m' x n x t.

• .. note:: – The m and m' are different by definition. The m is the number of models, while the m' is the number of uncertainty estimations. For example, if two MVE or evidential models are provided, both m and m' are two. However, for an ensemble of two models, m' would be one (even though m = 2).

Parameters:

• dataloader (torch.utils.data.DataLoader)

• models (Iterable[chemprop.models.MPNN] | Iterable[chemprop.models.MolAtomBondMPNN])

• trainer (lightning.pytorch.Trainer)

Return type:

tuple[torch.Tensor, torch.Tensor] | tuple[tuple[torch.Tensor | None, torch.Tensor | None, torch.Tensor | None], tuple[torch.Tensor | None, torch.Tensor | None, torch.Tensor | None]]

class chemprop.uncertainty.MVEEstimator

Bases: UncertaintyEstimator

Class that estimates prediction means and variances (MVE). [nix1994]

References

[nix1994] (1,2)

Nix, D. A.; Weigend, A. S. “Estimating the mean and variance of the target probability distribution.” Proceedings of 1994 IEEE International Conference on Neural Networks, 1994 https://doi.org/10.1109/icnn.1994.374138

__call__(dataloader, models, trainer)

Calculate the uncalibrated predictions and uncertainties for the dataloader.

dataloader: DataLoader

the dataloader used for model predictions and uncertainty predictions

models: Iterable[MPNN] | Iterable[MolAtomBondMPNN]

the models used for model predictions and uncertainty predictions. If using MolAtomBondMPNN models, the uncertainty estimator will return preds and uncs for each of the mole, atom, and bond predictions and uncertainties.

trainer: pl.Trainer

an instance of the Trainer used to manage model inference

Returns:

• preds (Tensor) – the model predictions, with shape varying by task type:

  • regression/binary classification: m x n x t

  • multiclass classification: m x n x t x c, where m is the number of models,

  n is the number of inputs, t is the number of tasks, and c is the number of classes.

• uncs (Tensor) – the predicted uncertainties, with shapes of m' x n x t.

• .. note:: – The m and m' are different by definition. The m is the number of models, while the m' is the number of uncertainty estimations. For example, if two MVE or evidential models are provided, both m and m' are two. However, for an ensemble of two models, m' would be one (even though m = 2).

Parameters:

• dataloader (torch.utils.data.DataLoader)

• models (Iterable[chemprop.models.MPNN] | Iterable[chemprop.models.MolAtomBondMPNN])

• trainer (lightning.pytorch.Trainer)

Return type:

tuple[torch.Tensor, torch.Tensor] | tuple[tuple[torch.Tensor | None, torch.Tensor | None, torch.Tensor | None], tuple[torch.Tensor | None, torch.Tensor | None, torch.Tensor | None]]

class chemprop.uncertainty.NoUncertaintyEstimator

Bases: UncertaintyEstimator

A helper class for making model predictions and associated uncertainty predictions.

__call__(dataloader, models, trainer)

Calculate the uncalibrated predictions and uncertainties for the dataloader.

dataloader: DataLoader

the dataloader used for model predictions and uncertainty predictions

models: Iterable[MPNN] | Iterable[MolAtomBondMPNN]

the models used for model predictions and uncertainty predictions. If using MolAtomBondMPNN models, the uncertainty estimator will return preds and uncs for each of the mole, atom, and bond predictions and uncertainties.

trainer: pl.Trainer

an instance of the Trainer used to manage model inference

Returns:

• preds (Tensor) – the model predictions, with shape varying by task type:

  • regression/binary classification: m x n x t

  • multiclass classification: m x n x t x c, where m is the number of models,

  n is the number of inputs, t is the number of tasks, and c is the number of classes.

• uncs (Tensor) – the predicted uncertainties, with shapes of m' x n x t.

• .. note:: – The m and m' are different by definition. The m is the number of models, while the m' is the number of uncertainty estimations. For example, if two MVE or evidential models are provided, both m and m' are two. However, for an ensemble of two models, m' would be one (even though m = 2).

Parameters:

• dataloader (torch.utils.data.DataLoader)

• models (Iterable[chemprop.models.MPNN] | Iterable[chemprop.models.MolAtomBondMPNN])

• trainer (lightning.pytorch.Trainer)

Return type:

tuple[torch.Tensor, torch.Tensor] | tuple[tuple[torch.Tensor | None, torch.Tensor | None, torch.Tensor | None], tuple[torch.Tensor | None, torch.Tensor | None, torch.Tensor | None]]

class chemprop.uncertainty.QuantileRegressionEstimator

Bases: UncertaintyEstimator

A helper class for making model predictions and associated uncertainty predictions.

__call__(dataloader, models, trainer)

Calculate the uncalibrated predictions and uncertainties for the dataloader.

dataloader: DataLoader

the dataloader used for model predictions and uncertainty predictions

models: Iterable[MPNN] | Iterable[MolAtomBondMPNN]

the models used for model predictions and uncertainty predictions. If using MolAtomBondMPNN models, the uncertainty estimator will return preds and uncs for each of the mole, atom, and bond predictions and uncertainties.

trainer: pl.Trainer

an instance of the Trainer used to manage model inference

Returns:

• preds (Tensor) – the model predictions, with shape varying by task type:

  • regression/binary classification: m x n x t

  • multiclass classification: m x n x t x c, where m is the number of models,

  n is the number of inputs, t is the number of tasks, and c is the number of classes.

• uncs (Tensor) – the predicted uncertainties, with shapes of m' x n x t.

• .. note:: – The m and m' are different by definition. The m is the number of models, while the m' is the number of uncertainty estimations. For example, if two MVE or evidential models are provided, both m and m' are two. However, for an ensemble of two models, m' would be one (even though m = 2).

Parameters:

• dataloader (torch.utils.data.DataLoader)

• models (Iterable[chemprop.models.MPNN] | Iterable[chemprop.models.MolAtomBondMPNN])

• trainer (lightning.pytorch.Trainer)

Return type:

tuple[torch.Tensor, torch.Tensor] | tuple[tuple[torch.Tensor | None, torch.Tensor | None, torch.Tensor | None], tuple[torch.Tensor | None, torch.Tensor | None, torch.Tensor | None]]

class chemprop.uncertainty.UncertaintyEstimator

Bases: abc.ABC

A helper class for making model predictions and associated uncertainty predictions.

abstractmethod __call__(dataloader, models, trainer)

Calculate the uncalibrated predictions and uncertainties for the dataloader.

dataloader: DataLoader

the dataloader used for model predictions and uncertainty predictions

models: Iterable[MPNN] | Iterable[MolAtomBondMPNN]

the models used for model predictions and uncertainty predictions. If using MolAtomBondMPNN models, the uncertainty estimator will return preds and uncs for each of the mole, atom, and bond predictions and uncertainties.

trainer: pl.Trainer

an instance of the Trainer used to manage model inference

Returns:

• preds (Tensor) – the model predictions, with shape varying by task type:

  • regression/binary classification: m x n x t

  • multiclass classification: m x n x t x c, where m is the number of models,

  n is the number of inputs, t is the number of tasks, and c is the number of classes.

• uncs (Tensor) – the predicted uncertainties, with shapes of m' x n x t.

• .. note:: – The m and m' are different by definition. The m is the number of models, while the m' is the number of uncertainty estimations. For example, if two MVE or evidential models are provided, both m and m' are two. However, for an ensemble of two models, m' would be one (even though m = 2).

Parameters:

• dataloader (torch.utils.data.DataLoader)

• models (Iterable[chemprop.models.MPNN] | Iterable[chemprop.models.MolAtomBondMPNN])

• trainer (lightning.pytorch.Trainer)

Return type:

tuple[torch.Tensor, torch.Tensor] | tuple[tuple[torch.Tensor | None, torch.Tensor | None, torch.Tensor | None], tuple[torch.Tensor | None, torch.Tensor | None, torch.Tensor | None]]

chemprop.uncertainty.UncertaintyEstimatorRegistry

class chemprop.uncertainty.BinaryClassificationEvaluator

Bases: abc.ABC

Evaluates the quality of uncertainty estimates in binary classification tasks.

abstractmethod evaluate(uncs, targets, mask)

Evaluate the performance of uncertainty predictions against the model target values.

Parameters:

• uncs (Tensor) – the predicted uncertainties (i.e., the predicted probability of class 1) of the shape of n x t, where n is the number of input molecules/reactions, and t is the number of tasks.

• targets (Tensor) – a tensor of the shape n x t

• mask (Tensor) – a tensor of the shape n x t indicating whether the given values should be used in the evaluation

Returns:

a tensor of the shape t containing the evaluated metrics

Return type:

Tensor

class chemprop.uncertainty.CalibrationAreaEvaluator

Bases: RegressionEvaluator

A class for evaluating regression uncertainty values based on how they deviate from perfect calibration on an observed-probability versus expected-probability plot.

evaluate(preds, uncs, targets, mask, num_bins=100)

Evaluate the performance of uncertainty predictions against the model target values.

Parameters:

• preds (Tensor) – the predictions for regression tasks. It is a tensor of the shape of n x t, where n is the number of input molecules/reactions, and t is the number of tasks.

• uncs (Tensor) – the predicted uncertainties (variance) of the shape of n x t

• targets (Tensor) – a tensor of the shape n x t

• mask (Tensor) – a tensor of the shape n x t indicating whether the given values should be used in the evaluation

• num_bins (int, default=100) – the number of bins to discretize the [0, 1] interval

Returns:

a tensor of the shape t containing the evaluated metrics

Return type:

Tensor

class chemprop.uncertainty.ExpectedNormalizedErrorEvaluator

Bases: RegressionEvaluator

A class that evaluates uncertainty performance by binning together clusters of predictions and comparing the average predicted variance of the clusters against the RMSE of the cluster. [1]_

\[\mathrm{ENCE} = \frac{1}{N} \sum_{i=1}^{N} \frac{|\mathrm{RMV}_i - \mathrm{RMSE}_i|}{\mathrm{RMV}_i}\]

where \(N\) is the number of bins, \(\mathrm{RMV}_i\) is the root of the mean uncertainty over the \(i\)-th bin and \(\mathrm{RMSE}_i\) is the root mean square error over the \(i\)-th bin. This discrepancy is further normalized by the uncertainty over the bin, \(\mathrm{RMV}_i\), because the error is expected to be naturally higher as the uncertainty increases.

References

[1]

Levi, D.; Gispan, L.; Giladi, N.; Fetaya, E. “Evaluating and Calibrating Uncertainty Prediction in Regression Tasks.” Sensors, 2022, 22(15), 5540. https://www.mdpi.com/1424-8220/22/15/5540

evaluate(preds, uncs, targets, mask, num_bins=100)

Evaluate the performance of uncertainty predictions against the model target values.

Parameters:

• preds (Tensor) – the predictions for regression tasks. It is a tensor of the shape of n x t, where n is the number of input molecules/reactions, and t is the number of tasks.

• uncs (Tensor) – the predicted uncertainties (variance) of the shape of n x t

• targets (Tensor) – a tensor of the shape n x t

• mask (Tensor) – a tensor of the shape n x t indicating whether the given values should be used in the evaluation

• num_bins (int, default=100) – the number of bins the data are divided into

Returns:

a tensor of the shape t containing the evaluated metrics

Return type:

Tensor

class chemprop.uncertainty.MulticlassClassificationEvaluator

Bases: abc.ABC

Evaluates the quality of uncertainty estimates in multiclass classification tasks.

abstractmethod evaluate(uncs, targets, mask)

Evaluate the performance of uncertainty predictions against the model target values.

Parameters:

• uncs (Tensor) – the predicted uncertainties (i.e., the predicted probabilities for each class) of the shape of n x t x c, where n is the number of input molecules/reactions, t is the number of tasks, and c is the number of classes.

• targets (Tensor) – a tensor of the shape n x t

• mask (Tensor) – a tensor of the shape n x t indicating whether the given values should be used in the evaluation

Returns:

a tensor of the shape t containing the evaluated metrics

Return type:

Tensor

class chemprop.uncertainty.MulticlassConformalEvaluator

Bases: MulticlassClassificationEvaluator

Evaluate the coverage of conformal prediction for multiclass classification datasets.

\[\Pr (Y_{\text{test}} \in C(X_{\text{test}}))\]

where the \(C(X_{\text{test}}) \subset \{1 \mathrel{.\,.} K\}\) is a prediction set of possible labels .

evaluate(uncs, targets, mask)

Evaluate the performance of uncertainty predictions against the model target values.

Parameters:

• uncs (Tensor) – the predicted uncertainties (i.e., the predicted probabilities for each class) of the shape of n x t x c, where n is the number of input molecules/reactions, t is the number of tasks, and c is the number of classes.

• targets (Tensor) – a tensor of the shape n x t

• mask (Tensor) – a tensor of the shape n x t indicating whether the given values should be used in the evaluation

Returns:

a tensor of the shape t containing the evaluated metrics

Return type:

Tensor

class chemprop.uncertainty.MultilabelConformalEvaluator

Bases: BinaryClassificationEvaluator

Evaluate the coverage of conformal prediction for binary classification datasets with multiple labels.

\[\Pr \left( \hat{\mathcal C}_{\text{in}}(X) \subseteq \mathcal Y \subseteq \hat{\mathcal C}_{\text{out}}(X) \right)\]

where the in-set \(\hat{\mathcal C}_\text{in}\) is contained by the set of true labels \(\mathcal Y\) and \(\mathcal Y\) is contained within the out-set \(\hat{\mathcal C}_\text{out}\).

evaluate(uncs, targets, mask)

Evaluate the performance of uncertainty predictions against the model target values.

Parameters:

• uncs (Tensor) – the predicted uncertainties (i.e., the predicted probability of class 1) of the shape of n x t, where n is the number of input molecules/reactions, and t is the number of tasks.

• targets (Tensor) – a tensor of the shape n x t

• mask (Tensor) – a tensor of the shape n x t indicating whether the given values should be used in the evaluation

Returns:

a tensor of the shape t containing the evaluated metrics

Return type:

Tensor

class chemprop.uncertainty.NLLClassEvaluator

Bases: BinaryClassificationEvaluator

Evaluate uncertainty values for binary classification datasets using the mean negative-log-likelihood of the targets given the assigned probabilities from the model:

\[\mathrm{NLL} = -\log(\hat{y} \cdot y + (1 - \hat{y}) \cdot (1 - y))\]

where \(y\) is the true binary label (0 or 1), and \(\hat{y}\) is the predicted probability associated with the class label 1.

The function returns a tensor containing the mean NLL for each task.

evaluate(uncs, targets, mask)

Evaluate the performance of uncertainty predictions against the model target values.

Parameters:

• uncs (Tensor) – the predicted uncertainties (i.e., the predicted probability of class 1) of the shape of n x t, where n is the number of input molecules/reactions, and t is the number of tasks.

• targets (Tensor) – a tensor of the shape n x t

• mask (Tensor) – a tensor of the shape n x t indicating whether the given values should be used in the evaluation

Returns:

a tensor of the shape t containing the evaluated metrics

Return type:

Tensor

class chemprop.uncertainty.NLLMulticlassEvaluator

Bases: MulticlassClassificationEvaluator

Evaluate uncertainty values for multiclass classification datasets using the mean negative-log-likelihood of the targets given the assigned probabilities from the model:

\[\mathrm{NLL} = -\log(p_{y_i})\]

where \(p_{y_i}\) is the predicted probability for the true class \(y_i\), calculated as:

\[p_{y_i} = \sum_{k=1}^{K} \mathbb{1}(y_i = k) \cdot p_k\]

Here: \(K\) is the total number of classes, \(\mathbb{1}(y_i = k)\) is the indicator function that is 1 when the true class \(y_i\) equals class \(k\), and 0 otherwise, and \(p_k\) is the predicted probability for class \(k\).

The function returns a tensor containing the mean NLL for each task.

evaluate(uncs, targets, mask)

Evaluate the performance of uncertainty predictions against the model target values.

Parameters:

• uncs (Tensor) – the predicted uncertainties (i.e., the predicted probabilities for each class) of the shape of n x t x c, where n is the number of input molecules/reactions, t is the number of tasks, and c is the number of classes.

• targets (Tensor) – a tensor of the shape n x t

• mask (Tensor) – a tensor of the shape n x t indicating whether the given values should be used in the evaluation

Returns:

a tensor of the shape t containing the evaluated metrics

Return type:

Tensor

class chemprop.uncertainty.NLLRegressionEvaluator

Bases: RegressionEvaluator

Evaluate uncertainty values for regression datasets using the mean negative-log-likelihood of the targets given the probability distributions estimated by the model:

\[\mathrm{NLL}(y, \hat y) = \frac{1}{2} \log(2 \pi \sigma^2) + \frac{(y - \hat{y})^2}{2 \sigma^2}\]

where \(\hat{y}\) is the predicted value, \(y\) is the true value, and \(\sigma^2\) is the predicted uncertainty (variance).

The function returns a tensor containing the mean NLL for each task.

evaluate(preds, uncs, targets, mask)

Evaluate the performance of uncertainty predictions against the model target values.

Parameters:

• preds (Tensor) – the predictions for regression tasks. It is a tensor of the shape of n x t, where n is the number of input molecules/reactions, and t is the number of tasks.

• uncs (Tensor) – the predicted uncertainties of the shape of n x t

• targets (Tensor) – a tensor of the shape n x t

• mask (Tensor) – a tensor of the shape n x t indicating whether the given values should be used in the evaluation

Returns:

a tensor of the shape t containing the evaluated metrics

Return type:

Tensor

class chemprop.uncertainty.RegressionConformalEvaluator

Bases: RegressionEvaluator

Evaluate the coverage of conformal prediction for regression datasets.

\[\Pr (Y_{\text{test}} \in C(X_{\text{test}}))\]

where the \(C(X_{\text{test}})\) is the predicted interval.

evaluate(preds, uncs, targets, mask)

Evaluate the performance of uncertainty predictions against the model target values.

Parameters:

• preds (Tensor) – the predictions for regression tasks. It is a tensor of the shape of n x t, where n is the number of input molecules/reactions, and t is the number of tasks.

• uncs (Tensor) – the predicted uncertainties of the shape of n x t

• targets (Tensor) – a tensor of the shape n x t

• mask (Tensor) – a tensor of the shape n x t indicating whether the given values should be used in the evaluation

Returns:

a tensor of the shape t containing the evaluated metrics

Return type:

Tensor

class chemprop.uncertainty.RegressionEvaluator

Bases: abc.ABC

Evaluates the quality of uncertainty estimates in regression tasks.

abstractmethod evaluate(preds, uncs, targets, mask)

Evaluate the performance of uncertainty predictions against the model target values.

Parameters:

• preds (Tensor) – the predictions for regression tasks. It is a tensor of the shape of n x t, where n is the number of input molecules/reactions, and t is the number of tasks.

• uncs (Tensor) – the predicted uncertainties of the shape of n x t

• targets (Tensor) – a tensor of the shape n x t

• mask (Tensor) – a tensor of the shape n x t indicating whether the given values should be used in the evaluation

Returns:

a tensor of the shape t containing the evaluated metrics

Return type:

Tensor

class chemprop.uncertainty.SpearmanEvaluator

Bases: RegressionEvaluator

Evaluate the Spearman rank correlation coefficient between the uncertainties and errors in the model predictions.

The correlation coefficient returns a value in the [-1, 1] range, with better scores closer to 1 observed when the uncertainty values are predictive of the rank ordering of the errors in the model prediction.

evaluate(preds, uncs, targets, mask)

Evaluate the performance of uncertainty predictions against the model target values.

Parameters:

• preds (Tensor) – the predictions for regression tasks. It is a tensor of the shape of n x t, where n is the number of input molecules/reactions, and t is the number of tasks.

• uncs (Tensor) – the predicted uncertainties of the shape of n x t

• targets (Tensor) – a tensor of the shape n x t

• mask (Tensor) – a tensor of the shape n x t indicating whether the given values should be used in the evaluation

Returns:

a tensor of the shape t containing the evaluated metrics

Return type:

Tensor

chemprop.uncertainty.UncertaintyEvaluatorRegistry