chemprop.uncertainty.estimator ============================== .. py:module:: chemprop.uncertainty.estimator Attributes ---------- .. autoapisummary:: chemprop.uncertainty.estimator.UncertaintyEstimatorRegistry Classes ------- .. autoapisummary:: chemprop.uncertainty.estimator.UncertaintyEstimator chemprop.uncertainty.estimator.NoUncertaintyEstimator chemprop.uncertainty.estimator.MVEEstimator chemprop.uncertainty.estimator.EnsembleEstimator chemprop.uncertainty.estimator.ClassEstimator chemprop.uncertainty.estimator.EvidentialTotalEstimator chemprop.uncertainty.estimator.EvidentialEpistemicEstimator chemprop.uncertainty.estimator.EvidentialAleatoricEstimator chemprop.uncertainty.estimator.DropoutEstimator chemprop.uncertainty.estimator.ClassificationDirichletEstimator chemprop.uncertainty.estimator.MulticlassDirichletEstimator chemprop.uncertainty.estimator.QuantileRegressionEstimator Module Contents --------------- .. py:class:: UncertaintyEstimator Bases: :py:obj:`abc.ABC` A helper class for making model predictions and associated uncertainty predictions. .. py:method:: __call__(dataloader, models, trainer) :abstractmethod: Calculate the uncalibrated predictions and uncertainties for the dataloader. dataloader: DataLoader the dataloader used for model predictions and uncertainty predictions models: Iterable[MPNN] | Iterable[MolAtomBondMPNN] the models used for model predictions and uncertainty predictions. If using MolAtomBondMPNN models, the uncertainty estimator will return preds and uncs for each of the mole, atom, and bond predictions and uncertainties. trainer: pl.Trainer an instance of the :class:`~lightning.pytorch.trainer.trainer.Trainer` used to manage model inference :returns: * **preds** (*Tensor*) -- the model predictions, with shape varying by task type: * regression/binary classification: ``m x n x t`` * multiclass classification: ``m x n x t x c``, where ``m`` is the number of models, ``n`` is the number of inputs, ``t`` is the number of tasks, and ``c`` is the number of classes. * **uncs** (*Tensor*) -- the predicted uncertainties, with shapes of ``m' x n x t``. * *.. note::* -- The ``m`` and ``m'`` are different by definition. The ``m`` is the number of models, while the ``m'`` is the number of uncertainty estimations. For example, if two MVE or evidential models are provided, both ``m`` and ``m'`` are two. However, for an ensemble of two models, ``m'`` would be one (even though ``m = 2``). .. py:data:: UncertaintyEstimatorRegistry .. py:class:: NoUncertaintyEstimator Bases: :py:obj:`UncertaintyEstimator` A helper class for making model predictions and associated uncertainty predictions. .. py:method:: __call__(dataloader, models, trainer) Calculate the uncalibrated predictions and uncertainties for the dataloader. dataloader: DataLoader the dataloader used for model predictions and uncertainty predictions models: Iterable[MPNN] | Iterable[MolAtomBondMPNN] the models used for model predictions and uncertainty predictions. If using MolAtomBondMPNN models, the uncertainty estimator will return preds and uncs for each of the mole, atom, and bond predictions and uncertainties. trainer: pl.Trainer an instance of the :class:`~lightning.pytorch.trainer.trainer.Trainer` used to manage model inference :returns: * **preds** (*Tensor*) -- the model predictions, with shape varying by task type: * regression/binary classification: ``m x n x t`` * multiclass classification: ``m x n x t x c``, where ``m`` is the number of models, ``n`` is the number of inputs, ``t`` is the number of tasks, and ``c`` is the number of classes. * **uncs** (*Tensor*) -- the predicted uncertainties, with shapes of ``m' x n x t``. * *.. note::* -- The ``m`` and ``m'`` are different by definition. The ``m`` is the number of models, while the ``m'`` is the number of uncertainty estimations. For example, if two MVE or evidential models are provided, both ``m`` and ``m'`` are two. However, for an ensemble of two models, ``m'`` would be one (even though ``m = 2``). .. py:class:: MVEEstimator Bases: :py:obj:`UncertaintyEstimator` Class that estimates prediction means and variances (MVE). [nix1994]_ .. rubric:: References .. [nix1994] Nix, D. A.; Weigend, A. S. "Estimating the mean and variance of the target probability distribution." Proceedings of 1994 IEEE International Conference on Neural Networks, 1994 https://doi.org/10.1109/icnn.1994.374138 .. py:method:: __call__(dataloader, models, trainer) Calculate the uncalibrated predictions and uncertainties for the dataloader. dataloader: DataLoader the dataloader used for model predictions and uncertainty predictions models: Iterable[MPNN] | Iterable[MolAtomBondMPNN] the models used for model predictions and uncertainty predictions. If using MolAtomBondMPNN models, the uncertainty estimator will return preds and uncs for each of the mole, atom, and bond predictions and uncertainties. trainer: pl.Trainer an instance of the :class:`~lightning.pytorch.trainer.trainer.Trainer` used to manage model inference :returns: * **preds** (*Tensor*) -- the model predictions, with shape varying by task type: * regression/binary classification: ``m x n x t`` * multiclass classification: ``m x n x t x c``, where ``m`` is the number of models, ``n`` is the number of inputs, ``t`` is the number of tasks, and ``c`` is the number of classes. * **uncs** (*Tensor*) -- the predicted uncertainties, with shapes of ``m' x n x t``. * *.. note::* -- The ``m`` and ``m'`` are different by definition. The ``m`` is the number of models, while the ``m'`` is the number of uncertainty estimations. For example, if two MVE or evidential models are provided, both ``m`` and ``m'`` are two. However, for an ensemble of two models, ``m'`` would be one (even though ``m = 2``). .. py:class:: EnsembleEstimator Bases: :py:obj:`UncertaintyEstimator` Class that predicts the uncertainty of predictions based on the variance in predictions among an ensemble's submodels. .. py:method:: __call__(dataloader, models, trainer) Calculate the uncalibrated predictions and uncertainties for the dataloader. dataloader: DataLoader the dataloader used for model predictions and uncertainty predictions models: Iterable[MPNN] | Iterable[MolAtomBondMPNN] the models used for model predictions and uncertainty predictions. If using MolAtomBondMPNN models, the uncertainty estimator will return preds and uncs for each of the mole, atom, and bond predictions and uncertainties. trainer: pl.Trainer an instance of the :class:`~lightning.pytorch.trainer.trainer.Trainer` used to manage model inference :returns: * **preds** (*Tensor*) -- the model predictions, with shape varying by task type: * regression/binary classification: ``m x n x t`` * multiclass classification: ``m x n x t x c``, where ``m`` is the number of models, ``n`` is the number of inputs, ``t`` is the number of tasks, and ``c`` is the number of classes. * **uncs** (*Tensor*) -- the predicted uncertainties, with shapes of ``m' x n x t``. * *.. note::* -- The ``m`` and ``m'`` are different by definition. The ``m`` is the number of models, while the ``m'`` is the number of uncertainty estimations. For example, if two MVE or evidential models are provided, both ``m`` and ``m'`` are two. However, for an ensemble of two models, ``m'`` would be one (even though ``m = 2``). .. py:class:: ClassEstimator Bases: :py:obj:`UncertaintyEstimator` A helper class for making model predictions and associated uncertainty predictions. .. py:method:: __call__(dataloader, models, trainer) Calculate the uncalibrated predictions and uncertainties for the dataloader. dataloader: DataLoader the dataloader used for model predictions and uncertainty predictions models: Iterable[MPNN] | Iterable[MolAtomBondMPNN] the models used for model predictions and uncertainty predictions. If using MolAtomBondMPNN models, the uncertainty estimator will return preds and uncs for each of the mole, atom, and bond predictions and uncertainties. trainer: pl.Trainer an instance of the :class:`~lightning.pytorch.trainer.trainer.Trainer` used to manage model inference :returns: * **preds** (*Tensor*) -- the model predictions, with shape varying by task type: * regression/binary classification: ``m x n x t`` * multiclass classification: ``m x n x t x c``, where ``m`` is the number of models, ``n`` is the number of inputs, ``t`` is the number of tasks, and ``c`` is the number of classes. * **uncs** (*Tensor*) -- the predicted uncertainties, with shapes of ``m' x n x t``. * *.. note::* -- The ``m`` and ``m'`` are different by definition. The ``m`` is the number of models, while the ``m'`` is the number of uncertainty estimations. For example, if two MVE or evidential models are provided, both ``m`` and ``m'`` are two. However, for an ensemble of two models, ``m'`` would be one (even though ``m = 2``). .. py:class:: EvidentialTotalEstimator Bases: :py:obj:`UncertaintyEstimator` Class that predicts the total evidential uncertainty based on hyperparameters of the evidential distribution [amini2020]_. .. rubric:: References .. [amini2020] Amini, A.; Schwarting, W.; Soleimany, A.; Rus, D. "Deep Evidential Regression". NeurIPS, 2020. https://arxiv.org/abs/1910.02600 .. py:method:: __call__(dataloader, models, trainer) Calculate the uncalibrated predictions and uncertainties for the dataloader. dataloader: DataLoader the dataloader used for model predictions and uncertainty predictions models: Iterable[MPNN] | Iterable[MolAtomBondMPNN] the models used for model predictions and uncertainty predictions. If using MolAtomBondMPNN models, the uncertainty estimator will return preds and uncs for each of the mole, atom, and bond predictions and uncertainties. trainer: pl.Trainer an instance of the :class:`~lightning.pytorch.trainer.trainer.Trainer` used to manage model inference :returns: * **preds** (*Tensor*) -- the model predictions, with shape varying by task type: * regression/binary classification: ``m x n x t`` * multiclass classification: ``m x n x t x c``, where ``m`` is the number of models, ``n`` is the number of inputs, ``t`` is the number of tasks, and ``c`` is the number of classes. * **uncs** (*Tensor*) -- the predicted uncertainties, with shapes of ``m' x n x t``. * *.. note::* -- The ``m`` and ``m'`` are different by definition. The ``m`` is the number of models, while the ``m'`` is the number of uncertainty estimations. For example, if two MVE or evidential models are provided, both ``m`` and ``m'`` are two. However, for an ensemble of two models, ``m'`` would be one (even though ``m = 2``). .. py:class:: EvidentialEpistemicEstimator Bases: :py:obj:`UncertaintyEstimator` Class that predicts the epistemic evidential uncertainty based on hyperparameters of the evidential distribution. .. py:method:: __call__(dataloader, models, trainer) Calculate the uncalibrated predictions and uncertainties for the dataloader. dataloader: DataLoader the dataloader used for model predictions and uncertainty predictions models: Iterable[MPNN] | Iterable[MolAtomBondMPNN] the models used for model predictions and uncertainty predictions. If using MolAtomBondMPNN models, the uncertainty estimator will return preds and uncs for each of the mole, atom, and bond predictions and uncertainties. trainer: pl.Trainer an instance of the :class:`~lightning.pytorch.trainer.trainer.Trainer` used to manage model inference :returns: * **preds** (*Tensor*) -- the model predictions, with shape varying by task type: * regression/binary classification: ``m x n x t`` * multiclass classification: ``m x n x t x c``, where ``m`` is the number of models, ``n`` is the number of inputs, ``t`` is the number of tasks, and ``c`` is the number of classes. * **uncs** (*Tensor*) -- the predicted uncertainties, with shapes of ``m' x n x t``. * *.. note::* -- The ``m`` and ``m'`` are different by definition. The ``m`` is the number of models, while the ``m'`` is the number of uncertainty estimations. For example, if two MVE or evidential models are provided, both ``m`` and ``m'`` are two. However, for an ensemble of two models, ``m'`` would be one (even though ``m = 2``). .. py:class:: EvidentialAleatoricEstimator Bases: :py:obj:`UncertaintyEstimator` Class that predicts the aleatoric evidential uncertainty based on hyperparameters of the evidential distribution. .. py:method:: __call__(dataloader, models, trainer) Calculate the uncalibrated predictions and uncertainties for the dataloader. dataloader: DataLoader the dataloader used for model predictions and uncertainty predictions models: Iterable[MPNN] | Iterable[MolAtomBondMPNN] the models used for model predictions and uncertainty predictions. If using MolAtomBondMPNN models, the uncertainty estimator will return preds and uncs for each of the mole, atom, and bond predictions and uncertainties. trainer: pl.Trainer an instance of the :class:`~lightning.pytorch.trainer.trainer.Trainer` used to manage model inference :returns: * **preds** (*Tensor*) -- the model predictions, with shape varying by task type: * regression/binary classification: ``m x n x t`` * multiclass classification: ``m x n x t x c``, where ``m`` is the number of models, ``n`` is the number of inputs, ``t`` is the number of tasks, and ``c`` is the number of classes. * **uncs** (*Tensor*) -- the predicted uncertainties, with shapes of ``m' x n x t``. * *.. note::* -- The ``m`` and ``m'`` are different by definition. The ``m`` is the number of models, while the ``m'`` is the number of uncertainty estimations. For example, if two MVE or evidential models are provided, both ``m`` and ``m'`` are two. However, for an ensemble of two models, ``m'`` would be one (even though ``m = 2``). .. py:class:: DropoutEstimator(ensemble_size, dropout = None) Bases: :py:obj:`UncertaintyEstimator` A :class:`DropoutEstimator` creates a virtual ensemble of models via Monte Carlo dropout with the provided model [gal2016]_. :param ensemble_size: The number of samples to draw for the ensemble. :type ensemble_size: int :param dropout: The probability of dropping out units in the dropout layers. If unspecified, the training probability is used, which is prefered but not possible if the model was not trained with dropout (i.e. p=0). :type dropout: float | None .. rubric:: References .. [gal2016] Gal, Y.; Ghahramani, Z. "Dropout as a bayesian approximation: Representing model uncertainty in deep learning." International conference on machine learning. PMLR, 2016. https://arxiv.org/abs/1506.02142 .. py:attribute:: ensemble_size .. py:attribute:: dropout :value: None .. py:method:: __call__(dataloader, models, trainer) Calculate the uncalibrated predictions and uncertainties for the dataloader. dataloader: DataLoader the dataloader used for model predictions and uncertainty predictions models: Iterable[MPNN] | Iterable[MolAtomBondMPNN] the models used for model predictions and uncertainty predictions. If using MolAtomBondMPNN models, the uncertainty estimator will return preds and uncs for each of the mole, atom, and bond predictions and uncertainties. trainer: pl.Trainer an instance of the :class:`~lightning.pytorch.trainer.trainer.Trainer` used to manage model inference :returns: * **preds** (*Tensor*) -- the model predictions, with shape varying by task type: * regression/binary classification: ``m x n x t`` * multiclass classification: ``m x n x t x c``, where ``m`` is the number of models, ``n`` is the number of inputs, ``t`` is the number of tasks, and ``c`` is the number of classes. * **uncs** (*Tensor*) -- the predicted uncertainties, with shapes of ``m' x n x t``. * *.. note::* -- The ``m`` and ``m'`` are different by definition. The ``m`` is the number of models, while the ``m'`` is the number of uncertainty estimations. For example, if two MVE or evidential models are provided, both ``m`` and ``m'`` are two. However, for an ensemble of two models, ``m'`` would be one (even though ``m = 2``). .. py:class:: ClassificationDirichletEstimator Bases: :py:obj:`UncertaintyEstimator` A :class:`ClassificationDirichletEstimator` predicts an amount of 'evidence' for both the negative class and the positive class as described in [sensoy2018]_. The class probabilities and the uncertainty are calculated based on the evidence. .. math:: S = \sum_{i=1}^K \alpha_i p_i = \alpha_i / S u = K / S where :math:`K` is the number of classes, :math:`\alpha_i` is the evidence for class :math:`i`, :math:`p_i` is the probability of class :math:`i`, and :math:`u` is the uncertainty. .. rubric:: References .. [sensoy2018] Sensoy, M.; Kaplan, L.; Kandemir, M. "Evidential deep learning to quantify classification uncertainty." NeurIPS, 2018, 31. https://doi.org/10.48550/arXiv.1806.01768 .. py:method:: __call__(dataloader, models, trainer) Calculate the uncalibrated predictions and uncertainties for the dataloader. dataloader: DataLoader the dataloader used for model predictions and uncertainty predictions models: Iterable[MPNN] | Iterable[MolAtomBondMPNN] the models used for model predictions and uncertainty predictions. If using MolAtomBondMPNN models, the uncertainty estimator will return preds and uncs for each of the mole, atom, and bond predictions and uncertainties. trainer: pl.Trainer an instance of the :class:`~lightning.pytorch.trainer.trainer.Trainer` used to manage model inference :returns: * **preds** (*Tensor*) -- the model predictions, with shape varying by task type: * regression/binary classification: ``m x n x t`` * multiclass classification: ``m x n x t x c``, where ``m`` is the number of models, ``n`` is the number of inputs, ``t`` is the number of tasks, and ``c`` is the number of classes. * **uncs** (*Tensor*) -- the predicted uncertainties, with shapes of ``m' x n x t``. * *.. note::* -- The ``m`` and ``m'`` are different by definition. The ``m`` is the number of models, while the ``m'`` is the number of uncertainty estimations. For example, if two MVE or evidential models are provided, both ``m`` and ``m'`` are two. However, for an ensemble of two models, ``m'`` would be one (even though ``m = 2``). .. py:class:: MulticlassDirichletEstimator Bases: :py:obj:`UncertaintyEstimator` A :class:`MulticlassDirichletEstimator` predicts an amount of 'evidence' for each class as described in [sensoy2018]_. The class probabilities and the uncertainty are calculated based on the evidence. .. math:: S = \sum_{i=1}^K \alpha_i p_i = \alpha_i / S u = K / S where :math:`K` is the number of classes, :math:`\alpha_i` is the evidence for class :math:`i`, :math:`p_i` is the probability of class :math:`i`, and :math:`u` is the uncertainty. .. rubric:: References .. [sensoy2018] Sensoy, M.; Kaplan, L.; Kandemir, M. "Evidential deep learning to quantify classification uncertainty." NeurIPS, 2018, 31. https://doi.org/10.48550/arXiv.1806.01768 .. py:method:: __call__(dataloader, models, trainer) Calculate the uncalibrated predictions and uncertainties for the dataloader. dataloader: DataLoader the dataloader used for model predictions and uncertainty predictions models: Iterable[MPNN] | Iterable[MolAtomBondMPNN] the models used for model predictions and uncertainty predictions. If using MolAtomBondMPNN models, the uncertainty estimator will return preds and uncs for each of the mole, atom, and bond predictions and uncertainties. trainer: pl.Trainer an instance of the :class:`~lightning.pytorch.trainer.trainer.Trainer` used to manage model inference :returns: * **preds** (*Tensor*) -- the model predictions, with shape varying by task type: * regression/binary classification: ``m x n x t`` * multiclass classification: ``m x n x t x c``, where ``m`` is the number of models, ``n`` is the number of inputs, ``t`` is the number of tasks, and ``c`` is the number of classes. * **uncs** (*Tensor*) -- the predicted uncertainties, with shapes of ``m' x n x t``. * *.. note::* -- The ``m`` and ``m'`` are different by definition. The ``m`` is the number of models, while the ``m'`` is the number of uncertainty estimations. For example, if two MVE or evidential models are provided, both ``m`` and ``m'`` are two. However, for an ensemble of two models, ``m'`` would be one (even though ``m = 2``). .. py:class:: QuantileRegressionEstimator Bases: :py:obj:`UncertaintyEstimator` A helper class for making model predictions and associated uncertainty predictions. .. py:method:: __call__(dataloader, models, trainer) Calculate the uncalibrated predictions and uncertainties for the dataloader. dataloader: DataLoader the dataloader used for model predictions and uncertainty predictions models: Iterable[MPNN] | Iterable[MolAtomBondMPNN] the models used for model predictions and uncertainty predictions. If using MolAtomBondMPNN models, the uncertainty estimator will return preds and uncs for each of the mole, atom, and bond predictions and uncertainties. trainer: pl.Trainer an instance of the :class:`~lightning.pytorch.trainer.trainer.Trainer` used to manage model inference :returns: * **preds** (*Tensor*) -- the model predictions, with shape varying by task type: * regression/binary classification: ``m x n x t`` * multiclass classification: ``m x n x t x c``, where ``m`` is the number of models, ``n`` is the number of inputs, ``t`` is the number of tasks, and ``c`` is the number of classes. * **uncs** (*Tensor*) -- the predicted uncertainties, with shapes of ``m' x n x t``. * *.. note::* -- The ``m`` and ``m'`` are different by definition. The ``m`` is the number of models, while the ``m'`` is the number of uncertainty estimations. For example, if two MVE or evidential models are provided, both ``m`` and ``m'`` are two. However, for an ensemble of two models, ``m'`` would be one (even though ``m = 2``).