Calibration#

Calibration of confidence signals: measure it, correct it, and bound it.

A robot that routes between modules (vision, language, a human model, a planner) or decides whether to ask a person before acting relies on the confidence each module reports. A confidence is calibrated when, among all the times a module reports 0.8, it is right about 80% of the time. This module provides the standard tools to check and repair that property.

Measures: reliability_curve(), expected_calibration_error(), max_calibration_error(), brier_score().
Recalibration: PlattScaling (binary), TemperatureScaling (multi-class), IsotonicCalibration (monotone, non-parametric).
Guarantees: SplitConformalClassifier (prediction sets with a coverage guarantee) and clopper_pearson() (exact binomial bounds), combined in CalibrationMonitor, which tracks a module’s outcomes online and returns a lower confidence bound on its success probability.

Examples

>>> import numpy as np
>>> from quantum_mind.applications.calibration import expected_calibration_error, IsotonicCalibration
>>> rng = np.random.default_rng(0)
>>> p = rng.uniform(0.3, 0.9, 5000)                 # true success probabilities
>>> y = rng.random(5000) < p                         # outcomes
>>> over = np.clip(p + 0.15, 0, 1)                   # an over-confident module
>>> expected_calibration_error(over, y) > 0.1          # reports about 0.15 too much
True
>>> fixed = IsotonicCalibration().fit(over, y).transform(over)
>>> expected_calibration_error(fixed, y) < 0.03
True

Classes

`PlattScaling`	Logistic recalibration of a binary confidence: \(\sigma(a\,\mathrm{logit}(c) + b)\).
`TemperatureScaling`	Multi-class recalibration by one temperature: \(\mathrm{softmax}(\log p / T)\).
`IsotonicCalibration`	Monotone, non-parametric recalibration by the pool-adjacent-violators algorithm.
`SplitConformalClassifier`	Split conformal prediction sets for any probabilistic classifier.
`CalibrationMonitor`	Online calibration check of one module, with a guaranteed lower bound on its success rate.
`AdaptiveConformalSets`	Prediction sets for a person's next answer that keep their coverage while the person changes.

Functions

`reliability_curve`	Data of a reliability diagram.
`expected_calibration_error`	Expected calibration error (ECE).
`max_calibration_error`	Largest calibration gap over the non-empty bins.
`brier_score`	Brier score (mean squared error of probabilities).
`clopper_pearson`	Exact (Clopper-Pearson) confidence interval for a success probability.

class PlattScaling(a=1.0, b=0.0)[source]#

Bases: object

Logistic recalibration of a binary confidence: \(\sigma(a\,\mathrm{logit}(c) + b)\).

Variables:

b (a,) – Slope and intercept, fitted by maximum likelihood (a = 1, b = 0 is the identity).

Parameters:

a (float)
b (float)

fit(conf, outcome)[source]#

Fit the slope and intercept.

Parameters:

conf (array_like) – Confidences in [0, 1].
outcome (array_like) – 0/1 outcomes.

Returns:

PlattScaling – self.

transform(conf)[source]#

Recalibrated confidences.

Parameters:: conf (array_like) – Confidences in [0, 1].
Returns:: numpy.ndarray

class TemperatureScaling(T=1.0)[source]#

Bases: object

Multi-class recalibration by one temperature: \(\mathrm{softmax}(\log p / T)\).

Variables:: T (float) – Temperature (> 1 softens over-confident predictions, < 1 sharpens under-confident ones).
Parameters:: T (float)

fit(prob, labels)[source]#

Fit the temperature by minimising the negative log-likelihood.

Parameters:

prob (array_like) – Class probabilities, shape (n, k).
labels (array_like) – True classes 0..k-1.

Returns:

TemperatureScaling – self.

transform(prob)[source]#

Recalibrated class probabilities.

Parameters:: prob (array_like) – Shape (n, k).
Returns:: numpy.ndarray

class IsotonicCalibration(x_=None, y_=None)[source]#

Bases: object

Monotone, non-parametric recalibration by the pool-adjacent-violators algorithm.

Variables:

y (x,) – Knots of the fitted non-decreasing step function (interpolated linearly between knots).

Parameters:

x_ (ndarray)
y_ (ndarray)

fit(conf, outcome)[source]#

Fit the monotone map from confidence to success rate.

Parameters:

conf (array_like) – Confidences in [0, 1].
outcome (array_like) – 0/1 outcomes.

Returns:

IsotonicCalibration – self.

transform(conf)[source]#

Recalibrated confidences.

Parameters:: conf (array_like)
Returns:: numpy.ndarray

class SplitConformalClassifier(alpha=0.1, qhat=None)[source]#

Bases: object

Split conformal prediction sets for any probabilistic classifier.

After fit() on held-out calibration data, the prediction set of a new input contains the true class with probability at least \(1 - \alpha\) (marginally, for exchangeable data), whatever the classifier. A robot can act when the set has one element and ask when it has several.

Variables:

alpha (float) – Allowed miscoverage, for example 0.1 for 90% coverage.
qhat (float) – Fitted threshold on the score \(1 - p_{\text{true}}\).

Parameters:

alpha (float)
qhat (float)

fit(prob, labels)[source]#

Compute the conformal threshold from calibration data.

Parameters:

prob (array_like) – Class probabilities on the calibration set, shape (n, k).
labels (array_like) – True classes.

Returns:

SplitConformalClassifier – self.

predict_sets(prob)[source]#

Prediction sets.

Parameters:: prob (array_like) – Class probabilities, shape (n, k).
Returns:: numpy.ndarray – Boolean mask (n, k): class j is in the set of sample i when mask[i, j].

class CalibrationMonitor(bins=10, alpha=0.05, min_count=10)[source]#

Bases: object

Online calibration check of one module, with a guaranteed lower bound on its success rate.

Outcomes are binned by reported confidence. For a new confidence the monitor returns the recalibrated success estimate of its bin and a Clopper-Pearson lower bound, which a risk-aware decision rule (quantum_mind.applications.robotics.risk_aware_ask_or_act()) can use instead of the raw confidence.

Parameters:

bins (int, optional) – Number of confidence bins.
alpha (float, optional) – Error level of the bounds.
min_count (int, optional) – Below this many outcomes in a bin, the raw confidence is returned as the estimate.

Examples

>>> from quantum_mind.applications.calibration import CalibrationMonitor
>>> m = CalibrationMonitor()
>>> for c, ok in [(0.9, 1), (0.9, 0), (0.9, 1), (0.9, 0)] * 10:
...     m.update(c, ok)
>>> est, lower = m.estimate(0.9)
>>> round(est, 2), lower < 0.5
(0.5, True)

update(conf, success)[source]#

Record one outcome.

Parameters:

conf (float) – Confidence the module reported.
success (bool or int) – Whether the module turned out to be right.

estimate(conf)[source]#

Recalibrated success estimate and lower bound for a reported confidence.

Parameters:

conf (float)

Returns:

estimate (float) – Observed success rate of the bin (the raw confidence while the bin has too few outcomes).
lower (float) – Clopper-Pearson lower bound (0 while the bin is empty).

ece()[source]#

Expected calibration error of the outcomes recorded so far (bin centres as confidences).

Returns:: float

class AdaptiveConformalSets(alpha=0.1, gamma=0.01, calibration_scores=None)[source]#

Bases: object

Prediction sets for a person’s next answer that keep their coverage while the person changes.

Split conformal prediction (SplitConformalClassifier) assumes the calibration answers and the new answer are exchangeable. Within one interaction they are not: trust, attention and the person’s view of the task drift. Adaptive conformal inference (Gibbs and Candès, 2021) keeps a running miscoverage level \(\alpha_t\) and corrects it after every answer,

\[\alpha_{t+1} = \alpha_t + \gamma\,(\alpha - \mathrm{err}_t),\]

where \(\mathrm{err}_t = 1\) if the answer fell outside the set. Whatever the sequence of answers, the long-run miscoverage satisfies

\[\Bigl|\frac1T \sum_{t=1}^T \mathrm{err}_t - \alpha\Bigr| \le \frac{\max(\alpha_1, 1 - \alpha_1) + \gamma}{\gamma T}.\]

The probabilities can come from any model of the person, for example the posterior predictive of OnlinePersonModel. A robot acts when the set holds one answer and asks when it holds several.

Parameters:

alpha (float, optional) – Target miscoverage (0.1 for 90% coverage).
gamma (float, optional) – Step size of the correction.
calibration_scores (array_like, optional) – Scores \(1 - p_{\text{true}}\) from earlier people, used before this person has answered.

Variables:

alpha_t (float) – Current working miscoverage level.
errors (list of int) – 1 for every answer outside its set.

Examples

>>> import numpy as np
>>> aci = AdaptiveConformalSets(alpha=0.2, gamma=0.05, calibration_scores=[0.1, 0.3, 0.5, 0.7])
>>> s = aci.predict_set([0.7, 0.2, 0.1])
>>> bool(s[0])                   # the most likely answer is in the set
True
>>> aci.update([0.7, 0.2, 0.1], 1)

property threshold#

current threshold on the score (inf: every answer, -inf: none).

Type:: float

predict_set(prob)[source]#

Answers in the prediction set.

Parameters:: prob (array_like) – Predicted probability of each answer.
Returns:: numpy.ndarray – Boolean mask over answers.

should_ask(prob)[source]#: bool: True when more than one answer is in the set.

update(prob, answer)[source]#

Record the observed answer and correct the working level.

Parameters:

prob (array_like) – The probabilities used for this answer’s prediction set.
answer (int) – Index of the observed answer.

property miscoverage#

fraction of answers that fell outside their sets so far.

Type:: float

bound()[source]#: float: the guaranteed bound on |miscoverage - alpha| after the answers so far.

reliability_curve(conf, outcome, bins=10)[source]#

Data of a reliability diagram.

Parameters:

conf (array_like) – Reported confidences in [0, 1].
outcome (array_like) – 1 if the prediction was right (or the action succeeded), 0 otherwise.
bins (int, optional) – Number of equal-width confidence bins.

Returns:

dict – confidence (mean confidence per bin), accuracy (success rate per bin), count and edges. Empty bins hold nan. A calibrated module lies on the diagonal.

expected_calibration_error(conf, outcome, bins=10)[source]#

Expected calibration error (ECE).

Parameters:

conf (array_like) – Confidences in [0, 1].
outcome (array_like) – 0/1 outcomes.
bins (int, optional) – Number of bins.

Returns:

float – \(\sum_b \frac{n_b}{N} |\mathrm{acc}_b - \mathrm{conf}_b|\); 0 for a perfectly calibrated module.

max_calibration_error(conf, outcome, bins=10)[source]#

Largest calibration gap over the non-empty bins.

Parameters:

conf (array_like) – Confidences.
outcome (array_like) – 0/1 outcomes.
bins (int, optional) – Number of bins.

Returns:

float

brier_score(prob, outcome)[source]#

Brier score (mean squared error of probabilities).

Parameters:

prob (array_like) – Binary confidences, shape (n,), or class probabilities, shape (n, k).
outcome (array_like) – 0/1 outcomes, shape (n,), or class labels 0..k-1 for the multi-class case.

Returns:

float – Lower is better; it rewards both calibration and sharpness.

clopper_pearson(successes, trials, alpha=0.05)[source]#

Exact (Clopper-Pearson) confidence interval for a success probability.

Parameters:

successes (int) – Number of successes.
trials (int) – Number of trials.
alpha (float, optional) – Two-sided error level.

Returns:

tuple of float – (lower, upper); (0, 1) when there are no trials.