Paired statistics

Paired statistical tests for model-comparison experiments.

Functions

paired_bootstrap_difference	Estimate a paired bootstrap CI for metric(A) - metric(B).
paired_permutation_test	Run a paired randomisation test by swapping model labels per observation.
paired_effect_size	Return standardised paired loss difference for log loss or Brier score.
holm_bonferroni	Apply Holm's step-down correction and return adjusted p-values.
compare_models	Compare every pair of named probability predictions with paired tests.

paired_bootstrap_difference(y_true, p_a, p_b, metric, n_boot=2000, confidence=0.95, rng=None, chunk_size=256)

Estimate a paired bootstrap CI for metric(A) - metric(B).

paired_permutation_test(y_true, p_a, p_b, metric, n_permutations=5000, rng=None)

Run a paired randomisation test by swapping model labels per observation.

paired_effect_size(y_true, p_a, p_b, metric)

Return standardised paired loss difference for log loss or Brier score.

holm_bonferroni(p_values, alpha=0.05)

Apply Holm’s step-down correction and return adjusted p-values.

compare_models(y_true, predictions, metric, n_boot=2000, n_permutations=5000, seed=0)

Compare every pair of named probability predictions with paired tests.