A Bootstrap Confidence Interval for a Metric

A single number — 0.83 AUC — says nothing about how much you'd trust it on a different sample of the same size. The bootstrap answers that without assuming any particular distribution, and it has exactly one step that is easy to get backwards.

Write bootstrap_ci(y_true, y_pred, metric_fn, n_boot, seed, alpha).

1. The point estimate is metric_fn(y_true, y_pred) on the original, unresampled data.
2. Seed one random generator: rng = np.random.default_rng(seed).
3. Repeat n_boot times: draw n row indices with replacement using rng, and apply the same drawn indices to both y_true and y_pred together — never resample them independently. Compute metric_fn on that resampled pair.
4. Return (point_estimate, lower, upper), where lower and upper are the 100 · alpha / 2 and 100 · (1 − alpha / 2) percentiles of the n_boot bootstrap scores — the percentile bootstrap. For alpha = 0.05 that is a 95% interval: the 2.5th and 97.5th percentiles, not the 5th and 95th.