A model that is accurate and a model whose probabilities mean something are different things. If you are going to act on 0.8: approve the loan, skip the review, page a human. Then eight out of ten of those had better be positive.
Write expected_calibration_error(y_true, y_prob, n_bins=10).
Split the interval [0, 1] into n_bins equal-width bins. Put each prediction in the bin its probability falls into. For every non-empty bin, compare the fraction of positives (accuracy) with the mean predicted probability (confidence), and take the absolute difference. Return the sum of those differences, each weighted by the share of all samples that bin holds.
You have read the opening. The rest is free with a Bezen account, and no card is needed.
Behind the sign-in