A model that is accurate and a model whose probabilities mean something are different things. If you are going to act on 0.8: approve the loan, skip the review, page a human. Then eight out of ten of those had better be positive.
Write expected_calibration_error(y_true, y_prob, n_bins=10).
Split the interval [0, 1] into n_bins equal-width bins. Put each prediction in the bin its probability falls into. For every non-empty bin, compare the fraction of positives (accuracy) with the mean predicted probability (confidence), and take the absolute difference. Return the sum of those differences, each weighted by the share of all samples that bin holds.
$$ECE = \sum_{b} \frac{n_b}{N} \, \bigl| \, \text{acc}(b) - \text{conf}(b) \, \bigr|$$
A perfectly calibrated model scores 0. A model that says 0.9 and is right half the time scores 0.4.
Build the architecture on a canvas: place the components, configure them, connect them into a data flow, and write a short reason for each one. The AI reviewer grades your design against a rubric written specifically for this problem.
Minimum 5 components · needs a wide desktop screen