Batch Normalization stabilizes layer inputs to speed up deep network training

Training deep neural networks is challenging because the distribution of inputs to each layer changes as the parameters of preceding layers update during training. This phenomenon, called internal covariate shift, forces the network to continuously adapt to shifting input distributions, slowing convergence and requiring careful initialization and low learning rates.

Internal covariate shift is problematic especially for layers with saturating nonlinearities like sigmoid, where inputs moving into saturation cause vanishing gradients and slow training. While ReLU activations and careful initialization partially mitigate this, stabilizing the input distributions to each layer could accelerate training.

The authors propose that fixing the distribution of inputs to each layer throughout training would reduce internal covariate shift, improving gradient flow and enabling faster, more stable training. This motivates a method to normalize layer inputs during training to maintain stable distributions.