Daily papers

Saturday, 14 November 2026 · Weekend classic from 2012 · 5 min read

Dropout reduces overfitting by randomly omitting neurons during training

Dropout trains neural networks by randomly disabling half of the hidden units on each training case, preventing co-adaptation and improving generalization. This method sets new records on benchmarks like MNIST, TIMIT, CIFAR-10, and ImageNet.

Based on the paper Improving neural networks by preventing co-adaptation of feature detectors by Geoffrey E. Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever and 1 other

Overfitting due to co-adaptation in large neural networks

Large feedforward neural networks trained on limited data often overfit, performing well on training data but poorly on unseen test data. This happens because feature detectors (hidden units) co-adapt, meaning they rely on the presence of specific other units to function correctly.

Co-adaptation leads to feature detectors that are only useful in certain internal contexts, reducing the network's ability to generalize to new inputs. Preventing these co-adaptations is key to improving test performance.

Traditional regularization methods like L2 weight decay reduce overfitting by shrinking weights but do not directly address co-adaptation among neurons.

Sign in to read the whole paper

You have read the opening. The rest is free with a Bezen account, and no card is needed.

In the rest of this paper

  • How the method works, step by step, with a flow chart
  • How it was tested, what was found, and where it falls short
Dropout reduces overfitting by randomly omitting neurons during training · Bezen