Sunday, 1 November 2026 · Weekend classic from 2021 · 5 min read
CLIP learns versatile image representations from 400M image-text pairs enabling zero-shot transfer
CLIP trains an image and text encoder jointly on 400 million image-text pairs using a contrastive loss to predict matching pairs, enabling zero-shot classification competitive with supervised models on 30+ vision tasks without task-specific training.
Based on the paper Learning Transferable Visual Models From Natural Language Supervision by Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh and 8 others
Challenges of fixed-label supervision and the promise of natural language supervision
Traditional computer vision models are trained to predict a fixed set of object categories using labeled datasets like ImageNet, limiting their ability to generalize to new concepts without additional labeled data.
Natural language supervision, learning from raw text paired with images, offers a broader and scalable source of information, potentially enabling models to understand and recognize a wider variety of visual concepts.
Previous attempts at learning from image captions or tags have shown promise but suffered from limited dataset size, inefficient training objectives, or restricted class sets, resulting in poor zero-shot transfer performance compared to supervised baselines.
Sign in to read the whole paper
You have read the opening. The rest is free with a Bezen account, and no card is needed.
In the rest of this paper
- How the method works, step by step, with a flow chart
- How it was tested, what was found, and where it falls short