BERT pre-trains deep bidirectional Transformers for versatile NLP tasks

Pre-training language representations has improved many natural language processing (NLP) tasks, from sentence-level tasks like natural language inference to token-level tasks like named entity recognition and question answering. However, existing pre-training methods typically use unidirectional language models that only condition on left or right context, limiting their ability to capture full sentence context.

Feature-based approaches like ELMo concatenate independently trained left-to-right and right-to-left models, while fine-tuning approaches like OpenAI GPT use unidirectional left-to-right Transformers. Both approaches restrict the architecture and limit the power of learned representations, especially for token-level tasks requiring context from both sides.

A key challenge is to pre-train deep bidirectional representations that condition on both left and right context simultaneously, enabling richer understanding and better transfer to downstream tasks without complex task-specific architectures.