Google publishes BERT and releases pretrained models
BERT unified many language-understanding tasks around a bidirectional Transformer encoder, followed by a release of code and pretrained checkpoints.
What happened
Event details
On October 11, 2018, Google researchers submitted the BERT paper. BERT uses masked language modeling to incorporate context from both sides of each token, together with next-sentence prediction during pretraining. Downstream systems typically add a small output layer and fine-tune the full model. The paper reported BERT-base with 110 million parameters and BERT-large with 340 million, and set new results on 11 natural-language-processing tasks. On November 2, Google announced the release of its TensorFlow code and several pretrained checkpoints.
Assessment
Why it matters
BERT established pretrained representations followed by task-specific fine-tuning as a dominant approach to natural-language understanding.
Availability
Access notes
The paper appeared on October 11; Google released the TensorFlow code and pretrained checkpoints in early November.