OpenAI introduces GPT-2 and begins a staged release
OpenAI introduces GPT-2, a family scaling to 1.5B parameters, but initially releases only a smaller model because of misuse concerns.
EVENT ARCHIVE
Events
English reports are listed here. AI-generated drafts are clearly labeled until an editor reviews them.
AI-assisted · reviewedImpact 89/100
Google publishes BERT and releases pretrained models
BERT unified many language-understanding tasks around a bidirectional Transformer encoder, followed by a release of code and pretrained checkpoints.
OpenAI releases a generative pre-trained Transformer, later known as GPT-1
OpenAI releases a two-stage transfer-learning system: pretrain a Transformer with language modeling, then adapt it to several language-understanding tasks with limited task-specific changes.
ELMo introduces deep contextualized word representations
The AllenNLP team introduces ELMo, which derives context-dependent word representations from a bidirectional LSTM language model, and releases an implementation and pretrained weights.
Google researchers publish the Transformer paper
The paper introduced an encoder–decoder architecture built entirely on attention, replacing recurrent computation with parallelizable self-attention.