NVIDIA releases Megatron-Turing NLG
Microsoft and NVIDIA jointly train a 530-billion-parameter dense language model.
SEARCH
Search English reports
English reports are listed here. AI-generated drafts are clearly labeled until an editor reviews them.
AI-assisted · reviewedImpact 45/100
OpenAI releases Codex
OpenAI releases a code-generation model fine-tuned from GPT-3 and used in the early GitHub Copilot.
AI21 Labs launches Jurassic-1
AI21 Labs launched the Jurassic-1 language model family and opened its API and web interface through the AI21 Studio open beta.
EleutherAI releases GPT-J
EleutherAI releases an early open 6-billion-parameter model that sees broad academic use.
Google first introduces LaMDA
Google introduces LaMDA for open-domain dialogue at I/O 2021; the model remains an internal research system without public weights or a general-purpose API.
EleutherAI releases GPT-Neo
EleutherAI releases an early open alternative to GPT-3.
Naver releases HyperCLOVA
Naver releases a large language model optimized for Korean.
Google publishes Switch Transformer, scaling a sparse model to 1.6T parameters
Switch Transformer routed each token to a single expert, scaling total sparse parameters to about 1.6 trillion without activating the full model for every token.
OpenAI introduces the 12B text-to-image model DALL·E
OpenAI introduces DALL·E, a 12B autoregressive Transformer that models text and discrete image tokens as one sequence.
OpenAI releases the CLIP vision-language model
OpenAI released CLIP, a vision-language model trained with natural-language supervision, together with its paper, code, and pretrained models.
Meta publishes BART research and pretrained models
Meta published the BART denoising sequence-to-sequence model and made code and pretrained weights available through fairseq.
GShard paper demonstrates a sparse-expert translation model with more than 600B parameters
Google researchers introduce GShard automatic sharding and use it to train a multilingual translation MoE with more than 600B total parameters on 2,048 TPU v3 accelerators.
OpenAI opens GPT-3 access through a private API beta
OpenAI launches a private beta for a general-purpose text-in, text-out API running GPT-3 family models, offering controlled access instead of open weights.
GPT-3 demonstrates in-context learning at 175B parameters
OpenAI publishes GPT-3 research showing that a 175B-parameter model can switch tasks from instructions and a few examples in the prompt without updating its weights.
Microsoft releases Turing-NLG as a limited academic demo
Microsoft introduced the 17-billion-parameter Turing-NLG and offered a private demo to a small group of academic users.
Google Research introduces the open-domain conversational model Meena
Google Research introduced the 2.6-billion-parameter open-domain conversational model Meena but did not release an external demo or model weights.
OpenAI releases the full GPT-2 1.5B weights
OpenAI releases code and model weights for the largest GPT-2 configuration, concluding its nine-month staged-release experiment.
Google publishes T5 and releases models, code, and C4
T5 cast a wide range of natural-language-processing tasks into a text-to-text format and released a model family ranging from 60 million to 11 billion parameters.
Facebook AI Research releases RoBERTa
Facebook AI Research retrains BERT with more data, larger batches, longer training, dynamic masking, and no next-sentence prediction, then releases the paper, weights, and fairseq implementation.
XLNet paper and XLNet-Large weights are released
Researchers from Google Brain and Carnegie Mellon University introduce XLNet and release the XLNet-Large checkpoint with TensorFlow code.