OpenAI releases a generative pre-trained Transformer, later known as GPT-1
OpenAI releases a two-stage transfer-learning system: pretrain a Transformer with language modeling, then adapt it to several language-understanding tasks with limited task-specific changes.
What happened
Event details
On June 11, 2018, OpenAI released the paper, model, and code for Improving Language Understanding by Generative Pre-Training. The system first trained a general language model on BooksCorpus, a collection of more than 7,000 unpublished books, then fine-tuned it for supervised tasks. Differences among tasks were handled mainly through input transformations instead of separate network designs. The paper reported then-state-of-the-art results on 9 of 12 evaluations. The work did not invent the Transformer or pretraining, but it combined generative pretraining, a shared backbone, and downstream fine-tuning into a scalable recipe that became the starting point of the GPT line. OpenAI disclosed a 12-layer decoder, 768-dimensional states, 12 attention heads, and 512-token training sequences. The first-party sources used here do not explicitly state a total parameter count, so the catalog no longer presents 117M as a confirmed field.
Assessment
Why it matters
GPT-1 combined generative language-model pretraining and task fine-tuning into a shared method and became the technical starting point of the GPT series. Its research influence was durable, while its direct market and policy impact at launch was limited.
Availability
Access notes
OpenAI released the paper, model, and implementation.