XLNet paper and XLNet-Large weights are released
Researchers from Google Brain and Carnegie Mellon University introduce XLNet and release the XLNet-Large checkpoint with TensorFlow code.
What happened
Event details
The XLNet paper introduced generalized autoregressive pretraining. Its objective takes an expectation over permutations of the token factorization order, allowing the model to use bidirectional context without inserting masked tokens that are absent during downstream use. XLNet also incorporates Transformer-XL relative positions and segment recurrence, and uses two-stream attention to prevent target-position leakage. Under comparable settings, the authors reported gains over BERT across 20 tasks covering question answering, natural-language inference, sentiment analysis and document ranking. The June 19 release included the XLNet-Large checkpoint and code; XLNet-Base followed on July 16. XLNet was an important branch of pretraining research, although its long-term ecosystem did not become as broad as BERT's.
Assessment
Why it matters
XLNet combined permutation language modeling, bidirectional context and Transformer-XL recurrence, and had a clear impact on language-understanding research, but did not become a lasting ecosystem standard on the scale of BERT.
Availability
Access notes
The initial release included the XLNet-Large checkpoint and TensorFlow code; the Base checkpoint followed in July 2019.