ELMo introduces deep contextualized word representations
The AllenNLP team introduces ELMo, which derives context-dependent word representations from a bidirectional LSTM language model, and releases an implementation and pretrained weights.
What happened
Event details
Researchers from the Allen Institute for AI and the University of Washington published the ELMo paper. Instead of assigning one fixed vector to each word, ELMo derives contextualized representations from the internal states of a deep bidirectional language model, capturing lexical, syntactic and semantic information as well as polysemy. The original architecture uses a character CNN and two-layer forward and backward LSTMs; it is not a Transformer. Adding ELMo to existing systems improved results across six NLP task families, including question answering, textual entailment and sentiment analysis. The official TensorFlow implementation and pretrained weights were made public, helping establish pretrained contextual representations as reusable components before BERT.
Assessment
Why it matters
ELMo shifted word representations from static vectors to contextual representations and showed that internal states from a pretrained language model could improve many downstream NLP systems.
Availability
Access notes
The paper, official implementation and English pretrained weights are publicly available.