Google publishes Switch Transformer, scaling a sparse model to 1.6T parameters
Switch Transformer routed each token to a single expert, scaling total sparse parameters to about 1.6 trillion without activating the full model for every token.
What happened
Event details
On January 11, 2021, Google researchers submitted the Switch Transformer paper. The method simplified mixture-of-experts routing by sending each token to one expert in a sparse feed-forward layer, reducing communication and implementation complexity. The paper reported up to a sevenfold pretraining speedup in experiments based on T5-Base and T5-Large. Its largest Switch-C configuration used 2,048 experts and about 1.6 trillion total parameters, with roughly four times the pretraining speed of T5-XXL. Because the model is sparse, that total parameter count is not equivalent to the computation of a similarly sized dense model.
Assessment
Why it matters
Switch Transformer showed that single-expert routing could simplify and scale sparse MoE training, influencing later expert-model architectures and systems work.
Availability
Access notes
The paper is public, and Google provides pretrained checkpoints including the 2,048-expert Switch-C configuration.