GShard paper demonstrates a sparse-expert translation model with more than 600B parameters
Google researchers introduce GShard automatic sharding and use it to train a multilingual translation MoE with more than 600B total parameters on 2,048 TPU v3 accelerators.
What happened
Event details
GShard is a set of lightweight parallel-computation annotation APIs together with an extension to the XLA compiler. It lets researchers express several sharding patterns with limited changes to model code. The paper used this system to train a multilingual machine-translation Transformer whose selected feed-forward layers were replaced by sparsely gated mixtures of experts. The largest experiment had more than 600B total parameters and was trained on 2,048 TPU v3 accelerators for four days to translate from 100 languages into English. Its significance lies in advancing automatic sharding and sparse-expert scaling; the total parameter count does not mean that every parameter is active for every token. This was a research disclosure, with no public model weights, training code or API.
Assessment
Why it matters
GShard joined automatic sharding with sparse-expert training, pushed the reported Transformer scale beyond 600B parameters and became a direct technical precedent for later MoE and large-model parallelism work; its immediate product impact was limited.
Availability
Access notes
The paper was public, but the experimental model, training code and weights were not released, and no public API was offered.