xAI releases Grok-1 weights and architecture
xAI released the 314-billion-parameter Grok-1 base-model weights and network architecture under Apache 2.0.
What happened
Event details
On March 17, 2024, xAI released Grok-1's base-model weights and network architecture under Apache 2.0. Grok-1 has 314 billion parameters and uses a mixture-of-experts architecture with eight experts. Two experts are selected for each token, activating roughly 25% of the weights. Its maximum context length is 8,192 tokens. The checkpoint comes from the pretraining phase that ended in October 2023 and was not fine-tuned for dialogue or another specific task. The official repository provides JAX examples for loading and sampling, but warns that the full model needs substantial GPU memory. Its reference MoE implementation favors correctness over inference efficiency. The release therefore offers unusually detailed access while remaining difficult to deploy.
Assessment
Why it matters
Grok-1 was an unusually large open-weight mixture-of-experts release. The full weights, architecture details, and Apache 2.0 license supported research and reuse, but the 314-billion-parameter footprint and unoptimized reference implementation limited practical access to teams with substantial GPU resources.
Availability
Access notes
The weights, network architecture, and JAX examples are available under Apache 2.0. This is a pretrained base checkpoint, not an instruction-tuned chat assistant.