Frontier Pre-training at Microsoft AI | Microsoft AI
The MoE model has 80 billion active and 2.8 trillion total parameters, trained on 16k GB200 GPUs over 38 days across 50 trillion tokens.
- Achieves 4.2x to 12.3x higher pre-training FLOPs efficiency than DeepSeek-V4-Pro-Base across held-out evaluations.
- Trained natively on vision via an encoder-free VLM architecture, with multimodal data accounting for 18% of the 50T training tokens.
- Maintained an average 21% model FLOPs utilization normalized to 2.5 PFLOPS in BF16 and 89% goodput.
- Employs multi-round dropless MoE with static memory allocation and an optimized implementation of the Muon optimizer.
- Pre-trained entirely on in-house organic data without distillation for eventual licensing to enterprise customers via Foundry.
Microsoft gains an in-house, frontier-scale multimodal base model to deploy across its product suite and license directly to enterprise customers.
