The Creators of DeepSpeed | Generated by AI
Question: Who invented DeepSpeed?
Answer:
DeepSpeed was created by the Microsoft Research AI at Scale team, not by a single individual. It was open-sourced in May 2020 as a PyTorch optimization library for training and serving very large neural networks. (Wikipedia)
The key people behind the original DeepSpeed project include:
- Jeff Rasley
- Samyam Rajbhandari
- Olatunji Ruwase
- Yuxiong He
These four authors wrote the original ZeRO (Zero Redundancy Optimizer) paper, which is the core innovation that made DeepSpeed famous. ZeRO partitions optimizer states, gradients, and parameters across GPUs, dramatically reducing memory usage while preserving the same model semantics. Most of DeepSpeed’s later features (ZeRO-Offload, ZeRO-Infinity, Ulysses, MoE support, etc.) build on this foundation. (Microsoft)
Leadership
The researcher most closely associated with the overall vision of DeepSpeed is Yuxiong He. He leads the Microsoft Research AI at Scale effort and is a senior author on many of the major DeepSpeed papers, including:
- ZeRO
- ZeRO-Offload
- ZeRO-Infinity
- DeepSpeed-MoE
- DeepSpeed Inference
- ZeRO++
Timeline
| Year | Milestone |
|---|---|
| 2020 | DeepSpeed open sourced by Microsoft Research |
| 2020 | ZeRO optimizer introduced |
| 2021 | ZeRO-Offload and ZeRO-Infinity |
| 2022 | DeepSpeed Inference and DeepSpeed-MoE |
| 2023–2024 | Ulysses, ZeRO++, Universal Checkpointing, Domino, and other scaling optimizations |
Today, DeepSpeed remains one of the three dominant distributed training frameworks for PyTorch, alongside:
- PyTorch Distributed
- NVIDIA Megatron-LM
- DeepSpeed
References:
Generated by AI. Curating and sharing still takes effort.
If you find it useful, feel free to
donate.
WeChat: @lzwjava
·
X: @lzwjava ·
Say hi 👋