GPT-6 (2025)
26/Mar/2024: Former Google engineer Kyle Corbitt (Twitter, 26/Mar/2024): ‘Spoke to a Microsoft engineer on the GPT-6 training cluster project. He kvetched about the pain they’re having provisioning infiniband-class links between GPUs in different regions. Me: “why not just colocate the cluster in one region?” Him: “Oh yeah we tried that first. We can’t put more than 100K H100s in a single state without bringing down the power grid.” 🤯’
Spoke to a Microsoft engineer on the GPT-6 training cluster project. He kvetched about the pain they're having provisioning infiniband-class links between GPUs in different regions.
70 megawatts (MW) is a significant amount of power. To put it in perspective, this is roughly the amount of electricity needed to power a small city or a large town.
For example, a city with around 70,000 households, assuming an average household consumes about 1 kW consistently, would require approximately 70 MW of power.
