In a world where allocations of “Hopper” H100 GPUs coming out of Nvidia’s factories are going out well into 2024, and the allocations for the impending “Antares” MI300X and MI300A GPUs are probably long since spoken for, anyone trying to build a GPU cluster to power a large language model for training or inference has to think outside of the box.
For those who want to buy and control their own iron, that might mean going to Intel for Gaudi 2 accelerators, or trying to get some Nvidia A100s or AMD MI250Xs, or even going to Cerebras Systems, SambaNova Systems, Groq, or Graphcore, Or it may mean paying a 2.5X to 3X premium to run LLMs on a big cloud – and don’t assume you can get reserved instance capacity there, either.
Or, you can build a cluster that makes use of Nvidia’s tweaked “Lovelace” L40S accelerator and employing composable infrastructure from Liqid to get a much higher ratio of GPU to CPU compute to boot. Provided your LLM can do its AI training without very much FP64 double precision floating point – and clearly, there are models that make do with FP32 and lower precision because none of the exotic AI engines above have any FP64 math at all. Portions of their workloads suffer a performance degradation because of this, but they make up for it in other ways. (Computer architecture is always about tradeoffs, isn’t it?)
For years now, composable infrastructure supplier Liqid has been stressing how composable GPU enclosures and composable I/O infrastructure linking them to servers in a cluster allow companies to share GPUs across different workloads with different configurations and therefore allow the overall GPU utilization to be driven upwards. This is a way to get more work out of the GPU money that you have spent. Which is important. But these days, you need to be able to get a lot of work out of whatever GPUs you can get your hands on.