Skimpy HBM Memory Opens Up The Way For AI Inference Memory Godbox
Generative AI is arguably the most complex application that humankind has ever created, and the math behind it is incredibly complex even if the results are simple enough to understand. GenAI also it has some serious bottlenecks when it comes to memory bandwidth and memory capacity, and these bottlenecks could be the driver for the adoption of memory godboxes that a number of different companies have been trying to bring to market over the past several years.
Generally, these memory servers use the CXL protocol to extend the main memory of systems, pooling many terabytes of DDR main memory so it can act as a relatively fast and fat cache for the wickedly high bandwidth but relatively low capacity HBM stacked memory that is commonly wrapped around GPUs and other kinds of XPU accelerators for AI workloads.
Enfabrica, with its new Emfasys memory cluster, is the latest to deliver a memory godbox in production, and KV caches for speeding up AI inference for ever-more-complex queries could turn out to be the killer application that Enfabrica and its peers have been waiting for.
