In the longest of runs, say within the next five to ten years, in the large datacenters of the world, the server chassis as we know it will no longer

The Tipping Point For PCI-Express Fabrics

submited by
Style Pass
2021-09-17 16:30:05

In the longest of runs, say within the next five to ten years, in the large datacenters of the world, the server chassis as we know it will no longer exist. We aren’t talking about the bent metal here. We are talking about ending the tyranny of the motherboard, which limits the scale and scope of the unit of compute in the datacenter.

We have been on the disaggregation and composability crusade for over two decades, and have been heartened to see so much progress in recent years. New fabrics have been laid atop various bus and network interconnects to lay the foundational work for creating true disaggregation and composability as we have dreamed of it for so long. We encapsulated our thoughts about why this all matters, and in the current context, back in March in an essay called The Future Of Infrastructure Is Fluid, and we are not going to repeat all of those strategic thoughts now. This is about tactics and getting a report from the field for one of the innovators in disaggregation and composability. Namely, GigaIO.

Fundamentally, PCI-Express fabrics take Ethernet and InfiniBand out of the picture, literally chopping the network interface card and all of its protocol and transport out and replacing it with a PCI-Express adapter card to extend the server bus out to any one of a number of PCI-Express switch topologies. To this fabric is added Non-Transparent Bridging (NTB), which allows two devices connected in a point-to-point fashion over PCI-Express to see into each other’s memory and I/O. We discussed this when GigaIO dropped out of stealth in October 2019 and was still working with relatively unimpressive PCI-Express 3.0 switching, which was fairly limited in the aggregate bandwidth of the devices and the port counts possible (what is called radix in the industry). InfiniBand and Ethernet ASICs have much higher bandwidth and much longer links between devices, but they also have much higher latencies.

Leave a Comment