I clustered four Framework Mainboards to test huge LLMs
Imagine my surprise when Nirav Patel, Framework's founder and CEO, was at Open Sauce a couple weeks ago, and wanted to talk! He said they had seen my Project Mini Rack posts earlier this year and thought it was the perfect application to try out their new AMD Ryzen AI Max+ 395-powered Mainboard, as it's mini ITX dimensions fit inside a 10" rack.
Framework sent over four pre-release Mainboards1 (along with four of their new Power Supplies, and Noctua CPU Fan kits), and also worked with DeskPi to get my hands on a new 2U Mini ITX tray for the Framework Desktop—which I installed in the black RackMate T1 mini rack you see at the top of this post.
I have a video going over everything, including cluster setup, single node and cluster AI inference performance, and my conversation with Nirav—you can watch that below:
I spent equal time running benchmarks as I did getting the benchmarks to run. I tested (and automated clustered setup for) Exo, llama.cpp RPC, and distributed-llama.
