M5Stack Adds Large Language Model Support to Its Offerings with the 3.2 TOPS LLM Module

Modular electronics specialist M5Stack has announced its latest hardware release, a module that aims to add large language model (LLM) artificial intelligence (AI) to your builds: the sensibly-named M5Stack LLM Module.

"[The LLM Module] is an integrated offline large language model (LLM) inference module designed for terminal devices that require efficient and intelligent interaction," M5Stack says of the hardware in question. "Whether for smart homes, voice assistants, or industrial control, Module LLM provides a smooth and natural AI experience without relying on the cloud, ensuring privacy and stability."

Aiming to ride the current boom in LLM technology, in which user queries are broken down into "tokens" and the most likely tokens returned in response to create an answer-shaped object that may, if everything goes well, also be correct, the LLM Module — brought to our attention by Linux Gizmos — is powered by an Axera AX630C system-on-chip. This combines two Arm Cortex-A53 cores running at up to 1.2GHz with an in-house neural processing unit (NPU) delivering 3.2 tera-operations per second (TOPS) of compute at INT8 precision rising as high as 12.8 TOPS if you drop to INT4 precision.

This, combined with 3GB of memory dedicated to the NPU with 1GB left for the operating system installed on a 32GB eMMC module, is enough to run smaller large language models entirely on-device — while drawing, the company claims, as little as 1.5W. The module includes an integrated microphone with wake-word and speech recognition models pre-loaded, and a speaker to serve as an output through an integrated text-to-speech model. The eMMC comes with an unspecified version of Canonical's Ubuntu Linux pre-loaded, and can be upgraded through a microSD Card slot.

Leave a Comment