An technical guide to architecting large ML pipelines on Snowflake warehouses

There are several kinds of machine learning pipelines. Each requires different architectural choices because of different expectations around latency, volume and environment capabilities. The constraints for serving models used in real-time threat detection in video feeds are very different from those for batch information extraction across hundreds of millions of documents.

This post focuses on the latter: Large-scale batch inferences utilizing Snowflake. At Modelbit we’ve helped many customers perform inferences on the data in their Snowflake warehouses, and we’ve learned a lot along the way. If you’re thinking about building your own ML pipeline on Snowflake, then this post is for you!

Below we’ll share hard-learned lessons and recommendations for how you should (and shouldn’t) build your own ML pipeline on Snowflake. And at the end we’ll share some predictions for where we think this technology is going and what the future holds.

When we’re talking about an ML pipeline in Snowflake, we’re talking about something dbt could orchestrate. Such a pipeline would process hundreds of millions of rows through dozens or more different ML models to create a collection of tables containing the inference results.

Leave a Comment