The Future of the Modern Data Stack

The Modern Data Stack is quickly picking up steam in tech circles as the go-to cloud data architecture, and although its popularity has been quickly rising, it can be ambiguously defined at times. In this blog post we’ll discuss what it is, how it came to be, and where we see it going in the future. Regardless of whether you’re new to the modern data stack or have been an early adopter, there should be something of interest for everyone. 

The Modern Data Stack commonly refers to a collection of technologies that comprise a cloud-native data platform, generally leveraged to reduce the complexity in running a traditional data platform. The individual components are not fixed, but they typically include: 

The goal is to make data actionable by reducing the time it takes for data to become useful to data workers in an organization. Gone are the days where it takes weeks for data to land in your company’s analytical warehouse after creation. Now it happens in hours or minutes. Companies that go down the path of the modern data stack adopt the technology as it fits their needs – i.e. you don’t necessarily need every component, and some may opt for other technologies, like Airflow, Dagster, or Prefect for an orchestration layer. A simple sample architecture is illustrated below.

Simply having a data platform in the cloud does not make it a “modern data stack.” In fact, I would wager to bet that most cloud architectures really fail to meet the categorization. Things like lift-and-shifted platforms, cloud data lakes, and bespoke solutions often fail to really capture the essence of the modern data stack and often feel as clunky as their on-premises cousins. So what makes something part of the modern data stack? If we look latitudinally across technologies in this ecosystem, we’ll begin to notice that they share some common properties that get at the core of the modern data stack. I’ll propose the following as key capabilities of technology in the modern data stack: 

Leave a Comment