The Remarkable Utility of Dataflow Computing
We build computer systems around abstractions. The right abstractions are timeless and versatile, meaning they apply in many different settings. Such abstractions make developers’ lives easier, and allow system developers and researchers to optimize performance “below the hood”, by changing systems without breaking the abstractions. In this post, I will argue that the abstraction of dataflow computing is a remarkably powerful one. Mapping computations into dataflow graphs has given us better, more fault-tolerant and scalable distributed systems, better compilers, and better databases. Dataflow also has the potential to enable new system designs that solve other — seemingly unrelated — real-world problems, such as making it easier and cheaper to comply with privacy legislation like the European Union’s GDPR, or California’s recent CCPA.
The key idea behind dataflow computing is to express computation as a dataflow graph. Each vertex of a dataflow graph represents an operation, and each edge represents a data dependency. For example, a dataflow graph for the function f(x, y, z) = x * y + z might look like this:
