This InfinyOn Data Stack is designed to handle complex data processing workflows, allowing for customization and scalability through various programmi

SDF Architecture Diagram

submited by
Style Pass
2024-11-05 15:00:37

This InfinyOn Data Stack is designed to handle complex data processing workflows, allowing for customization and scalability through various programming languages and system primitives.

At the core, the InfinyOn data stack combines data streaming and state management technologies. Fluvio handles event streaming, persistence, replication, and connections with external data sources and target destinations. SDF manages state objects such as SQL tables and key-value documents.

Services, while omitted from the diagram, define the composition of a subsection of the data-pipelines. Services define the data source, chain of operations, state objects, and the destination.

Operators are functions and primitives that provide access to data streaming records and stateful objects. There are two types of operators for transformations and window processing. The first type performs record-by-record operations such as filter, map, filter-map, or flat-map. The second allows the user to shape the records into a bounded context for window processing. These window operations are computing aggregates, aka materialized views.

Operators generate functional APIs with placeholders for the user to apply their custom business logic. Custom logic can apply transformations, perform call-outs for third-party applications, build state objects, and more. See the examples in github for more details. The functions support programming languages like Rust and Python (coming soon) or SQL queries.

Leave a Comment